June 15, 2016: The case against upstream packaging
June 16, 2016: The case against upstream packaging (postscript)

Original The case against upstream packaging The case against upstream packaging (postscript) Editable
version 1 & 2 of 2


Lisandro Damián Nicanor Pérez Meyer (2016-06-15-15-13-19-798)

With my Debian maintainer hat on: thanks for writing this :)


Disclaimers, et al. These opinions are not at all related to Arch Linux, though maintaining Arch packages has certainly colored my opinions. I've been a volunteer maintainer for years now, because I feel it helps people. I have a bias towards the value that maintainers and maintaining packages provide.


While most people are taking talking about the new future of Universal Linux Packages from a technical standpoint, the social perspective isn't getting much attention. Frankly any technical issue can be solved by a better "universal package" but the social changes can't be improved by emailing a patch. So let's examine the relationship between upstream developers, distribution maintainers and end users. With a focus on how we got here and where we might go next.

jjacky (2016-06-15-15-39-30-195)

Small typo I think, should read (1st sentence) : "..are talking about.."

Great piece.


Normally I really like the word 'upstream'. But it has a lot of connotations and doesn't suit the tone of this essay. To define it, upstream is generally used as a collective pronoun to mean "the group of people who gives us software". They are people who fix bugs, add features and occasionally make mistakes. They have the best interests of their users and their software at heart. They are good people who always mean well and take pride in their work. Upstream is not a single group, but spread among everyone who writes software.

"Upstream", however seems to be a mostly-FOSS concept, largely a side effect of the selective pressures that distributions provide. Since this essay goes into what a weakening of distributions would look like I'm going to use the original, older, uglier, more agnostic term "Independent Software Vendor" or ISV.

Why say ISV? Why does this word matter? "Upstream" is used both collectively for FOSS as a whole and singularly when talking about a single project. This re-use of the word is a small flaw as it suggests all software developers are fundamentally the same. This is not the case: some have very different values. Being "upstream" also has an implicit social contract attached to it. The "I", for independent, is to remind us that this is not always the case. And the "V" is for vendor, a reminder that sometimes these are businesses who consider profitability to overrule all other considerations.

Obviously this does a disservice to those who volunteer their time and intelligence without compensation, and do consider software quality and user happiness to be important. For the purposes of this essay I'm painting everyone with the "ISV" brush, even those pillars of the community who deserve better. But we are heading towards a future where the nice guys don't win.

Let us start with the (recent) past that kicked off the whole idea of "ISVs can package stuff" and "maybe we don't need maintainers".

The Cargo Cult of App

The promise: Phone app stores came out of nowhere and suddenly had millions of developers appear out of the woodwork overnight! If we make our distribution more like phone app stores then we get millions of application developers!


The reality: App Stores have been a nearly unmitigated disaster for users. Supposedly one of Linux's failings is that there is too much pointless choice, too many K and G versions of things and it divides developer efforts. Why have so many window managers and text editors? App Stores have the same problem. With the traditional FOSS model, there are a hundred different programs and each program is missing a different feature. In the App Store, there are a hundred different programs doing the same thing but each screws the user over in a slightly different way. Spying? Ads? Battery sucking rookie mistakes? Battery sucking bitcoin mining botfarm? Take your pick.

kk (2016-06-15-18-37-38-082)

Disaster for some developers as well. Get caught on the wrong side of the review process and be unable to reach a human for an explanation.


The promise: But Linux is different! No one does those horrible things on Linux! And no one ever will!

The reality: But is it different? Somehow we've gotten this idea that everyone who writes software for Linux does it for the warm fuzzies. No Linux ISV tries to take advantage of people, and none of them ever had. Linux has been around for 20 years. Everyone has been ethical and trustworthy the whole time. Clearly they have proven themselves and will continue to be ethical and trustworthy. Alternatively, Linux is somehow immune to this darker side of human nature.

Maintainers are the Difference

Linux-related news might suggest maintainers are useless. Either they are bikeshedding on mailing lists or closing bugs as WONTFIX, or introducing security vulnerabilities in software they don't understand, or generally being a BOFH. Of course this is wrong - we simply want to use the best operating system possible. So what does a maintainer actually do?

Mostly a maintainer is a packager: they take software from an ISV and repackage it for a distribution's packaging system. And here is where some people feel is a huge waste. Why package the same software a hundred times for a hundred different distributions? (There is good reason for this "waste", but let me finish describing everything else a maintainer does.)


More fundamentally, the maintainer is the primary line of defence and interaction between users and developers. Maintainers shield developers from uninformed users, allowing the devs to write software with less support overhead. Non-bugs are caught and filtered out. Low-quality bugs reported to the distribution's tracker often becomes a good bug when the maintainer reports it upstream. Without distros the ISV would have to carry the full weight of support. Here's an example:

commenter (2016-06-16-01-54-21-850)

Although as a whole I think this article makes a number of good points, you here as an Arch maintainer almost lost me, ex-Arch user. Nearly every time I reported a non-packaging bug on the Arch bugzilla I was told to "report this upstream" and/or saw bugs closed as "upstream".

And in fact, this is standing Arch policy. Not one I ever agreed with, and for the precise reason described in the paragraph I'm commenting on, but this then at this point did give me the impression that you were writing an article unsynchronized with (your specific) reality.

kk (2016-06-16-08-51-11-042)

Arch encourages its users to be more active. If a dev asked you to report a bug upstream yourself, then the dev has given you two indirect compliments. 1) You have found a real bug and 2) You are far enough along to do it for us.

Personally I always try ask people if they want to report it upstream or if I should do it for them.


<NewUser> I've never gotten OpenSCAD to work on my computer, so I don't know anything particular to it.
<@teepee> NewUser: we could help with that (hopefully) :)
<NewUser> I'm gonna upgrade my system software when Slackware proper is RTMed. We'll see then.
<@teepee> ahh, cool, slackware is still around
* teepee used that like 15 years ago
<NewUser> Not a lot. I'm also installing Arch.
<@teepee> ahh, with arch, that's easy, you just complain to keenerd if it's not working
<keenerd> teepee: Spot on :-)
<@teepee> :)
<@teepee> it's always good to know there's someone who cares

Maintainers also shield users from developers, offering a layer of quality control. At the very least, all the software in a distribution should play friendly with any other piece of software in the repositories. Maintainers are responsible for that, as well as choosing a subset of the (subjectively) best software FOSS has to offer. Maintainers will disable features that they feel act in bad faith. Maintainers' greatest power is the ability to outright say "This is not good enough for our users" and consequently punish an ISV by either patching out the offensive part or in extreme cases removing the software from the repositories. ISVs know this and so don't act out. After 20 years of enforced good behavior this has lead to the idea of ISVs as "the benevolent upstream developer." This is why Linux doesn't have spyware, doesn't come with browser toolbars, doesn't bundle limited trials, doesn't nag you to purchase and doesn't pummel you with advertising.

It is simply not possible for abusive software to exist in Linux today. No maintainer would willingly release it. If a maintainer ever got a reputation for releasing abusive software they would be stripped of their privileges. Someone who cares steps up and becomes a maintainer instead. The system is aggressively self correcting. It has been like this for twenty years, so much to the point that few Linux ISVs even consider being unethical. And Linux users have a very hard time imagining that any ISV could ever act in bad faith.


Maintainers Provide Consistency

Joe Julian (2016-06-16-08-42-01-621)

Yes. Unfortunately for Gluster, most maintainers seem to keep around versions with severe bugs and never upgrade the packages to bugfixed versions or even backport fixes.

Sure it's consistent, but it's consistently broken. We package our own packages so that our users have the chance to upgrade to a non-broken version.

Arch, of course, is the exception where new versions are packaged within a day or two of release. Even then, however, it's always the bleeding edge version. There's no mechanism to stick with a specific branch.

kk (2016-06-16-08-57-12-075)

I'm sorry to hear that. Have you been able to find any maintainers who use it themselves and are bothered by the out-of-dateness?

Arch does have mechanisms for using older versions. There is a glusterfs34 package in the AUR for example. Admittedly you can't have it installed side by side with the latest glusterfs 3.7 package but that could be fixed.


For the sake of argument, let us presume that all ISVs are ethical and hold the interests of the community as their utmost ideal. Would this utopia still require maintainers to provide quality control? Yes, because no one can agree on what "quality" is beyond "does it build and run". Any good distribution is not about the file extension on the package - it is about the decisions the software implicitly makes. A distribution is a world view provided by the maintainers and shared by the users. In other words, maintainers don't just ship packages. They provide a certain flavor of consistent and preferred opinions to a group of users. (As an aside, this is why users of different distros often don't see eye-to-eye.)

Could ISVs possibly make everyone happy? They can't. It is impossible. What happens when two groups of users expect different things from the same software? To give an example, imagine an ultralight distribution like Puppy. They want the software as slim as possible. Recompile as i386, it'll save 5%. Chop out the built-in-help, we'll use the website. Get rid of the autosave-every-0.1-seconds, it'll wear out the flash drives. Now imagine an Ubuntu user. They want every bell and whistle enabled. In fact, could you add integration for our new notification and desktop search services? And then imagine the paranoid LFS user. They don't trust any binary package and insist on getting full sources, reviewing them and building them From Scratch.

What is an ISV to do? They can try to package half a dozen different flavors of their software. Now a hypothetical cross-distro Universal Packaging Format is not saving them any effort because they need to make (and test, and support) a different build target for each demographic of users. Or the ISV can ignore some users, making a subset of them unhappy.

If people are unhappy enough they will repackage the software in a more suitable manner. Now the users have unwittingly become maintainers. The moment someone starts improving 3rd party software, they become a maintainer.

And if a maintainer shares their packages, they'll attract like-minded users who appreciate the work as well as like-minded maintainers who can share the workload. And thus a distribution is born. It can't be fought. A monoculture will breed diversity.


To clarify a few points often heard:


  • The promise: Sandboxing makes you immune to bad ISVs.
  • The reality: It protects you from some things and not others. App Stores have proven that ISVs will try to get away with as much abusive behavior as possible. Sandboxing protects against many threats, but does not suddenly make people ethical. Distributions with human maintainers are more likely to make mistakes than an OS-enforced sandbox, but human maintainers are also more likely to sniff out the dumb stunts that ISVs try to pull. No human maintainer is ever going to willingly package a flashlight app which phones home. From source anyway. I imagine sandboxing will be used as an additional layer of safety in most distros (eventually) but mostly for the purposes of protecting from honest mistakes
creshal (2016-06-16-04-29-29-729)

I never understood why the implicit consensus is that we can't have sandboxing without an app store, while OpenBSD adds more and more sandboxing features to regular packages just fine.



  • The promise: By cutting out the maintainer middle-man, security fixes can be pushed out to users faster.
  • The reality: Often maintainers are faster, particularly with anything using shared libraries. A distro can update a shared library once and all software is now safe, without needing to rebuild every piece of software using it. Most of the sandboxes do not share libraries and so you are waiting on the slowest ISV to rebuild their software. If they ever do. ISVs are notorious for using ancient libraries with extensive out-of-tree patches and never migrating the patches forward.
Federico (2016-06-16-02-24-46-251)

Also, ISVs abandon projects and their software is not automatically removed from "app stores". Distributions either maintain or remove such packages.


  • The promise: A universal package means that ISVs don't need be aware of every distro and its unique quirks.
  • The reality: They don't need to and never have. Put out the sources with a makefile. The maintainers can handle the rest. It's what we do. Heck, the makefile is optional. We can write one. (But please merge it when we send a pull request!)
  • The promise: App Stores are a re-spin of the traditional distribution repository. One central location, easy search, easy install, automatic updates. Its practically the same thing.
  • The reality: They are still sorely lacking. No app store has the slightest concept of dependencies. They often aren't even dogfooded for the operating system's software. And of course anything that passes an automated scan can get into an app store. Distributions provide extensive curation that you can't get without an army of volunteers.
  • The promise: Make an App Store and a million devs will appear to fill it with software.
  • The reality: Smartphones had a large user base before they had any apps. Get the users first and the devs will follow. Thinking an app store will bring in more developers is cargo cult logic at its finest. It will however bring in a wave of opportunistic software that was previously barred from Linux. There are entire frameworks designs to wrap an Android or iOS application with malware, integrating ads or analytics with a single click. That simply wouldn't make it past a maintainer.
  • The promise: A common packaging effort will improve the quality of software on Linux, because packaging for every distro is too hard.
  • The reality: Handwriting is too hard. I have created a solution, the "keyboard". Now people will no longer write dumb things because the hard part is solved. Making packaging "easy" will not improve the quality. Just like how legible writing has no relation on writing intelligently, the mechanics of packaging has no relation on the quality of software. Write good software, release sources, and every distro will package it for you.
  • The promise: ISVs releasing packages means less duplicated work between distros.
  • The reality: Distros exist because users can't agree on how exactly we want software. A universal package will not be good enough for some percentage of them. The actual act of packaging is the fastest part. The slowest part (fixing bugs that upstream hasn't) is shared between distributions because all of our patches are public.
  • The promise: Many embarrassing bugs will be avoided, because ISVs are smarter and more experienced than maintainers.
  • The reality: Oh boy the stories I could tell about the things I've seen. But out of courtesy, I won't name names. Needless to say they are as dumb and human as we maintainers are. If you want, think of maintainers as outside consultants who will find the craziest thing an organization is doing. Except we don't charge $200/hour for what amounts to best-practice common sense.


  • The promise: Software directly from the ISV has a complete chain of trust and is better for users.
  • The reality: This is only true if the ISV won't try to sneak something in. If you don't trust your maintainers, You don't need to have blind trust in your maintainers, but don't get rid of them either. Instead, Verify verify their work with your own builds and examine the chain of trust the distribution provides. Distros are usually way more open about the chain of trust than ISVs are.

The sentence “ If you don't trust your maintainers, don't get rid of them.” is confusing, because it appears to present that as the solution to untrustworthy maintainers.

I would prefer this and the following sentence to be re-written. Suggestion: “You don't need to have blind trust in your maintainers, but don't get rid of them either. Instead, verify their work with your own builds […]”.

kk (2016-06-16-10-51-46-084)

That is perfect, thank you.


Yes, all of this will happen.


I don't mean to pick on Shuttleworth, Shuttleworth but he has explicitly stated that the whole point of their new universal packages is to take control away from the maintainers/users and give it control to the ISV ISVs. If you believe "control" is a finite resource then giving implies taking from elsewhere. Most likely from the maintainers and users. To quote:

Mark Shuttleworth (2016-06-15-23-48-32-574)

Maintainers DO matter. Maintainers make the system work, and they make a huge difference in the world. Without maintainers we would not have the rich and diverse portfolio of innovation that Linux represents today.

Please don't suggest that I am unappreciative or lack insight into the benefits to the world of strong maintainership - quite the reverse, Ubuntu has focused on ways to attract new kinds of maintainers to strengthen the pool of talent doing this, and I admire similar efforts elsewhere such as yours in Arch.

The point of snaps is not to disempower maintainers.

The point of snaps is not to substitute for the base operating system.

The point is to enable an ISV who DOES want to reach the widest range of users, and support their diverse choices of Linux, to do so in a tractable way. Nobody has suggested that snaps are the only way to do things going forward - don't misread headlines in the media as policy goals for me or anybody else, they are just ways to get attention to the article!

There are many apps that are explicitly packaged for Ubuntu and perhaps also another distribution, but not available for the long tail of distros that are interesting but don't have a lot of users. Snaps solve that problem. They are a useful addition to maintainership, they are not a threat to maintainership.


"With snappy Ubuntu, Owncloud can publish exactly what they want you to use as a snappy package, and can update that for you directly, in a safe transactional manner with full support for rolling back. I think upstream developers are going to love being in complete control of their app on snappy Ubuntu Core."

Not only will they love it, some are probably already cackling with glee at the prospects of "complete control".


Alternative Solutions

Patrick Burroughs (Celti) (2016-06-15-21-33-32-131)

Gotta chime in here as a sort of weird mutant hybrid of both worlds:

I'm a dedicated Arch Linux user, and started maintaining unofficial Arch Linux packages for a certain ISV's applications (Tiliado's Nuvola Player) — and somewhere along the line I'm now a member of that ISV for the official purpose of maintaining those packages.

I think there's an entire spectrum between the two ends, really, and the community policing doesn't stop just because the packages are provided by an ISV.

kk (2016-06-16-09-09-30-206)

Not at all strange. You are exactly what I hope will become more common, individuals who understand and care about both sides. Arch is a little unique in how much "maintainer-ish" work an unaffiliated person can accomplish however.


No, I'm not just going to carp about how change is bad. Here are some other ways to achieve the same goals.

Let's say your distro wants to attract developers. Suse's method has been to set up a comprehensive build service. Now the ISV doesn't have to maintain a distro-specific toolchain, the distro will do that for them. And if it builds on the build server, the distro promises that it'll run across their operating system. And Suse should be commended for supporting several distros beyond their own. Of course these services are not cheap. In exchange for hosting the service, the distribution retains their ability to curate. If an ISV acts in bad faith and refuses to meet the distro's standards, then the distro can de-list their packages. Suddenly the ISV needs to spin up a tool chain and hope that people will google them instead of using the integrated search and proceed to "sideload" their software.

Let's say you don't have the infrastructure for a giant build server, but you still want it to be easy for devs to get software into the distro. First and foremost, your distro should be easy to contact. Put in big letters on the home page something like "Want your software in the official repo? Send an email to this address and our maintainers will look at your proposal and get back to you soon."


Let's say you're an ISV and you want your software in a distro. You could learn how to make and release packages, but it is much less effort to cooperate with the distro and find a maintainer who will champion your software. And keep in mind this is a purely social solution to a supposedly technical problem. Supporting a dozen distros doesn't require learning a dozen build systems, just making a dozen friends at different distros. Note they don't even have to be an actual maintainer - any trustworthy person familiar with a distribution can do this job. Admittedly if you're an ISV who does proprietary software you are going to need some really good friends.

Samuel A. Falvo II (2016-06-16-05-37-45-426)

There's the crux and primary motivation for Snappy: proprietary software. If you want millions and millions of apps for your platform (which, for Canonical, might well be just for bragging rights), you don't want to impede the process with fickle maintainer negotiations.


Maybe finding friends and contacting distros is hard. Perhaps we need a matchmaking organization that can pair up potential maintainers with ISVs.

But let's presume you are an ISV who writes proprietary software. In this case, please don't release binaries. Give the maintainers source code and they'll do everything for you. You might think it is impossible to be proprietary and release sources, but you would be wrong. Microsoft has some brilliant lawyers who have drafted licenses which let you keep proprietary levels of control.

post_install() { echo $message; }


Maintainers, distributions and all the variety of "wasted duplicated effort" is the only reason the system has worked. Maintainers are the ambassadors between users and developers. Together we have formed a beautiful symbiosis with FOSS programmers because we could freely tweak, build and distribute their software. We will always disagree, do something about it, attract users, attract volunteers and formalize these groups as distributions. Putting packaging in the hands of those who write software (instead of those who use it) will inevitably lead to a proliferation of bad actors, as has been demonstrated in every other major operating system lacking strong community curation.

commenter (2016-06-16-02-08-38-435)

On the desktop though, the system has NOT worked. All of Linux hasn't.



Thanks to Ivy Foster for reviewing drafts of this.

commenter (2016-06-16-02-30-05-169)

One issue fully missing from this article is closed-source software. Maybe not important for Debian users, but most of us tend to not mind some forms. It, I feel, would be the primary user of snaps.

xav (2016-06-16-05-11-06-921)

Where do I +1 you ?



## post_post_install()

Wow! Around 15k unique visitors in the first 24 hours! A small thanks to Rackspace and a big thanks to static site generators for making that possible. Naturally with so many eyeballs there will be some feedback. Here are the /r/linux and the Hacker News threads.

Some thoughts about the most common reactions I've seen. For those who gave positive support and talked about technologies that give more power to users, thank you. I'm mostly going to focus on the negatives though. None of these are actual quotes, but amalgams of discontent.

> "This essay is extremely self serving and exaggerates the importance of maintainers."

Well yes. I did put the disclaimer at the top. You were warned.

> "That is nice and all but distros just don't have the manpower to do a good job, so I'll use ISV packages." > > "There's no distro package for anything I want to use, so I'll use ISV packages." > > "I don't want to be stuck on five year old software, so I'll use ISV packages." > > "I don't want to update my entire OS just to get a new version of one program, so I'll use ISV packages."

Turning to an ISV package as a primary solution to distro problems is not the answer. Another wrong answer is the change distros because the grass is greener. If you are not happy with your distro the solution is always the same: become more active in it. All of the aggressively-self-correcting mechanisms are powered by disappointment. Channel yours productively. Here is the general distro-agnostic sequence:

1. Learn the packaging tools. Your distro has made all of these available and well documented. Start by simply re-building the official packages, then customizing the official packages, building/customizing unofficial packages, and finally making new packages for software not in the distro. 2. Share your new packages. Possibly your custom ones too. 3. Actively work to improve your distro as much as you can. Write and submit patches, mostly. Fix bugs, both distro and upstream. Create tools that make the other points on this list easier/faster/safer. 4. After doing this for a few years, you will either want to become a maintainer (and people will want you to become one) or you will have decided it just isn't the distro for you.

As a side note, switching to a rolling distribution was the most common suggestion. While on average this means you get new software earlier it does have its own headaches. (The "update my entire OS" complaint.) I suspect the best solution for release-based OSes is for semi-official 3rd party repositories. These would be run by actual OS maintainers (not random folk in a user repo) and simply offer more recent versions of in-demand pieces of software. Obviously they would not be held to the same reliability standards as the rest of the release, but they should have the same level of safety and trustworthiness. I would not be surprised if many of these already exist but are not well publicized.

> "No one is saying that all packages should be replaced with universal packages from upstream."

Actually people have said that. I started assembling the draft of this essay three years ago. That was around the time containerizing stuff was just taking off. There were numerous articles with clickbait titles about how we wouldn't need distributions any more. And numerous people appeared to non-sarcastically support the idea. It is worth noting that Mark Shuttleworth also does not agree with clickbait headlines: "Nobody has suggested that snaps are the only way to do things going forward - don't misread headlines in the media as policy goals for me or anybody else, they are just ways to get attention to the article!" And conversely I feel the universal package technology is worth having and using. If nothing else, competition and variety are both good.

> "Nix or Guix would solve all of this."

Your are absolutely correct! A fun project would be to figure out how to treat a distro repository as a giant collection of binary-only packages. Import the entire distribution and you instantly get a mostly familiar system but with all the futuristic transactional technology. Binary packages kind of cut across the grain of these tools, so I suspect this will not be easy.

> "Here's a bunch of examples where $distro messed up."

Well yes, I did say we are still human. It is worth reading through the comment threads for specifics but I had expected people to come up with better examples than OpenSSL and the Ubuntu Lens. I was particularly amused by the person who suggested that it was a failure of maintainers that Heartbleed was detected by professionals at a security firm first. They are the professionals, I would hope they are better at it. Our job in security is to make sure 100% of installations are patched as fast as possible.

Ubuntu and Debian were mentioned the most. I suspect this is not because they are worse, but because they are popular.

Where it becomes tricky is when the distro is also an ISV. Most distributions hate writing software, even when it is seen as a pillar of their existence. When I started using Arch, /etc/rc.local was the reason to use Arch. We dropped it faster than you could say "good riddance" after a replacement we liked came around.

But some distros are in the business of becoming more and more of an ISV. Or occasionally an ISV will decide they need to have a distro. This leads to conflicts of interest and ethically questionable monetization decisions that never go over well with the community. Mint seems to be the counterexample of a good ISV/distro hybrid. They somehow manage to be an ethical upstream developer, listen to people and collect donations without making anyone too upset.

> "Linux has mainstream malware and ad-ware."

Okay, no one actually said this directly. But they could have. The Opera browser for many years shipped a free banner-ad supported version. I even used it, because it was just so fast on a Pentium 2. And even after Opera did away with the ads, they had a license that made redistribution on mirrors (let alone modification) impossible. I was going to use this as an example, but it was more than a decade ago.

And finally, my apologies to Mr. Shuttleworth. He commented here and even personally emailed me within 12 hours of this going up. I tried very hard to keep the essay completely agnostic with regards to specific technologies, but he had the first pull quote I found and was the only person I named explicitly. This was lazy of me - I should have found similar quotes from all the major platforms.