Maintainers Matter

Disclaimers, et al. These opinions are not at all related to Arch Linux, though maintaining Arch packages has certainly colored my opinions. I've been a volunteer maintainer for years now, because I feel it helps people. I have a bias towards the value that maintainers and maintaining packages provide.

Normally I really like the word 'upstream'. But it has a lot of connotations and doesn't suit the tone of this essay. To define it, upstream is generally used as a collective pronoun to mean "the group of people who gives us software". They are people who fix bugs, add features and occasionally make mistakes. They have the best interests of their users and their software at heart. They are good people who always mean well and take pride in their work. Upstream is not a single group, but spread among everyone who writes software.

"Upstream", however seems to be a mostly-FOSS concept, largely a side effect of the selective pressures that distributions provide. Since this essay goes into what a weakening of distributions would look like I'm going to use the original, older, uglier, more agnostic term "Independent Software Vendor" or ISV.

Why say ISV? Why does this word matter? "Upstream" is used both collectively for FOSS as a whole and singularly when talking about a single project. This re-use of the word is a small flaw as it suggests all software developers are fundamentally the same. This is not the case: some have very different values. Being "upstream" also has an implicit social contract attached to it. The "I", for independent, is to remind us that this is not always the case. And the "V" is for vendor, a reminder that sometimes these are businesses who consider profitability to overrule all other considerations.

Obviously this does a disservice to those who volunteer their time and intelligence without compensation, and do consider software quality and user happiness to be important. For the purposes of this essay I'm painting everyone with the "ISV" brush, even those pillars of the community who deserve better. But we are heading towards a future where the nice guys don't win.

Let us start with the (recent) past that kicked off the whole idea of "ISVs can package stuff" and "maybe we don't need maintainers".

The Cargo Cult of App

The reality: App Stores have been a nearly unmitigated disaster for users. Supposedly one of Linux's failings is that there is too much pointless choice, too many K and G versions of things and it divides developer efforts. Why have so many window managers and text editors? App Stores have the same problem. With the traditional FOSS model, there are a hundred different programs and each program is missing a different feature. In the App Store, there are a hundred different programs doing the same thing but each screws the user over in a slightly different way. Spying? Ads? Battery sucking rookie mistakes? Battery sucking bitcoin mining botfarm? Take your pick.

The promise: But Linux is different! No one does those horrible things on Linux! And no one ever will!

Maintainers are the Difference

Linux-related news might suggest maintainers are useless. Either they are bikeshedding on mailing lists or closing bugs as WONTFIX, or introducing security vulnerabilities in software they don't understand, or generally being a BOFH. Of course this is wrong - we simply want to use the best operating system possible. So what does a maintainer actually do?

Mostly a maintainer is a packager: they take software from an ISV and repackage it for a distribution's packaging system. And here is where some people feel is a huge waste. Why package the same software a hundred times for a hundred different distributions? (There is good reason for this "waste", but let me finish describing everything else a maintainer does.)

More fundamentally, the maintainer is the primary line of defence and interaction between users and developers. Maintainers shield developers from uninformed users, allowing the devs to write software with less support overhead. Non-bugs are caught and filtered out. Low-quality bugs reported to the distribution's tracker often becomes a good bug when the maintainer reports it upstream. Without distros the ISV would have to carry the full weight of support. Here's an example:

<NewUser> I've never gotten OpenSCAD to work on my computer, so I don't know anything particular to it.
<@teepee> NewUser: we could help with that (hopefully) :)
<NewUser> I'm gonna upgrade my system software when Slackware proper is RTMed. We'll see then.
<@teepee> ahh, cool, slackware is still around
* teepee used that like 15 years ago
<NewUser> Not a lot. I'm also installing Arch.
<@teepee> ahh, with arch, that's easy, you just complain to keenerd if it's not working
<keenerd> teepee: Spot on :-)
<@teepee> :)
<@teepee> it's always good to know there's someone who cares

Maintainers also shield users from developers, offering a layer of quality control. At the very least, all the software in a distribution should play friendly with any other piece of software in the repositories. Maintainers are responsible for that, as well as choosing a subset of the (subjectively) best software FOSS has to offer. Maintainers will disable features that they feel act in bad faith. Maintainers' greatest power is the ability to outright say "This is not good enough for our users" and consequently punish an ISV by either patching out the offensive part or in extreme cases removing the software from the repositories. ISVs know this and so don't act out. After 20 years of enforced good behavior this has lead to the idea of ISVs as "the benevolent upstream developer." This is why Linux doesn't have spyware, doesn't come with browser toolbars, doesn't bundle limited trials, doesn't nag you to purchase and doesn't pummel you with advertising.

Maintainers Provide Consistency

For the sake of argument, let us presume that all ISVs are ethical and hold the interests of the community as their utmost ideal. Would this utopia still require maintainers to provide quality control? Yes, because no one can agree on what "quality" is beyond "does it build and run". Any good distribution is not about the file extension on the package - it is about the decisions the software implicitly makes. A distribution is a world view provided by the maintainers and shared by the users. In other words, maintainers don't just ship packages. They provide a certain flavor of consistent and preferred opinions to a group of users. (As an aside, this is why users of different distros often don't see eye-to-eye.)

What is an ISV to do? They can try to package half a dozen different flavors of their software. Now a hypothetical cross-distro Universal Packaging Format is not saving them any effort because they need to make (and test, and support) a different build target for each demographic of users. Or the ISV can ignore some users, making a subset of them unhappy.

If people are unhappy enough they will repackage the software in a more suitable manner. Now the users have unwittingly become maintainers. The moment someone starts improving 3rd party software, they become a maintainer.

And if a maintainer shares their packages, they'll attract like-minded users who appreciate the work as well as like-minded maintainers who can share the workload. And thus a distribution is born. It can't be fought. A monoculture will breed diversity.

Myths

To clarify a few points often heard:

• The promise: Sandboxing makes you immune to bad ISVs.
• The reality: It protects you from some things and not others. App Stores have proven that ISVs will try to get away with as much abusive behavior as possible. Sandboxing protects against many threats, but does not suddenly make people ethical. Distributions with human maintainers are more likely to make mistakes than an OS-enforced sandbox, but human maintainers are also more likely to sniff out the dumb stunts that ISVs try to pull. No human maintainer is ever going to willingly package a flashlight app which phones home. From source anyway. I imagine sandboxing will be used as an additional layer of safety in most distros (eventually) but mostly for the purposes of protecting from honest mistakes.

• The promise: By cutting out the maintainer middle-man, security fixes can be pushed out to users faster.
• The reality: Often maintainers are faster, particularly with anything using shared libraries. A distro can update a shared library once and all software is now safe, without needing to rebuild every piece of software using it. Most of the sandboxes do not share libraries and so you are waiting on the slowest ISV to rebuild their software. If they ever do. ISVs are notorious for using ancient libraries with extensive out-of-tree patches and never migrating the patches forward.

• The promise: A common packaging effort will improve the quality of software on Linux, because packaging for every distro is too hard.
• The reality: Handwriting is too hard. I have created a solution, the "keyboard". Now people will no longer write dumb things because the hard part is solved. Making packaging "easy" will not improve the quality. Just like how legible writing has no relation on writing intelligently, the mechanics of packaging has no relation on the quality of software. Write good software, release sources, and every distro will package it for you.

• The promise: ISVs releasing packages means less duplicated work between distros.
• The reality: Distros exist because users can't agree on how exactly we want software. A universal package will not be good enough for some percentage of them. The actual act of packaging is the fastest part. The slowest part (fixing bugs that upstream hasn't) is shared between distributions because all of our patches are public.

• The promise: Many embarrassing bugs will be avoided, because ISVs are smarter and more experienced than maintainers.
• The reality: Oh boy the stories I could tell about the things I've seen. But out of courtesy, I won't name names. Needless to say they are as dumb and human as we maintainers are. If you want, think of maintainers as outside consultants who will find the craziest thing an organization is doing. Except we don't charge $200/hour for what amounts to best-practice common sense.

• The promise: Software directly from the ISV has a complete chain of trust and is better for users.
• The reality: This is only true if the ISV won't try to sneak something in. You don't need to have blind trust in your maintainers, but don't get rid of them either. Instead, verify their work with your own builds and examine the chain of trust the distribution provides. Distros are usually way more open about the chain of trust than ISVs are.

Yes, all of this will happen.

I don't mean to pick on Shuttleworth, but he has explicitly stated that the whole point of their new universal packages is to give control to the ISVs. If you believe "control" is a finite resource then giving implies taking from elsewhere. Most likely from the maintainers and users. To quote:

Not only will they love it, some are probably already cackling with glee at the prospects of "complete control".

Alternative Solutions

No, I'm not just going to carp about how change is bad. Here are some other ways to achieve the same goals.

Let's say your distro wants to attract developers. Suse's method has been to set up a comprehensive build service. Now the ISV doesn't have to maintain a distro-specific toolchain, the distro will do that for them. And if it builds on the build server, the distro promises that it'll run across their operating system. And Suse should be commended for supporting several distros beyond their own. Of course these services are not cheap. In exchange for hosting the service, the distribution retains their ability to curate. If an ISV acts in bad faith and refuses to meet the distro's standards, then the distro can de-list their packages. Suddenly the ISV needs to spin up a tool chain and hope that people will google them instead of using the integrated search and proceed to "sideload" their software.

Let's say you don't have the infrastructure for a giant build server, but you still want it to be easy for devs to get software into the distro. First and foremost, your distro should be easy to contact. Put in big letters on the home page something like "Want your software in the official repo? Send an email to this address and our maintainers will look at your proposal and get back to you soon."

Let's say you're an ISV and you want your software in a distro. You could learn how to make and release packages, but it is much less effort to cooperate with the distro and find a maintainer who will champion your software. And keep in mind this is a purely social solution to a supposedly technical problem. Supporting a dozen distros doesn't require learning a dozen build systems, just making a dozen friends at different distros. Note they don't even have to be an actual maintainer - any trustworthy person familiar with a distribution can do this job. Admittedly if you're an ISV who does proprietary software you are going to need some really good friends.

But let's presume you are an ISV who writes proprietary software. In this case, please don't release binaries. Give the maintainers source code and they'll do everything for you. You might think it is impossible to be proprietary and release sources, but you would be wrong. Microsoft has some brilliant lawyers who have drafted licenses which let you keep proprietary levels of control.

post_install() { echo $message; }

Maintainers, distributions and all the variety of "wasted duplicated effort" is the only reason the system has worked. Maintainers are the ambassadors between users and developers. Together we have formed a beautiful symbiosis with FOSS programmers because we could freely tweak, build and distribute their software. We will always disagree, do something about it, attract users, attract volunteers and formalize these groups as distributions. Putting packaging in the hands of those who write software (instead of those who use it) will inevitably lead to a proliferation of bad actors, as has been demonstrated in every other major operating system lacking strong community curation.

Thanks to Ivy Foster for reviewing drafts of this.

post_post_install()

Some thoughts about the most common reactions I've seen. For those who gave positive support and talked about technologies that give more power to users, thank you. I'm mostly going to focus on the negatives though. None of these are actual quotes, but amalgams of discontent.

"This essay is extremely self serving and exaggerates the importance of maintainers."

Well yes. I did put the disclaimer at the top. You were warned.

As a side note, switching to a rolling distribution was the most common suggestion. While on average this means you get new software earlier it does have its own headaches. (The "update my entire OS" complaint.) I suspect the best solution for release-based OSes is for semi-official 3rd party repositories. These would be run by actual OS maintainers (not random folk in a user repo) and simply offer more recent versions of in-demand pieces of software. Obviously they would not be held to the same reliability standards as the rest of the release, but they should have the same level of safety and trustworthiness. I would not be surprised if many of these already exist but are not well publicized.

"No one is saying that all packages should be replaced with universal packages from upstream."

Actually people have said that. I started assembling the draft of this essay three years ago. That was around the time containerizing stuff was just taking off. There were numerous articles with clickbait titles about how we wouldn't need distributions any more. And numerous people appeared to non-sarcastically support the idea. It is worth noting that Mark Shuttleworth also does not agree with clickbait headlines: "Nobody has suggested that snaps are the only way to do things going forward - don't misread headlines in the media as policy goals for me or anybody else, they are just ways to get attention to the article!" And conversely I feel the universal package technology is worth having and using. If nothing else, competition and variety are both good.

"Nix or Guix would solve all of this."

"Here's a bunch of examples where $distro messed up."

Well yes, I did say we are still human. It is worth reading through the comment threads for specifics but I had expected people to come up with better examples than OpenSSL and the Ubuntu Lens. I was particularly amused by the person who suggested that it was a failure of maintainers that Heartbleed was detected by professionals at a security firm first. They are the professionals, I would hope they are better at it. Our job in security is to make sure 100% of installations are patched as fast as possible.

Ubuntu and Debian were mentioned the most. I suspect this is not because they are worse, but because they are popular.

Where it becomes tricky is when the distro is also an ISV. Most distributions hate writing software, even when it is seen as a pillar of their existence. When I started using Arch, /etc/rc.local was the reason to use Arch. We dropped it faster than you could say "good riddance" after a replacement we liked came around.

But some distros are in the business of becoming more and more of an ISV. Or occasionally an ISV will decide they need to have a distro. This leads to conflicts of interest and ethically questionable monetization decisions that never go over well with the community. Mint seems to be the counterexample of a good ISV/distro hybrid. They somehow manage to be an ethical upstream developer, listen to people and collect donations without making anyone too upset.

"Linux has mainstream malware and ad-ware."

Okay, no one actually said this directly. But they could have. The Opera browser for many years shipped a free banner-ad supported version. I even used it, because it was just so fast on a Pentium 2. And even after Opera did away with the ads, they had a license that made redistribution on mirrors (let alone modification) impossible. I was going to use this as an example, but it was more than a decade ago.