April, 3, 2024 archives

A popular response to the attempted backdooring of the XZ Utils has been people like Tim Bray talking about the maintenance of open source projects and how to pay for them.

When I transitioned from leading the web development team at MySQL to an engineering position in the server team, I spent the first year as a maintenance engineer. I blogged a little about the results of that one year and calculated that I had fixed approximately one reported bug per working day.

But you’ll also notice that I had to heap some praise on Sergei Golubchik who reviewed fixes for even more bugs than I had fixed. (He also was responsible for working on new features. He is extremely talented, and I’m not surprised to see he’s the chief architect at MariaDB.)

That sort of reviewing and pulling in patches is a critical component of maintaining an open source project, and a big problem is that is not all that fun. Writing code? Fun. Fixing bugs? Often fun. Reviewing changes, merging them in, and making releases? A lot less fun. (Building tools to do that? More fun, and can sidetrack people from doing the less-fun part.)

It is also a lot different for projects with a lot of developers, a small crowd of developers, and just a few developers. The process that a patch goes through to make it into the Linux kernel doesn’t necessarily scale down to a project with just a few part-time developers, and vice versa. A long time ago, I made some noise about how MySQL might want to adopt something that looked more like the Linux kernel system of pulling up changes rather than what was the existing system of many developers pushing into the main tree, and nobody seemed very interested.

Anyway, as people think about creating ways of paying people to maintain open source software, I think it is very important to make sure they don’t inadvertently create a system that bullies existing open source project maintainers to make them focus on the less-fun aspects to developing software, because that’s kind of how we got into this latest mess.

You already see that happening with supposed-to-be-helpful supply chain tools demanding that projects jump through hoops to be certified, or packaging tools trying to push their build configuration into projects (with an extra layer of crypto nonsense), or a $3 trillion dollar company demanding a “high priority” bug fix from volunteers.

I am curious to see where these discussions lead, because there is certainly not one easy solution that is going to work everywhere. It will also be interesting to see how quickly they lose steam as we get some distance from the XZ Utils backdoor experience.

(Also, I’m still looking for work, and I’m willing to do the less-fun stuff if the pay is right.)

I have been thinking more about bullying in the open source community being a security problem and how part of the process of getting the XZ backdoor into play was playing into the mental health struggles of the original maintainer.

It reminded me of a flawed system that I found having a large impact on my mental health while running our store. Like any diligent small business, we claimed our listings on Yelp and Google and the other major spots where people leave reviews.

I can still feel my stomach dropping when I would get a notification from Yelp that said someone had left a review. Thosee notifications didn’t give the single most critical bit of information: how many stars. Was it going to be a nasty one-star review, or was it someone leaving a wonderful five-star review? No clue! First you have to log in to the Yelp for Business website to see how the rest of your day would feel.

It was the “we need to talk later today” message from your boss that anyone could lob at me at any moment, maybe without even knowing they were doing it.

This was also brought to mind when I read about Daniel Stenberg’s experience with AI-generated security bug reports for curl and besides the aggravation of the reports being nonsense, I could only imagine that adrenaline spike when the report first comes in.

And bringing this back to something I linked to yesterday, where someone from Microsoft reported an issue with ffmpeg as “high priority”. Again, I can imagine one of the maintainers reading this request and feeling that adrenaline spike, that anxiety trigger.

It turns out that it wasn’t even a bug in the project, the reporter just didn’t know the right command line options to use. (I will also point out that they also said they were going to provide updates on whether the free support they received worked, but then they never did.)

Maybe the ffpmeg maintainers and Daniel are good at shrugging this stuff off. I thought I was, but one of the lessons that I learned from running our store is that the effects are cumulative even when individually small, and it is a good idea to figure out how to combat it.

And while there won’t be any simple technical solutions to these human problems in open source, it is very important to make sure whatever tools are being built don’t create these same sorts of anxiety spikes. We need to make sure to build kindness into the tools, whatever they may be.