Unpopular Opinion Penguin #1: Sweet and Sour
I’m starting this section on my blog called Unpopular Opinion Penguin, where I talk about, well, unpopular opinions. There are a couple of things that I want to talk about, and this is the first one.
When ZNX was announced and made available, we never thought that it would cause the amount of undeserved backlash that it received. We were excited to bring something new to the table, but unfortunately, what we found is that the world was not ready for it.
Today, I’m going to talk about that.
Origins
When we set out to create Nitrux (in its 2017 incarnation), we wanted to create something different, something that was indeed ours. We had a rough idea of how we wanted our system to be available on as many form factors as possible. We didn’t want to be just another distribution of the bunch. So we brainstormed ideas that we wanted our Linux distribution to have, one of these ideas was to structure the distribution such a way that it the idea of a package manager was not an integral part of it. We had just recently started to get involved with AppImages, and we thought how would an operating system work if it followed the same concept.
It was the 26th of March 2018, and we published a short article about a new tool that we were working on called “ZNX.” In this article, we compared it to libostree, the technology developed by Red Hat, Inc. that powers OSTree and Flatpak, and what was possibly the closest piece of software in use that was similar to what we wanted to do that was available in the Linux desktop at the time.
With ZNX, we wanted to provide the benefits of libostree. Still, without the complexity of using it and without the possibility of our solution to depend on Systemd, we wanted a more straightforward program. At the moment, ZNX was still a concept. It was until later that year we finally started using it.
By the 19th of April 2018, we published a second short article describing the features of ZNX. We were thrilled that in such a short amount of time, a functional prototype was already available for testing. In the article, its author describes it as “znx follows the UNIX philosophy guidelines. It’s a simple tool that does one thing and does it well: it manages several distributions within a single storage device.” thus, it was clear, ZNX manages Linux distributions, it does not install them.
Seemingly, this description was the first thing that confused people; if it’s not an installer, what is it then?.
Continuing with the description of ZNX, the article reads, “[…]It’s written in the POSIX shell script, which makes it easy to get it working.” How could something as significant in its scope as this be written in a couple of months by a small team of software developers and using the shell script language?. One word, simplicity.
We fast-forwarded to two months to the 19th of June 2018 when we published another short article with more information about how ZNX works. In it, we shed more light on how ZNX is using existing tools to perform its tasks.
And, even though ZNX only required three programs (sgdisks, GRUB, and zsync), it’s the title that very likely causes the most controversy, possibly upsetting more than one person. It was a bold claim, but it wasn’t a lie. ZNX was providing the management it advertised, users could add, remove, update and downgrade Linux distributions, using a single tool, independent of their package manager or init system, but it wasn’t complete yet.
On the 20th of July 2018, we published a short article about what was a missing feature, data persistency. By this point, no one could have blamed users for thinking that ZNX was a program that was similar to YUBI, WUBI, MultiBoot USB, Linux Live USB, Universal USB Installer or the recently released Ventoy.
And because of that, we published an article explaining why that wasn’t the case since ZNX didn’t save user data to a file, but to the device itself.
With znx you get more than just putting an ISO image on a storage device.[…]It’s a more elaborated (though simple) and fine tool.
On the 12th of September 2018, after we attended Akademy 2018, we published a short article with instructions on how to start using ZNX. With this article, it was the first time that we used the word deploy in the context of ZNX to refer to the addition of ISO files through the ZNX mechanism, and this was seemingly the second thing that confused people; so it “deploys” Linux distributions? So is it a container?.
An assertion to which the answer is, no, it’s not a container. A container runs in userspace on top of an operating system kernel (It means you can run different Linux systems (containers) on a single host. For example, you can run an RHEL and a SUSE container on an Ubuntu server. The Ubuntu Server can be a virtual machine or a physical host.), and of course, ZNX does not do this.
So it’s not an installer, and it’s not a container, then what is it?.
We followed the previous article with another describing how to update and remove the deployed systems using ZNX. With the functionality of ZNX in place by this moment, we were ready to make the transition to use it; we decided that it would be a good idea to upload an ISO that used ZNX by default, and doing that meant, no traditional installer.
On the 6th of October 2018, we published an article where we announced a pre-release ISO. In it, we make the statement that “znx follows the concept of AppImages, which is that software isn’t “installed,” but it is instead deployed.”
And the article proceeds to explain why that is the case.
[…]instead of unpacking the squashfs file (the compressed filesystem that holds the actual OS) and extracting the contents to the target device (thus “installing” the OS the traditional way). The OS remains as one file which can be updated using a zsync file for differential updates
And yet, what stood out the most was that only EFI/UEFI computers were supported, ZNX was not going to work with MBR and Legacy BIOS hardware, something that, once again, outraged more than one person.
Then, the day finally comes, on the 27th of October 2018, we finally start using ZNX on our official releases when we published the announcement of version 1.1.0 of Nitrux.
Little did we know what was coming.
Post-release mayhem
It was the 30th of October 2018 when we published an article explaining why the newest release of Nitrux wouldn’t work on MBR drives and Legacy BIOS computers. However, this did little to clear up the ensuing confusion. Suddenly we were overwhelmed with comments about how the ISO didn’t work, how ZNX didn’t work, how users couldn’t install the new ISO, etc.
These reactions prompted us to improve our documentation, or rather, create documentation and continue to strengthen ZNX.
Throughout 2019 the source code of ZNX was refactored and improved. We thought it would be a good idea to create another article comparing the workflow of ZNX with that of a traditional installer. And on the 28th of January 2019, we published a new piece describing that workflow.
On the 18th of March 2019, Distrowatch mentioned ZNX on it’s Weekly Issue, something that at the moment we felt was a sweet moment of victory, but that quickly turned sour (there’s more about this topic, but that would be for another occasion).
While we expected that with these articles, it would be more precise what ZNX was about, it had almost zero impact on people’s perception. The majority of people downloading Nitrux would still ask where the installer was. One thing was sure, ZNX was so far ahead that the idea seemed almost incomprehensible for people that had not been following its development.
As a last-ditch effort, on the 19 of August 2019, we published a new article detailing how ZNX works, with the idea being that it was clear as possible without falling too much into tech-talk. Once again, it had zero impact on clearing up whatever misconceptions people had about it.
To the vast majority, however, ZNX simply did not work at all because it did not behave like an installer, no matter how much we tried to explain it.
On the 17th of December 2019, I was interviewed by the popular Linux website It’s FOSS. In the interview where I talk about the history of Nitrux, I mention the intended idea and workflow of ZNX. And this, once again, did nothing and possibly angered even more people for whatever reason.
I believe one reason as to why ZNX hasn’t become more popular is that because everyone is so used to the concept of package managers and installers that ZNX is alien to them.
I was also starting to think that people would hate whatever we did to amend ZNX perception regardless.
In this interview, I also make a piece of information available, and that was that for months we had a problem with a program that we use in our tooling and that it was causing an issue with our ISO files. However, even though I explained what the problem was, everyone attributed this issue of the ISO files unfairly to ZNX.
We had already mentioned this in our social media, but as it seems people didn’t notice or care at this point, they would just blame ZNX.
The / cause of the problem
I’m leaving the best to the end, but for this part, I’m focusing on what we did wrong.
The very first thing that I would say that we did wrong was not to have the appropriate documentation when ZNX was released. I mean, there was a wiki, but it didn’t contain a lot of information. We later improved that wiki and also created more documentation such as the Compendium and the FAQ, which were available on the Nitrux website. But it possibly was too late.
The second thing that we did wrong was not to do a better job finding the problem with the ISO files to prevent people from blaming ZNX for something that it had nothing to do with it. When we finally found and fixed the problem, the damage was done, people had already put the blame entirely on ZNX, and we couldn’t reverse that.
The third thing that we did wrong was making and releasing ZNX GUI. During the development of ZNX, we thought that because it is a CLI program, many users wouldn’t want to use it, and so we made a GUI for it. That idea was not wrong in itself, the problem came when we picked the absolute wrong tool for the job, and instead of using a proper GUI framework, we utilized KDialog.
KDialog is described as “kdialog allows you to display dialog boxes from shell scripts.” We figured that’s perfect, the ZNX source code is in the shell script language, so this makes sense. But then, KDialog is meant for a minimal scope of use cases, that is, well, displaying dialog windows (and notifications). And therefore, ZNX GUI made extensive use of dialog windows, which doesn’t sound like a problem at first, and it didn’t mean that ZNX GUI didn’t work, but, that ZNX GUI was an absolute chore to use.
To put an example, to everyone using Nitrux (at this time), ZNX GUI was their introduction to ZNX, to deploy an ISO using ZNX, it would take precisely two steps (init and deploy), but, with ZNX GUI it would take eight steps to deploy an image. Another example is, ZNX does have verbose output, it does display informative success and error messages, it shows help content, but ZNX GUI did none of that.
ZNX GUI was a mistake, and it did more bad than good to ZNX. ZNX GUI was not ZNX, but because it was the only graphical thing that said ZNX, the association was apparent. To the people that used it, ZNX GUI was terrible, and therefore, ZNX was awful. People didn’t care that it was made with KDialog, or that even in its latest incarnation it was more sensible (albeit still tedious to use because of KDialog’s scope and inherent limitations), people also didn’t care that ZNX could be executed from the Terminal, people simply blamed ZNX.
The fourth thing that we did wrong was to introduce new terminology, such as deployments. When we implemented ZNX, we introduced new words in our vocabulary, and the first was initialization, which meant in the context of ZNX (automatically) formatting a storage device, creating the partitions on the device, and adding the labels to these partitions. Then deployments, which meant creating a directory structure with the name of the system that was entered by the user and downloading or copying an ISO file and moving it to this directory structure.
Using the word install even if it would’ve been factually wrong could’ve saved us many problems.
The fifth thing that we did wrong was to not package the ZNX AppImage appropriately when it was released. Upon release, we were providing users of an AppImage to start using ZNX, however, that AppImage had a major flaw in that it was not utilizing the utilities provided by util-linux that were included in the AppImage, prompting many people to once again, put the blame on ZNX because to their eyes it wasn’t that the AppImage had a problem, to them ZNX didn’t work.
Eventually, we resolved the problems, but alas, it was too little too late.
Unpopular Opinion
Now comes the best bit.
Even though by mid to late 2019, we had pretty extensive information about ZNX, we made the erroneous assumption that people would read it, as it turns out people would completely ignore it, we added a big, bright, blue button to the website and a chunky block of text right above the download button that redirected to the Compendium. Still, even then, that did not work at all.
Primarily this was caused by the fact that we made the mention that we created ISO files from Ubuntu sources. So people would download them and would think that it was, well, Ubuntu, that it worked like Ubuntu, that it would ‘install’ like Ubuntu, suffice to say that was wrong.
Another thing is that many blogs and sites that reported on Nitrux did not mention ZNX or the Compendium and the FAQ and ended up contributing to the already ensuing confusion about what ZNX was, why there was no (traditional) installer, why ZNX did things the way that it did, what was its purpose, etc.
One such site, being Distrowatch, which does link the Compendium but not the FAQ (both removed for obvious reasons) but that it never mentioned ZNX in the description.
Then, I mean, for over 25-ish years, Linux distributions have been using installers to have users install them to their computers, imagine trying to change that in the span of a little over two years, yeah that did not sit well with many people.
In late 2019 the package manager (dpkg and APT) was removed from the released ISO files, which led everyone to think that ZNX (somehow) prevented the use of a package manager. Quite frankly, at this moment, I was strongly confused as to why people had come to that conclusion. It was as if every previous version that did have the package manager and that was using ZNX suddenly never happened.
The removal of the package manager simply was because of various reasons.
The first was that because the point was to use AppImages to add and remove end-user applications, it would’ve made a package manager redundant for that use case. The second was that because ZNX was meant to be used to manage (add, remove, upgrade, downgrade, rollback) the system, having a package manager do that would’ve also been redundant.
The third was that we had added Homebrew (a package manager that is very similar to Pacman) and were working on PNX (basically Pacman, which I do like), both of which installed software to the Home directory and not the root, the point being that the Root was immutable with the exceptions of the directories enabled in the Overlay. That would also mean that (when available), other package managers like Nix (which I also like) could’ve been available to let users utilize those package managers and still not break the system.
Finally, the idea of using a new FHS (mostly) for aesthetic purposes but also for restructuring how the system is put together, the latter would be something for the far future. Something similar to Google’s Fuscia.
In the beginning, I mention libostree, OSTree, and Flatpak but, what I didn’t mention was that there are two distributions that already apply the very same concept we pursued, and they’re Fedora Silverblue and Endless OS. As a matter of fact, Fedora Silverblue is pretty much the closest. It’s an immutable system, with delta updates, rollbacks, by using rpm-ostree and uses Flatpak to deploy end-user software. Obviously, Fedora Silverblue has existed for many years prior to Nitrux, but the point is that the concept was already in use.
Well, technically, there’s Kubic from openSUSE, but Silverblue and Endless are meant for the desktop, and Kubic isn’t.
And I bring this up because perhaps surprisingly (to me anyway) I do not see the same amount of backlash (and on occasions hate) that ZNX gathered. Generally, it’s quite the contrary, garnering praise and hype.
It sure isn’t easy trying to change the status quo as an underdog.
What’s ironic is that just some days ago, I found out about the existence of frzr (I think it’s pronounced as frozer) and what was my surprise when I read the README, “frzr is a deployment and automatic update mechanism for operating systems. It deploys pre-built systems via read-only btrfs subvolumes, thus ensuring safe and atomic updates that never interrupt the user.” and it’s part of the GamerOS distribution.
I mention it not because of its similarity to ZNX but because I swear to god that it will not be as unfairly treated as ZNX was. For the most part, because the distribution itself is based on Arch Linux which, from the looks of it guarantees some kind of leverage or protection when it comes to people disliking something. Unlike, you know, if you used Ubuntu, as we did.
To quote Alanis Morrisette in her 1996 hit single.
“And isn’t it ironic… don’t you think.
A little too ironic… and, yeah, I really do think… “
Oh, well.
Conclusion
ZNX was a part of a greater plan for the future, but due to the unfair, and undeserved amount of backlash, well, it’s a future that slowly fades away.
So, there you have it, an unpopular opinion.