Docker, Jails and All That Jazz

Different means of isolating applications and processes existed for decades, starting from chroot, through FreeBSD jails and Solaris zones to the most recent invention – Docker. The point is, all of those technologies serve slightly different purposes, despite the fact that people insist on arranging them into a linear, progressing hierarchy. For instance, chroots (or chroot jails) are meant to restrict the trapped user to a sub-directory, ideally isolated from the root of the host filesystem. It is a common, lightweight approach to isolate file access in FTP server daemons since it is fairly easy to implement. However, it lacks certain abstractions available in more advanced tools like jails and zones. So far, I have used chroots, Docker and FreeBSD jails, and I would like to share some key observations I made regarding these technologies.

Traditional chroot jails are extremely basic, nevertheless useful partially for that very reason. If all you want is a nested directory hierarchy for building software packages or similar purposes, chroots are good enough. Unfortunately, they do not prevent secondary access to the host environment via another chroot, neither do they limit resource consumption by the chroot’ed application. Therefore, care should be taken when using chroots in an Internet-fronting environment. The FTP context is moderately safe since a chroot-trapped user is limited by the FTP protocol. In general, chroots are most useful when they constitute a subsystem in a larger, more complex tool.

FreeBSD jails were developed as an improvement to the venerable UNIX chroot technology. In practice, they contain a complete FreeBSD userland, fully transparent to the host system, yet restricted by the host system kernel from within. From the outside they’re no different than an unpacked system tarball or disc image. The great thing about jails, though, is that they’re an integral part of FreeBSD and as such can be easily managed in a multitude of ways. Resource handling? Easy via rctl. Installing packages? The pkg package manager has a “-j” flag specifically for that purpose. What about enabling and starting/stopping daemons? Again, service has a “-j” flag and so does sysrc to control lines in /etc/rc.conf. The only tricky bit is network management, but this too can be easily scripted due to the modular nature of FreeBSD. The main issue with jails is that they’re a FreeBSD-only technology. This unfortunately restricts adoption, but so would any other technology requiring an intimate relationship with the operating system.

Docker is a lot more modern and similarly to FreeBSD jails, it relies on features specific to a single platform, namely Linux. It leverages kernel cgroups for privilege separation and a virtual filesystem (overlay2, previously aufs) for managing the underlying disk volume. Compared to FreeBSD jails, Docker containers are a lot more abstracted and communication is typically carried out through the docker daemon. In the case of docker volumes, I find it slightly distressing that the only access to the data is via the daemon. Another thing that differentiates it from FreeBSD jails is the default access level. The root user in Docker containers has absolute control over the data in the container, and on top of it can perform network-related actions which should be reserved to the host system. The biggest issue however arises when mounting host directories in the container. An unprivileged user in the container has root access to host directories. Frankly, FreeBSD jails entail similar risks and therefore the official announcement in the Handbook:

Important:

Jails are a powerful tool, but they are not a security panacea. While it is not possible for a jailed process to break out on its own, there are several ways in which an unprivileged user outside the jail can cooperate with a privileged user inside the jail to obtain elevated privileges in the host environment.

Most of these attacks can be mitigated by ensuring that the jail root is not accessible to unprivileged users in the host environment. As a general rule, untrusted users with privileged access to a jail should not be given access to the host environment.

 

Despite all that and the fact that it’s the most recent of before-mentioned technologies, it quickly evoked significant appeal (aka hype) and is now both widely used, and integrated with many open-source tools and platforms like OpenStack, Cockpit and GitLab. In fact, nowadays it is very popular to provide ready-to-use Docker containers as part of the standard software package distribution. Security, as usual, is an afterthought since spinning up Docker containers is so trivial that it should be illegal.

The question still stands, though – Why are some technologies considered superior to others? From this competition I would immediately exclude chroot, since it is more of a tool for developing higher-level isolation concepts. However, what about Docker and FreeBSD jails? Why is Docker considered better? A general problematic trend I see in software development is trying to make tools do things they were not designed to do. Yes, oftentimes the product is successful and we’re amazed by the tool’s robustness. Alas, when it fails, we complain about its uselessness. That’s rather unfair, no? Jails are an extension of the chroot concept and are meant to hold long-running processes like database servers, Web servers, etc. You create a ZFS sub-volume, unpack a complete FreeBSD userland into it and route it to the Internet via the host packet filter. Thanks to PF’s design, it is easy to control data packet flow via TCP/IP down to a single packet. The ZFS sub-volume can be selectively backed up via per-volume snapshots, stored on another ZFS array or re-deployed at will. Hell, if we want, we can make it transient and completely obliterate it after the application finished running, a la Docker. The FreeBSD package building infrastructure, poudriere, makes heavy use of semi-transient jails. All of this seems sounds so trivial, n’est ce pas? Yet, the adoption compared to Docker is simply not there, because Docker was overly hyped and demonstrated to be easier than figuring out FreeBSD + jails for the average user.

This is where Docker wins – widespread integration and adoption. It is theoretically much easier (and faster!) for an average Joe to roll out a Docker container from an official pre-built image. Basically, almost any server appliance can be deployed as a Docker container. That doesn’t mean that Docker is necessarily better. Applications won’t work equally well on all container images due to differences in configuration and library versioning (say, CentOS vs Debian). More complex setups still require customization and writing dockerfiles is no easier or less clunky than deploying FreeBSD jails. Even more so, after some tinkering, I have to admit that setting up a FreeBSD jail with iocage was far easier than getting around Docker. The file system transparency inherent to jails is a big boon here. Regardless, orchestration tools like Chef or Ansible will help in both cases equally well. Furthermore, Docker was not designed for permanent storage from the get-go, but rather received this feature much later. Docker volumes are quite stable, but I still find it more convenient to create a ZFS volume and share it with the container rather than play around with Docker volumes. Again, this approach is much more transparent. Nevertheless, thanks to the overall hype, Docker was tested and used in production many times over already.

In conclusion, both Docker and FreeBSD jails have their respective niches in the IT ecosystem. For quick testing and deployment of lightweight applications (simple Python daemons, etc.) – Docker. For full-blown processes like a PostgreSQL or MySQL server, which require a dedicated IP or a mix of daemons – jails. If jails are not available, there is always KVM or a similar hardware virtualization platform which similarly emulates a complete ecosystem. The important thing is to understand how to choose the tool most suitable for a given use case. There is no better nor best, after all.

Advertisements

DevOps Oversimplified

To begin with, DevOps is one of those newfangled expressions in the IT ecosystem, which are currently hip and trendy, but cause a grimace on the faces of older system administrators. Before, we relied on cron and the flimsy magic of throwaway Shell scripts to manage system daemons. Now, we have Ansible, Docker and other sexy new tools which bridge the gap between software development and deployment. I still have some reservations towards Docker, though I really enjoy using Ansible in my pseudo-sysadmin workflow. The truth is that all of these tools have their place and are astonishingly powerful when used correctly. As a person who worked with multiple UNIX-like and non-UNIX operating systems, I can definitely claim that your millage may vary. Different scenarios require different tools and no tool is universal enough to work on all operating systems. Portability is a bit of an Eldorado which we all aim for, though never really reach. The other aspect is skepticism. I shudder every time I see adoption before comprehension. People notice something is cool, then just take it and use it right away, completely ignoring the How and What if. That is essentially what this article is all about!

Paper magazines are a bit of a dying concept, especially in the IT world. Stack Overflow is a Google / DuckDuckGo search away (to the left from your local booze supplier, can’t miss it!). Nevertheless, they’re an important part of my childhood and buying them in a way supports the open-source cause. Therefore, I purchase an issue from time to time. Unfortunately…

itsatrap

The article on Amazon Web Services (AWS) and managing virtual machine images looked interesting at a glance. I never worked with the AWS, therefore I decided to give it a go. The first thing that hit me was choosing Ubuntu 16.04 LTS as the guest operating system. I have nothing against Ubuntu as I use it at work as well, but what most people forget is that Debian and Ubuntu have quirks, which other Linux distributions lack. Most importantly, though – What is the use case? That is the first question which should be addressed when choosing a distribution. In fact, I would argue that if the article is meant to be useful professionally, CentOS, RedHat or openSUSE are better picks. That’s being needlessly pedantic, of course. Chances are the addressee of the article used Ubuntu before and has some experience with it.

Later, I find the phrase the package manager holds <package>. Personally, I have never seen this expression. Everyone talks/writes/types about repositories (repos). The only thing the package manager holds is a cache of the repository state to know which packages are available without having to ping the repository every single time. Again, that’s merely nitpicking. The thing that really disturbed me appeared several lines later:


pip install awscli

In 99% of the cases, this command will fail as it is typically executed from a regular user’s login session. By default, the pip script disallows installation of external Python libraries system-wide, and for perfectly valid reasons. If it somehow works, however, we’re in deep trouble. The worst that can happen is overwriting system-level Python libraries. For a more Windows-like reference, imagine altering the default DirectX installation. Pandemonium. A slightly better way to do this is:


pip install --user awscli

This will guarantee that libraries are installed into the user’s $HOME/.local/lib sub-directory and the wrapper scripts into $HOME/.local/bin. An even better way to do it is:


python2 -m pip install --user awscli

Thus, we specify the Python version for which we want to download the library and our approach is portable across multiple UNIX-like operating systems. PIP, as executed via “pip install”, is merely a wrapper script, which may or may not be offered by a Linux distribution and may or may not point to Python 2. Hell, in some cases it might even be buggy! The assumptions in the article went downhill hereafter. They made it really difficult for me to trust anything the writer stated unless tested thoroughly. The Internet is full of AWS horror stories already.

What this article demonstrates and what I think is extremely hurtful to the Linux ecosystem is bad practices and naive assumptions. Every Linux distribution is a Ubuntu, every Shell is Bash, etc. All of it, because people are encouraged to disregard core aspects of UNIX system management. One of my recent woes was when a company provided printer drivers as a URL to a Shell installation script. The recommended procedure was downloading the script via cURL and piping it directly to Bash with elevated privileges. Absolute madness!

OpenSUSE – a Server Perspective

This entry is a bit of an update to How I migrate(d) to OpenSUSE and Why , but also a short story about my experience with openSUSE since I had started using it to power our servers a couple of months ago . My general impression of openSUSE didn’t really change much, but at least now it’s supported by real-life experience.

I work as a junior system administrator / Python software developer / database administrator / other. I am primarily responsible for data storage and the well-being of our network infrastructure. When I started, the main operating system was Ubuntu Server 16.04 LTS with some machines accidentally running Ubuntu Server 17.10, which hit end-of-life soon after. Since I had to migrate the 17.10 machines, I considered switching our infrastructure to a more mature server environment like CentOS or openSUSE. Ubuntu Server 16.04 LTS is actually quite solid and I have nothing against it, however the sysadmin culture favors other operating systems and Debian-based distributions tend to introduce Debian-specific tweaks, which are non-portable. I worked with CentOS before as it was running on our structural biology workstations back when I was still a biologist. As a server platform CentOS is simply incredible. Many of the open-source projects developing server applications and appliances target CentOS primarily because of this. Unfortunately, CentOS packages are typically very stale, missing certain improvements unless they’re backported. Also, all of the mentioned server applications are installed from manually downloaded RPM packages, which puts the extra burden of having to track them on the administrator. What if there existed a platform that is neither stale, nor requires manually tracking the additional installed software…? It turns out such a platform indeed exists!

Fast-forward a couple of months and almost all of our KVM virtual machines run openSUSE 42.3 or 15.0 Leap. It wasn’t without hurdles, of course. Firstly, I absolutely loathe the idea of a GUI-first installer. Especially when it’s coded in a scripting language like Ruby. In addition, due to its complexity, it is prone to breakage and lags the serial console horribly. However, to be perfectly fair, it also handles quite some unusual partitioning regimes and a discrete selection of installed packages. After the first boot everything runs smoothly. The firewall is properly set up to allow only SSH connections so remote management is a given. My advice: create a single image and clone it, remembering to refresh the SSH keys, otherwise the OpenSSH client will complain. Also, change the IP address if it’s static or else the router will reject connections. For that purpose YaST2 is the perfect tool. In fact, thanks to additional modules (available in the repositories) it aides full system management, from drafting firewall rules to setting up Samba and database instance control. It has saved my life many times already and frankly, it is one of the features which brought me to openSUSE in the first place. What I really like about openSUSE is also the vastness of additional repositories. Our workflow revolves around time-series data and for that purpose we often rely on tools from the TICK and ELK software stacks. Of course, the stacks need to be relatively up-to-date so that we have access to the latest stable code base. Regular repositories in long-term support distributions never offer that (except for FreeBSD, I suppose). Moreover, for ease of use and upgrading we require repositories, not one-off DEB/RPM packages. The openSUSE Open Build System provides exactly that! SQL database servers and extensions? The server:database repository. HTTP servers? The server:http repository. And so on and so forth. In essence, it is as simple as adding the repository URL either through YaST2 or on the command line through zypper and we’re good to go. No more rogue packages polluting our long-running systems. Finally, the more standard software packages available in the official openSUSE repositories are of an acceptable vintage. I was slightly disappointed by Python 3.4, since I tend to overuse features introduced in Python 3.5, however openSUSE 15.0 is already out and I will migrate once I am sure it is stable enough.

To sum up, I can definitely recommend openSUSE Leap as a server platform. With fantastic tools such as YaST2 and the Open Build System, it is a pleasure to administer. BTRFS is making progress, however I would avoid it unless file system level compression is necessary. Unfortunately, it often is. As openSUSE is strongly against ZFS due to clashing licenses, BTRFS is almost the only option available. Alongside Leap, I tried Tumbleweed as a developer platform. Alas, it rolls too fast for me and the frequent massive package upgrades are simply overwhelming. Also, software stacks can break if they aren’t completely in sync. Despite the fact that the openSUSE project sports the amazing OpenQA testing platform, some bugs still get through. Nevertheless, running openSUSE in both development AND deployment is extremely tempting! Definitely beats the Fedora/CentOS combo.

We Are Developers 2018 – Day 2

Day 2 of the We Are Developers World Congress is up (at least for me, since I don’t have enough stamina for both the after-party and another full day of talks). Compared to day 1, I made some progress on the food and water front. The local grocery store, Hofer proved extremely useful. Armed with bacon buns and non-sparkling water I was ready for more developer-flavored bliss!

Alas, the first presentation was slightly disappointing. Instead of a talk about accelerated learning, I got a lecture on how learning works, from which I learned nothing. Thankfully, the second talk fully compensated for the shortcomings of the first one. Enter Brenda Romero – one of the legends of game development (think Wizardry 1-8). This talk was doubly important for me, because I would really love to join the game development “circus”, but I’m not yet sure whether I have the guts (or a “more-than-mellow” liver). I’m still not sure, but the take-home message was crystal clear – just do it! Brenda had a lot of important things to say regarding not giving up and not taking comments from others too personally. The audience can be brutal and vicious, and the gaming industry itself is tough. At least I know what I’m up against!

20180517_100610.jpg

Brenda Romero (centre) talking about her childhood toy assembling endeavors

Numero tertio was a continuation of game development goodness. I originally intended to attend the AI talk by Lassi Kurkijarvi, but John Romero. I don’t think I need to say more to anyone who at least heard of Quake or Doom. It was not a replay of last year’s talk, mind you! Rather, we got a full story of Doom’s development, which to me was both interesting and inspiring. John Romero is an amazing game developer and the pace at which he, John Carmack and other programmers at idSoftware produced Doom was simply dazzling. While modern games are of course a lot more complex, developers from the early 1990s didn’t have the tools, such as SDKs or version control we now possess.

20180517_110340

John Romero (centre) on developing and shipping Doom

 

Later on, it just spiraled! I lost track of the talks a bit, since there was some major reshuffling in the schedule. The presentation from Tessa Mero on ChatOps at Cisco was quite interesting. I do use Slack and various IRC clients, but a greater need for ChatOps and its integration with the software development cycle is definitely there. I wasn’t fully aware of that, to be completely honest. Next, Tereza Iofciu from mytaxi gave us a tour of machine learning and showed us the importance of computer algorithms in predictive cab distribution planning. It wasn’t about self-driving cars or reducing manpower, but rather about reducing the load on drivers and improving clientele’s satisfaction. Computer-accelerated supply-demand, so to speak.

In the afternoon I took an accidental detour to a book-signing event hosted by John and Brenda Romero. Not only did I get a chance to talk to them personally (*heavy breathing!*), but also got a copy of Masters of Doom signed (*more heavy breathing!*). John said that if I read it, I’ll definitely get into game development professionally. I’m completely embracing the idea as I type this. One of the last talks I attended was given by Yan Cui on how he used the Akka actor model implementation (together with Netty) to solve latency issues in a mobile multiplayer game (MMO specifically). Obviously, it was a success and his convincing speech makes me want to try it out. It’s about concurrency, but without the overhead of traditional multiprocessing and/or multithreading. Although I don’t code in C# just yet, there is a Python implementation of Akka, which was recently recommended to me.

20180517_154747.jpg

Yan Cui (centre) explaining message relays in the actor model of concurrent programming

In summary, it was great to meet like-minded folks and actually talk to fellow game developers, who like challenges and don’t shy away from trying out new approaches to software design. Perhaps that’s what I’m looking for – challenges? Stay tuned for more exciting impressions from day 3 of the Congress!

 

We Are Developers 2018 – Day 1

To begin with, I attended the We Are Developers World Congress last year (2017) and I was quite amazed by it. I got to see John Romero, the legend of game development and author of titles such as Wolfenstein 3D, Doom and Quake I. Actually, the congress inspired me so much that I decided to finally part with my scientific career and pursue a life as a software developer and/or system administrator (a bit of both in reality). To the point, though. The We Are Developers World Congress is a fairly novel venture and even the Internet knows very little about it outside of  the main website and single blog posts. It hasn’t become a tradition just yet, thereby media coverage is patchy at best. Considering that it grows exponentially in its grandeur (2000 attendees last year, 8000 registered attendees this year!), I decided to cover it myself.

wearedevelopers_2017_logo

wearedevelopers_2018_logo

The logo from the 2017 edition (above) and the logo from the 2018 edition (below)

The Congress started with a treat already – a fireside chat between Monty Munford and Stephen Gary Wozniak (Steve Wozniak, The Great Woz). It was intended as a casual interview, but The Woz proved to be exactly the person as depicted in the 2013 movie with Ashton Kutcher entitled Jobs. Steve Wozniak is extremely chatty and simply adores talking about himself, therefore it was only natural for him to dominate the discussion. Slightly to the detriment of the “chat” aspect of the event. I enjoyed it nevertheless. Many important points were raised – the economy of social media (Should we not get a fair share of the profit made by Facebook and Google off our personal data?), the “I” in “Artificial Intelligence” (It’s not really “intelligence” if it’s programmed!), Elon Musk (Tesla fails to deliver, year after year…), etc. It was somewhat surprising to witness that Steve Wozniak hasn’t really changed since the crazy ventures of his teen years with Steven Paul Jobs. Quite the amazing spirit!

20180516_102109.jpg

Monty Munford (left) having a fireside chat with the Great Woz (right)

The fireside chat was followed by an interesting talk from Joseph Sirosh from Microsoft. He talked about the various machine learning tools offered as part of Microsoft’s Azure hosting platform. To be honest, I am extremely skeptical regarding Microsoft’s ideas, especially when it concerns open-source software, supposedly open to the public. Microsoft has a disappointing track record of using the embrace, extend, extinguish tactic against promising software projects and a sinusoidal quality trend of its flagship product – Windows. Accordingly, I took the with a bucket of salt approach. The mood among other attendees was similarly negative. Unnecessarily, though! Azure’s machine learning tools seemed very promising in the end. I do consider using them for some of my projects.

After the lunch break I joined the Headless CMS track, and after the initial slightly disappointing talk, I was enthusiastic about Jeremiah Lee and his JSON API idea. REST APIs are a big part of the Web nowadays, ever increasingly so. We do need a slightly more elaborate and efficient data format standard built on-top of the venerable JSON. At that point I realized that unlike the Web development track last year, this time programming language animosities were absent. The implementation is irrelevant to  the standard if we all agree on its importance!  The last talk in the Headless CMS track I attended was given by Kaz Sato from Google. The topic being machine learning again, but this time leveraging Google’s AutoML platform and TensorFlow. Machine learning is actually one of the main themes of this year’s edition of the We Are Developers World Congress. It’s very clear that we need it!

20180516_113233.jpg

Joseph Sirosh (centre), showcasing MS Azure AI services and APIs

To sum up, based on the various talks I attended, I begin to form a vision regarding the future of computers. We started with humongous, clunky mainframes and progressed into the personal computer era with the contributions from Steve Wozniak, Steve Jobs and many others. However, the dichotomy returns. Computers turn into mobile “enabling” devices, which aid us in our daily tasks and ease our interaction with the world (and each other). Heaps of data at our fingertips! However, we need a back-end, an infrastructure of powerful serves to store data and organize it in an accessible way. In-between is of course a robust network interface which carries the data from the back-end to us, the clients/users.

 

How I migrate(d) to OpenSUSE and Why

I’m a die hard FreeBSD fan. I simply love it! It rubs me the right (UNIX) way. Through trials and tribulations I managed to make it do things it was possibly not designed to do. ZFS? Amazeballs. Cool factor over 9000! However, all of that came at a tremendous cost in energy and time. I reached a point when I don’t want to spend time manually configuring everything and needing to invent ways of automatizing things which should work out-of-the-box. Furthermore, most FreeBSD tools are not compatible with other operating systems, therefore learning FreeBSD (or any other BSD variant, for that matter) locks me in FreeBSD. Despite many incompatibilities, this is not the case with Linux. On a side note, the ZFS on Linux project was a great idea. The Linux ecosystem badly needed a mature storage-oriented filesystem, such as ZFS. BTRFS to me at least “is not there yet”. Other tools, such as containers were reinvented in some many different ways that Linux has outpaced FreeBSD many times over. Importantly, Linux tools were tested in many more real life scenarios and are in general more streamlined. For automation, this is crucial. Again, I don’t want to tinker with virtually every tool I intend to use. Neither do I want to read pages and pages of technical documents to get a simple container running. More so, I should not be forced to, since that’s terribly unproductive. Finally, I like to run the same operating system on most of my computers (be it i386, x86_64 or ARM). FreeBSD support for many desktop and laptop subsystems is spotty at best…

Enter OpenSUSE!

green_lizard

Cute lizard stock photo. Courtesy of the Interweb.

Seemingly, OpenSUSE addresses all of the above issues. True, ZFS support is not reliable and there are no plans to the contrary. The problem is as always licensing. BTRFS is still buggy enough to throw a surprise blow where it hurts the most. Personally, I don’t run RAID 5/6 setups, but that’s BTRFS’ biggest weakness right now. That and occasional “oh shit!” moments. Regardless, I think I’ll need to get used to it. Lots of backups, coffee and prayer – the bread & butter of a sysadmin. On the up side, this is virtually the only concern I have regarding OpenSUSE.

The clear positives:

  • Centralized system management via YaST2 (printers, bootloader, kernel parameters, virtual machines, databases, network servers, etc.). A command-line interface is also available for headless appliances. This is absolutely indispensable.
  • Access to extra software packages via semi-official repositories. Every tool or framework I needed was easily found. This is a much more scalable approach than the Debian/Ubuntu way of downloading ready .deb packages from vendors and having to watch out for updates. Big plus.
  • Impressive versatility. OpenSUSE is theoretically a desktop-oriented platform, though thanks to the many frameworks it offers, it works equally well on servers. In addition, there is the developer-centric rolling-release flavor, Tumbleweed, which tries to follow upstream projects closely. Very important when relying on core libraries like pandas or numpy in Python.

So far, I’ve switched my main desktop machines over to OpenSUSE, but I’m also testing its capabilities as a KVM host and database server. Wish me luck!

The Ubuntu Conundrum

Ubuntu is perhaps the most popular Linux-based operating system, however for that very reason it has as many proponents as enemies. I myself use Ubuntu (Xubuntu 16.04 LTS, to be exact) at work, both as a development platform and to host services in libvirt/KVM virtual machines (Ubuntu Server 16.04 LTS there). It performs alright and so far hasn’t let us down, though we haven’t been using it through more than 2 releases so we’re unable to gauge its reliability properly. On more personal grounds I can say it works splendidly on my early-2011 MacBook Pro 15″ with faulty AMD graphics and has since the very beginning (out-of-the-box, as one might say). Singular package upgrades don’t bring about all of the regressions people profess so fervently. However, I can understand where the hate is coming from and I admit it is partially justified.

Product and popularity
For whatever reason human psychology dictates that we equate quality with popularity. If something is extremely popular, it simply must be good, right? Wrong. Completely. A product is popular, because someone with enough resources made it visible to as many consumers as possible. The product was made popular. Quality is a useful, but clearly secondary measure. A good anecdote is the long gone rivalry between VHS and Betamax. We all remember VHS, though most of us do not remember Betamax, which was technically superior. However, it lost the popularity race and will forever be remembered as the second best or not remembered at all. Now, this is not to say that Ubuntu is in any way inferior…

Ubuntu, the (non)universal operating system
The main issue with Ubuntu is that it succeeded as a more open operating system alternative to Windows and macOS X, however did not solve the underlying problem – computer literacy. Of course, not every computer user has to be a geek and hack the kernel. However, when I see that Ubuntu users address their PC-related issues with the same shamanism and hocus pocus as in Windows, my soul twists in convulsions. We did not flee from closed-source operating systems only to change the names of our favorite tools and the look of our graphical user interfaces, though observing current trends, I might be terribly wrong. The other problem is that Ubuntu’s popularity has become self-perpetuating. It’s popular, because it’s popular. Many tutorials online and in magazines assume that if one uses Linux, he or she surely runs Ubuntu on all of his or her computers. This is extremely hurtful to the entirety of the Linux ecosystem, because neither Debian nor Ubuntu represent standard Linux. Both of those systems introduce a number of configuration improvements to applications, which are not defined in upstream documentation and absent in other distributions (so-called Debianisms). Therefore, Ubuntu being a universal operating system is more of a publicity gimmick than a fact. Especially, considering that on servers, SLES (SUSE Linux Enterprise edition), CentOS and Red Hat clearly dominate.

The solution?
I would say it’s high time we begin showing newcomers that there is an amazing world of Linux beyond Ubuntu. To that end, I have a couple of suggestions for specific needs and distributions covering those needs. Related questions come up often in the Linux Facebook group and around the Internet, but get answered superficially via click-bait articles listing top 10 distributions in 2017/18. Not exactly useful. Anyhow, the list:

  • Software development:
    – Fedora (up-to-date packages and developer-centric tools like COPR)
    – Arch Linux (up-to-date with a wide range of packages via AUR and vanilla package configuration for simplicity)
    – openSUSE Tumbleweed (up-to-date with a rolling, snapshot based release cycle, but sharing the Leap / SLES high-quality management tools like YaST2)
  • Servers:
    – openSUSE Leap (3-year long support life cycle, high-quality management tools like YaST2 and straightforward server + database + VM configuration)
    – CentOS (binary compatible with Red Hat Enterprise Linux)
    – FreeBSD (ZFS hard drive pool management + snapshots, reliable service/database separation via jails, rock solid base system)
  • Easy-to-use:
    – Manjaro Linux (based on Arch Linux, with lots of straightforward graphical configuration tools, multiple installable kernels, etc.)
    – Fedora (not only for developers!)
    – openSUSE Leap (for similar reasons as above + a streamlined, user-friendly installer)
  • For learning Linux:
    – Gentoo (painful at first, but extremely flexible with discrete software feature selection at compile-time via USE flags)
    – Arch Linux (Keep It Simple Stupid; no hand-holding, but with high-quality documentation to make the learning curve less steep)
    – CRUX (similar to Gentoo, but without the useful scripts; basically, vanilla Linux with a very simple package manager)
  • For learning BSDs:
    – FreeBSD (as mentioned above)
    – OpenBSD (strong emphasis on code-correctness, system engineering and network management)
    – DragonflyBSD (pioneering data storage and multi-processor systems)