CRUX – Take It, Make It, Break It!

CRUX is a rather unknown distribution that escaped the swallowing waves of major GNU/Linux trends, like systemd and pulseaudio. It consists of the bare essential components, in their simplest form, that make GNU/Linux a bit more than just the kernel. Services are handled by shell scripts and are meant to start/stop/restart processes spawned in daemon mode, like dhcpcd, crond, etc. The syntax is well-established in Unix-like operating systems and easy to follow. Esentially, anyone could write their own service init script.

Package management, as we know it from other GNU/Linux distributions, was originally not part of CRUX. Much like Gentoo and BSD systems it employs a source-based ports tree located in /usr/ports (though Gentoo has it elsewhere). Building, adding and removing packages is achieved via ‘pkgmk’, ‘pkgadd’ and ‘pkgrm’, respectively. The main tool is ‘ports’ for checking differences between the local and remote trees and notifying which applications may require an update. In recent years, however ‘prt-get’ was written to simplify the management process, especially package dependency resolution. Though I found running ‘ports -u -d’ and ‘pkgmk -d -u’ or ‘pkgmk -d -i’ fun at first, for more complex applications the dependency web and often ‘not found’ errors would get to me. Nevertheless, the BSD feels are surely there. I’m loving it!

Speaking of ports management, there are 3 central repositories that offer the most basic tools and applications – core, opt and xorg. If one wants to go with a minimal system, everything is within those repositories, even utilities like mplayer and feh. Therefore, it is possible to build a rather functional desktop, without the need for extra software. However, in case core, opt or xorg don’t contain all that we need, there is always the ‘contrib’, ‘compat-32’ and user repositories to fill the gaps. Shortly browsing through the full ports collection using either ‘portdbc’ or the website, I noticed a lot is actually available. If not, writing a Pkgfile (basic text file containing information on source code whereabouts, version number, description, etc.) is as simple if not simpler than producing a PKGBUILD (Arch Linux) or ebuild (Gentoo Linux). Speaking of the latter, CRUX is less barebones than Gentoo, because the base system already contains all of the standard tools to get things running. This is favorable, as less things have to be compiled from source, which by the way seems faster than on Gentoo Linux.

Boot time is always a bit of a red herring argument and I would not mention it, unless I were impressed. I can tell you, I really am. Getting from the GRUB2 screen to the login prompt takes 5-10 seconds. Since CRUX doesn’t really use a service/process management system like openRC or systemd, the initial log is quite short and somehow resembles that found in FreeBSD. Connecting USB ports, mounting drives, attaching drivers to devices and the likes.

On the downside, everything must be configured by the user. For instance, I tried to install xfce4 as my desktop environment. For some functionalities I had to produce proper polkit .rules myself as the polkit package did not contain any. This is not really the fault of CRUX, but rather that of the software developers who did not add the necessary lines to the sources. Somewhat strange, since those .rules are always necessary for ‘shutdown’, ‘suspend/resume’ and ‘reboot’ desktop buttons to work. Another thing was the kernel for CRUX 3.2 (version 4.1.13). Surprisingly, the .config lacked full information on the graphics stack, though this should have been available since kernel version 3.x at least. In addition, every time I reinstalled and compiled its sources, something would go wrong after reboot. Of course, new kernel sources can be promptly downloaded, from for instance,  during install and a new kernel compiled. Nevertheless, this is tricky business and one can get easily lost in the plethora of modules and features.

To sum it up, I think CRUX works as advertised. It’s relatively easy to install if someone already went through Arch Linux and the painstik ritual of Gentoo. Software selection is modest, though sufficient to build a functional operating system. I guess the only gripes I would have are against the intrinsic chaos of GNU/Linux rather than CRUX itself. Lastly, it does not promise mind-boggling user-friendliness like Ubuntu or openSUSE and really stays away from the Red Hat extravaganza.

Make GNU/Linux Great Again?

When I escaped from Windows to GNU/Linux I felt that the GNU/Linux ecosystem is like a remote, secluded beach. A place of tranquility, solitude and retreat from the outside world. Only the initiated ones could come, so that the beach remained well-kept and untainted. Unfortunately, this is no longer the case to me. Some came and thought ‘Oh, what a lovely place! Let’s build a resort here so that many more may flock and enjoy it!’. On paper this sounded great. Unfortunately, with people arrived noise, odor and litter…

As a GNU/Linux user I am a proponent of the ‘tried and tested’. I like simple setups with a minimal number of components and services that are mandatory for my PC to achieve the intended functionalities. I loath stacks of UIs/APIs and infinite towers of front-ends. Understandably, there is a need to separate the user from the hardware to prevent inadvertent damage. However, going several levels deep is a choice, not an obligation.

As many know and most should know, Linux is merely the kernel. The components that sit on top of it are interchangeable and as a whole, the GNU/Linux operating system can be built in various ways. There are several init architectures, including the legacy upstart, sysVinit, systemd, epoch and openRC. Same goes for the bootloader – LILO (now defunct), GRUB2, syslinux, etc. Why then do we only hear of systemd nowadays?

In short, it’s a lobby/agenda of companies that financially support GNU/Linux development. Systemd was spawned by Red Hat as an alternative to the old and ugly Unix ways of the past, which in their opinion are old and ugly, because they require more effort to maintain. Dubious logic, followed by dubious assurance of increased performance and simplified maintenance. Per reference, shell-based init scripts are so simple to write that even a high-schooler would easily be able to handle them.

However, for the resort builders it doesn’t matter. They see opportunity, invest and make profit. Freedom is a financially unsustainable luxury they have no need for. They want a new beach where they can sell their ice cream and pop. This money-driven cancer infests the whole GNU/Linux ecosystem and finds no satiation. The biggest problem is that it kills GNU/Linux’s modularity, the biggest selling point! If desktop environments start to depend on service management utilities, we have a problem…

Fortunately, there are ways and means I have been exploring recently. For instance, one doesn’t need the Gnome Network Manager for wireless network management, as wpa_supplicant’s GTK-based GUI does it perfectly well. In addition, there are GNU/Linux distributions that still appreciate the Unix ways, like Void Linux, CRUX and Gentoo Linux. We can and definitely should make GNU/Linux great again. The place of freedom and tranquility it used to be.

Per my contribution I decided to thoroughly explore CRUX and see how well one can build a GNU/Linux distribution on top of it.

War on Inconsistency II

In my former study of Gentoo Linux I encountered certain problems with GNU/Linux that are the more apparent as one strays from mainstream distributions, such as Ubuntu, Fedora/Red Hat and OpenSUSE/SUSE. Those problems do not exist in the BSD world, where the core operating system is an intrinsically functional platform for add-on applications. As laid out in the FreeBSD Handbook, it’s a chaos vs order issue. Chaos, unfortunately, gives rise to inconsistency. Dedoimedo has a great comparison of Ubuntu flavors illustrating that.

A typical user expects their operating system of choice to just work. Would that be the case for GNU/Linux, selecting a favorite distribution would rely primarily on visuals. While slightly undermining flexibility, it would mean every end-user would be allowed to enjoy the GNU/Linux ecosystem fully and not worry about driver issues, being vastly different on each distribution. One such quite ridiculous case was when the proprietary Broadcom driver failed to work on Debian, but worked perfectly fine on Ubuntu. Theoretically, the same driver. Practically, different everything else.

For now a lot of companies support only the top 3-5 GNU/Linux distributions. They cannot and will not extend that support to the whole GNU/Linux ecosystem, because they actually care that their products work well, even if only on a small number of most popular platforms. The issue could be partially addressed by releasing the source code of the product, so that patches and adjustments can be applied. Then, revenue may still be generated via technical support and various other forms of counseling. It is perfectly viable and companies do so. Another way would be to work on some level of consistency within the Linux kernel and the basic userland (compilers, command-line monitoring and diagnostic tools, partitioners, etc). Frankly, it’s quite ridiculous that the very base of the operating system (the kernel) can differ so much between distributions that troubleshooting driver issues is but a lesson in futility.

To be perfectly clear, no flexibility has to be sacrificed. It would be enough to agree on the basic modules, features and tools, still making the setup perfectly viable to build upon. In addition, nothing new needs to be created, because all of the components are already present. Not such a bad idea, right?

Date with the Gentoo Oxen – Part Deux


As part of my journey to the sane and minimal of GNU/Linux, I decided to revisit Larry’s neighborhood of Gentoo Linux. I tried it for some days in the past, though without much success. Perhaps I was merely not ready for a certain level of complexity. After having spent many more hours recently, I sadly reached a similar conclusion. To avoid a full-scale rant, I decided to put down the strong and weak points of Gentoo Linux as I see them in my tired, watery eyes.

Why Yes:

  • Almost perfect control of what will end up in the kernel and installed software. Especially useful when we want to avoid certain unwanted, but persistent components like dbus, systemd, pulseaudio, etc.
  • Minimal, tailored setups (see above), essential for embedded or low-powered systems.
  • Broadening our understanding of GNU/Linux in general.
  • Testing various compilation parameters as part of day-to-day developers’ work.
  • L33t feel of being able to tackle the most vicious maintenance issues (relevant at least to me!).


Why the hell No:

  • Building packages from source code is time-consuming and should be done by build-farms, rather than personal computers.
  • Compiling badly written code produces binaries compiled from badly written code. Unless core issues are addressed, compiling alone doesn’t help much.
  • With currently accessible hardware, the overall performance gain is minimal. We’re talking <5% net gain on average.
  • Number of Gentoo installations = number of Gentoo users. Reproducibility is highly relevant to bug-tracking and in Gentoo Linux it’s immensely difficult to achieve.
  • Kernel trimming is a two-sided sword – often 3rd party devices/drivers may fail to work, because we forgot to add just this one module/feature. This applies to the whole of GNU/Linux, though.
  • Non-standard software behavior and often permission problems. For instance, why does the ‘lspci’ command require superuser elevation? Why does my X session fail to start with timeouts on xauth and the .Xauthority file lock-up?
  • Scattered configurations – ALSA can be configured through at least 2-3 files system-wide. This, however is more of a GNU/Linux issue than anything else.
  • Features unintentionally left out might be useful in the future. This involves more compilation rounds if they are.
  • Some features cannot be avoided due to package design. For instance, KDE wallpapers needing almost the full KDE desktop due to some minor requirement for animations.

The unfortunate take-home of my modest assessment is that it’s not worth it. For instance, during kernel compilation I accidentally left out parameters which were later needed for my wireless driver in an absolutely non-trivial way. In addition, updates can be quite lengthy and require leaving the computer on overnight. Personally, I don’t consider this to be standard XXI-century practice, unless we’re developers and this is what we do out of fancy or for a living.

The distinct advantage of operating systems, such as Arch Linux or FreeBSD is that source file adjustment and compilation is optional and only required when we want to tinker with program components. Heck, at this point one could even try CRUX Linux, because it’s so much simpler. In addition, it seems to be a tad more KISS-savvy than Gentoo Linux.

Over and out!

Finding Unix in GNU/Linux

In the spirit of Unix claimed thus far, I decided to explore GNU/Linux options that would provide similar capabilities as FreeBSD. Of course, FreeBSD is absolutely irreplaceable and nothing can beat the organized nature of ‘true’ Unices. Nevertheless, there are some GNU/Linux distributions spawned from the soul of Unix Prime, which probably deserve attention. Let’s see what gives!

  • Arch Linux was created with Unix and the KISS (Keep It Simple Stupid) principle in mind. This is visible in the minimal nature of the base system, ports-like package management (binary packages with an optional source-only ABS tree for development) and the AUR (Arch Linux User Repository). It’s fantastic for quick, flexible setups and keeping everything on the cutting edge. The only major gripe I have is systemd and the occasional reliance on its components. True, one can install OpenRC or any other init system from the repositories or beyond (or use the OpenRC edition of Manjaro Linux). Alas, even then some systemd components have to remain for compatibility measures. In my eyes and the eyes of many, systemd is strongly against Unix principles and its merits are rather superfluous.
  • Slackware is a bit of a hermit turtle in the GNU/Linux world. It has existed since time immemorial and remained much unchanged. Although Pulse Audio was introduced to mitigate some Bluetooth sound compatibility issues, Slackware still tries to adhere to its Unix-heavy principles. Oddly enough, slack does not mean that the distribution is easy to use. Quite the contrary, it’s the lead developer who was slacking and did not produce a proper package manager for Slackware Linux. More of an inside joke than an actual insult, of course.
  • CRUX Linux is probably the closest one can get to a BSD equivalent in GNU/Linux land. It’s basically a BSD distribution built with GNU tools on top of the Linux kernel. We get the all-too-familiar /etc/rc.conf main configuration file, in addition to a classical ports tree and services handling. The downside is that CRUX Linux does not involve absolutely any hand-holding in terms of configuration (except for the initial repository setup + some default system configs to get the OS booting) and the selection of packages is extremely scarce in comparison to the likes of other GNU/Linux distribution.
  • Gentoo Linux is like CRUX Linux on steroids. Instead of a BSD-style init system we have OpenRC with its own set of services and controls. Knowing and maintaining all of the various GNU/Linux init systems is tricky business, especially that most far exceed the simplicity of the traditional Unix design. Gentoo Linux is primarily source-based and much like in the case of Slackware and CRUX Linux, almost everything has to be build locally. This reduces bandwidth, though increases PC usage dramatically. Feel the hard drive succumbing under the constant read-write onslaught! Unlike its source-ish cousins, Gentoo offers additional customization means in the form of USE flags. The majority of software packages can be produced just the way we want them (dbus support? systemd support? pulseaudio support?).

As GNU/Linux is only Unix-like, it has traded some of the clean design merits for user-friendly blink. This is good for the end-user, though a nightmare for the more-inclined, who now have to learn how each GNU/Linux distribution handles things. As Canonical and Red Hat have topped the ecosystem, using anything besides xUbuntu or CentOS/RHEL is down to personal preference and more personal struggle. On the other side of Planet FOSS, the BSD crowd keeps pushing sane and useful software forward. To each their own, I guess.

I will probably give Gentoo Linux and a revamped Arch Linux with OpenRC a try. I am a geeky masochist after all and scrolling waterfalls of C/C++ code turn me on.

Simplicity Is the Ultimate Sophistication

Originally coined by Leonardo da Vinci, later accepted and implemented by Steven Paul Jobs as part of the Mac OS X credo. In the advent of Ubuntu’s shoddy release called Xenial Xerus and its heavy repercussions on Linux Mint 17.3, I decided to revisit this beautiful maxim. I do not intend to go on another rant, yet the need and willingness are definitely there.

The new Ubuntu 16.04 LTS breaks many things, as have observed Roger from Dedoimedo and my father (both highly influential figures to me). The problem though is not that it breaks things. Rather, that the things broken severely cripple user-experience and Long Term Support releases should display improved reliability from the get-go. I was honestly looking forward to 16.04, since 14.04 had been a decently solid release. Now, I have no intention to even touch (as in, the ‘touch’ Unix command) Xenial Xerus. Ubuntu’s new release painfully shows a couple of things that are wrong with Ubuntu and potentially with the whole of the GNU/Linux ecosystem:

  1. Catering to too broad an audience makes the operating system unsuitable for any single individual. This is one of the major problems with Windows and seeing it happen to a GNU/Linux distribution pains me.
  2. Diminishing quality of the base installation. I agree it’s completely impossible to please everyone (see, point 1.) and I am not expecting every piece of software in the repositories to work flawlessly. However, if parts of the base install don’t work as intended, although they clearly did in past releases, one really starts to wonder about quality assurance…
  3. Lack of consistency between flavors of Ubuntu. The point was originally made by Roger from Dedoimedo. Until personally tested, it is not clear whether each flavor of Ubuntu will support our hardware equally well. In former times I used to think that the major difference between flavors was merely the selection of apps and the desktop environment. Guess I was wrong. This is slightly off-putting, because it means there may be a lack of sync between developer groups responsible for each flavor.

Point 1. is a philosophical issue and cannot be treated in terms of good and bad. Unfortunately, the mythical End User, much like Big Foot has never shown itself, so an educated guess is all we can bargain for. Points 2. and 3. display symptoms of improper management. I myself had problems with Lubuntu 16.04 shortly after release. Some directory paths were already restructured for the upcoming move to LXQT, though PCManFM was still looking for the wallpaper in the defunct LXDE directory. Amending the issue took me approximately 5 minutes. In all seriousness, how difficult can it be to configure the wallpaper properly out-of-the-box…?

The beauty of an operating system is defined in two things – A) it looks just about right (clean, clear and non-intrusive design) and B) things just work (minimal, yet functional setup). Ubuntu would have far less problems if the developers focused more on getting a small subset of simple apps and scripts to work and NOT change them for the sake of change. Building up from a solid base is much easier to carry out afterwards. Many other GNU/Linux distributions suffer from similar problems, though not all of them. Somehow I never had issues with Manjaro XFCE, even when using development snapshots. I reckon keeping it simple at the core eventually pays off.

‘Getting the Job Done’ Dilemmas

A WordPress user, codeinfig, brought my attention to what I was trying to express in my previous blog entry, but somehow failed to name. The patched together arguments I’ve been tossing around recently illustrate a major issue in software design – abstraction vs obstruction. We build abstraction layers to tie together smaller system components (or features) or to provide an easy to use interface for ourselves and others. However, abstraction layers tend to stack ad infinitum, because we all like having our own, right? Unfortunately, handling abstraction layers becomes more and more difficult with each level. Thus, obstruction is born.

I kept asking myself – why do we even need abstraction layers? I personally prefer working on the very core of the operating system, almost bare metal (though I don’t speak binary yet). The BSDs intrinsically allow this, much to my appreciation. The reason is simple – there is barely any need for abstraction layers. Don’t get me wrong, though. I have nothing against abstraction. It’s often useful in preventing hardware or data damage when getting up close and personal with our PCs. No one wants to intentionally fry their graphics (unless it’s a stress test), correct? Abstractions are good as long as their well-documented and transparent enough.

That’s the problem the GNU/Linux ecosystem is facing right now. There is tons of abstractions that are neither well-documented nor transparent. There are tons of obstructions. Proponents say that systemd units are more structured and more clearly written than legacy rc scripts. Yet they somehow generate more problems, bugs and are in general harder to work with for seasoned Unix administrators. Why binary logs? What was wrong with text files? Are we slowly entering the .bat zone of Windows OS? Pulse Audio suffers from similar problems. It started extremely buggy, reached a somewhat stable state…and still doesn’t solve the supposed problems ALSA caused. One can go on and on, there is hardly any point.

Linas Veptas laid it out perfectly in his open letter to Debian and Ubuntu developers. The root cause of the problem are abstractions turned obstructions, spawned by people who think it’s reasonable to search for a needle in a haystack using a huge industry-grade electromagnet. This we already know. The symptoms are more down to earth, though. Linus and many other Unix IT people/admins/greybeards want to keep on using GNU/Linux, but more importantly get the job done. This is impossible, because good is swapped for an utopian view of better, whether the vast GNU/Linux populace wants it or not. When one spends hours on fixing so-called modern inventions, getting things to work and then getting a good night’s sleep is really all that matters. Why do you think the Devuan project was launched? The developers just want to use their favorite GNU/Linux distribution, Debian without the commitment to nonsense.

Alas, the obstruction avalanche has already caught momentum. The reason is mentioned along the lines in an assay by Steve Litt. People are flawed beings and most of us don’t parse incoming information through a sanity-control mechanism. Microsoft and Apple  tell us we need complexity, because everything nowadays is more complex and they can take care of that everything for us. So that we don’t have to worry, of course! In practice obscurity never solved any problems and never will. That’s just the way of things.