Just Linux Things

As a follow-up to my previous post, I noticed that a lot of the recent open-source technologies are extremely Linux-centric. It all started around the time GNOME3 and systemd were introduced. Both heavily rely on facilities present in Linux-based operating systems, making them difficult to port to other Unix platforms like the BSDs. These technologies are also strongly promoted to entice prospective developers. While I understand the need for platform-centric efficiency inherently tied to Linux-specific features (cgroups, etc.), it is also important not to ignore the rest of the IT ecosphere. Yes, I mean even Windows, which is normally not considered on equal terms as Unix, but is relevant when talking about C# or .NET applications.

A recent example of this trend is Docker. Containers are now the new pink and everyone wants to get a piece of the pie. Docker barely reached release 1.x, yet some companies already make claims about “widespread adoption”. I thought industries prefer stable and tested-for-years solutions. I find this new craze odd the least. As expected, Docker is a Linux thing. While the containers are indeed OS-level on GNU/Linux systems, much like LXC (Linux Containers), they’re not on Windows and neither on MacOS X. Oddly enough, the latter uses a Unix virtual machine manager xhyve (based on FreeBSD’s bhyve). Therefore, despite the fact that developer interfaces are similar or even identical, the engines running underneath will have a substantial overhead on non-Linux systems. At that point one might consider whether native and more established solutions are not already available and more suitable for multi-container setups. On FreeBSD we have Jails and a ton of Jail managers to make one’s life easier. I have a feeling that Jails on FreeBSD would do a lot better than Docker containers on GNU/Linux. Not to mention that a FreeBSD base system is a lot slimmer than whatever one could consider a base system in GNU/Linux world. “Widespread adoption” seems to be lacking, because most of world’s servers run GNU/Linux.

Another weird trend I notice is identifying everything Linux-related with Ubuntu, as if Ubuntu was the only True Linux Distribution. I often read articles that claim to touch on Linux, but in reality discuss Ubuntu. A square is a rectangle, but not all rectangles are squares! That’s so obvious, no? This “Linux = Ubuntu” assumption hurts the whole ecosystem quite a lot. People learn how to use Ubuntu, think they’ve mastered Linux. Then, they’re dropped into a den full of Gentoos and CentOS’, and they end up suffering. Ouch! A fallout of this worrying trend is the fact that people deploy Ubuntu in places where a more lightweight GNU/Linux distribution would be a lot more suitable. That’s one of the reasons why Docker switched from Ubuntu to Alpine Linux eventually. With the wealth and diversity of the GNU/Linux ecosystem, one doesn’t even need to go far.

It’s Time for FreeBSD!

For the last couple of weeks I have been delving deeper into the arcane arts of FreeBSD, paying extra attention to containers (jails), local/remote package distribution (poudriere) and storage utilities (ZFS RAID, mirrors). Truth be told, with every ounce of practical knowledge I was increasingly impressed by this Unix-like operating system. It’s nothing short of amazing, really! The irony lies in the fact that FreeBSD is an underdog in the Unix world (less than OpenBSD or Illumos, but still), despite the fact that it excels as a server environment and established many technologies currently in focus (process isolation, efficient networking and firewalls, data storage, etc.) years ago already. GNU/Linux is picking up pace, but it still has a long way to go.

I feel traditional Unices got the “base system + additional applications” separation properly. One might think it’s just a matter of personal taste – the “order vs chaos” debate going on for ages. The truth is that this separation is not only reasonable, but also extremely useful when one begins to treat the operating system as more than a mere Internet browser or music player. Order is paramount to organizing and securing data from ill-intent or hardware failure. I really appreciate the use of /usr/local, /var/db, /var/cache and other typical Unix volumes on FreeBSD, as it makes the system more predictable, and therefore hustle-free. When we have a multitude of systems to care about, hustle-free becomes a necessity. With the new container technologies like Docker it’s a realistic scenario – 1 host system serving N guest systems. One doesn’t need to run a server farm to get a taste of that.

This is basically where (and when) FreeBSD comes in. It’s a neatly organized Unix-like system with great storage capabilities (ZFS), process isolation for guest systems (jails and bhyve), network routing (ipfw, pf, etc.) and package distribution (synth, poudriere, etc.). And everything is fully integrated! The “pkg” package manager knows what jails are and do, and can easily install programs to them. Poudriere coordinates building new packages, using jail containers so that the host system is not compromised. These packages can later be distributed via HTTP/HTTPS/FTP remotely or locally via the file protocol. Such low-level integration is somewhat foreign to the GNU/Linux world, though among server distributions like OpenSUSE, CentOS or Ubuntu Server it is constantly improving.

Still, whenever I think about the divide between BSD and GNU/Linux, I see a tall brick wall that both sides are struggling to tear down. FreeBSD wants to become more desktop-oriented, while GNU/Linux is trying to reinforce its server roots. Difficult to tell whether this is good or bad. The BSDs do indeed excel as server systems, as recently highlighted in a NASA study. GNU/Linux is more suited for heavy computation and leisure. The brick wall has plenty of nicks, yet it stands strong. Maybe there is a “third option”? Why not let each do the job they do best? What I mean to say is that FreeBSD has its place in the world and the time is ripe to truly begin to appreciate it!

FreeBSD-Debian ZFS Migration

Since the Zettabyte File System (ZFS) is steadily getting more and more stable on non-Solaris and non-FreeBSD systems, I decided to put my data pool created for the previous entry to the test. In principle, it should be possible to migrate a pool from one operating system to another. Imagine the following scenario – a company is getting new hardware and/or new IT experts and needs to migrate to a different OS. In my case it was from FreeBSD to Debian and vice versa. All data volumes were located in a single pool, but depending on the size of the company, it might be several pools instead. Before even thinking of migrating it is first important to make sure that all I/O related to the pool(s) to be migrated was stopped. When the coast is clear we can “zpool export <pool>” and begin our exodus to another operating system.

From FreeBSD to Debian
After exporting the zdata pool I installed Debian Testing/Stretch onto the system-bound SSD drive. ZFS is not part of the base installation, hence all pool imports need to be done after the system is ready and the zfs kernel module is built from the zfs-dkms and spl-dkms packages. apt resolves all dependencies properly so the only weak link is potential issues with building ZFS on GNU/Linux. Should no problems occur, we can proceed with importing the ZFS pool. GNU/Linux is cautious and warns the user about duplicate partitions/volumes. Those will not be mounted, even if the pool itself is imported successfully. Thankfully, conflicts can be resolved instantly by using a transition partition/drive to move data around. Once that’s done, our ZFS pool is ready for new writes. Notice that the content of /usr/local/ will undergo major changes as FreeBSD uses it for storing installed ports/packages and their configurations. In addition, /var/db will contain the pkg sqlite database with all registered packages. While this does not specifically interfere with either apt or Debian (apt configurations are in /var/lib and cached .deb packages in /var/cache/apt/archives), it’s important to take notice of.

From Debian to FreeBSD
Here, the migration is slightly smoother. The “bsdinstall” FreeBSD installer is designed in a more server-centric fashion (and ZFS is integral to the base system) so the ZFS pool can be connected and imported even before the first boot into the new system. The downside is that FreeBSD does not warn about “overmounting” system partitions from the zdata pool so it’s relatively easy to bork the fresh installation. Also, /var/cache will contain loads of unwanted directories and /usr/src, /usr/obj, /usr/ports and /usr/local need to be populated anew just like during a brand new FreeBSD installation.

Either way, the migration process is not too difficult and definitely not horrendously time-consuming. Should the user/administrator have PostgreSQL, MySQL or other SQL-like databases in /var/db, extra steps might need to be taken to ascertain forward and backward compatibility of the database packages. In the end, it’s a matter of knowing what each OS places where. FreeBSD is structured in a very intuitive and safe (from an administrator’s point of view) way. Debian, just like any other GNU/Linux distribution is a bit more chaotic, hence more caution is required. Both are good in their own regard, hence my incentive for migration testing.

FreeBSD – SSD + 2xHDD ZFS Installation

I recently got an extra 2 TB hard drive for my mighty (cough, cough, maybe some 9-10 years ago) HP Z200 workstation running FreeBSD 11.0-RELEASE so I decided to finally build a proper 2-drive RAID (Redundant Array of Independent Disks) mirror. I read the zfs and zpool manual pages (manpages) thoroughly on top of the related FreeBSD Handbook chapters and got to work. Since I also have a 160GB SSD inside that PC, some tinkering was required. The main issue was that SSD drives make use of TRIM for improved block device balancing. UFS provides TRIM support, but ZFS does not. Initially, I thought of having two separate ZFS pools – zroot for root-on-zfs and boot snapshots on the SSD and zdata for high volume data partitions like /usr and /var on the 2-drive array. However, after careful considerations I came up with a simpler partitioning scheme:

160GB Intel SSD:
MBR -> BSD:
141G     freebsd-ufs   (TRIM enabled; mounted as “/”)
8G         freebsd-swap

zdata mirrored array on 1.5T Seagate Barracuda + 2T WD Caviar Green:
GPT:
1.32T freebsd-zfs (on each drive)

With such a partitioning scheme I lost boot snapshots, though it was a lot easier to install the OS as I could rely on the standard FreeBSD installation procedure (bsdinstall) entirely. First, I performed a standard installation via bsdinstall onto the SSD. Next, I created a 2-drive ZFS pool and named it “zdata” following the Handbook. I made sure that all parent partitions like /usr and /var are mounted from the SSD and only the variable and expandable sub-directories like /var/db, /usr/ports, /usr/src, /usr/local, etc. are placed on the ZFS pool. Since each of those required a parent directory in the ZFS pool, I used /zdata/usr and /zdata/var, respectively. That way the /usr and /var mountpoints did not get overridden with empty /usr and /var directories from the ZFS pool. This protects the core system from getting wiped if one of the ZFS drives fails. In addition, the system can be reinstalled easily and the ZFS pool added later without major setbacks. The trick is that all ports are installed to /usr/local and the package manager database  is in the /var/db directory. Flexible, easy and extremely well documented.

Just to clarify, the above is no rocket science and can be done very easily with the tools immediately available in the core FreeBSD installation. This should really be highlighted more as apart from the descendants of Solaris, FreeBSD is the only operating system that offers such capabilities out-of-the-box. GNU/Linux systems have their own RAID and volume management tools, but they’re definitely not as established as ZFS. The GNU/Linux alternative to ZFS is btrfs, as it too combines a volume manager with a file system. However, key features like RAID-5/6 are still unstable and no GNU/Linux distribution offers btrfs-only setups.

FreeBSD – There and Back Again

I guess it should be no surprise that I returned to FreeBSD once more. One of the reasons I originally started learning C was to be able to help writing/fixing wireless drivers for FreeBSD. Although I haven’t reached that point of proficiency just yet, I feel FreeBSD is truly the place I belong after all. From the intrinsic order of a cathedral, through good programming practices and complete documentation to great system-level tools (jails, zfs, bhyve, etc.). Reading the most recent issue of Admin: Network & Security made it even clearer to me. GNU/Linux is growing strong in the server sector, with new GUI-driven tools and frameworks for container management. Personally, I think that’s awesome! It’s a win for the whole open-source world. However, Unix is more than just GNU/Linux – people often forget about Solaris/OpenIndiana and the various BSD-based operating systems (FreeBSD, OpenBSD, NetBSD, DragonflyBSD, etc.). They too are great server platforms, one can learn a lot from. There are scenarios in which a non-Linux Unix is more suitable for the same or similar tasks.

When I get hyped about a specific GNU/Linux distribution, I often consider how many and what people use that distribution. I typically stay clear of user-friendly distros for beginners. While I used to be a beginner also, the discussions in their forums and/or IRC channels don’t get me involved. Usually, the gist is that someone didn’t manage to accomplish something, because they decided not to read the documentation or not search the Web / said forums for a solution. I understand, we are there to help after all. However, the original poster needs to put in some effort, otherwise our aid is for naught. The other sort of issues appears after major releases. Something gets broken, because it was unintentionally changed and messed up a setup of this 1 in a 100 user. I’m then as infuriated as the sufferer, because these issues should not happen in the first place. Problems and inexperience drive me away from user-friendly distributions, but then again, these distributions garner the most users. We hear about OpenSUSE, Ubuntu, etc., but not so much about Gentoo, CRUX and others. It’s a conundrum I cannot solve. I end up gritting my teeth and plunging head first into the fray. Regrets come sooner than later, though.

Then, quite obviously I turn to FreeBSD once more. It’s extremely solid and doesn’t break, even when running -STABLE. When I don’t know something, I fire up man something from the command-line or read the respective chapter in the FreeBSD Handbook. Seldom, but still, for unanswered or new problems I ask in the forums. In most cases it turns out that the answer was indeed in the Handbook, just not in the chapter I would expect it to be. Fair game. One of the major concerns is hardware support. Agreed, it’s a tad behind GNU/Linux and Windows/MacOS X. However, the hardware that works, does so without a hitch. No forgotten or flaky drivers for common devices. It’s a matter of preference, but I’d rather have a narrower selection of compatible hardware I can trust anytime, than an empty claim that it works, there is a driver for it, while in reality it doesn’t. What about the popularity I mentioned earlier? There are quite some people on the many IRC channels and companies/organizations of importance in the world proactively choose FreeBSD for their servers. Recently, NASA decided to use FreeBSD for their project. Many more success stories are out there and they’re definitely a credit to FreeBSD’s outstanding quality. However, there is very little self-promotion compared to company-sponsored GNU/Linux distributions. The quality of FreeBSD seems to stem more from the honest work of community members, than merely writing about it (which I’m committing right now…). In that respect, FreeBSD is more community-driven than any GNU/Linux distribution that makes such claims.

I might change my mind at some point, but for now I’m happy to be back to FreeBSD. It’s secure, doesn’t break, keeps my data safe and helps me get the job done fairly quickly. All things considered, I would choose it anytime as my go-to server platform. Hope more people begin thinking alike.

Gentoo vs FreeBSD

I have been running FreeBSD 11-STABLE for some time, though I felt I need to switch to a GNU/Linux environment again. It so happens that we use CentOS 7.2 at work. In addition, FreeBSD is unfortunately not well suited for handling multiple Python versions with unusual libraries. Therefore, I needed something for developers, though not overloaded with fluff like Fedora (yet it would play perfectly with the CentOS environment at work). At this point, I considered either Arch Linux or Gentoo. After running Arch for a short while, I realized I’m a “tweaker” by nature. That’s what let me enjoy the FreeBSD Ports tree so much after all. The minimalism of Arch Linux is great, though tweaking comes with a price of having to locally fork packages and track them with desired features. Therefore, I required something as malleable as FreeBSD in “Linux Land”.

What might be obvious by now, I chose Gentoo. Thanks to my Bash scripts and a pre-defined kernel .config, it took me merely an evening to get things running on a measly ASUS ultrabook (my current mobile FOSS machine). I then began to realize how much FreeBSD and Gentoo have in common and where they differ. I thought it might be useful to some if I share my observations. Especially, that there is the Gentoo/FreeBSD project for those of us who prefer a more Unix-like kernel, yet don’t want to part with the GNU userland.

First off, the Ports Tree I gently mentioned as my prime fancy earlier. What changed is that I started using ports-mgmt/portmaster on FreeBSD more extensively. In many ways it’s similar to Gentoo’s Portage. The make.conf file for global configurations is present in both Unix-like operating systems as well, though it differs in purpose and syntax obviously. I believe the biggest difference, however is that FreeBSD flags need to be tracked on a per package basis, while Gentoo heavily relies on more centralized settings. Regardless, FreeBSD has a slight advantage by providing pre-built binary packages for most ports, unlike Gentoo. Sometimes building software from source can be annoying, specifically when the builds fail for reasons we cannot control.

Secondly, one can run Gentoo in a FreeBSD way – as a server and/or local distribution system. Portage has built-in software distribution features, while FreeBSD offers ports-mgmt/poudriere for that purpose. Where Gentoo falls short is FreeBSD’s jails, bhyve and outstanding firewall solutions (PF and IPFW). No wonder, though. “True” Unix-like operating systems, namely Solaris and the BSDs had these features integrated early on. GNU/Linux is just now experiencing a renaissance of container technologies and virtualization. Furthermore, ZFS on Linux still has some licensing issues which prevent clean kernel integration. For that reason alone Solaris/OpenIndiana/Illumos or FreeBSD are better picks in terms of extensive data storage.

Thirdly, Gentoo and FreeBSD share some BSD and GNU utilities, though that’s more an aspect of the open-source community, than either of the systems. Let’s face it, FreeBSD belongs to the Unix-like BSD world, while Gentoo is a GNU/Linux distribution. However, it is feasible to reproduce the FreeBSD “feel” in Gentoo to some extent. For instance, we can try to rely on as many BSD tools as possible. Also, GCC can be replaced with Clang for software builds if one so prefers. I am actually eager to test this in a Linux LXC container or a simple chroot environment soon.

Looking at the discussed point again, I’m beginning to wonder whether choosing a GNU/Linux distribution as my primary OS was such a good idea. Probably not, though as in the case of Debian earlier, it’s a matter of use scenarios. Gentoo gives me the flexibility and hardware+software support I need on mobile devices. Also, similarly to FreeBSD, it is extremely “sane” and values stability over Arch’s bleeding edge. I guess I will bear with the long compile times somehow, though on a proper workstation I would much prefer to run any of the BSDs instead.

Debian vs FreeBSD?

After great, though slightly lacking experience with FreeBSD on the desktop, I decided to look for the “next best thing” in Linux Land. In the end I chose Debian, and for good reasons, I think. There is so much that Debian and FreeBSD have in common that my observations might be useful to anyone considering switching between the two.

First, shortly on the dissimilarities between Linux and the classic Unices, such as FreeBSD, OpenBSD, NetBSD, Solaris and others. The main contesting point is the definition of the base system. The original Unix paradigm calls for tight integration between the kernel and core tool facilities. After all, the kernel means nothing without the userland interfaces that turn cryptic system calls into actual functionalities. Therefore, BSD systems are tightly developed as a whole (kernel + core userland). Linux was historically only the kernel, initially written by Linus Torvalds. The userland (often of GNU origin) was written later. However, while the core tool set is developed by different project teams, each GNU/Linux (GNU userland + Linux kernel) distribution defines some concept of a base system. Therefore, the Unix vs Linux differences are more historical from today’s perspective. An interesting essay by Eric S. Raymond (“The Cathedral and the Bazaar”) elaborates on the top-down (Unix) and bottom-up (Linux) approaches to software development and maintenance in regards to the base system. Below, a brief “compare and contrast” for the sake of completeness:

• File systems – UFS, ZFS (FreeBSD) vs ext4, xfs, btrfs, etc. (Debian); this does not cover file systems compatible with both Unix-like OS’ (ntfs, fat*, etc.)
• Init systems – BSD-style init scripts (FreeBSD) vs systemd (Debian) or sysVinit (Devuan)
• Package management – ports and binary packages (FreeBSD) vs binary packages and source tree used for development (Debian)
• Sound systems – modified OSS (FreeBSD) vs ALSA and/or PulseAudio (Debian)
• Hard drive naming –  adaXsYN or adaXpY for SATA and PATA drives, daXsY for USB drives (FreeBSD) vs sdNX for SATA or USB drives and hdNX for PATA drives (Debian) where X, Y are integers and N is a character
• Releases – CURRENT, STABLE, RELEASE (FreeBSD) vs Unstable, Testing, Stable (Debian)
• Containers and virtualization – jails and bhyve (FreeBSD) vs LXC, KVM, etc. (Debian)
• Other differences I did not mention

Important to note, Debian features listed above are common to most GNU/Linux operating systems. For instance, KVM is a Red Hat invention. Also, the differences between FreeBSD and GNU/Linux have blurred over the years. For instance, FreeBSD offers a lot of GNU tools as part of the Ports Tree, while ZFS and some useful BSD commands were ported to GNU/Linux. Can one then be considered better than the other?

In the end, it all boils down to specific use cases, I believe. Although both Debian and FreeBSD target servers, their strong points are substantially different. FreeBSD relies heavily on data protection and consistency (ZFS), but also slim virtualization (bhyve) and process isolation (jails). If those features are important to us and clearly they should be if we are running a multitude of network-facing services, FreeBSD is clearly the better choice. However, if we want a more general-purpose operating system that is consistent across multiple platforms (mini-computers, servers, workstations, desktops, etc.), Debian would be more suitable. The edge that the bazaar/Lego Linux design has is that one can tailor the operating system to specific needs without the need to use different Unix kernels. Furthermore, hardware and tech support for Linux-based operating systems are clearly more prevalent. It’s down to the availability of resources, though it is rather important to mention that a good server platform can potentially make a reliable and secure operating system for home use as well. Something worth remembering!