Different means of isolating applications and processes existed for decades, starting from chroot, through FreeBSD jails and Solaris zones to the most recent invention – Docker. The point is, all of those technologies serve slightly different purposes, despite the fact that people insist on arranging them into a linear, progressing hierarchy. For instance, chroots (or chroot jails) are meant to restrict the trapped user to a sub-directory, ideally isolated from the root of the host filesystem. It is a common, lightweight approach to isolate file access in FTP server daemons since it is fairly easy to implement. However, it lacks certain abstractions available in more advanced tools like jails and zones. So far, I have used chroots, Docker and FreeBSD jails, and I would like to share some key observations I made regarding these technologies.
Traditional chroot jails are extremely basic, nevertheless useful partially for that very reason. If all you want is a nested directory hierarchy for building software packages or similar purposes, chroots are good enough. Unfortunately, they do not prevent secondary access to the host environment via another chroot, neither do they limit resource consumption by the chroot’ed application. Therefore, care should be taken when using chroots in an Internet-fronting environment. The FTP context is moderately safe since a chroot-trapped user is limited by the FTP protocol. In general, chroots are most useful when they constitute a subsystem in a larger, more complex tool.
FreeBSD jails were developed as an improvement to the venerable UNIX chroot technology. In practice, they contain a complete FreeBSD userland, fully transparent to the host system, yet restricted by the host system kernel from within. From the outside they’re no different than an unpacked system tarball or disc image. The great thing about jails, though, is that they’re an integral part of FreeBSD and as such can be easily managed in a multitude of ways. Resource handling? Easy via rctl. Installing packages? The pkg package manager has a “-j” flag specifically for that purpose. What about enabling and starting/stopping daemons? Again, service has a “-j” flag and so does sysrc to control lines in /etc/rc.conf. The only tricky bit is network management, but this too can be easily scripted due to the modular nature of FreeBSD. The main issue with jails is that they’re a FreeBSD-only technology. This unfortunately restricts adoption, but so would any other technology requiring an intimate relationship with the operating system.
Docker is a lot more modern and similarly to FreeBSD jails, it relies on features specific to a single platform, namely Linux. It leverages kernel cgroups for privilege separation and a virtual filesystem (overlay2, previously aufs) for managing the underlying disk volume. Compared to FreeBSD jails, Docker containers are a lot more abstracted and communication is typically carried out through the docker daemon. In the case of docker volumes, I find it slightly distressing that the only access to the data is via the daemon. Another thing that differentiates it from FreeBSD jails is the default access level. The root user in Docker containers has absolute control over the data in the container, and on top of it can perform network-related actions which should be reserved to the host system. The biggest issue however arises when mounting host directories in the container. An unprivileged user in the container has root access to host directories. Frankly, FreeBSD jails entail similar risks and therefore the official announcement in the Handbook:
Jails are a powerful tool, but they are not a security panacea. While it is not possible for a jailed process to break out on its own, there are several ways in which an unprivileged user outside the jail can cooperate with a privileged user inside the jail to obtain elevated privileges in the host environment.
Most of these attacks can be mitigated by ensuring that the jail root is not accessible to unprivileged users in the host environment. As a general rule, untrusted users with privileged access to a jail should not be given access to the host environment.”
Despite all that and the fact that it’s the most recent of before-mentioned technologies, it quickly evoked significant appeal (aka hype) and is now both widely used, and integrated with many open-source tools and platforms like OpenStack, Cockpit and GitLab. In fact, nowadays it is very popular to provide ready-to-use Docker containers as part of the standard software package distribution. Security, as usual, is an afterthought since spinning up Docker containers is so trivial that it should be illegal.
The question still stands, though – Why are some technologies considered superior to others? From this competition I would immediately exclude chroot, since it is more of a tool for developing higher-level isolation concepts. However, what about Docker and FreeBSD jails? Why is Docker considered better? A general problematic trend I see in software development is trying to make tools do things they were not designed to do. Yes, oftentimes the product is successful and we’re amazed by the tool’s robustness. Alas, when it fails, we complain about its uselessness. That’s rather unfair, no? Jails are an extension of the chroot concept and are meant to hold long-running processes like database servers, Web servers, etc. You create a ZFS sub-volume, unpack a complete FreeBSD userland into it and route it to the Internet via the host packet filter. Thanks to PF’s design, it is easy to control data packet flow via TCP/IP down to a single packet. The ZFS sub-volume can be selectively backed up via per-volume snapshots, stored on another ZFS array or re-deployed at will. Hell, if we want, we can make it transient and completely obliterate it after the application finished running, a la Docker. The FreeBSD package building infrastructure, poudriere, makes heavy use of semi-transient jails. All of this seems sounds so trivial, n’est ce pas? Yet, the adoption compared to Docker is simply not there, because Docker was overly hyped and demonstrated to be easier than figuring out FreeBSD + jails for the average user.
This is where Docker wins – widespread integration and adoption. It is theoretically much easier (and faster!) for an average Joe to roll out a Docker container from an official pre-built image. Basically, almost any server appliance can be deployed as a Docker container. That doesn’t mean that Docker is necessarily better. Applications won’t work equally well on all container images due to differences in configuration and library versioning (say, CentOS vs Debian). More complex setups still require customization and writing dockerfiles is no easier or less clunky than deploying FreeBSD jails. Even more so, after some tinkering, I have to admit that setting up a FreeBSD jail with iocage was far easier than getting around Docker. The file system transparency inherent to jails is a big boon here. Regardless, orchestration tools like Chef or Ansible will help in both cases equally well. Furthermore, Docker was not designed for permanent storage from the get-go, but rather received this feature much later. Docker volumes are quite stable, but I still find it more convenient to create a ZFS volume and share it with the container rather than play around with Docker volumes. Again, this approach is much more transparent. Nevertheless, thanks to the overall hype, Docker was tested and used in production many times over already.
In conclusion, both Docker and FreeBSD jails have their respective niches in the IT ecosystem. For quick testing and deployment of lightweight applications (simple Python daemons, etc.) – Docker. For full-blown processes like a PostgreSQL or MySQL server, which require a dedicated IP or a mix of daemons – jails. If jails are not available, there is always KVM or a similar hardware virtualization platform which similarly emulates a complete ecosystem. The important thing is to understand how to choose the tool most suitable for a given use case. There is no better nor best, after all.