Systemd, or How I Learned to Stop Worrying and Love Newness

Working in the IT world where things are not yet fully connected, integrated, and automated leaves a plenty of opportunities for engineers to get their hands dirty by hard DevOps work. Big Data Engineering is such a field of work where there is a lot of cogs lying around waiting to be integrated. In other words, DevOps and, in this particular case, linux are inevitable and inseparable parts of Big Data Engineering. This is a story about one of those cogs.

Systemd is a relatively new addition to the Linux ecosystem. It is a system and service manager that is compatible with SysV and LSB init scripts and can work as a drop-in replacement for sysvinit. Systemd’s inception was followed by a huge controversy. The flame war is not yet fully extinguished, but systemd has become a part of many Linux distributions and, in my opinion, is here to stay. This post describes my encounter with systemd.

Prelude

The story started naively, as it usually does - another day, another DevOps task. This time, creating an Ansible role for deploying OS metrics , one of our tools at SmartCat Labs , was needed.

OS metrics is one of the goodies in SmartCat’s monitoring toolbox. It is an OS metrics data shipper or, in project terms, a reporter that sends collected data as a stream of events to Riemann server. More about the monitoring stack we use can be found in a separate blog post ― Monitoring stack for distributed systems .

So far, so good.

Plot

OS metrics is a python project. To start it manually, a python command like the following has to be executed.

$ python start.py -h <riemann server address>

As we want the OS metrics running as a system service, we need to provide an extra step to ‘daemonize’ it. The recommended way to do this in Ansible is to use the service module . It is an Ansible core module responsible for controlling services on remote hosts that supports a variety of init systems (BSD init, OpenRC, SysV, Solaris SMF, systemd, upstart).Theservicemodule provides a nice abstracted way of service handling. An Ansible task for starting could look like this:

- name: Start OS Metrics service service: name=osmetrics enabled=yes state=started

This is not enough to have a service up and running. Depending on the underlay init system, a service script or definition is needed.Even without a conciseness thought, the reflex was to provide a SysV init script. Old habits die slowly, and the /etc/init.d/mantra is carved deeply into the gray matter. So, I extended the task like this:

- name: Install OS Metrics service init script template: src=osmetrics.j2 dest=/etc/init.d/osmetrics mode=664 register: init-script - name: Restart OS Metrics service service: name=osmetrics enabled=yes state=restarted when: init-script|changed

Then, I started looking for an init script, suitable for starting/stopping python process. One can have a favorite SysV init script, carefully tailored over the years and used in many situations (my precious!). I, on the other hand, belong to those that look for it out there every time. The thing is that you can never find a perfect one; it’s so elusive. You could end up with a SystemV init script template like this or this and boldly go where no man has gone before. But in the end it could resemble something like this (as it was in my case):

#!/bin/bash # # chkconfig: 35 90 12 # description: Foo server # # Get function from functions library . /etc/init.d/functions # Start the service FOO start() { initlog -c "echo -n Starting FOO server: " /path/to/FOO & ### Create the lock file ### touch /var/lock/subsys/FOO success $"FOO server startup" echo } # Restart the service FOO stop() { initlog -c "echo -n Stopping FOO server: " killproc FOO ### Now, delete the lock file ### rm -f /var/lock/subsys/FOO echo } ### main logic ### case "$1" in start) start ;; stop) stop ;; status) status FOO ;; restart|reload|condrestart) stop start ;; *) echo $"Usage: $0 {start|stop|restart|reload|status}" exit 1 esac exit 0

Your mileage may vary depending on the distribution, distribution version, personal preferences and so on. But in the end, it looks very familiar and, what’s most important, it is simple and easily understandable in every of its incarnations. Right? In some cases, yes. In some other cases, you are supposed to meditate and live an ascetic life for several years in order to achieve the spiritual level needed for understanding it.

Once a decent init script was in place, I proceeded with the Ansible role using a CentOS 7 minimal Vagrant box as the test platform (CentOS 7 was the target platform). However, it started complaining right away that theservice unit filehad been changed and thatsystemctl daemon-reloadhas to be executed.

Newness

It didn’t take me long to realize that CentOS 7 is using systemd to manage system services. Even if interacting with the init V script directly,/etc/init.d/functionsprovides a way for systemd to hijack the process.

A close inspection of/etc/init.d/functionsrevealed thatSYSTEMCTL_SKIP_REDIRECTvariable can be used to avoid systemd completely. For example, you could add a line like this...

SYSTEMCTL_SKIP_REDIRECT=1

...into your init script before executing. /etc/init.d/functionsand you should be good.

But I decided to embrace the newness and to give it a try to prove itself, despite all the horrible things that could be found googling for systemd. Most of these accusing posts were published in 2014 when systemd was introduced in some of the popular Linux distributions. Since then, systemd has gained a wider acceptance. Now there are plenty of overviews, documentation, and how-to resources. For example, this DigitalOcean community tutorial or RedHat System Administration Guide provide very good introductions on the subject, as well as basic usage instructions.

Unfolding Systemd is a system and service manager for Linux operating systems. Except for daemons, it can control different system tasks such as file system mount points, timers, socket and D-Bus activation. Systemd introduced the concept of units (systemd units). Each unit represents a single systemd responsibility/task. The list of available unit types can be found