How to Manage Systemd Services and Units on ECS with the Systemctl Command
By Alexandru Andrei, Alibaba Cloud Community Blog author.
Almost all major Linux distributions have adopted systemd. Since
systemctl is an important component of systemd, you will often need to use this command to control various parts of your system. This utility is just one of many provided by the systemd software suite. Here are just a few examples of other programs included in this suite:
localectl. Some names immediately suggest what the tool does. You can easily guess that with
hostnamectl you can manage the hostname of your ECS instance, and with
timedatectl, you can adjust time and date settings. But while
systemctl suggests that the tool can manage the system, it's not obvious what parts of the system it can tweak. So, let's explore that first.
What Can You Manage with systemctl?
Most of the actions you perform with this command, revolve around units, which can be of various types, like service, socket, target, timer, and others (you will learn what these are, later in this article). Here’s a short list of what you will usually use systemctl for:
- Manage units (start, stop, restart, enable, disable, etc.)
- Analyze running units, see which ones work, which failed, check their status in various ways.
- Edit unit files that control the lifecycle of a process (services), or when an action will take place (timers), how a filesystem should be mounted (mount units), etc.
- Reboot and poweroff the system. You can also sleep/standby or hibernate, but not on servers; this only applies to home computers.
- Switch to a different target (previously called runlevel), or change the default target the system boots into. This is very rarely used on servers, though, because the
multi-user.targetis almost always the only one you need as the active/default boot target.
Most of the times, you will work with service units, a type of systemd unit that manages the lifecycle of a process. They control what programs start on your system, when they start, the command line switches passed to the applications, what happens when the process crashes (does it automatically restart?), and many other things, that you can read about in this systemd service manual.
Example of the Anatomy of a systemd Unit File
If you host your website with Nginx, you will have an
nginx.service file. This will be pulled in by your package manager when you install the utility. In this file, there are instructions for systemd, telling it how to start the nginx daemon process (ExecStart), what commands to execute to stop it (ExecStop), to reload its configuration (ExecReload), and so on. Here's how
nginx.service looks like on Ubuntu 18.04:
# Stop dance for nginx
# ExecStop sends SIGSTOP (graceful stop) to the nginx process.
# If, after 5s (--retry QUIT/5) nginx is still running, systemd takes control
# and sends SIGTERM (fast shutdown) to the main process.
# After another 5s (TimeoutStopSec=5), and if nginx is alive, systemd sends
# SIGKILL to all the remaining processes in the process group (KillMode=mixed).
# nginx signals reference doc:
Description=A high performance web server and a reverse proxy server
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/n
With systemctl, you can disable nginx from automatically starting up. You can restart the process, stop it, inspect its status, and much more. Practical examples will be offered in the rest of the article, as they are much easier to understand than theory alone.
As systemd (which systemctl is a part of) constantly evolves, some things may be changed or added in the future. This tutorial is based on systemd version 237 that was available on Ubuntu 18.04.2 LTS, at the time of writing.
You can check your current version of systemd with:
systemd Unit Types
In the commands offered as examples, you will see actions performed on services, which are just one type of unit. Although you will work with services most often, you may also need to touch other types of units, such as timers or sockets. The commands are similar though. For example, if you start a service with
systemctl start unit.service, you start a timer with
systemctl start unit.timer. Basically, the only thing that changes is the name of the unit.
The default location where unit files are placed is
/lib/systemd/system. For example, when you install the
openssh-server package, three files will be placed in
You can take a look at the current contents of this directory with
When you edit a unit with
systemctl edit, the override unit will be stored in
Here’s a short summary of the purpose of important unit types.
- Service — Usually manages the lifecycle of a daemon, like the SSH server, a web server like nginx or Apache, cron daemon, etc. But it is not limited to daemon/server utilities. These units can manage any kind of program/process, according to the instructions contained in the file. When does it start, what happens when it crashes, how it should restart, what should happen when it stops, are just a few of the things a service unit can control.
- Target — Contains a list of units, needed to achieve a certain system state. For example,
multi-user.targetis a common unit of this type. This groups together all the units that should be started to launch a multi-user environment, which is a fancy way of describing an operating system that multiple users can connect to and utilize it at the same time. If you run
systemctl list-dependencies, you can easily see how such targets work (press "q" when you want to quit, if you launched that command).
- Socket — With this, systemd can start a service only when there is a demand for it. For example, a visitor tries to connect to your server’s IP address, on port 1234. The daemon that should listen on that port is not currently running, maybe to save resources. However, systemd will listen for incoming connections on this port. As soon as it gets a request, it looks for the instructions in the corresponding
.socketfile, to start a particular service. This, in turn, launches the daemon that should listen on that port, and the request gets passed from systemd to the daemon (what info systemd received on its socket, it will pass to the daemon after it starts). Besides saving resources, this can also speed up the boot process, as services that are required later won't be unnecessarily launched at boot time. Furthermore, it helps the machine avoid having to load programs that are rarely used. This means that some daemons/utilities may run just once every few days, and only run for a few minutes (e.g., a remote backup request may trigger this, only when necessary). Besides network sockets, these unit types can also make use of IPC sockets or FIFO files, which means that even a program that needs another program can just send a request on a local (IPC) socket and start up the utility it requires.
- Timer — This does a similar job to cron. Simply put, an active timer unit contains information about what to do/run at a specific future time. Normally, these are periodic jobs that need to run at a specific time, daily, on particular days, weekly, or monthly, etc. Timers are not limited to specific times though. Instead of running a service every day at 12AM, you can set it to run one hour after the machine boots, for example. So, instead of having a fixed date and/or time, they have a relative trigger time that depends on another event.
You may notice other types of units, such as: device, mount, automount, path, swap, scope, slice. Most of them are automatically generated. For example, most
.mount units are generated after the system mounts filesystems listed in
/etc/fstab. Because of their (usually) dynamic/automatic nature, you will almost never need to touch these units.
To read the manual of a specific type of unit, append the unit type name after
man systemd.. For example:
Press “q” to quit the manual.
How to List Units Managed by systemd
It’s useful to maximize your terminal window before you continue. Some output is very wide and won’t fit properly, or get misaligned if your SSH client window is not wide enough. Run this command:
This shows you which units are loaded and active. You can scroll through this output with your arrow keys, or PAGE UP and PAGE DOWN. Press “q” to quit.
Note that some units are considered active even if a process has been started but it is not currently running anymore. Here is an example:
systemd-journal-flush.service loaded active exited Flush Journal to Persistent Storage
In this case, the systemd logging utility has been called to write the contents of logs stored in volatile memory, to disk. So, you can consider that the service is “active” because it has done its job and its effects continue even after the process has exited (logs are still stored on disk).
Other processes need to be currently running to be considered active. For example, this is the case for the SSH daemon that you use to connect to your instance.
ssh.service loaded active running OpenBSD Secure Shell server
You can easily recognize the type of a unit by its extension. “ssh.service” is obviously a service type unit, while “syslog.socket” is a socket type unit. The names you use in these commands are the actual names of the files that encapsulate the contents of that unit (some text that instructs systemd about everything the unit should do). These files are stored in
/lib/systemd/system, as mentioned earlier in this article. You will see how to explore their contents, later on.
How to Filter Units Listed by systemctl
To also list units that are inactive, run
systemctl list-units --all
Sometimes, you will want to filter units by their type. To list only those of the timer type:
systemctl --type timer list-units
Currently, you can choose one of the following types, to pass after the
--type command line switch:
How to Check Status of Units with systemctl
You can also list units, organized in a dependency tree:
Press “q” to quit this view.
Here, you can see how target unit types group units together. You can also easily see which units are active, which are inactive, and which are in a failed state, signaled by a green circle, gray and red one, respectively.
Such commands are used to give you a general view of the system and statuses of multiple units. But you can further explore the status of a particular unit, to get more information about it. Earlier, you saw that the SSH daemon is managed by the “ssh.service” unit. Explore its status with
systemctl status ssh.service
Here, you can see if the service is actively running (its daemon/process is currently active). You can also see if it’s enabled, which means, if it automatically starts at boot. If it’s disabled, the service can still be started, but manually, with a command entered by the user (or by socket-based activation). If you see something like this in the status
Loaded: masked (/dev/null; bad)
it means that the service is disabled and cannot be started manually either (a kind of “forced disable”; will be explained later).
Process: 673 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
shows you the process ID (PID) number (673 in this case) and the command used to start the service, or, in this case, the command used before starting the service,
/usr/sbin/sshd -t. The main service runs
/usr/sbin/sshd -D $SSHD_OPTS afterwards. You will see later how you can explore unit files to see exactly what they do.
To get a list of all unit files available on the system, run
Here, you can easily see which units are enabled, disabled, or masked. Static units are technically not “enabled” (to autostart at boot), but they’re usually started automatically ,either way, by something else. For example, “apt-daily.service” is started by “apt-daily.timer”.
How to Manage Units with systemctl
The commands in the next sections require superuser/root privileges. This means you either have to log in as the root user, or prefix the commands with “sudo”, if you’re logged in as a regular user (with sudo privileges). Example: if you’re not logged in as root, “systemctl disable nginx.service” should become “sudo systemctl disable nginx.service”.
Most of the times, you will manage services with systemctl, to start, stop, enable, disable, restart them. But these operations work on most of the other types of units, such as timers, sockets, and so on. The commands for most actions are very straightforward, just pass the name of the action you want to perform to systemctl. For example,
systemctl disable nginx.service
would disable the nginx HTTP server from automatically starting up after booting the system.
To re-enable it:
systemctl enable nginx.service
Make Sure a Unit Can Never Start (Mask Unit)
Sometimes though, even if you stop a unit and disable it from starting at boot, you will notice that the unit may start “by itself,” later on. This happens in the case of units that are configured in such a way, that they may be launched by other units or utilities in case of necessity (e.g., socket-based activation). To make sure that the unit cannot be started under any circumstances, you can force-disable it with
systemctl mask nginx.service
nginx.service file to
/dev/null, so that when systemd or any other utility tries to access the unit file, it gets redirected to
/dev/null, which contains nothing (it's a virtual empty/void file). Thus, the unit cannot be started anymore since null contents means systemd has no instructions available on how to run/manage the unit.
To unmask it, so that you can re-enable it at boot, start it manually, or let dependencies automatically start it again:
systemctl unmask nginx.service
In some cases, you may get an error like this one when trying to mask a file:
Failed to mask unit: File /etc/systemd/system/nginx.service already exists.
In this case, you should first move the override file somewhere else, or remove it if you’re sure you don’t need it. For example, first move the file to the
mv /etc/systemd/system/nginx.service /root
Afterwards, you can mask the unit.
Manage Unit Running State with systemctl
To start a unit:
systemctl start nginx.service
And to stop it:
systemctl stop nginx.service
If this doesn’t work, because the program is blocked, became unresponsive in some way, you can forcefully stop it with:
systemctl kill -s SIGKILL nameof.service
Since this is a very abrupt end to the program code, use it as a last resort. The process won’t have a chance to close files and connections properly, finish current jobs, save progress, etc.
Restarting a unit is useful if it’s currently in a failed state and you fixed whatever was causing it. There are other scenarios where it can be useful though, for example, after you make configuration changes.
systemctl restart nginx.service
Note that this (usually) completely stops (quits) the process and starts it again, which means that in the short time it takes to relaunch, some connections might get refused, active connections get dropped, and generally, whatever jobs it was running will get interrupted. That’s why, instead of restarting, you have the option of reloading a unit. If the daemon/process included in the unit supports a so-called “graceful restart,” it will reload its new configuration without interrupting active jobs or refusing incoming ones.
systemctl reload nginx.service
How to View and Edit Units with systemctl
You can display the contents of a unit file with
systemctl cat, for example:
systemctl cat ssh.service
To edit the file, use
systemctl edit --full ssh.service
Press CTRL+X to exit the editor. If you make changes to the file and want to save them, press CTRL+X, then press “y” and then ENTER.
If, later on, you decide that your changes aren’t necessary anymore, or, you made a mistake and don’t remember the previous, original contents, you can easily undo the changes with
systemctl revert ssh.service
This will restore the “factory default” version, the original file that your distribution shipped with when you installed the package that contained the particular unit.
How to Reboot, Shutdown, Standby and Hibernate with systemctl
On servers, you will normally only use the first two operations:
to reboot, and
to shutdown the machine/virtual server.
The next two operations are irrelevant to ECS instances since these are virtual, not physical machines. You are getting a portion of CPU time, memory and other types of resources, from a real server. You don’t own or control the hardware, so, you can’t put an ECS instance into standby mode. However, for reference, for those that use Linux-based distributions on their home device(s), you can put it/them in standby mode, to save power, and quickly resume work later, with:
Another way to preserve the running state of your device — -keep the same open windows, text typed in, programs working in the background and the progress they made — -is to hibernate it. This is useful when you have little battery left on a laptop, or you just want to completely power off a computer and resume exactly where you left off when you next boot it. Since hibernating saves the entire machine state to disk and then completely powers off the device, even if your battery runs out entirely, you can resume work as soon as you get access to a power source.
Note: Hibernate will fail if your swap isn’t large enough to save memory contents. Hibernate can also be problematic sometimes if you use proprietary video drivers. Specifically, when you resume boot later, you might get stuck with a black screen, because the video card fails to re-initialize properly. Test out the hibernate feature a few times, before using it to save and resume important work. If it works the first time, it doesn’t mean it will work every time. Some proprietary drivers just randomly fail upon resuming from hibernate. Use open-source drivers when possible.
Another fancy feature is to save RAM contents to disk (hibernate) and then put the computer in standby. This is useful because the computer can resume much faster if it doesn’t lose power, since it resumes directly from RAM. However, if it loses power (e.g., battery runs out while in standby), the computer can still resume work by reading the RAM contents it saved to disk. This is called a hybrid sleep and can be entered into with:
These are the scenarios for which systemctl is most commonly used for. You can also read the systemctl manual, online, which often gets updated with new information after the utility receives new features.