Linux instances and other *nix (Unix-like) based operating systems in general (e.g. the BSD releases) come with great logging utilities preinstalled. These catch all of the important boot, kernel, system and program messages, errors, warnings, notices, and everything that might be useful to help monitor and debug the software running on the platform. Traditionally, this has been handled by
rsyslog daemons. Each version brought incremental improvements,
rsyslog being the log daemon that has been used by most of the Linux based operating systems in the latest years.
Since most of the popular Linux distributions have migrated to systemd, there is an ongoing shift to another logging daemon,
rsyslog separates log messages to different files such as
/var/log/syslog and so on,
journald centralizes everything in one place. It also uses the binary format instead of the text format to store data. This can offer some advantages. For example, it makes it easier to see how different events from multiple system areas relate, since messages are sorted by time and displayed in a single log. This can make it easier to connect the dots and follow the chain of errors to see where a particular problem started. Filtering and sorting messages, changing output type and various tasks are also easier to do because of the way
systemd-journald stores data. Another helpful feature is that the utility used to display logs, automatically formats text, highlighting important pieces of information in bold letters or different colors; e.g. errors are displayed in red, making it easier to notice them as you're scrolling through text.
Note: The steps in this tutorial have been tested on Debian and Ubuntu. Although highly unlikely, other distributions may require adapting the commands presented here. This can happen if the distribution uses an older version of systemd or defines different paths for configurations files. Some commands require administrator privileges, so you should be logged in as the root user or prefix the command with
sudo if required.
Enable Persistent Logs
On most Linux distributions,
rsyslog is running in parallel with systemd's logging daemon.
rsyslog log files are kept in the
/var/log directory. For this reason, you will usually find that
systemd-journald will only keep volatile log messages for the current boot (logs are kept in memory instead of disk). To enable persistent logging, so that you can use
journalctl to consult logs from previous boots, check the following:
grep Storage /etc/systemd/journald.conf
If the output is
#Storage=auto, it means that if the directory
/var/log/journal exists, then journald will keep persistent logs at that location. If it doesn't exist, it will keep logs only in volatile memory, which will be lost when rebooting or shutting down the instance.
Check if the directory exists:
If it exists you can skip creating it and restarting the journal daemon. If it doesn’t, create it, to keep logs on disk:
mkdir -p /var/log/journal
Then restart the daemon to apply the changes:
systemctl restart systemd-journald
Storage is set to another value such as
none, you can edit the configuration file and set it to
sed -i '/Storage/ c\Storage=persistent' /etc/systemd/journald.conf
The command above searches for any line containing the string “Storage” and replaces it with “Storage=persistent”.
Restart the journal to apply the changes:
systemctl restart systemd-journald
To learn more about the values you can set in
Press “q” to quit the manual.
You can edit the configuration file with:
Check the variables that set limits on how many log archives can be kept on the system. Here’s an example from the Debian 9 operating system:
This limits the amount of log files that can be kept to 100. You can also limit by disk space if you want to. To edit a variable, delete the preceding # sign. “#” signals that the line is commented and not active, but these commented lines show you the default values that are used even if not set in this file. To change the maximum number of journal files to be kept around, you would edit the file like this:
You would then press CTRL+X, then “y”, then ENTER, to save the file and you would apply the changes with:
systemctl restart systemd-journald
Log messages are prefixed by date and time of event. But for most people, the timezone used on an Alibaba Cloud ECS instance won’t coincide with the timezone the user is currently located in. Check the timezone set on the instance with:
To see the available timezones you can apply:
You can press PAGE UP and PAGE DOWN to navigate this list. Press “q” to quit.
The command for setting the timezone is
timedatectl set-timezone Contintent/City. The name of the continent and name of city have to start with capital letters. Example command:
timedatectl set-timezone Europe/Amsterdam. Now the
journalctl utility will show the timestamps of events in your local time.
Filtering Journal Output
The command you use to interact with journal files is
journalctl. To be able to read the system journals, you either have to be logged in as the root user or prefix the command with
sudo, if you're logged in as a regular user that has sudo privileges. Another way to view system log files, from a regular user account, is to add that user to the
systemd-journal group. You can do this with a command such as
adduser name_of_user systemd-journal. Then log out if you're already logged in as that user and log back in. Now you can view journal files without having to prefix the command with sudo and enter your password.
If you run:
The available journal files will be concatenated and displayed to the user. The
less utility is used to display the text, which means you can use some of its keyboard shortcuts. For example, to go to the end of the text displayed press >. To go back to the beginning you can press <.
You can search for text from the position that is currently displayed, and continuing down, by pressing /, entering the text you want to search for, e.g. “ssh” and then pressing ENTER. Matched text will be highlighted in white. Pressing n will search for the next occurrence and N will search for the previous. To search upwards instead of downwards, you can press ? and then enter your text. Press q to quit the utility. Consult
man less for more information.
Filter by Boot Number
Every time you boot your system, a new boot entry is created in the journal. You can see all of the available ones with:
root@instance1:~# journalctl --list-boots
-4 b995cd57ae72431ba9be9560f28f80f7 Wed 2018-08-08 22:59:57 EEST—Wed 2018-08-08 23:00:04 EEST
-3 9b069b7c07a74e1a9d07d7ba830c22a8 Wed 2018-08-08 23:00:07 EEST—Wed 2018-08-08 23:32:35 EEST
-2 99439be7b99a4f67ad7b36768a508adc Wed 2018-08-08 23:32:39 EEST—Wed 2018-08-08 23:33:31 EEST
-1 0a261487baad44a394f86475ee5392f3 Wed 2018-08-08 23:33:35 EEST—Wed 2018-08-08 23:33:52 EEST
0 6ee25c9b1cd94c7aaed4db674c14d431 Wed 2018-08-08 23:33:57 EEST—Wed 2018-08-08 23:49:17 EEST
They’re all prefixed by a number, starting from 0, which means current boot. “-1” signifies, one boot ago, “-2” two boots ago and so on. To view messages from one particular boot you can use
journalctl -b 6ee25c9b1cd94c7aaed4db674c14d431. But this requires copy-pasting that series of characters. You can also pick a boot by using its index number:
journalctl -b -1
Press “q” to quit.
Filter by Time
To filter by time of events, you can use the
-U command line switches.
-S can be replaced with
You can use
year-month-day hour:minute:second as arguments. Or you can use just the date part or just the time part. You can also type
today instead of
yesterday are also understood. And probably the most useful type of time arguments are relative values such as
-1h which refers to 1 hour ago.
To see messages since the end of January 2018:
journalctl -S 2018-01-31
To see messages generated since midnight:
journalctl -S 00:00
journalctl -S today
To see messages from the last hour:
journalctl -S -1h
Such commands cut content from the beginning. But you may want to cut from the end. If you don’t want to see messages from the last hour:
journalctl -U -1h
These filters can be combined. If you want to see only the content generated yesterday, removing everything else:
journalctl -S yesterday -U 00:00
Filter by Priority
Filtering by message priority can often be useful, especially in cases where you need to debug something. When you’re looking for things that are malfunctioning, normal status messages are of no use. To only list errors:
journalctl -p err
To list warnings:
journalctl -p warning
Other types of priority levels are: alert, crit, debug, emerg, err, info, notice, warning. These are explained in the syslog manual which can be consulted with
man 3 syslog.
You can combine different types of filters together. To get errors generated in the last hour:
journalctl -p err -S -1h
Filter by Unit, File or Device Path
Sometimes you will only want to see logs associated with a particular service on your system. For example, maybe you want to see everything related to the SSH daemon:
journalctl -u ssh.service
To see what services generate log messages, you can type
journalctl -u at the terminal, add a space after
-uand then press TAB two times.
To filter messages based on the running process’ file path:
If you want to see log messages related to a device, for example the first virtual disk on your system:
To show only logs generated by the Linux kernel:
Control journalctl Output
By default, the journal is displayed starting with the oldest entry available. But most of the times, you will want to see the recent entries first. To do this, you can use the
-e switch in a command such as
journalctl -b 0 -e to go to the end of the journal from the last boot. This will also limit the output to 1000 lines. To display more lines you can use the
-n switch. To see the output from the last boot, go to the end of it and limit the number of lines to 10000:
journalctl -b 0 -e -n 10000
At times, you may find yourself trying to reproduce errors or test functionality. Instead of looking at what happened in the past, in such cases you’ll want to see log output live and then interact with a program so that it generates events you can see in real time. The journalctl utility has a “follow” option:
To exit this mode press CTRL+C.
journalctl output can also be reversed so that messages are displayed starting with the newest entry and ending with the oldest:
journalctl -o json
To make it “pretty”, meaning also easily readable by humans:
journalctl -o json-pretty
To see all the details the logging daemon actually has about each event:
journalctl -o verbose
Log messages can also be enriched by the so-called Message Catalog. For specific events found in the catalog, this will add explanations, possible solutions, links, or any kind of additional information that might be useful to the user.
Manage Disk Space Used by the Journal
To see how much storage space the journal files are using:
To free up space, you have two options, either to delete files based on time, or delete files based on how much space they are using. The following commands will only delete archived journal files. Actively used journal files won’t be touched. To delete anything older than a certain time:
You can also use “days”, “weeks” and “years”.
To delete based on how much disk space is used:
This will keep at most 2 gigabytes of archived journal files and delete the rest.
M can be used for megabytes.
To learn more about journalctl, consult the manual with