systemdジャーナルを使用した一時的な問題のトラブルシューティング

ほとんどの場合、すべてのジャーナルエントリを一覧表示して手動で検索する必要はなく、望ましくもありません。特定のサービスに関連するエントリを探す場合もあれば、特定の時間に発生したエントリを探す場合もあります。 journalctl コマンドは、検索したいデータのみを表示できる強力なオプションを提供します。

-list-bootsから始めます オプション。ジャーナルエントリが存在する期間中のすべてのブートを一覧表示します。 journalctl.confに注意してください ファイルは、ジャーナルエントリが特定の経過時間に達した後、またはジャーナルが使用するストレージデバイス（HDD / SSD）スペースが指定された最大量に達した後に破棄されることを指定する場合があります：

[root@testvm1 ~]# journalctl --list-boots 

-10 dcb6dcc0658e4a8d8c781c21a2c6360d Mon 2020-06-08 07:47:20 EDT—Mon 2020-06-08 07:53:05 EDT

 -9 7d61951a85f445c5946774aaae8bc2a4 Fri 2020-07-03 15:50:06 EDT—Fri 2020-07-03 18:21:23 EDT

 -8 1b3a847577e544b4a2679fe576b62206 Fri 2020-07-03 18:21:58 EDT—Fri 2020-07-03 22:10:54 EDT

 -7 5fef01a3568743af99118107ca6f61ae Fri 2020-07-03 22:18:41 EDT—Sat 2020-07-04 06:50:19 EDT

 -6 6e944f99ebd9405984090f829a927fa4 Sat 2020-07-04 07:33:25 EDT—Sat 2020-07-04 07:58:59 EDT

 -5 ec8b0c81ca4744b1bad071bdef432959 Sat 2020-07-04 08:12:06 EDT—Sat 2020-07-04 09:12:47 EDT

 -4 cb173ec716824e21b87ccf6cb43a9a99 Sat 2020-07-04 10:19:53 EDT—Sat 2020-07-04 11:31:03 EDT

 -3 4fe354a893194409843aa9623a36dbb0 Sat 2020-07-04 07:59:58 EDT—Sun 2020-07-05 09:39:30 EDT

 -2 06fb81f1b29e4f68af9860844668446c Mon 2020-07-06 06:27:05 EDT—Mon 2020-07-13 08:50:06 EDT

 -1 33dbf8e6b9de4113a591c4f487d0ac37 Mon 2020-07-13 04:50:33 EDT—Thu 2020-07-16 13:49:32 EDT

  0 75c9b70913934748b5396b3b172bee50 Mon 2020-07-20 08:43:01 EDT—Fri 2020-07-24 12:44:06 EDT

最新のブートIDが下部に表示されます。これは、長くてランダムな16進数です。このデータを使用して、特定のブートのジャーナルを表示できます。これは、左端の列のブートオフセット番号または2番目の列のUUIDを使用して指定できます。このコマンドは、オフセットが -2のブートインスタンスのジャーナルを表示します。 -現在のブートから2番目に前のブート：

[root@testvm1 ~]# journalctl -b -2

-- Logs begin at Mon 2020-06-08 07:47:20 EDT, end at Fri 2020-07-24 12:44:06 EDT. --

Jul 06 06:27:05 testvm1.both.org kernel: Linux version 5.7.6-201.fc32.x86_64 ([email protected]) (gcc version 10.1>

Jul 06 06:27:05 testvm1.both.org kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.7.6-201.fc32.x86_64 root=/dev/mapper/VG01-root ro >

Jul 06 06:27:05 testvm1.both.org kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'

Jul 06 06:27:05 testvm1.both.org kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'

<SNIP>

Or you could use the UUID for the desired boot. The offset numbers change after each boot, but the UUID does not:

[root@testvm1 ~]# journalctl -b 06fb81f1b29e4f68af9860844668446c

The -u option allows you to select specific units to examine. You can use a unit name or a pattern for matching, and you can use this option multiple times to match multiple units or patterns. In this example, I used it in combination with -b to show chronyd journal entries for the current boot:

[root@testvm1 ~]# journalctl -u chronyd -b

-- Logs begin at Mon 2020-06-08 07:47:20 EDT, end at Sun 2020-07-26 09:10:47 EDT. --

Jul 20 12:43:31 testvm1.both.org systemd[1]: Starting NTP client/server...

Jul 20 12:43:31 testvm1.both.org chronyd[811]: chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCD>

Jul 20 12:43:31 testvm1.both.org chronyd[811]: Frequency -0.021 +/- 0.560 ppm read from /var/lib/chrony/drift

Jul 20 12:43:31 testvm1.both.org chronyd[811]: Using right/UTC timezone to obtain leap second data

Jul 20 12:43:31 testvm1.both.org systemd[1]: Started NTP client/server.

Jul 20 12:44:00 testvm1.both.org chronyd[811]: Selected source 192.168.0.52

Jul 20 12:44:00 testvm1.both.org chronyd[811]: System clock TAI offset set to 37 seconds

Jul 20 12:44:00 testvm1.both.org chronyd[811]: System clock wrong by 1.412227 seconds, adjustment started

Jul 20 12:44:01 testvm1.both.org chronyd[811]: System clock was stepped by 1.412227 seconds

[root@testvm1 ~]#

Suppose you want to look at events that were recorded between two arbitrary times. You can also use -S (--since ) and -U (--until ) to specify the beginning and ending times. The following command displays journal entries starting at 15:36:00 on July 24, 2020, through the current time:

[root@testvm1 ~]# journalctl -S "2020-07-24 15:36:00"

And this command displays all journal entries starting at 15:36:00 on July 24, 2020, until 16:00:00 on July 25:

[root@testvm1 ~]# journalctl -S "2020-07-24 15:36:00" -U "2020-07-25 16:00:00"

This command combines -S , -U , and -u to give journal entries for the NetworkManager service unit starting at 15:36:00 on July 24, 2020, until 16:00:00 on July 25:

[root@testvm1 ~]# journalctl -S "2020-07-24 15:36:00" -U "2020-07-25 16:00:00" -u NetworkManager

Some syslog facilities, such as cron, auth, mail, daemon, user, and more, can be viewed with the --facility オプション。 You can use --facility=help to list the available facilities. In this example, the mail facility is not the Sendmail service that would be used for an email service, but the local client used by Linux to send email to root as event notifications. Sendmail actually has two parts, the server, which (for Fedora and related distributions) is not installed by default, and the client, which is always installed so that it can be used to deliver system emails to local recipients, especially root:

[root@testvm1 ~]# journalctl --facility=mail

The journalctl man page lists all the options that can be used to narrow searches. The table below summarizes some of the options I use most frequently. Most of these options can be used in various combinations to further narrow a search. Refer to my previous article Analyzing systemd calendar and timespans for details on creating and testing timestamps, as well as important tips like using quotes around timestamps.

Options to narrow searches of the journal

Other interesting options

The journalctl program offers some other interesting options, as well. These are useful for refining the data search, specifying how the journal data is displayed, and managing the journal files.

Additional interesting journalctl options

I'll explore some of these entries below. More options can be found in the journalctl man page.

Journal files

If you have not already, be sure to list the files in the journal directory on your host. Remember that the name of the directory containing the journal files is a long, random number. This directory contains multiple active and archived journal files, including some for users:

[root@david ~]# cd /var/log/journal/ad8f29ed15044f8ba0458c846300c2a4/

[root@david ad8f29ed15044f8ba0458c846300c2a4]# ll

total 352308

-rw-r-----+ 1 root systemd-journal  33554432 May 28 13:07 system@0c91aaef57c441859ea5e421edff6528-0000000000000001-0005a6703120d448.journal

-rw-r-----+ 1 root systemd-journal 109051904 Jun 23 21:24 system@0c91aaef57c441859ea5e421edff6528-0000000000008238-0005a6b85e4e03c6.journal

-rw-r-----+ 1 root systemd-journal 100663296 Jul 21 18:39 system@0c91aaef57c441859ea5e421edff6528-0000000000021f3e-0005a8ca55efa66a.journal

-rw-r-----+ 1 root systemd-journal  41943040 Jul 30 09:34 system.journal

-rw-r-----+ 1 root systemd-journal   8388608 May 28 13:07 user-1000@037bcc7935234a5ea243b3af304fd08a-0000000000000c45-0005a6705ac48a3c.journal

-rw-r-----+ 1 root systemd-journal  16777216 Jun 23 21:24 user-1000@bc90cea5294447fba2c867dfe40530db-0000000000008266-0005a6b85e910761.journal

-rw-r-----+ 1 root systemd-journal  41943040 Jul 21 16:08 user-1000@bc90cea5294447fba2c867dfe40530db-0000000000021f4b-0005a8ca68c83ab7.journal

-rw-r-----+ 1 root systemd-journal   8388608 Jul 30 09:34 user-1000.journal

[root@david ad8f29ed15044f8ba0458c846300c2a4]#

You can see the user files in this listing for the user ID (UID) 1000, which is my Linux account. The --files option allows me to see the content of specified files, including the user files:

[root@david ad8f29ed15044f8ba0458c846300c2a4]# journalctl --file user-1000.journal

<SNIP>

Jul 29 14:13:32 david.both.org tumblerd[145509]: Registered thumbnailer /usr/bin/gdk-pixbuf-thumbnailer -s %s %u %o

Jul 29 14:13:32 david.both.org Thunar[2788]: ThunarThumbnailer: got 0 handle (Queue)

Jul 29 14:13:32 david.both.org Thunar[2788]: ThunarThumbnailer: got 0 handle (Error or Ready)

Jul 29 14:13:32 david.both.org Thunar[2788]: ThunarThumbnailer: got 0 handle (Finished)

Jul 29 14:15:33 david.both.org tumblerd[145552]: error: Broken zip file structure

Jul 29 14:20:34 david.both.org systemd[2466]: dbus-:[email protected]: Succeeded.

Jul 29 14:34:17 david.both.org systemd[2466]: Starting Cleanup of User's Temporary Files and Directories...

Jul 29 14:34:17 david.both.org systemd[2466]: systemd-tmpfiles-clean.service: Succeeded.

Jul 29 14:34:17 david.both.org systemd[2466]: Finished Cleanup of User's Temporary Files and Directories.

Jul 29 14:48:26 david.both.org systemd[2466]: Started dbus-:[email protected].

Jul 29 14:48:26 david.both.org tumblerd[145875]: Registered thumbnailer gsf-office-thumbnailer -i %i -o %o -s %s

<SNIP>

This output shows, among other things, temporary file cleanup for the UID1000 user. Data relating to individual users may be helpful in locating the root cause of problems originating in user space. I found a number of interesting entries in this output. Try it on your host and see what you find.

Adding journal entries

It can be useful to add your own entries to the journal. This is accomplished with the systemd-cat program that allows piping the STDOUT of a command or program to the journal. This command can be used as part of a pipeline on the command line or in a script:

[root@testvm1 ~]# echo "Hello world" | systemd-cat -p info -t myprog

[root@testvm1 ~]# journalctl -n 10

Jul 27 09:01:01 testvm1.both.org CROND[976442]: (root) CMD (run-parts /etc/cron.hourly)

Jul 27 09:01:01 testvm1.both.org run-parts[976445]: (/etc/cron.hourly) starting 0anacron

Jul 27 09:01:01 testvm1.both.org run-parts[976451]: (/etc/cron.hourly) finished 0anacron

Jul 27 09:07:53 testvm1.both.org unknown[976501]: Hello world

Jul 27 09:10:47 testvm1.both.org systemd[1]: Starting system activity accounting tool...

Jul 27 09:10:47 testvm1.both.org systemd[1]: sysstat-collect.service: Succeeded.

Jul 27 09:10:47 testvm1.both.org systemd[1]: Finished system activity accounting tool.

Jul 27 09:10:47 testvm1.both.org audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/syst>

Jul 27 09:10:47 testvm1.both.org audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/syste>

Jul 27 09:17:10 testvm1.both.org myprog[976516]: Hello world

[root@testvm1 ~]#

The -p option specifies a priority, emerg, alert, crit, err, warning, notice, info, debug, or a value between 0 and 7 that represents each of those named levels. These priority values are the same as those defined by syslog(3) 。 The default is info. -t option is an identifier, which can be any arbitrary short string, such as a program or script name. This string can be used for searches by the journalctl コマンド：

[root@testvm1 ~]# journalctl -t myprog

-- Logs begin at Mon 2020-06-08 07:47:20 EDT, end at Mon 2020-07-27 09:21:57 EDT. --

Jul 27 09:17:10 testvm1.both.org myprog[976516]: Hello world

[root@testvm1 ~]#

Journal management

I use the --disk-usage option to check on journal sizes, along with other commands relating to disk usage, to ensure that my /var filesystem is not filling up:

[root@testvm1 ~]# journalctl --disk-usage 

Archived and active journals take up 136.0M in the file system.

[root@testvm1 ~]#

The disk usage for the journals on the testvm1 host is about 136MB. The result on my primary workstation is 328MB, and the host I use for my firewall and router uses 2.8GB for the journals. Journal sizes depend greatly on the host's use and daily run time. My physical hosts all run 24x7.

The /etc/systemd/journald.conf file can be used to configure the journal file sizes, rotation, and retention times to meet any needs not met by the default settings. You can also configure the journal storage location—you can specify a directory on the storage device or whether to store everything in RAM, which is volatile storage. If the journals are stored in RAM, they will not be persistent between boots.

The default time unit in the journald.conf file is seconds, but it can be overridden using the suffixes year , month , week , day , h , or m 。

Suppose you want to limit the total amount of storage space allocated to journal files to 1GB, store all journal entries in persistent storage, keep a maximum of 10 files, and delete any journal archive files that are more than a month old. You can configure this in /etc/systemd/journald.conf using:

SystemMaxUse=1G

Storage=persistent 

SystemMaxFiles=10

MaxRetentionSec=1month

By default, the SystemMaxUse is 10% of available disk space. The default settings have been fine for the systems I work with, and I have not needed to change any of them. The journald.conf man page states that the time-based settings for specifying how long to store journal entries in a single file or to retain older files are normally not necessary. This is because file number and size configurations usually force rotation and deletion of old files before any time settings might be needed.

The SystemKeepFree option ensures a specific amount of space is kept free for other data. Many databases and application programs use the /var filesystem to store data, so ensuring enough available storage space may be critical in systems with smaller hard drives and minimum space allocated to /var 。

If you make changes to this configuration, be sure to monitor the results carefully for an appropriate period of time to ensure they are performing as expected.

Journal file rotation

The journal files are typically rotated automatically based upon the configuration in the /etc/systemd/journald.conf ファイル。 Files are rotated whenever one of the specified conditions is met. So if, for example, the space allocated to journal files is exceeded, the oldest file(s) is deleted, the active file is made into an archive, and a new active file is created.

Journal files can also be rotated manually. I suggest using the --flush option to ensure current data is moved to persistent storage before you run the command:

[root@testvm1 ~]# journalctl --rotate

There is another way to purge old journal files without performing a file rotation. The vacuum-size= , vacuum-files= , and vacuum-time= commands can be used to delete old archive files down to a specified total size, number of files, or prior time. The option values consider only the archive files and not the active ones, so the resulting reduction in total file size might be somewhat less than expected.

The following command purges old archive files so that only ones that are less than one month old are left. You can use the s , m , h , days , months , weeks , and years suffixes:

[root@testvm1 ~]# journalctl --vacuum-time=1month 

This command deletes all archive files except for the four most recent ones. If there are fewer than four archive files, nothing will happen, and the original number of files remains:

[root@testvm1 ~]# journalctl --vacuum-files=4

This last vacuum command deletes archive files until 200MB or less of archived files are left:

[root@testvm1 ~]# journalctl --vacuum-size=200M

Only complete files are deleted. The vacuum commands do not truncate archive files to meet the specification. They also work only on archive files, not active ones.

Final thoughts

This article looked at using the journalctl command to extract various types of data from the systemd journal in different formats. It also explored managing journal files and how to add entries to the log from commands and scripts.

The systemd journal system provides a significant amount of metadata and context for entries compared to the old syslogd program. This additional data and the context available from the other journal entries around the time of an incident can help the sysadmin locate and resolve problems much faster than having to search multiple syslog files.

In my opinion, the journalctl command meets the Unix philosophy that programs should do one thing and do it well. The only thing journalctl does is extract data from the journal and provide many options for selecting and formatting that data. At about 85K, it is not very big. Of course, that does not include shared libraries, but those are, by definition, shared with other programs.

You should now have enough information to use the systemd journal more effectively. If you would like to know more than what I covered here, look in the man pages for journalctl and systemd-cat 。

Resources

There is a great deal of information about systemd available on the internet, but much is terse, obtuse, or even misleading. In addition to the resources mentioned in this article, the following webpages offer more detailed and reliable information about systemd startup. This list has grown since I started this series of articles to reflect the research I have done.

• DigitalOcean has a very good article about journalctl and how to view and manipulate systemd logs.
• The Fedora Project has a good, practical guide to systemd. It has pretty much everything you need to know in order to configure, manage, and maintain a Fedora computer using systemd.
• The Fedora Project also has a good cheat sheet that cross-references the old SystemV commands to comparable systemd ones.
• Red Hat documentation contains a good description of the Unit file structure as well as other important information.
• For detailed technical information about systemd and the reasons for creating it, check out Freedesktop.org's description of systemd.
• Linux.com's "More systemd fun" offers more advanced systemd information and tips.

There is also a series of deeply technical articles for Linux sysadmins by Lennart Poettering, the designer and primary developer of systemd. These articles were written between April 2010 and September 2011, but they are just as relevant now as they were then. Much of everything else good that has been written about systemd and its ecosystem is based on these papers.

• Rethinking PID 1
• systemd for Administrators, Part I
• systemd for Administrators, Part II
• systemd for Administrators, Part III
• systemd for Administrators, Part IV
• systemd for Administrators, Part V
• systemd for Administrators, Part VI
• systemd for Administrators, Part VII
• systemd for Administrators, Part VIII
• systemd for Administrators, Part IX
• systemd for Administrators, Part X
• systemd for Administrators, Part XI