Fatmawati Achmad Zaenuri / Shutterstock
Linux system logging changed with the introduction of systemd. Learn how to use the journalctl command to read and filter system log messages.
Centralized logging
No stranger to controversy, the systemd system and service manager introduces a significant change to the way system logs are collected. Logs were previously located in different locations on the file system depending on the service or daemon that created them. But they all had one thing in common. These were plain text files.
With systemd, all system, boot and kernel log files are collected and managed by a central and dedicated logging solution. The format in which they are stored is binary. This notably facilitates the extraction of data in different formats, such as JSON, as we see it.
it can also facilitate cross-reference to related information that would have previously been recorded in separate log files. Since data is now kept in a single log, data from multiple sources of interest can be selected and displayed in a single list of interleaved entries.
journalctl is the tool used to work with the newspaper.
journalctl With No Frills
You can call journalctl without command line parameters:
journalctl
journalctl displays the entire journal, with the oldest entries at the top of the list. The list is displayed at least, allowing you to paginate and search using the usual navigation functions of less. You can also use the Left Arrow and Right Arrow keys to scroll sideways to read wide log entries.
Press the End key to go directly to the bottom of the list and to the most recent log entries.
Press Ctrl + C to exit.
RELATED: How to use the less command on Linux
Although journalctl can be called without using sudo, you will make sure you see all the details in the newspaper if you use sudo.
sudo journalctl
If you need to, you can have journalctl send its output to the terminal window instead of less, using the –no-pager option.
sudo journalctl – no-pager
The output scrolls quickly through the terminal window and you return to the command prompt.
To limit the number of rows returned by journalctl, use the -n (rows) option. Let’s ask for ten output lines:
sudo journalctl -n 10
After log updates
To have journalctl display the most recent entries as they arrive in the journal, use the -f (follow) option.
sudo journalctl -f
The new entry has a time stamp of 07:09:07. When a new activity takes place, the new entries are added at the bottom of the screen. Almost real-time updates – cool!
At 7:09:59 am, an application called geek-app injected a log entry into the log that said “New message from HTG”.
Changing the display format
Since the journal is a binary file, the data it contains must be translated or analyzed into text before it can be displayed to you. With different analyzers, different output formats can be created from the same binary source data. Journalctl can use several different formats.
The default output is the short format, which is very similar to the classic system log format. To explicitly request the short format, use the -o (output) option with the short modifier.
sudo journalctl -n 10 -o short-full
From left to right, the fields are:
Time of creation of the message, in local time.
The host name.
The name of the process. This is the process that generated the message.
The newspaper message.
To get a full timestamp, use the short-full modifier:
sudo journalctl -n 10 -o short-full
The date and time formats in this output are the format in which you must provide the dates and times when you select log messages by period, as we will see soon.
To see all of the metadata that accompanies each log message, use the detail modifier.
sudo journalctl -n 10 -o verbose
The are many possible areas, but it is rare that all fields are present in a message.
One area to discuss is the Priority field. In this example, it has a value of 6. The value represents the importance of the message:
0: Emergency. The system is unusable.
1: Alert. A condition has been reported and needs to be corrected immediately.
2: Critical. This covers crashes, coredumps, and major failures in main applications.
3: Fault. An error has been reported, but is not considered to be serious.
4: Warning. Calls a condition to your attention which, if ignored, can become a mistake.
5: Note. Used to report unusual events, but not errors.
6: Information. Regular operational messages. These do not require any action.
seven: Debugging. Messages placed in applications to facilitate debugging.
If you want the output to be presented as properly formed JavaScript object notation (JSON), use the json modifier:
sudo journalctl -n 10 -o json
Each message is properly wrapped as a well-formed JSON object and displays one message per line of output.
To have JSON output nice print, use the json-pretty modifier.
sudo journalctl -n 10 -o json-pretty
Each JSON object is split into multiple lines, with each name-value pair on a new line.
To display only log entry messages, without timestamp or other metadata, use the cat modifier:
sudo journalctl -n 10 -o cat
This display format can make it difficult to identify the process that triggered the log event, although some messages contain a clue.
Selection of log messages by period
To limit journalctl output to a period of interest, use the -S (from) and -U (to) options.
To see log entries from a particular time and date, use this command:
sudo journalctl -S “2020-91-12 07:00:00”
The display contains only messages arrived after the date and time of the order.
To define a period for which you want to create a report, use the options -S (from) and -U (until) simultaneously. This command examines log messages over a 15-minute period:
sudo journalctl -S “2020-91-12 07:00:00” -U “2020-91-12 07:15:00”
It’s a great combination if you know something strange happened on your system and pretty much when it happened.
Use of relative time periods
You can use relative addressing when selecting your periods. That means you can say things like “show me all the events from one day until now”. This is exactly what this command means. The “d” means “day” and the “-1” means a day in the past.
sudo journalctl -S -1d
Log messages are listed from 00:00:00 yesterday until “now”.
If you want to investigate a recent event, you can specify a relative period measured in hours. Here we take a look at the latest log messages:
sudo journalctl -S -1h
Messages from the last hour are displayed for you. You can also use “m” to define relative time periods measured in minutes and “w” for weeks.
journalctl understands today, yesterday and tomorrow. These modifiers provide a convenient way to specify common time periods. To see all the events that occurred yesterday, use this command:
sudo journalctl -S yesterday
All log events that occurred yesterday, until midnight 00:00:00, are retrieved and displayed for you.
To see all the log messages received so far, use this command:
sudo journalctl -S today
Everything from 00:00:00 until the time of issue of the command is displayed.
You can mix the different period modifiers. To see everything from two days to the start of the day, use this command:
sudo journalctl -S -2d -U today
Everything from the day before yesterday to today is retrieved and displayed.
Selection of log messages by data fields
You can search for log messages which correspond to a wide range of journal fields. These searches try to find matches in the metadata attached to each message. It is recommended that you refer to the list of fields and choose the ones that will be most useful to you.
Keep in mind that an application completes or not all fields depends entirely on the authors of the application. You cannot guarantee that each field will be completed.
All log field modifiers are used in the same way. We will use some of them in our examples below. To find log messages for a specific application, use the _COMM (command) modifier. If you also use the -f (follow) option, journalctl will follow new messages from this application as they arrive.
sudo journalctl -f _COMM = geek application
You can search for log entries using the Process ID of the process that generated the log message. Use the ps command to find the identifier of the daemon process or the application you are going to search for.
sudo journalctl _PID = 751
On the machine used to search for this item, the SSH the demon is process 751.
You can also search by user id. This is the user ID of the person who launched the application or command, or who owns the process.
sudo journalctl _UID = 1000
All messages associated with any other user ID are filtered. Only messages related to user 1000 are displayed:
Another way to find log messages related to a specific application is to provide the path to the executable.
sudo journalctl / usr / bin / anacron
All the anacron scheduler log messages are retrieved and displayed.
To facilitate the search, we can ask journalctl to list all the values it contains, for one of the fields in the journal.
To see the user IDs for which journalctl has logged log messages, use the -F (fields) option and pass the _UID field identifier.
journalctl -F _UID
Let’s do it again and let’s look at it Group id (GID):
journalctl -F _GID
You can do it with one of the log field identifiers.
List of kernel messages
There is an integrated way to quickly isolate messages from the kernel. You don’t have to research and isolate them yourself. The -k (kernel) option suppresses all other messages and gives you an instant view of the kernel log entries.
sudo journalctl -k
Highlighting reflects the importance of the message, according to the values in the Priority field.
Review of startup messages
Journalctl has you covered if you have a startup problem that you want to investigate. You may have added new hardware and it is unresponsive, or a previously functioning hardware component no longer works after your last system upgrade.
To see the log entries related to your last startup, use the -b (startup) option:
journalctl -b
The log entries for the last start are displayed for you.
When we say “last boot,” we mean the boot process that brought your computer to life for your currently connected session. To see previous starts, you can use a number to tell journalctl which start you are interested in. To see the third previous boot, use this command:
journalctl -b 3
Generally, if you had a problem and had to restart your machine, you are interested in a previous boot sequence. It is therefore a current order form.
It’s easy to get involved in the boot sequence. To help you, we can ask journalctl to list the boots it has recorded in its log, using the –list-boots option.
journalctl –list-boots
You can identify the startup for which you want to see messages from the timestamp, then use the number in the left column to get the log messages for this startup sequence. You can also choose the 32-bit boot ID and pass it to journalctl.
sudo journalctl -b 1f00248226ed4ab9a1abac86e0d540d7
The boot sequence log messages that we requested are retrieved and displayed.
Log hard disk space management
Of course, the log and all of its log messages are stored on your hard drive. This means that they will take up space on the hard drive. To see the space occupied by the log, use the –disk-usage option.
journalctl –disk-usage
With today’s hard drives, 152 MB, it is not much space, but for demonstration purposes, we will always reduce it. We can do this in two ways. The first is to set a size limit to which you want the log to be reduced. It will grow back, of course, but we can prune it now ready for this new growth.
We will use the wonderfully labeled option –vacuum-size, and pass the size to which we would like the log to be reduced. We will ask you for 100MB. The way to think about this is that we are asking journalctl to “throw out everything you can, but don’t go below 100MB”.
journalctl –vacuum-size = 100M
The other way to reduce the size of the log is to use the –vacuum-time option. This option tells journalctl to delete messages older than the period you provide on the command line. You can use days, weeks, months, and years in the period.
Let’s delete all messages older than a week:
journalctl –vacuum-time = 1weeks
Data vs information
Data is only useful if you can access and use it. Then it becomes useful information. The journalctl command is a flexible and sophisticated tool that allows you to access information of interest in different ways.
You can use just about any snippet of information you need to get into the log messages you need.