1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
|
=========================
Systemd Journal Support
=========================
One of the newer features in oslo.log is the ability to integrate with
the systemd journal service (journald) natively on newer Linux
systems. When using native journald support, additional metadata will
be logged on each log message in addition to the message itself, which
can later be used to do some interesting searching through your logs.
Enabling
========
In order to enable the support you must have Python bindings for
systemd installed. On Red Hat based systems, run::
yum install systemd-python
On Ubuntu/Debian based systems, run::
apt install python-systemd
If there is no native package for your distribution, or you are
running in a virtualenv, you can install with pip.::
pip install systemd-python
.. note::
There are also many non official systemd python modules on pypi,
with confusingly close names. Make sure you install `systemd-python
<https://pypi.org/project/systemd-python>`_.
After the package is installed, you must enable journald support
manually in all services that will be using it. Add the following to
the config files for all services:
.. code-block:: ini
[DEFAULT]
use_journal = True
In all relevant config files.
Extra Metadata
==============
Journald supports the concept of adding structured metadata in
addition to the log message in question. This makes it much easier to
take the output of journald and push it into other logging systems
like Elastic Search, without needing to regex guess relevant data. It
also allows you to search the journal by these fields using
``journalctl``.
We use this facility to add our own structured information, if it is
known at the time of logging the message.
CODE_FILE=, CODE_LINE=, CODE_FUNC=
The code location generating this message, if known. Contains the
source filename, the line number and the function name. (This is the
same as systemd uses)
THREAD_NAME=, PROCESS_NAME=
Information about the thread and process, if known. (This is the same
as systemd uses)
EXCEPTION_TEXT=, EXCEPTION_INFO=
Information about an exception, if an exception has been logged.
LOGGER_NAME=
The name of the python logger that emitted the log
message. Very often this is the module where the log message was
emitted from.
LOGGER_LEVEL=
The name of the python logging level, which allows seeing all
'ERROR' messages very easily without remembering how they are
translated to syslog priorities.
SYSLOG_IDENTIFIER=
The binary name identified for syslog compatibility. It will be the
basename of the process that emits the log messages
(e.g. ``nova-api``, ``neutron-l3-agent``)
PRIORITY=
The syslog priority (based on LOGGER_LEVEL), which allows syslog
style filtering of messages based on their priority (an
openstack.err log file for instance).
REQUEST_ID=
Most OpenStack services generate a unique ``request-id`` on every
REST API call, which is then passed between it's sub services as
that request is handled. For example, this can be very useful in
tracking the build of a nova server from the initial HTTP POST to
final VM create.
PROJECT_ID=, PROJECT_NAME=, USER_ID=, USER_NAME=
The keystone known user and project information about the
requestor. Both the id and name are provided for easier
searching. This can be used to understand when particular users or
projects are reporting issues in the environment.
Additional fields may be added over time. It is unlikely that fields
will be removed, but if so they will be deprecated for one release
cycle before that happens.
Using Journalctl
================
Because systemd is relatively new in the Linux ecosystem, it's worth
noting how one can effectively use journal control.
If you want to follow all the journal logs you would do so with::
journalctl -f
That's going to be nearly everything on your system, which you will
probably find overwhelming. You can limit this to a smaller number of
things using the ``SYSLOG_IDENTIFIER=``::
journalctl -f SYSLOG_IDENTIFIER=nova-compute SYSLOG_IDENTIFIER=neutron-l3-agent
Specifying a query parameter multiple times defaults to an ``OR``
operation, so that will show either nova-compute or neutron-l3-agent
logs.
You can also query by request id to see the entire flow of a REST
call::
journalctl REQUEST_ID=req-b1903300-77a8-401d-984c-8e7d17e4a15f
References
==========
- A complete list of the systemd journal fields is here, it is worth
making yourself familiar with them -
https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html
- The complete journalctl manual is worth reading, especially the
``-o`` parameter, as default displayed time resolution is only in
seconds (even though systemd internally is tracking microsecs) -
https://www.freedesktop.org/software/systemd/man/journalctl.html
- The guide for using systemd in devstack provides additional examples
of effective journalctl queries -
https://opendev.org/openstack/devstack/src/branch/master/doc/source/systemd.rst
|
