diff options
author | John Wilkins <john.wilkins@inktank.com> | 2013-04-17 18:18:10 -0700 |
---|---|---|
committer | John Wilkins <john.wilkins@inktank.com> | 2013-04-17 18:18:10 -0700 |
commit | 808ad25a28dd9b52dab75f6b29646e6d854b1047 (patch) | |
tree | d89d4d51eade8b54eabcbc17c626ea20a1ab36e0 | |
parent | 3c144e9b6cff3899b4e4be1e8a3049379ce776ff (diff) | |
download | ceph-808ad25a28dd9b52dab75f6b29646e6d854b1047.tar.gz |
doc: Removed fragmented logging info. Consolidated into one doc.
Logging was variously described in the ceph configuration document,
a configuration reference, and a section in operations. Since
logging and debugging are generally used with troubleshooting,
I consolidated the docs and placed them in the troubleshooting
section. Also fixed the example and provided additional detail.
fixes: #3804
Signed-off-by: John Wilkins <john.wilkins@inktank.com>
-rw-r--r-- | doc/rados/configuration/log-and-debug-ref.rst | 330 | ||||
-rw-r--r-- | doc/rados/operations/debug.rst | 62 | ||||
-rw-r--r-- | doc/rados/troubleshooting/log-and-debug.rst | 550 |
3 files changed, 550 insertions, 392 deletions
diff --git a/doc/rados/configuration/log-and-debug-ref.rst b/doc/rados/configuration/log-and-debug-ref.rst deleted file mode 100644 index 3e493934824..00000000000 --- a/doc/rados/configuration/log-and-debug-ref.rst +++ /dev/null @@ -1,330 +0,0 @@ -======================================== - Logging and Debugging Config Reference -======================================== - -.. important:: See `Debugging and Logging`_ for details on log rotation. - -Logging and debugging settings are not required, but you may override default -settings as needed. Ceph supports the following settings: - -.. _Debugging and Logging: ../../operations/debug - -Logging -======= - -``log file`` - -:Desription: The location of the logging file for your cluster. -:Type: String -:Required: No -:Default: ``/var/log/ceph/$cluster-$name.log`` - - -``log max new`` - -:Description: The maximum number of new log files. -:Type: Integer -:Required: No -:Default: ``1000`` - - -``log max recent`` - -:Description: The maximum number of recent events to include in a log file. -:Type: Integer -:Required: No -:Default: ``1000000`` - - -``log to stderr`` - -:Description: Determines if logging messages should appear in ``stderr``. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``err to stderr`` - -:Description: Determines if error messages should appear in ``stderr``. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``log to syslog`` - -:Description: Determines if logging messages should appear in ``syslog``. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``err to syslog`` - -:Description: Determines if error messages should appear in ``syslog``. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``log flush on exit`` - -:Description: Determines if Ceph should flush the log files after exit. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``clog to monitors`` - -:Description: Determines if ``clog`` messages should be sent to monitors. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``clog to syslog`` - -:Description: Determines if ``clog`` messages should be sent to syslog. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``mon cluster log to syslog`` - -:Description: Determines if the cluster log should be output to the syslog. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``mon cluster log file`` - -:Description: The location of the cluster's log file. -:Type: String -:Required: No -:Default: ``/var/log/ceph/$cluster.log`` - - - -OSD -=== - - -``osd debug drop ping probability`` - -:Description: ? -:Type: Double -:Required: No -:Default: 0 - - -``osd debug drop ping duration`` - -:Description: The duration ? -:Type: Integer -:Required: No -:Default: 0 - -``osd debug drop pg create probability`` - -:Description: -:Type: Integer -:Required: No -:Default: 0 - -``osd debug drop pg create duration`` - -:Description: ? -:Type: Double -:Required: No -:Default: 1 - -``osd preserve trimmed log`` - -:Description: ? -:Type: Boolean -:Required: No -:Default: ``false`` - -``osd tmapput sets uses tmap`` - -:Description: For debug only. ??? -:Type: Boolean -:Required: No -:Default: ``false`` - - -``osd min pg log entries`` - -:Description: The minimum number of log entries for placement groups. -:Type: 32-bit Unsigned Integer -:Required: No -:Default: 1000 - -``osd op log threshold`` - -:Description: How many op log messages to show up in one pass. -:Type: Integer -:Required: No -:Default: 5 - - - -Filestore -========= - -``filestore debug omap check`` - -:Description: Checks the ``omap``. This is an expensive operation. -:Type: Boolean -:Required: No -:Default: 0 - - -MDS -=== - - -``mds debug scatterstat`` - -:Description: ? -:Type: Boolean -:Required: No -:Default: ``false`` - - -``mds debug frag`` - -:Description: -:Type: Boolean -:Required: No -:Default: ``false`` - - -``mds debug auth pins`` - -:Description: ? -:Type: Boolean -:Required: No -:Default: ``false`` - - -``mds debug subtrees`` - -:Description: ? -:Type: Boolean -:Required: No -:Default: ``false`` - - - -RADOS Gateway -============= - - -``rgw log nonexistent bucket`` - -:Description: Should we log a non-existent buckets? -:Type: Boolean -:Required: No -:Default: ``false`` - - -``rgw log object name`` - -:Description: Should an object's name be logged. // man date to see codes (a subset are supported) -:Type: String -:Required: No -:Default: ``%Y-%m-%d-%H-%i-%n`` - - -``rgw log object name utc`` - -:Description: Object log name contains UTC? -:Type: Boolean -:Required: No -:Default: ``false`` - - -``rgw enable ops log`` - -:Description: Enables logging of every RGW operation. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``rgw enable usage log`` - -:Description: Enable logging of RGW's bandwidth usage. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``rgw usage log flush threshold`` - -:Description: Threshold to flush pending log data. -:Type: Integer -:Required: No -:Default: ``1024`` - - -``rgw usage log tick interval`` - -:Description: Flush pending log data every ``s`` seconds. -:Type: Integer -:Required: No -:Default: 30 - - -``rgw intent log object name`` - -:Description: -:Type: String -:Required: No -:Default: ``%Y-%m-%d-%i-%n`` - - -``rgw intent log object name utc`` - -:Description: Include a UTC timestamp in the intent log object name. -:Type: Boolean -:Required: No -:Default: ``false`` - -``rgw cluster root pool`` - -:Description: RADOS pool to store radosgw metadata for this instance -:Type: String -:Required: No -:Default: ``.rgw.root`` - -``rgw gc max objs`` - -:Description: Number of objects to collect garbage collection data -:Type: 32-bit Integer -:Default: 32 - -``rgw gc obj min wait`` - -:Description: Minimum time to wait before object's removal and its processing by the garbage collector -:Type: 32-bit Integer -:Default: 2 hours. ``2*60*60`` - -``rgw gc processor max time`` - -:Description: Max time for a single garbage collection process run -:Type: 32-bit Integer -:Default: 1 hour. ``60*60`` - -``rgw gc processor max period`` - -:Description: Max time between the beginning of two consecutive garbage collection processes run -:Type: 32-bit Integer -:Default: 1 hour. ``60*60`` - - diff --git a/doc/rados/operations/debug.rst b/doc/rados/operations/debug.rst deleted file mode 100644 index 21ab019627c..00000000000 --- a/doc/rados/operations/debug.rst +++ /dev/null @@ -1,62 +0,0 @@ -======================= - Debugging and Logging -======================= - -You may view Ceph log files under ``/var/log/ceph`` (the default location). - -.. important:: If you enable or increase the rate of Ceph logging, ensure - that you have sufficient disk space on your OS disk. Verbose logging - can generate over 1GB of data per hour. If your OS disk reaches its - capacity, the node will stop working. - -.. topic:: Accelerating Log Rotation - - If your OS disk is relatively full, you can accelerate log rotation by - modifying the Ceph log rotation file at ``/etc/logrotate.d/ceph``. Add - a size setting after the rotation frequency to accelerate log rotation - (via cronjob) if your logs exceed the size setting. For example, the - default setting looks like this:: - - rotate 7 - weekly - compress - sharedscripts - - Modify it by adding a ``size`` setting. :: - - rotate 7 - weekly - size 500M - compress - sharedscripts - - Then, start the crontab editor for your user space. :: - - crontab -e - - Finally, add an entry to check the ``etc/logrotate.d/ceph`` file. :: - - 30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1 - - The preceding example checks the ``etc/logrotate.d/ceph`` file every 30 minutes. - - -Ceph is still on the leading edge, so you may encounter situations that require -using Ceph's debugging and logging. To activate and configure Ceph's debug -logging, refer to `Ceph Logging and Debugging`_. For additional logging -settings, refer to the `Logging and Debugging Config Reference`_. - -.. _Ceph Logging and Debugging: ../../configuration/ceph-conf#ceph-logging-and-debugging -.. _Logging and Debugging Config Reference: ../../configuration/log-and-debug-ref - -You can change the logging settings at runtime so that you don't have to -stop and restart the cluster. Refer to `Ceph Configuration - Runtime Changes`_ -for additional details. - -Debugging may also require you to track down memory and threading issues. -You can run a single daemon, a type of daemon, or the whole cluster with -Valgrind. You should only use Valgrind when developing or debugging Ceph. -Valgrind is computationally expensive, and will slow down your system otherwise. -Valgrind messages are logged to ``stderr``. - -.. _Ceph Configuration - Runtime Changes: ../../configuration/ceph-conf#ceph-runtime-config diff --git a/doc/rados/troubleshooting/log-and-debug.rst b/doc/rados/troubleshooting/log-and-debug.rst new file mode 100644 index 00000000000..2f2e5e4abc0 --- /dev/null +++ b/doc/rados/troubleshooting/log-and-debug.rst @@ -0,0 +1,550 @@ +======================= + Logging and Debugging +======================= + +Typically, when you add debugging to your Ceph configuration, you do so at +runtime. You can also add Ceph debug logging to your Ceph configuration file if +you are encountering issues when starting your cluster. You may view Ceph log +files under ``/var/log/ceph`` (the default location). + +.. tip:: When debug output slows down your system, the latency can hide + race conditions. + +Logging is resource intensive. If you are encountering a problem in a specific +area of your cluster, enable logging for that area of the cluster. For example, +if your OSDs are running fine, but your metadata servers are not, you should +start by enabling debug logging for the specific metadata server instance(s) +giving you trouble. Enable logging for each subsystem as needed. + +.. important:: Verbose logging can generate over 1GB of data per hour. If your + OS disk reaches its capacity, the node will stop working. + +If you enable or increase the rate of Ceph logging, ensure that you have +sufficient disk space on your OS disk. See `Accelerating Log Rotation`_ for +details on rotating log files. When your system is running well, remove +unnecessary debugging settings to ensure your cluster runs optimally. Logging +debug output messages is relatively slow, and a waste of resources when +operating your cluster. + +See `Subsystem, Log and Debug Settings`_ for details on available settings. + +Runtime +======= + +If you would like to see the configuration settings at runtime, you must log +in to a host with a running daemon and execute the following:: + + ceph --admin-daemon {/path/to/admin/socket} config show | less + ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | less + +To activate Ceph's debugging output (*i.e.*, ``dout()``) at runtime, use the +``ceph tell`` command to inject arguments into the runtime configuration:: + + ceph {daemon-type} tell {daemon id or *} injectargs '--{name} {value} [--{name} {value}]' + +Replace ``{daemon-type}`` with one of ``osd``, ``mon`` or ``mds``. You may apply +the runtime setting to all daemons of a particular type with ``*``, or specify +a specific daemon's ID (i.e., its number or letter). For example, to increase +debug logging for a ``ceph-osd`` daemon named ``osd.0``, execute the following:: + + ceph osd tell 0 injectargs '--debug_osd 0/5' + +The ``ceph tell`` command goes through the monitors. If you cannot bind to the +monitor, you can still make the change by logging into the host of the daemon +whose configuration you'd like to change using ``ceph --admin-daemon``. +For example:: + + sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set debug_osd 0/5 + +See `Subsystem, Log and Debug Settings`_ for details on available settings. + + +Boot Time +========= + +To activate Ceph's debugging output (*i.e.*, ``dout()``) at boot time, you must +add settings to your Ceph configuration file. Subsystems common to each daemon +may be set under ``[global]`` in your configuration file. Subsystems for +particular daemons are set under the daemon section in your configuration file +(*e.g.*, ``[mon]``, ``[osd]``, ``[mds]``). For example:: + + [global] + debug ms = 1/5 + + [mon] + debug mon = 20 + debug paxos = 1/5 + debug auth = 2 + + [osd] + debug osd = 1/5 + debug filestore = 1/5 + debug journal = 1 + debug monc = 5/20 + + [mds] + debug mds = 1 + debug mds balancer = 1 + debug mds log = 1 + debug mds migrator = 1 + + +See `Subsystem, Log and Debug Settings`_ for details. + + +Accelerating Log Rotation +========================= + +If your OS disk is relatively full, you can accelerate log rotation by modifying +the Ceph log rotation file at ``/etc/logrotate.d/ceph``. Add a size setting +after the rotation frequency to accelerate log rotation (via cronjob) if your +logs exceed the size setting. For example, the default setting looks like +this:: + + rotate 7 + weekly + compress + sharedscripts + +Modify it by adding a ``size`` setting. :: + + rotate 7 + weekly + size 500M + compress + sharedscripts + +Then, start the crontab editor for your user space. :: + + crontab -e + +Finally, add an entry to check the ``etc/logrotate.d/ceph`` file. :: + + 30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1 + +The preceding example checks the ``etc/logrotate.d/ceph`` file every 30 minutes. + + +Valgrind +======== + +Debugging may also require you to track down memory and threading issues. +You can run a single daemon, a type of daemon, or the whole cluster with +Valgrind. You should only use Valgrind when developing or debugging Ceph. +Valgrind is computationally expensive, and will slow down your system otherwise. +Valgrind messages are logged to ``stderr``. + + +Subsystem, Log and Debug Settings +================================= + +In most cases, you will enable debug logging output via subsystems. + +Ceph Subsystems +--------------- + +Each subsystem has a logging level for its output logs, and for its logs +in-memory. You may set different values for each of these subsystems by setting +a log file level and a memory level for debug logging. Ceph's logging levels +operate on a scale of ``1`` to ``20``, where ``1`` is terse and ``20`` is +verbose. + +A debug logging setting can take a single value for the log level and the +memory level, which sets them both as the same value. For example, if you +specify ``debug ms = 5``, Ceph will treat it as a log level and a memory level +of ``5``. You may also specify them separately. The first setting is the log +level, and the second setting is the memory level. You must separate them with +a forward slash (/). For example, if you want to set the ``ms`` subsystem's +debug logging level to ``1`` and its memory level to ``5``, you would specify it +as ``debug ms = 1/5``. For example: + + + +.. code-block:: ini + + debug {subsystem} = {log-level}/{memory-level} + #for example + debug mds log = 1/20 + + +The following table provides a list of Ceph subsystems and their default log and +memory levels. Once you complete your logging efforts, restore the subsystems +to their default level or to a level suitable for normal operations. + + ++--------------------+-----------+--------------+ +| Subsystem | Log Level | Memory Level | ++====================+===========+==============+ +| ``default`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``lockdep`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``context`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``crush`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds balancer`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds locker`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds log`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds log expire`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds migrator`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``buffer`` | 0 | 0 | ++--------------------+-----------+--------------+ +| ``timer`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``filer`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``objecter`` | 0 | 0 | ++--------------------+-----------+--------------+ +| ``rados`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``rbd`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``journaler`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``objectcacher`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``client`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``osd`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``optracker`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``objclass`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``filestore`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``journal`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``ms`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``mon`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``monc`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``paxos`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``tp`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``auth`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``finisher`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``heartbeatmap`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``perfcounter`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``rgw`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``hadoop`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``asok`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``throttle`` | 1 | 5 | ++--------------------+-----------+--------------+ + + +Logging Settings +---------------- + +Logging and debugging settings are not required in a Ceph configuration file, +but you may override default settings as needed. Ceph supports the following +settings: + + +``log file`` + +:Desription: The location of the logging file for your cluster. +:Type: String +:Required: No +:Default: ``/var/log/ceph/$cluster-$name.log`` + + +``log max new`` + +:Description: The maximum number of new log files. +:Type: Integer +:Required: No +:Default: ``1000`` + + +``log max recent`` + +:Description: The maximum number of recent events to include in a log file. +:Type: Integer +:Required: No +:Default: ``1000000`` + + +``log to stderr`` + +:Description: Determines if logging messages should appear in ``stderr``. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``err to stderr`` + +:Description: Determines if error messages should appear in ``stderr``. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``log to syslog`` + +:Description: Determines if logging messages should appear in ``syslog``. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``err to syslog`` + +:Description: Determines if error messages should appear in ``syslog``. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``log flush on exit`` + +:Description: Determines if Ceph should flush the log files after exit. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``clog to monitors`` + +:Description: Determines if ``clog`` messages should be sent to monitors. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``clog to syslog`` + +:Description: Determines if ``clog`` messages should be sent to syslog. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``mon cluster log to syslog`` + +:Description: Determines if the cluster log should be output to the syslog. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``mon cluster log file`` + +:Description: The location of the cluster's log file. +:Type: String +:Required: No +:Default: ``/var/log/ceph/$cluster.log`` + + + +OSD +--- + + +``osd debug drop ping probability`` + +:Description: ? +:Type: Double +:Required: No +:Default: 0 + + +``osd debug drop ping duration`` + +:Description: +:Type: Integer +:Required: No +:Default: 0 + +``osd debug drop pg create probability`` + +:Description: +:Type: Integer +:Required: No +:Default: 0 + +``osd debug drop pg create duration`` + +:Description: ? +:Type: Double +:Required: No +:Default: 1 + +``osd preserve trimmed log`` + +:Description: Preserves trimmed logs after trimming. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``osd tmapput sets uses tmap`` + +:Description: Uses ``tmap``. For debug only. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``osd min pg log entries`` + +:Description: The minimum number of log entries for placement groups. +:Type: 32-bit Unsigned Integer +:Required: No +:Default: 1000 + + +``osd op log threshold`` + +:Description: How many op log messages to show up in one pass. +:Type: Integer +:Required: No +:Default: 5 + + + +Filestore +--------- + +``filestore debug omap check`` + +:Description: Debugging check on synchronization. This is an expensive operation. +:Type: Boolean +:Required: No +:Default: 0 + + +MDS +--- + + +``mds debug scatterstat`` + +:Description: Ceph will assert that various recursive stat invariants are true + (for developers only). + +:Type: Boolean +:Required: No +:Default: ``false`` + + +``mds debug frag`` + +:Description: Ceph will verify directory fragmentation invariants when + convenient (developers only). + +:Type: Boolean +:Required: No +:Default: ``false`` + + +``mds debug auth pins`` + +:Description: The debug auth pin invariants (for developers only). +:Type: Boolean +:Required: No +:Default: ``false`` + + +``mds debug subtrees`` + +:Description: The debug subtree invariants (for developers only). +:Type: Boolean +:Required: No +:Default: ``false`` + + + +RADOS Gateway +------------- + + +``rgw log nonexistent bucket`` + +:Description: Should we log a non-existent buckets? +:Type: Boolean +:Required: No +:Default: ``false`` + + +``rgw log object name`` + +:Description: Should an object's name be logged. // man date to see codes (a subset are supported) +:Type: String +:Required: No +:Default: ``%Y-%m-%d-%H-%i-%n`` + + +``rgw log object name utc`` + +:Description: Object log name contains UTC? +:Type: Boolean +:Required: No +:Default: ``false`` + + +``rgw enable ops log`` + +:Description: Enables logging of every RGW operation. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``rgw enable usage log`` + +:Description: Enable logging of RGW's bandwidth usage. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``rgw usage log flush threshold`` + +:Description: Threshold to flush pending log data. +:Type: Integer +:Required: No +:Default: ``1024`` + + +``rgw usage log tick interval`` + +:Description: Flush pending log data every ``s`` seconds. +:Type: Integer +:Required: No +:Default: 30 + + +``rgw intent log object name`` + +:Description: +:Type: String +:Required: No +:Default: ``%Y-%m-%d-%i-%n`` + + +``rgw intent log object name utc`` + +:Description: Include a UTC timestamp in the intent log object name. +:Type: Boolean +:Required: No +:Default: ``false`` |