Logstash Logback Encoder

!! This document applies to the next version under development.

See here for documentation on the latest released version.

Logstash Logback Encoder

Provides logback encoders, layouts, and appenders to log in JSON and other formats supported by Jackson.

Supports both regular LoggingEvents (logged through a Logger) and AccessEvents (logged via logback-access).

Originally written to support output in logstash's JSON format, but has evolved into a highly-configurable, general-purpose, structured logging mechanism for JSON and other Jackson dataformats. The structure of the output, and the data it contains, is fully configurable.

Including it in your project
Java Version Requirements
Usage
LoggingEvent Fields
AccessEvent Fields
- Standard Fields
- Header Fields
Customizing Jackson
Masking
Customizing Standard Field Names
Customizing Version
Customizing Timestamp
Customizing Logger Name Length
Customizing Stack Traces
Prefix/Suffix/Separator
Composite Encoder/Layout
Status Listeners

Including it in your project

Maven style:

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>6.3</version>
</dependency>
<!-- Your project must also directly depend on either logback-classic or logback-access.  For example: -->
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.2.3</version>
</dependency>

If you get ClassNotFoundException/NoClassDefFoundError/NoSuchMethodError at runtime, then ensure the required dependencies (and appropriate versions) as specified in the pom file from the maven repository exist on the runtime classpath. Specifically, the following need to be available on the runtime classpath:

jackson-databind / jackson-core / jackson-annotations
logback-core
logback-classic (required for logging LoggingEvents)
logback-access (required for logging AccessEvents)
slf4j-api

Older versions than the ones specified in the pom file might work, but the versions in the pom file are what testing has been performed against.

If you are using logstash-logback-encoder in a project (such as spring-boot) that also declares dependencies on any of the above libraries, you might need to tell maven explicitly which versions to use to avoid conflicts. You can do so using maven's dependencyManagement feature. For example, to ensure that maven doesn't pick different versions of logback-core, logback-classic, and logback-access, add this to your project's pom.xml

    <properties>
        <ch.qos.logback.version>1.2.3</ch.qos.logback.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>ch.qos.logback</groupId>
                <artifactId>logback-core</artifactId>
                <version>${ch.qos.logback.version}</version>
            </dependency>
            <dependency>
                <groupId>ch.qos.logback</groupId>
                <artifactId>logback-classic</artifactId>
                <version>${ch.qos.logback.version}</version>
            </dependency>
            <dependency>
                <groupId>ch.qos.logback</groupId>
                <artifactId>logback-access</artifactId>
                <version>${ch.qos.logback.version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

Java Version Requirements

logstash-logback-encoder	Minimum Java Version supported
>= 6.0	1.8
5.x	1.7
<= 4.x	1.6

Usage

To log using JSON format, you must configure logback to use either:

an appender provided by the logstash-logback-encoder library, OR
an appender provided by logback (or another library) with an encoder or layout provided by the logstash-logback-encoder library

The appenders, encoders, and layouts provided by the logstash-logback-encoder library are as follows:

Format	Protocol	Function	LoggingEvent	AccessEvent
Logstash JSON	Syslog/UDP	Appender	`LogstashUdpSocketAppender`	`LogstashAccessUdpSocketAppender`
Logstash JSON	TCP	Appender	`LogstashTcpSocketAppender`	`LogstashAccessTcpSocketAppender`
any	any	Appender	`LoggingEventAsyncDisruptorAppender`	`AccessEventAsyncDisruptorAppender`
Logstash JSON	any	Encoder	`LogstashEncoder`	`LogstashAccessEncoder`
Logstash JSON	any	Layout	`LogstashLayout`	`LogstashAccessLayout`
General JSON	any	Encoder	`LoggingEventCompositeJsonEncoder`	`AccessEventCompositeJsonEncoder`
General JSON	any	Layout	`LoggingEventCompositeJsonLayout`	`AccessEventCompositeJsonLayout`

These encoders/layouts can generally be used by any logback appender (such as RollingFileAppender).

The general composite JSON encoders/layouts can be used to output any JSON format/data by configuring them with various JSON providers. The Logstash encoders/layouts are really just extensions of the general composite JSON encoders/layouts with a pre-defined set of providers.

The logstash encoders/layouts are easier to configure if you want to use the standard logstash version 1 output format. Use the composite encoders/layouts if you want to heavily customize the output, or if you need to use logstash version 0 output.

The *AsyncDisruptorAppender appenders are similar to logback's AsyncAppender, except that a LMAX Disruptor RingBuffer is used as the queuing mechanism, as opposed to a BlockingQueue. These async appenders can delegate to any other underlying logback appender.

UDP Appender

To output JSON for LoggingEvents to a syslog/UDP channel, use the LogstashUdpSocketAppender with a LogstashLayout or LoggingEventCompositeJsonLayout in your logback.xml, like this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="stash" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
    <host>MyAwesomeSyslogServer</host>
    <!-- port is optional (default value shown) -->
    <port>514</port>
    <!-- layout is required -->
    <layout class="net.logstash.logback.layout.LogstashLayout"/>
  </appender>
  <root level="all">
    <appender-ref ref="stash" />
  </root>
</configuration>

You can further customize the JSON output by customizing the layout as described in later sections.

For example, to configure global custom fields, you can specify

  <appender name="stash" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
    <host>MyAwesomeSyslogServer</host>
    <!-- port is optional (default value shown) -->
    <port>514</port>
    <layout class="net.logstash.logback.layout.LogstashLayout">
      <customFields>{"appname":"myWebservice"}</customFields>
    </layout>
  </appender>

To output JSON for AccessEvents over UDP, use a LogstashAccessUdpSocketAppender with a LogstashAccessLayout or AccessEventCompositeJsonLayout in your logback-access.xml, like this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="stash" class="net.logstash.logback.appender.LogstashAccessUdpSocketAppender">
    <host>MyAwesomeSyslogServer</host>
    <!-- port is optional (default value shown) -->
    <port>514</port>

    <layout class="net.logstash.logback.layout.LogstashAccessLayout">
      <customFields>{"appname":"myWebservice"}</customFields>
    </layout>
  </appender>

  <appender-ref ref="stash" />
</configuration>

To receive syslog/UDP input in logstash, configure a syslog or udp input with the json codec in logstash's configuration like this:

input {
  syslog {
    codec => "json"
  }
}

TCP Appenders

To output JSON for LoggingEvents over TCP, use a LogstashTcpSocketAppender with a LogstashEncoder or LoggingEventCompositeJsonEncoder in your logback.xml, like this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
      <destination>127.0.0.1:4560</destination>

      <!-- encoder is required -->
      <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
  </appender>

  <root level="DEBUG">
      <appender-ref ref="stash" />
  </root>
</configuration>

To output JSON for AccessEvents over TCP, use a LogstashAccessTcpSocketAppender with a LogstashAccessEncoder or AccessEventCompositeJsonEncoder in your logback-access.xml, like this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="stash" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender">
      <destination>127.0.0.1:4560</destination>

      <!-- encoder is required -->
      <encoder class="net.logstash.logback.encoder.LogstashAccessEncoder" />
  </appender>

  <appender-ref ref="stash" />
</configuration>

The TCP appenders use an encoder, rather than a layout as the UDP appenders . You can use a Logstash*Encoder, *EventCompositeJsonEncoder, or any other logback encoder. All of the output formatting options are configured at the encoder level.

Internally, the TCP appenders are asynchronous (using the LMAX Disruptor RingBuffer). All the encoding and TCP communication is delegated to a single writer thread. There is no need to wrap the TCP appenders with another asynchronous appender (such as AsyncAppender or LoggingEventAsyncDisruptorAppender).

All the configuration parameters (except for sub-appender) of the async appenders are valid for TCP appenders. For example, waitStrategyType and ringBufferSize.

The TCP appenders will never block the logging thread. If the RingBuffer is full (e.g. due to slow network, etc), then events will be dropped.

The TCP appenders will automatically reconnect if the connection breaks. However, events may be lost before Java's socket realizes the connection has broken.

To receive TCP input in logstash, configure a tcp input with the json_lines codec in logstash's configuration like this:

input {
    tcp {
        port => 4560
        codec => json_lines
    }
}

In order to guarantee that logged messages have had a chance to be processed by the TCP appender, you'll need to cleanly shut down logback when your application exits.

Keep-alive

If events occur infrequently, and the connection breaks consistently due to a server-side idle timeout, then you can enable keep alive functionality by configuring a keepAliveDuration like this:

  <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
      ...
      <keepAliveDuration>5 minutes</keepAliveDuration>
  </appender>

When the keepAliveDuration is set, then a keep alive message will be sent if an event has not occurred for the length of the duration. The keep alive message defaults to the system's line separator, but can be changed by setting the keepAliveMessage property.

Multiple Destinations

The TCP appenders can be configured to try to connect to one of several destinations like this:

  <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
      <destination>destination1.domain.com:4560</destination>
      <destination>destination2.domain.com:4560</destination>
      <destination>destination3.domain.com:4560</destination>

      ...
  </appender>

or this:

  <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
      <destination>
          destination1.domain.com:4560,
          destination2.domain.com:4560,
          destination3.domain.com:4560
      </destination>

      ...
  </appender>

The appender uses a connectionStrategy to determine:

the order in which destination connections are attempted, and
when an established connection should be reestablished (to the next destination selected by the connection strategy).

Logs are only sent to one destination at a time (i.e. not all destinations). By default, the appender will stay connected to the connected destination until it breaks, or until the application is shut down. Some connection strategies can force a reconnect (see below). If a connection breaks, then the appender will attempt to connect to the next destination selected by the connection strategy.

The available connection strategies are as follows:

Strategy	Description
`preferPrimary`	(default) The first destination is considered the primary destination. Each additional destination is considered a secondary destination. This strategy prefers the primary destination, unless it is down. The appender will attempt to connect to each destination in the order in which they are configured. If a connection attempt fails, thes the appender will attempt to connect to the next destination. If a connection succeeds, and then closes before the `minConnectionTimeBeforePrimary` has elapsed, then the appender will attempt to connect to the next destination. If a connection succeeds, and then closes after the `minConnectionTimeBeforePrimary` has elapsed, then the appender will attempt to connect to the destinations in the order in which they are configured, starting at the first/primary destination. The `secondaryConnectionTTL` can be set to gracefully close connections to secondary destinations after a specific duration. This will force the the appender to reattempt to connect to the destinations in order again. The `secondaryConnectionTTL` value does not affect connections to the primary destination. The `minConnectionTimeBeforePrimary` (10 seconds by default) specifies the minimum amount of time that a sucessfully established connection must remain open before the next connection attempt will try the primary. i.e. If a connection stays open less than this amount of time, then the next connection attempt will attempt the next destination (instead of the primary). This is used to prevent a connection storm to the primary in the case the primary accepts a connection, and then immediately closes it. Example: <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender"> <destination>destination1.domain.com:4560</destination> <destination>destination2.domain.com:4560</destination> <destination>destination3.domain.com:4560</destination> <connectionStrategy> <preferPrimary> <secondaryConnectionTTL>5 minutes</secondaryConnectionTTL> </preferPrimary> </connectionStrategy> </appender>
`roundRobin`	This strategy attempts connections to the destination in round robin order. If a connection fails, the next destination is attempted. The `connectionTTL` can be set to gracefully close connections after a specific duration. This will force the the appender to reattempt to connect to the next destination. Example: <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender"> <destination>destination1.domain.com:4560</destination> <destination>destination2.domain.com:4560</destination> <destination>destination3.domain.com:4560</destination> <connectionStrategy> <roundRobin> <connectionTTL>5 minutes</connectionTTL> </roundRobin> </connectionStrategy> </appender>
`random`	This strategy attempts connections to the destination in a random order. If a connection fails, the next random destination is attempted. The `connectionTTL` can be set to gracefully close connections after a specific duration. This will force the the appender to reattempt to connect to the next random destination. Example: <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender"> <destination>destination1.domain.com:4560</destination> <destination>destination2.domain.com:4560</destination> <destination>destination3.domain.com:4560</destination> <connectionStrategy> <random> <connectionTTL>5 minutes</connectionTTL> </random> </connectionStrategy> </appender>

You can also use your own custom connection strategy by implementing the DestinationConnectionStrategy interface, and configuring the appender to use it like this:

  <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
      <destination>destination1.domain.com:4560</destination>
      <destination>destination2.domain.com:4560</destination>
      <destination>destination3.domain.com:4560</destination>
      <connectionStrategy class="your.package.YourDestinationConnectionStrategy">
      </connectionStrategy>
  </appender>

Reconnection Delay

By default, the TCP appender will wait 30 seconds between connection attempts to a single destination. The time between connection attempts to each destination is tracked separately.

This amount of time to delay can be changed by setting the reconnectionDelay field.

  <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
      ...
      <reconnectionDelay>1 second</reconnectionDelay>
  </appender>

Write buffer size

By default, a buffer size of 8192 is used to buffer socket output stream writes. You can adjust this by setting the appender's writeBufferSize.

  <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
      ...
      <writeBufferSize>16384</writeBufferSize>
  </appender>

Buffering can be disabled by setting the writeBufferSize to 0. Consider disabling the write buffer if you are concerned about losing data from the buffer for flaky connections. Disabling the buffer can potentially slow down the writer thread due to increased system calls, but in some environments, this does not seem to affect overall performance. See this discussion.

Write timeouts

If a destination stops reading from its socket input, but does not close the connection, then writes from the TCP appender will eventually backup, causing the ring buffer to backup, causing events to be dropped.

To detect this situation, you can enable a write timeout, so that "stuck" writes will eventually timeout, at which point the connection will be re-established. When the write buffer is enabled, any buffered data will be lost when the connection is reestablished.

By default there is no write timeout. To enable a write timeout, do the following:

  <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
      ...
      <writeTimeout>1 minute</writeTimeout>
  </appender>

Note that since the blocking java socket output stream used to send events does not have a concept of a write timeout, write timeouts are detected using a task scheduled periodically with the same frequency as the write timeout. For example, if the write timeout is set to 30 seconds, then a task will execute every 30 seconds to see if 30 seconds has elapsed since the start of the current write operation. Therefore, it is recommended to use longer write timeouts (e.g. > 30s, or minutes), rather than short write timeouts, so that this task does not execute too frequently. Also, this approach means that it could take up to two times the write timeout before a write timeout is detected.

SSL

To use SSL, add an <ssl> sub-element within the <appender> element for the LogstashTcpSocketAppender or LogstashAccessTcpSocketAppender.

See the logback manual for how to configure SSL. SSL for the Logstash*TcpSocketAppenders are configured the same way as logback's SSLSocketAppender.

For example, to enable SSL using the JVM's default keystore/truststore, do the following:

  <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
      ...

      <!-- Enable SSL using the JVM's default keystore/truststore -->
      <ssl/>
  </appender>

To use a different truststore, do the following:

  <appender name="stash" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender">
      ...

      <!-- Enable SSL and use a different truststore -->
      <ssl>
          <trustStore>
              <location>classpath:server.truststore</location>
              <password>${server.truststore.password}</password>
          </trustStore>
      </ssl>
  </appender>

All the customizations that logback offers (such as configuring cipher specs, protocols, algorithms, providers, etc.) are supported by the Logback*TcpSocketAppenders.

See the logstash documentation for the tcp input for how to configure it to use SSL.

Async Appenders

The *AsyncDisruptorAppender appenders are similar to logback's AsyncAppender, except that a LMAX Disruptor RingBuffer is used as the queuing mechanism, as opposed to a BlockingQueue. These async appenders can delegate to any other underlying logback appender.

For example:

  <appender name="async" class="net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender">
    <appender class="ch.qos.logback.core.rolling.RollingFileAppender">
       ...
    </appender>
  </appender>

The async appenders will never block the logging thread. If the RingBuffer is full (e.g. due to slow network, etc), then events will be dropped.

By default, the BlockingWaitStrategy is used by the worker thread spawned by this appender. The BlockingWaitStrategy minimizes CPU utilization, but results in slower latency and throughput. If you need faster latency and throughput (at the expense of higher CPU utilization), consider a different wait strategy offered by the disruptor.

!! Whichever wait strategy you choose, be sure to test and monitor CPU utilization, latency, and throughput to ensure it meets your needs. For example, in some configurations, SleepingWaitStrategy can consume 90% CPU utilization at rest.

The wait strategy can be configured on the async appender using the waitStrategyType parameter, like this:

  <appender name="async" class="net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender">
    <waitStrategyType>sleeping</waitStrategyType>
    <appender class="ch.qos.logback.core.rolling.RollingFileAppender">
       ...
    </appender>
  </appender>

The supported wait strategies are as follows:

Wait Strategy	Parameters	Implementation
`blocking`	none	`BlockingWaitStrategy`
`busySpin`	none	`BusySpinWaitStrategy`
`liteBlocking`	none	`LiteBlockingWaitStrategy`
`yielding`	none	`YieldingWaitStrategy`
sleeping{ retries, sleepTimeNs } e.g. `sleeping` or `sleeping{500,1000}`	`retries` - Number of times (integer) to spin before sleeping. (default = 200) `sleepTimeNs` - Time in nanoseconds to sleep each iteration after spinning (default = 100)	`SleepingWaitStrategy`
phasedBackoff{ spinTime, yieldTime, timeUnit, fallbackStrategy } e.g. `phasedBackoff{10,60,seconds,blocking}`	`spinTime` - Time to spin before yielding `yieldTime` - Time to yield before falling back to the `fallbackStrategy` `timeUnit` - Units of time for spin and yield timeouts. String name of a `TimeUnit` value (e.g. `seconds`) `fallbackStrategy` - String name of the wait strategy to fallback to after the timeouts have elapsed	`PhasedBackoffWaitStrategy`
timeoutBlocking{ timeout, timeUnit } e.g. `timeoutBlocking{1,minutes}`	`timeout` - Time to block before throwing an exception `timeUnit` - Units of time for timeout. String name of a `TimeUnit` value (e.g. `seconds`)	`TimeoutBlockingWaitStrategy`
liteTimeoutBlocking{ timeout, timeUnit } e.g. `liteTimeoutBlocking{1,minutes}`	`timeout` - Time to block before throwing an exception `timeUnit` - Units of time for timeout. String name of a `TimeUnit` value (e.g. `seconds`)	`LiteTimeoutBlockingWaitStrategy`

See AsyncDisruptorAppender for other configuration parameters (such as ringBufferSize, producerType, threadNamePrefix, daemon, and droppedWarnFrequency)

In order to guarantees that logged messages have had a chance to be processed by asynchronous appenders (including the TCP appender) and ensure background threads have been stopped, you'll need to cleanly shut down logback when your application exits.

Appender Listeners

Listeners can be registered to an appender to receive notifications for the appender lifecycle and event processing.

See the two listener interfaces for the types of notifications that can be received:

AppenderListener - basic notifications for the async appenders and udp appender.
TcpAppenderListener - extension of AppenderListener with additional TCP-specific notifications. Only works with the TCP appenders.

Some example use cases for a listener are:

Monitoring metrics for events per second, event processing durations, dropped events, connections successes / failures, etc.
Logging event processing errors to a different appender (that perhaps appends to a different destination).

A FailureSummaryLoggingAppenderListener is provided that will log a warning on the first success after a series of consecutive append/send/connect failures. The message includes summary details of the failures that occurred (such as the number of failures, duration of the failures, etc). To register it:

  <appender name="stash" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender">
      <listener class="net.logstash.logback.appender.listener.FailureSummaryLoggingAppenderListener"/>
  </appender>

To create your own listener, create a new class that extends one of the *ListenerImpl classes or directly implements the *Listener interface. Then register your listener class to an appender using the listener xml element like this:

  <appender name="stash" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender">
      ...

      <listener class="your.package.YourListenerClass">
          <yourListenerProperty>propertyValue</yourListenerProperty>
      </listener>
  </appender>

Multiple listeners can be registered by supplying multiple listener xml elements.

Encoders / Layouts

You can use any of the encoders/layouts provided by the logstash-logback-encoder library with other logback appenders.

For example, to output LoggingEvents to a file, use the LogstashEncoder with the RollingFileAppender in your logback.xml like this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="stash" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
      <level>info</level>
    </filter>
    <file>/some/path/to/your/file.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>/some/path/to/your/file.log.%d{yyyy-MM-dd}</fileNamePattern>
      <maxHistory>30</maxHistory>
    </rollingPolicy>
    <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
  </appender>
  <root level="all">
    <appender-ref ref="stash" />
  </root>
</configuration>

To log AccessEvents to a file, configure your logback-access.xml like this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="stash" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>/some/path/to/your/file.log</file>
    <encoder class="net.logstash.logback.encoder.LogstashAccessEncoder" />
  </appender>

  <appender-ref ref="stash" />
</configuration>

The LogstashLayout and LogstashAccessLayout can be configured the same way as the LogstashEncoder and LogstashAccessEncoder. All the other examples in this document use encoders, but the same options apply to the layouts as well.

To receive file input in logstash, configure a file input in logstash's configuration like this:

input {
  file {
    path => "/some/path/to/your/file.log"
    codec => "json"
  }
}

LoggingEvent Fields

The following sections describe the fields included in the JSON output by default for LoggingEvents written by the

LogstashEncoder
LogstashLayout, and
the logstash appenders

If you are using the composite encoders/layouts, then the fields written will vary by the providers you configure.

Standard Fields

These fields will appear in every LoggingEvent unless otherwise noted. The field names listed here are the default field names. The field names can be customized (see Customizing Standard Field Names).

Field	Description
`@timestamp`	Time of the log event. (`yyyy-MM-dd'T'HH:mm:ss.SSSZZ`) See customizing timestamp.
`@version`	Logstash format version (e.g. `1`) See customizing version.
`message`	Formatted log message of the event
`logger_name`	Name of the logger that logged the event
`thread_name`	Name of the thread that logged the event
`level`	String name of the level of the event
`level_value`	Integer value of the level of the event
`stack_trace`	(Only if a throwable was logged) The stacktrace of the throwable. Stackframes are separated by line endings.
`tags`	(Only if tags are found) The names of any markers not explicitly handled. (e.g. markers from `MarkerFactory.getMarker` will be included as tags, but the markers from `Markers` will not.) This can be fully disabled by specifying `<includeTags>false</includeTags>`, in the encoder/layout/appender configuration.

MDC fields

By default, each entry in the Mapped Diagnostic Context (MDC) (org.slf4j.MDC) will appear as a field in the LoggingEvent.

This can be fully disabled by specifying <includeMdc>false</includeMdc>, in the encoder/layout/appender configuration.

You can also configure specific entries in the MDC to be included or excluded as follows:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <includeMdcKeyName>key1ToInclude</includeMdcKeyName>
  <includeMdcKeyName>key2ToInclude</includeMdcKeyName>
</encoder>

or

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <excludeMdcKeyName>key1ToExclude</excludeMdcKeyName>
  <excludeMdcKeyName>key2ToExclude</excludeMdcKeyName>
</encoder>

When key names are specified for inclusion, then all other fields will be excluded. When key names are specified for exclusion, then all other fields will be included. It is a configuration error to specify both included and excluded key names.

Context fields

By default, each property of Logback's Context (ch.qos.logback.core.Context) will appear as a field in the LoggingEvent. This can be disabled by specifying <includeContext>false</includeContext> in the encoder/layout/appender configuration.

Note that logback versions prior to 1.1.10 included a HOSTNAME property by default in the context. As of logback 1.1.10, the HOSTNAME property is lazily calculated (see LOGBACK-1221), and will no longer be included by default.

Caller Info Fields

The encoder/layout/appender do not contain caller info by default. This can be costly to calculate and should be switched off for busy production environments.

To switch it on, add the includeCallerData property to the configuration.

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <includeCallerData>true</includeCallerData>
</encoder>

If the encoder is included inside an asynchronous appender, such as AsyncAppender, LoggingEventAsyncDisruptorAppender, or LogstashTcpSocketAppender, then includeCallerData must be set to true on the appender as well.

When switched on, the following fields will be included in the log event:

Field	Description
`caller_class_name`	Fully qualified class name of the class that logged the event
`caller_method_name`	Name of the method that logged the event
`caller_file_name`	Name of the file that logged the event
`caller_line_number`	Line number of the file where the event was logged

Custom Fields

In addition to the fields above, you can add other fields to the LoggingEvent either globally, or on an event-by-event basis.

Global Custom Fields

Add custom fields that will appear in every LoggingEvent like this :

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <customFields>{"appname":"myWebservice","roles":["customerorder","auth"],"buildinfo":{"version":"Version 0.1.0-SNAPSHOT","lastcommit":"75473700d5befa953c45f630c6d9105413c16fe1"}}</customFields>
</encoder>

or in an AccessEvent like this :

<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder">
  <customFields>{"appname":"myWebservice","roles":["customerorder","auth"],"buildinfo":{"version":"Version 0.1.0-SNAPSHOT","lastcommit":"75473700d5befa953c45f630c6d9105413c16fe1"}}</customFields>
</encoder>

Event-specific Custom Fields

When logging a message, you can add additional fields to the JSON output by using

structured arguments provided by StructuredArguments, OR
markers provided by Markers

The difference between the two is that

StructuredArguments are included in a the log event's formatted message (when the message has a parameter for the argument) AND in the JSON output.
- StructuredArguments will be included in the JSON output if using LogstashEncoder/Layout or if using composite encoders/layouts with the arguments provider.
Markers are only written to the JSON output, and NEVER to the log event's formatted message.
- Markers will be included in the JSON output if using LogstashEncoder/Layout or if using composite encoders/layouts with the logstashMarkers provider.

You can use StructuredArguments even if the message does not contain a parameter for the argument. However, in this case, the argument will only be written to the JSON output and not the formatted message (which is effectively the same behavior that the Markers provide). In general, you should use StructuredArguments, unless you have a static analyzer that flags parameter count / argument count mismatches.

Both StructuredArguments and Markers require constructing additional objects. Therefore, it is best practice to surround the log lines with logger.isXXXEnabled(), to avoid the object construction if the log level is disabled.

Examples using StructuredArguments:

import static net.logstash.logback.argument.StructuredArguments.*;

/*
 * Add "name":"value" to the JSON output,
 * but only add the value to the formatted message.
 *
 * The formatted message will be `log message value`
 */
logger.info("log message {}", value("name", "value"));

/*
 * Add "name":"value" to the JSON output,
 * and add name=value to the formatted message.
 *
 * The formatted message will be `log message name=value`
 */
logger.info("log message {}", keyValue("name", "value"));

/*
 * Add "name":"value" ONLY to the JSON output.
 *
 * Since there is no parameter for the argument,
 * the formatted message will NOT contain the key/value.
 *
 * If this looks funny to you or to static analyzers,
 * consider using Markers instead.
 */
logger.info("log message", keyValue("name", "value"));

/*
 * Add multiple key value pairs to both JSON and formatted message
 */
logger.info("log message {} {}", keyValue("name1", "value1"), keyValue("name2", "value2")));

/*
 * Add "name":"value" to the JSON output and
 * add name=[value] to the formatted message using a custom format.
 */
logger.info("log message {}", keyValue("name", "value", "{0}=[{1}]"));

/*
 * In the JSON output, values will be serialized by Jackson's ObjectMapper.
 * In the formatted message, values will follow the same behavior as logback
 * (formatting of an array or if not an array `toString()` is called).
 *
 * Add "foo":{...} to the JSON output and add `foo.toString()` to the formatted message:
 *
 * The formatted message will be `log message <result of foo.toString()>`
 */
Foo foo  = new Foo();
logger.info("log message {}", value("foo", foo));

/*
 * Add "name1":"value1","name2":"value2" to the JSON output by using a Map,
 * and add `myMap.toString()` to the formatted message.
 *
 * Note the values can be any object that can be serialized by Jackson's ObjectMapper
 * (e.g. other Maps, JsonNodes, numbers, arrays, etc)
 */
Map myMap = new HashMap();
myMap.put("name1", "value1");
myMap.put("name2", "value2");
logger.info("log message {}", entries(myMap));

/*
 * Add "array":[1,2,3] to the JSON output,
 * and array=[1,2,3] to the formatted message.
 */
logger.info("log message {}", array("array", 1, 2, 3));

/*
 * Add fields of any object that can be unwrapped by Jackson's UnwrappableBeanSerializer to the JSON output.
 * i.e. The fields of an object can be written directly into the JSON output.
 * This is similar to the @JsonUnwrapped annotation.
 *
 * The formatted message will contain `myobject.toString()`
 */
logger.info("log message {}", fields(myobject));

/*
 * In order to normalize a field object name, static helper methods can be created.
 * For example, `foo(Foo)` calls `value("foo" , foo)`
 */
logger.info("log message {}", foo(foo));

Abbreviated convenience methods are available for all the structured argument types. For example, instead of keyValue(key, value), you can use kv(key, value).

Examples using Markers:

import static net.logstash.logback.marker.Markers.*;

/*
 * Add "name":"value" to the JSON output.
 */
logger.info(append("name", "value"), "log message");

/*
 * Add "name1":"value1","name2":"value2" to the JSON output by using multiple markers.
 */
logger.info(append("name1", "value1").and(append("name2", "value2")), "log message");

/*
 * Add "name1":"value1","name2":"value2" to the JSON output by using a map.
 *
 * Note the values can be any object that can be serialized by Jackson's ObjectMapper
 * (e.g. other Maps, JsonNodes, numbers, arrays, etc)
 */
Map myMap = new HashMap();
myMap.put("name1", "value1");
myMap.put("name2", "value2");
logger.info(appendEntries(myMap), "log message");

/*
 * Add "array":[1,2,3] to the JSON output
 */
logger.info(appendArray("array", 1, 2, 3), "log message");

/*
 * Add "array":[1,2,3] to the JSON output by using raw json.
 * This allows you to use your own json seralization routine to construct the json output
 */
logger.info(appendRaw("array", "[1,2,3]"), "log message");

/*
 * Add any object that can be serialized by Jackson's ObjectMapper
 * (e.g. Maps, JsonNodes, numbers, arrays, etc)
 */
logger.info(append("object", myobject), "log message");

/*
 * Add fields of any object that can be unwrapped by Jackson's UnwrappableBeanSerializer.
 * i.e. The fields of an object can be written directly into the json output.
 * This is similar to the @JsonUnwrapped annotation.
 */
logger.info(appendFields(myobject), "log message");

See DEPRECATED.md for other deprecated ways of adding json to the output.

AccessEvent Fields

The following sections describe the fields included in the JSON output by default for AccessEvents written by the

LogstashAccessEncoder,
LogstashAccessLayout, and
the logstash access appenders.

If you are using the composite encoders/layouts, then the fields written will vary by the providers you configure.

Standard Fields

These fields will appear in every AccessEvent unless otherwise noted. The field names listed here are the default field names. The field names can be customized (see Customizing Standard Field Names).

Field	Description
`@timestamp`	Time of the log event. (`yyyy-MM-dd'T'HH:mm:ss.SSSZZ`) See customizing timestamp.
`@version`	Logstash format version (e.g. `1`) See customizing version.
`message`	Message in the form `${remoteHost} - ${remoteUser} [${timestamp}] "${requestUrl}" ${statusCode} ${contentLength}`
`method`	HTTP method
`protocol`	HTTP protocol
`status_code`	HTTP status code
`requested_url`	Request URL
`requested_uri`	Request URI
`remote_host`	Remote host
`remote_user`	Remote user
`content_length`	Content length
`elapsed_time`	Elapsed time in millis

Header Fields

Request and response headers are not logged by default, but can be enabled by specifying a field name for them, like this:

<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder">
  <fieldNames>
    <requestHeaders>request_headers</requestHeaders>
    <responseHeaders>response_headers</responseHeaders>
  </fieldNames>
</encoder>

See Customizing Standard Field Names) for more details.

To write the header names in lowercase (so that header names that only differ by case are treated the same), set lowerCaseFieldNames to true, like this:

<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder">
  <fieldNames>
    <requestHeaders>request_headers</requestHeaders>
    <responseHeaders>response_headers</responseHeaders>
  </fieldNames>
  <lowerCaseHeaderNames>true</lowerCaseHeaderNames>
</encoder>

Headers can be filtered via configuring the requestHeaderFilter and/or the responseHeaderFilter with a HeaderFilter, such as the IncludeExcludeHeaderFilter.

The IncludeExcludeHeaderFilter can be configured like this:

<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder">
  <fieldNames>
    <requestHeaders>request_headers</requestHeaders>
  </fieldNames>
  <requestHeaderFilter>
    <include>Content-Type</include>
  </requestHeaderFilter>
</encoder>

Custom filters implementing HeaderFilter can be used by specifying the filter class like this:

  <requestHeaderFilter class="your.package.YourFilterClass"/>

Customizing Jackson

Logstash-logback-encoder uses Jackson to encode log and access events.

Logstash-logback-encoder provides sensible defaults for Jackson, but gives you full control over the Jackson configuration.

For example, you can:

specify the data format
customize the JsonFactory and JsonGenerator
register jackson modules
configure character escapes

Data Format

JSON is used by default, but other data formats supported by Jackson can be used.

⚠️ When using non-JSON data formats, you must include the appropriate jackson dataformat library on the runtime classpath, typically via a maven/gradle dependency (e.g. for Smile, include jackson-dataformat-smile).

Decorators are provided for the following data formats:

To use one these formats, specify the <jsonFactoryDecorator> like this:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <jsonFactoryDecorator class="net.logstash.logback.decorate.smile.SmileJsonFactoryDecorator"/>
</encoder>

Other data formats can be used by implementing a custom net.logstash.logback.decorate.JsonFactoryDecorator.

The following decorators can be used to configure data-format-specific generator features:

For example:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <jsonFactoryDecorator class="net.logstash.logback.decorate.smile.SmileJsonFactoryDecorator"/>
  <jsonGeneratorDecorator class="net.logstash.logback.decorate.smile.SmileFeatureJsonGeneratorDecorator">
    <disable>WRITE_HEADER</disable>
  </jsonGeneratorDecorator>
</encoder>

Customizing JSON Factory and Generator

The JsonFactory and JsonGenerator used to write output can be customized by instances of:

For example, you could enable pretty printing by using the PrettyPrintingJsonGeneratorDecorator

Or customize object mapping like this:

public class ISO8601DateDecorator implements JsonFactoryDecorator  {

	@Override
	public JsonFactory decorate(JsonFactory factory) {
		ObjectMapper codec = (ObjectMapper) factory.getCodec();
		codec.setDateFormat(new ISO8601DateFormat());
		return factory;
	}
}

and then specify the decorators in the logback.xml file like this:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <jsonGeneratorDecorator class="net.logstash.logback.decorate.PrettyPrintingJsonGeneratorDecorator"/>
  <jsonFactoryDecorator class="your.package.ISO8601DateDecorator"/>
</encoder>

JsonFactory and JsonGenerator features can be enabled/disabled by using the FeatureJsonFactoryDecorator and FeatureJsonGeneratorDecorator, respectively. For example:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <jsonFactoryDecorator class="net.logstash.logback.decorate.FeatureJsonFactoryDecorator">
    <disable>USE_THREAD_LOCAL_FOR_BUFFER_RECYCLING</disable>
  </jsonFactoryDecorator>
  <jsonGeneratorDecorator class="net.logstash.logback.decorate.FeatureJsonGeneratorDecorator">
    <enable>WRITE_NUMBERS_AS_STRINGS</enable>
  </jsonGeneratorDecorator>
</encoder>

See the net.logstash.logback.decorate package and sub-packages for other decorators.

Registering Jackson Modules

By default, Jackson modules are dynamically registered via ObjectMapper.findAndRegisterModules().

Therefore, you just need to add jackson modules (e.g. jackson-datatype-jdk8) to the classpath, and they will be dynamically registered.

To disable automatic discovery, set <findAndRegisterJacksonModules>false</findAndRegisterJacksonModules> on the encoder/layout.

If you have a module that Jackson is not able to dynamically discover, you can register it manually via a JsonFactoryDecorator.

Customizing Character Escapes

By default, when a string is written as a JSON string value, any character not allowed in a JSON string will be escaped. For example, the newline character (ASCII 10) will be escaped as \n.

To customize these escape sequences, use the net.logstash.logback.decorate.CharacterEscapesJsonFactoryDecorator.

For example, if you want to use something other than \n as the escape sequence for the newline character, you can do the following:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <jsonFactoryDecorator class="net.logstash.logback.decorate.CharacterEscapesJsonFactoryDecorator">
    <escape>
      <targetCharacterCode>10</targetCharacterCode>
      <escapeSequence>\u2028</escapeSequence>
    </escape>
  </jsonFactoryDecorator>
</encoder>

You can also disable all the default escape sequences by specifying <includeStandardAsciiEscapesForJSON>false</includeStandardAsciiEscapesForJSON> on the CharacterEscapesJsonFactoryDecorator. If you do this, then you will need to register custom escapes for each character that is illegal in JSON string values. Otherwise, invalid JSON could be written.

Masking

The MaskingJsonGeneratorDecorator can be used to mask sensitive values (e.g. personally identifiable information (PII) or financial data).

Data to be masked can be identified by path and/or by value.

Identifying field values to mask by path

Paths of fields to mask can be specified in several ways, as shown in the following example:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <jsonGeneratorDecorator class="net.logstash.logback.mask.MaskingJsonGeneratorDecorator">

    <!--
      The default mask string can optionally be specified by <defaultMask>.
      When the default mask string is not specified, **** is used.
    -->
    <defaultMask>****</defaultMask>

    <!-- Field paths to mask added via <path> will use the default mask string -->
    <path>singleFieldName</path>
    <path>/absolute/path/to/mask</path>
    <path>partial/path/to/mask</path>
    <path>partial/path/with/*/wildcard</path>
    <path>tilde~0slash~1escapedPath</path>

    <!-- Multiple field paths can be specified as a comma separated string in the <paths> element. -->
    <paths>path1,path2,path3</paths>

    <!-- Field paths to mask added via <pathMask> can use a non-default mask string -->
    <pathMask>
      <path>some/path</path>
      <path>some/other/path</path>
      <mask>[masked]</mask>
    </pathMask>
    <pathMask>
      <paths>anotherFieldName,anotherFieldName2</paths>
      <mask>**anotherCustomMask**</mask>
    </pathMask>

    <!-- Field paths to mask can be supplied dynamically with an implementation of MaskingJsonGeneratorDecorator.PathMaskSupplier -->
    <pathMaskSupplier class="your.custom.PathMaskSupplierA"/>

    <!-- Custom implementations of net.logstash.logback.mask.FieldMasker
         can be used for more advanced masking behavior-->
    <fieldMasker class="your.custom.FieldMaskerA"/>
    <fieldMasker class="your.custom.FieldMaskerB"/>
  </jsonGeneratorDecorator>
</encoder>

See PathBasedFieldMasker for the path string format and more examples. But in general:

Paths follow a format similar to (but not exactly same as) a JSON Pointer.
Absolute paths start with / and are absolute to the root of the JSON output event (e.g. /@timestamp would mask the default timestamp field)
Partial paths do not start with / and match anywhere that path sequence is seen in the output.
A path with a single token (i.e. no / characters) will match all occurrences of a field with the given name
A wildcard token (*) will match anything at that location within the path
Use ~1 to escape / within a token
Use ~0 to escape ~ within a token

Identifying field values to mask by value

Specific values to be masked can be specified in several ways, as seen in the following example:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <jsonGeneratorDecorator class="net.logstash.logback.mask.MaskingJsonGeneratorDecorator">

    <!--
      The default mask string can optionally be specified by <defaultMask>.
      When the default mask string is not specified, **** is used.
    -->
    <defaultMask>****</defaultMask>

    <!-- Values to mask added via <value> will use the  default mask string -->
    <value>^foo$</value>
    <value>^bar$</value>

    <!-- Multiple values can be specified as a comma separated string in the <values> element. -->
    <values>^baz$,^blah$</values>

    <!-- Values to mask added via <valueMask> can use a non-default mask string
         The mask string here can reference regex capturing groups if needed -->
    <valueMask>
      <value>^(foo)-.*$</value>
      <value>^(bar)-.*$</value>
      <mask>\1****</mask>
    </valueMask>

    <!-- Values to mask can be supplied dynamically with an implementation of MaskingJsonGeneratorDecorator.ValueMaskSupplier -->
    <pathMaskSupplier class="your.custom.ValueMaskSupplierA"/>

    <!-- Custom implementations of net.logstash.logback.mask.ValueMasker
         can be used for more advanced masking behavior-->
    <valueMasker class="your.custom.ValueMaskerA"/>
    <valueMasker class="your.custom.ValueMaskerB"/>
  </jsonGeneratorDecorator>
</encoder>

Identifying data to mask by value is much more expensive than identifying data to mask by path. Therefore, prefer identifying data to mask by path.

Customizing Standard Field Names

The standard field names above for LoggingEvents and AccessEvents can be customized by using the fieldNamesconfiguration element in the encoder or appender configuration.

For example:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <fieldNames>
    <timestamp>time</timestamp>
    <message>msg</message>
    ...
  </fieldNames>
</encoder>

Prevent a field from being output by setting the field name to [ignore].

For LoggingEvents, see LogstashFieldNames for all the field names that can be customized. Each java field name in that class is the name of the xml element that you would use to specify the field name (e.g. logger, levelValue). Additionally, a separate set of shortened field names can be configured like this:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <fieldNames class="net.logstash.logback.fieldnames.ShortenedFieldNames"/>
</encoder>

For LoggingEvents, log the caller info, MDC properties, and context properties in sub-objects within the JSON event by specifying field names for caller, mdc, and context, respectively.

For AccessEvents, see LogstashAccessFieldNames for all the field names that can be customized. Each java field name in that class is the name of the xml element that you would use to specify the field name (e.g. fieldsMethod, fieldsProtocol).

Customizing Version

The version field value by default is the string value 1.

The value can be changed like this:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <version>2</version>
</encoder>

The value can be written as a number (instead of a string) like this:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <writeVersionAsInteger>true</writeVersionAsInteger>
</encoder>

Customizing Timestamp

By default, timestamps are written as string values in the format yyyy-MM-dd'T'HH:mm:ss.SSSZZ (e.g. 2018-04-28T22:23:59.164-07:00), in the default TimeZone of the host Java platform.

You can change the timezone like this:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <timeZone>UTC</timeZone>
</encoder>

The value of the timeZone element can be any string accepted by java's TimeZone.getTimeZone(String id) method.

You can change the pattern used like this:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <timestampPattern>yyyy-MM-dd'T'HH:mm:ss.SSS</timestampPattern>
</encoder>

Use these timestamp pattern values to output the timestamp as a unix timestamp (number of milliseconds since unix epoch).

[UNIX_TIMESTAMP_AS_NUMBER] - write the timestamp value as a numeric unix timestamp
[UNIX_TIMESTAMP_AS_STRING] - write the timestamp value as a string verion of the numeric unix timestamp

For example:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <timestampPattern>[UNIX_TIMESTAMP_AS_NUMBER]</timestampPattern>
</encoder>

Customizing Logger Name Length

For LoggingEvents, you can shorten the logger name field length similar to the normal pattern style of %logger{36}. Examples of how it is shortened can be found here

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <shortenedLoggerNameLength>36</shortenedLoggerNameLength>
</encoder>

Customizing Stack Traces

When logging exceptions, stack traces are formatted using logback's ExtendedThrowableProxyConverter by default. However, you can configure the encoder to use any ThrowableHandlingConverter to format stacktraces.

Note that the ThrowableHandlingConverter only applies to the exception passed as an extra argument to the log method, the way you normally log an exception in slf4j. Do NOT use structured arguments or markers for exceptions.

A powerful ShortenedThrowableConverter is included in the logstash-logback-encoder library to format stacktraces by:

Limiting the number of stackTraceElements per throwable (applies to each individual throwable. e.g. caused-bys and suppressed)
Limiting the total length in characters of the trace
Abbreviating class names
Filtering out consecutive unwanted stackTraceElements based on regular expressions.
Using evaluators to determine if the stacktrace should be logged.
Outputing in either 'normal' order (root-cause-last), or root-cause-first.
Computing and inlining hexadecimal hashes for each exception stack (more info).

For example:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
    <maxDepthPerThrowable>30</maxDepthPerThrowable>
    <maxLength>2048</maxLength>
    <shortenedClassNameLength>20</shortenedClassNameLength>
    <exclude>sun\.reflect\..*\.invoke.*</exclude>
    <exclude>net\.sf\.cglib\.proxy\.MethodProxy\.invoke</exclude>
    <evaluator class="myorg.MyCustomEvaluator"/>
    <rootCauseFirst>true</rootCauseFirst>
    <inlineHash>true</inlineHash>
  </throwableConverter>
</encoder>

ShortenedThrowableConverter can even be used within a PatternLayout to format stacktraces in any non-JSON logs you may have.

Prefix/Suffix/Separator

You can specify a prefix (written before the JSON object), suffix (written after the JSON object), and/or line separator (written after suffix), which may be required for the log pipeline you are using, such as:

If you are using the Common Event Expression (CEE) format for syslog, you need to add the @cee: prefix.
If you are using other syslog destinations, you might need to add the standard syslog headers.
If you are using Loggly, you might need to add your customer token.

For example, to add standard syslog headers for syslog over UDP, configure the following:

<configuration>
  <conversionRule conversionWord="syslogStart" converterClass="ch.qos.logback.classic.pattern.SyslogStartConverter"/>

  <appender name="stash" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
    <host>MyAwesomeSyslogServer</host>
    <!-- port is optional (default value shown) -->
    <port>514</port>
    <layout>
      <prefix class="ch.qos.logback.classic.PatternLayout">
        <pattern>%syslogStart{USER}</pattern>
      </prefix>
    </layout>
  </appender>

  ...
</configuration>

When using the LogstashEncoder, LogstashAccessEncoder or a composite encoder, the prefix is an Encoder, not a Layout, so you will need to wrap the prefix PatternLayout in a LayoutWrappingEncoder like this:

<configuration>
  ...
  <appender ...>
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      ...
      <prefix class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
        <layout class="ch.qos.logback.classic.PatternLayout">
          <pattern>@cee:</pattern>
        </layout>
      </prefix>    
    </encoder>
  </appender>
</configuration>

Note that logback's xml configuration reader will trim whitespace from xml element values. Therefore, if you want to end the prefix or suffix pattern with whitespace, first add the whitespace, and then add something like %mdc{keyThatDoesNotExist} after it. For example <pattern>your pattern %mdc{keyThatDoesNotExist}</pattern>. This will cause logback to output the whitespace as desired, and then a blank string for the MDC key that does not exist.

⚠️ If you encounter the following warning: A "net.logstash.logback.encoder.LogstashEncoder" object is not assignable to a "ch.qos.logback.core.Appender" variable., you are encountering a backwards incompatibilility introduced in logback 1.2.1. Please vote for LOGBACK-1326 and add a thumbs up to PR#383 to try to get this addressed in logback. In the meantime, the only solution is to downgrade logback-classic and logback-core to 1.2.0

The line separator, which is written after the suffix, can be specified as:

SYSTEM (uses the system default)
UNIX (uses \n)
WINDOWS (uses \r\n), or
any other string.

For example:

<configuration>
  ...
  <appender ...>
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      ...
      <lineSeparator>UNIX</lineSeparator>
    </encoder>
  </appender>
</configuration>

Composite Encoder/Layout

If you want greater flexibility in the JSON format and data included in LoggingEvents and AccessEvents, use the LoggingEventCompositeJsonEncoder and AccessEventCompositeJsonEncoder (or the corresponding layouts).

These encoders/layouts are composed of one or more JSON providers that contribute to the JSON output. No providers are configured by default in the composite encoders/layouts. You must add the ones you want.

For example:

<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
  <providers>
    <mdc/>
    <pattern>
      <pattern>
        {
          "timestamp": "%date{ISO8601}",
          "myCustomField": "fieldValue",
          "relative": "#asLong{%relative}"
        }
      </pattern>
    </pattern>
    <stackTrace>
      <throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
        <maxDepthPerThrowable>30</maxDepthPerThrowable>
        <maxLength>2048</maxLength>
        <shortenedClassNameLength>20</shortenedClassNameLength>
        <exclude>^sun\.reflect\..*\.invoke</exclude>
        <exclude>^net\.sf\.cglib\.proxy\.MethodProxy\.invoke</exclude>
        <evaluator class="myorg.MyCustomEvaluator"/>
        <rootCauseFirst>true</rootCauseFirst>
      </throwableConverter>
    </stackTrace>
  </providers>
</encoder>

The logstash-logback-encoder library contains many providers out-of-the-box, and you can even plug-in your own by extending JsonProvider. Each provider has its own configuration options to further customize it.

Providers for LoggingEvents

For LoggingEvents, the available providers and their configuration properties (defaults in parenthesis) are as follows:

Provider	Description/Properties
`timestamp`	Event timestamp `fieldName` - Output field name (`@timestamp`) `pattern` - Output format (`yyyy-MM-dd'T'HH:mm:ss.SSSZZ`) If set to `[UNIX_TIMESTAMP_AS_NUMBER]`, then the timestamp will be written as a numeric unix timestamp value If set to `[UNIX_TIMESTAMP_AS_STRING]`, then the timestamp will be written as a string unix timestamp value `timeZone` - Timezone (local timezone)
`version`	Logstash JSON format version `fieldName` - Output field name (`@version`) `version` - Output value (`1`) `writeAsInteger` - Write the version as a integer value (`false` = write as a string value)
`message`	Formatted log event message `fieldName` - Output field name (`message`)
`rawMessage`	Raw log event message, as opposed to formatted log where parameters are resolved `fieldName` - Output field name (`raw_message`)
`loggerName`	Name of the logger that logged the message `fieldName` - Output field name (`logger_name`) `shortenedLoggerNameLength` - Length to which the name will be attempted to be abbreviated (no abbreviation)
`threadName`	Name of the thread from which the event was logged `fieldName` - Output field name (`thread_name`)
`logLevel`	Logger level text (INFO, WARN, etc) `fieldName` - Output field name (`level`)
`logLevelValue`	Logger level numerical value `fieldName` - Output field name (`level_value`)
`callerData`	Outputs data about from where the logger was called (class/method/file/line) `fieldName` - Sub-object field name (no sub-object) `classFieldName` - Field name for class name (`caller_class_name`) `methodFieldName` - Field name for method name (`caller_method_name`) `fileFieldName` - Field name for file name (`caller_file_name`) `lineFieldName` - Field name for lin number (`caller_line_number`)
`stackTrace`	Stacktrace of any throwable logged with the event. Stackframes are separated by newline chars. `fieldName` - Output field name (`stack_trace`) `throwableConverter` - The `ThrowableHandlingConverter` to use to format the stacktrace (`stack_trace`)
`stackHash`	(Only if a throwable was logged) Computes and outputs a hexadecimal hash of the throwable stack. This helps identifying several occurrences of the same error (more info). `fieldName` - Output field name (`stack_hash`) `exclude` - Regular expression pattern matching stack trace elements to exclude when computing the error hash `exclusions` - Coma separated list of regular expression patterns matching stack trace elements to exclude when computing the error hash
`throwableClassName`	(Only if a throwable was logged) Outputs a field that contains the class name of the thrown Throwable. `fieldName` - Output field name (`throwable_class`) `useSimpleClassName` - When true, the throwable's simple class name will be used. When false, the fully qualified class name will be used. (`true`)
`throwableRootCauseClassName`	(Only if a throwable was logged) Outputs a field that contains the class name of the root cause of the thrown Throwable. `fieldName` - Output field name (`throwable_root_cause_class`) `useSimpleClassName` - When true, the throwable's simple class name will be used. When false, the fully qualified class name will be used. (`true`)
`context`	Outputs entries from logback's context `fieldName` - Sub-object field name (no sub-object)
`contextName`	Outputs the name of logback's context `fieldName` - Output field name (`context`)
`mdc`	Outputs entries from the Mapped Diagnostic Context (MDC). Will include all entries by default. When key names are specified for inclusion, then all other fields will be excluded. When key names are specified for exclusion, then all other fields will be included. It is a configuration error to specify both included and excluded key names. `fieldName` - Sub-object field name (no sub-object) `includeMdcKeyName` - Name of keys to include (all) `excludeMdcKeyName` - Name of keys to include (none)
`tags`	Outputs logback markers as a comma separated list `fieldName` - Output field name (`tags`)
`logstashMarkers`	Used to output Logstash Markers as specified in Event-specific Custom Fields
`nestedField`	Nests a JSON object under the configured fieldName. The nested object is populated by other providers added to this provider. See Nested JSON provider `fieldName` - Output field name `providers` - The providers that should populate the nested object.
`pattern`	Outputs fields from a configured JSON Object string, while substituting patterns supported by logback's `PatternLayout`. See Pattern JSON Provider `pattern` - JSON object string (no default) `omitEmptyFields` - whether to omit fields with empty values (`false`)
`arguments`	Outputs fields from the event arguments array. See Event-specific Custom Fields `fieldName` - Sub-object field name (no sub-object) `includeNonStructuredArguments` - Include arguments that are not an instance of `StructuredArgument`. (default=false) Object field name will be `nonStructuredArgumentsFieldPrefix` prepend to the argument index `nonStructuredArgumentsFieldPrefix` - Object field name prefix (default=arg)
`uuid`	Outputs random UUID as field value. Handy when you want to provide unique identifier for log lines `fieldName` - Output field name (`uuid`) `strategy` - UUID generation strategy (`random`). Supported options: `random` - for Type 4 UUID `time` - for Type 1 time based UUID `ethernet` - Only for 'time' strategy. When defined - MAC address to use for location part of UUID. Set it to `interface` value to use real underlying network interface or to specific values like `00:C0:F0:3D:5B:7C`
`sequence`	Outputs an incrementing sequence number for every log event. Useful for tracking pottential message loss during transport (eg. UDP) `fieldName` - Output field name (`sequence`)

Providers for AccessEvents

For AccessEvents, the available providers and their configuration properties (defaults in parenthesis) are as follows:

Provider	Description/Properties
`timestamp`	Event timestamp `fieldName` - Output field name (`@timestamp`) `pattern` - Output format (`yyyy-MM-dd'T'HH:mm:ss.SSSZZ`) If set to `[UNIX_TIMESTAMP_AS_NUMBER]`, then the timestamp will be written as a numeric unix timestamp value If set to `[UNIX_TIMESTAMP_AS_STRING]`, then the timestamp will be written as a string unix timestamp value `timeZone` - Timezone (local timezone)
`version`	Logstash JSON format version `fieldName` - Output field name (`@version`) `version` - Output value (`1`) `writeAsInteger` - Write the version as a integer value (`false` = write as a string value)
`message`	Message in the form `${remoteHost} - ${remoteUser} [${timestamp}] "${requestUrl}" ${statusCode} ${contentLength}` `fieldName` - Output field name (`message`)
`method`	HTTP method `fieldName` - Output field name (`method`)
`protocol`	HTTP protocol `fieldName` - Output field name (`protocol`)
`statusCode`	HTTP status code `fieldName` - Output field name (`status_code`)
`requestedUrl`	Requested URL `fieldName` - Output field name (`requested_url`)
`requestedUri`	Requested URI `fieldName` - Output field name (`requested_uri`)
`remoteHost`	Remote Host `fieldName` - Output field name (`remote_host`)
`remoteUser`	Remote User `fieldName` - Output field name (`remote_user`)
`contentLength`	Content length `fieldName` - Output field name (`content_length`)
`elapsedTime`	Elapsed time in milliseconds `fieldName` - Output field name (`elapsed_time`)
`requestHeaders`	Include the request headers `fieldName` - Output field name (no default, must be provided) `lowerCaseHeaderNames` - Write header names in lower case (`false`) `filter` - A filter to determine which headers to include/exclude. See `HeaderFilter` and `IncludeExcludeHeaderFilter`
`responseHeaders`	Include the response headers `fieldName` - Output field name (no default, must be provided) `lowerCaseHeaderNames` - Write header names in lower case (`false`) `filter` - A filter to determine which headers to include/exclude. See `HeaderFilter` and `IncludeExcludeHeaderFilter`
`nestedField`	Nests a JSON object under the configured fieldName. The nested object is populated by other providers added to this provider. See Nested JSON provider `fieldName` - Output field name `providers` - The providers that should populate the nested object.
`pattern`	Outputs fields from a configured JSON Object string, while substituting patterns supported by logback access's `PatternLayout`. See Pattern JSON Provider `pattern` - JSON object string (no default) `omitEmptyFields` - whether to omit fields with empty values (`false`)

Nested JSON Provider

Use the nestedField provider to create a sub-object in the JSON event output.

For example...

<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
  <providers>
    <timestamp/>
    <nestedField>
      <fieldName>fields</fieldName>
      <providers>
        <logLevel/>
      </providers>
    </nestedField>
  </providers>
</encoder>

...will produce something like...

{
  "@timestamp":"...",
  "fields":{
    "level": "DEBUG"
  }
}

Pattern JSON Provider

When used with a composite JSON encoder/layout, the pattern JSON provider can be used to define a template for a portion of the logged JSON output. The encoder/layout will populate values within the template. Every value in the template is treated as a pattern for logback's standard PatternLayout so it can be a combination of literal strings (for some constants) and various conversion specifiers (like %d for date).

The pattern string (configured within the pattern provider) must be a JSON Object. The contents of the JSON object are included within the logged JSON output.

This example...

<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
  <providers>
    <!-- provides the timestamp -->
    <timestamp/>

    <!-- provides the version -->
    <version/>

    <!-- provides the fields in the configured pattern -->
    <pattern>
      <!-- the pattern that defines what to include -->
      <pattern>
        { "level": "%level" }
      </pattern>
    </pattern>
  </providers>
</encoder>

... will produce something like...

{
  "@timestamp":"...",
  "@version": "1",
  "level": "DEBUG"
}

The real power comes from the fact that there are lots of standard conversion specifiers so you can customise what is logged and how. For example, you could log a single specific value from MDC with %mdc{mykey}. Or, for access logs, you could log a single request header with %i{User-Agent}.

You can use nested objects and arrays in your pattern.

If you use a null, number, or a boolean constant in a pattern, it will keep its type in the resulting JSON. However, only the text values are searched for conversion patterns. And, as these patterns are sent through PatternLayout, the result is always a string even for something which you may feel should be a number - like for %b (bytes sent, in access logs).

You can either deal with the type conversion on the logstash side or you may use special operations provided by this encoder. The operations are:

#asLong{...} - evaluates the pattern in curly braces and then converts resulting string to a long (or a null if conversion fails).
#asDouble{...} - evaluates the pattern in curly braces and then converts resulting string to a double (or a null if conversion fails).
#asJson{...} - evaluates the pattern in curly braces and then converts resulting string to json (or a null if conversion fails).
#tryJson{...} - evaluates the pattern in curly braces and then converts resulting string to json (or just the string if conversion fails).

So this example...

<pattern>
  {
    "line_str": "%line",
    "line_long": "#asLong{%line}",
    "has_message": "#asJson{%mdc{hasMessage}}",
    "json_message": "#asJson{%message}"
  }
</pattern>

... And this logging code...

MDC.put("hasMessage", "true");
LOGGER.info("{\"type\":\"example\",\"msg\":\"example of json message with type\"}");

...will produce something like...

{
  "line_str":"97",
  "line_long":97,
  "has_message":true,
  "json_message":{"type":"example","msg":"example of json message with type"}
}

Note that the value that is sent for line_long is a number even though in your pattern it is a quoted text. And the json_message field value is a json object, not a string.

Omitting fields with empty values

The pattern provider can be configured to omit fields with the following empty values:

null
empty string ("")
empty array ([])
empty object ({})
objects containing only fields with empty values
arrays containing only empty values

To omit fields with empty values, configure omitEmptyFields to true (default is false), like this:

<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
  <providers>
    <pattern>
      <omitEmptyFields>true</omitEmptyFields>
      <pattern>
        {
          "logger": "%logger",
          "level": "%level",
          "thread": "%thread",
          "message": "%message",
          "traceId": "%mdc{traceId}"
        }
      </pattern>
    </pattern>
  </providers>
</encoder>

If the MDC did not contain a traceId entry, then a JSON log event from the above pattern would not contain the traceId field...

{
  "logger": "com.example...",
  "level": "DEBUG",
  "thread": "exec-1",
  "message": "Hello World!"
}

LoggingEvent patterns

For LoggingEvents, patterns from logback-classic's PatternLayout are supported.

For example:

<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
  <providers>
    <timestamp/>
    <pattern>
      <pattern>
        {
        "custom_constant": "123",
        "tags": ["one", "two"],
        "logger": "%logger",
        "level": "%level",
        "thread": "%thread",
        "message": "%message",
...
        }
      </pattern>
    </pattern>
  </providers>
</encoder>

AccessEvent patterns

For AccessEvents, patterns from logback-access's PatternLayout are supported.

For example:

<encoder class="net.logstash.logback.encoder.AccessEventCompositeJsonEncoder">
  <providers>
    <pattern>
      <pattern>
        {
        "custom_constant": "123",
        "tags": ["one", "two"],
        "remote_ip": "%a",
        "status_code": "%s",
        "elapsed_time": "%D",
        "user_agent": "%i{User-Agent}",
        "accept": "%i{Accept}",
        "referer": "%i{Referer}",
        "session": "%requestCookie{JSESSIONID}",
...
        }
      </pattern>
    </pattern>
  </providers>
</encoder>

There is also a special operation that can be used with this AccessEvents:

#nullNA{...} - if the pattern in curly braces evaluates to a dash ("-"), it will be replaced with a null value.

You may want to use it because many of the PatternLayout conversion specifiers from logback-access will evaluate to "-" for non-existent value (for example for a cookie, header or a request attribute).

So the following pattern...

<pattern>
  {
    "default_cookie": "%requestCookie{MISSING}",
    "filtered_cookie": "#nullNA{%requestCookie{MISSING}}"
  }
</pattern>

...will produce...

{
  "default_cookie": "-",
  "filtered_cookie": null
}

Custom JSON Provider

You can create your own JSON provider by implementing the JsonProvider interface (or extending one of the existing classes that implements the JsonProvider interface).

Then, add the provider to a LoggingEventCompositeJsonEncoder like this:

<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
  <providers>
    ...
    <provider class="your.provider.YourJsonProvider">
        <!-- Any properties exposed by your provider can be set here -->
    </provider>
    ...
  </providers>
</encoder>

or a LogstashEncoder like this:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
    ...
    <provider class="your.provider.YourJsonProvider">
        <!-- Any properties exposed by your provider can be set here -->
    </provider>
    ...
</encoder>

You can do something similar for AccessEventCompositeJsonEncoder and LogstashAccessEncoder as well, if your JsonProvider handles IAccessEvents.

Status Listeners

During execution, the encoders/appenders/layouts provided in logstash-logback-encoder will add logback status messages to the logback StatusManager. These status messages are typically reported via a logback StatusListener.

Since the async appenders (especially the tcp appenders) report warnings and errors via the status manager, a default status listener that outputs WARN and ERROR level status messages to standard out will be registered on startup if a status listener has not already been registered. To disable the automatic registering of the default status listener by an appender, do one of the following:

register a different logback status listener, or
set <addDefaultStatusListener>false</addDefaultStatusListener in each async appender.

Profiling

Memory usage and performance of logstash-logback-encoder have been improved by addressing issues discovered with the help of the YourKit Java Profiler.

YourKit, LLC has graciously donated a free license of the YourKit Java Profiler to this open source project.

Name		Name	Last commit message	Last commit date
Latest commit History 687 Commits
.github		.github
.mvn/wrapper		.mvn/wrapper
src		src
.gitattributes		.gitattributes
.gitignore		.gitignore
CONTRIBUTORS.md		CONTRIBUTORS.md
DEPRECATED.md		DEPRECATED.md
LICENSE		LICENSE
README.md		README.md
license-header.txt		license-header.txt
mvnw		mvnw
mvnw.cmd		mvnw.cmd
pom.xml		pom.xml
stack-hash.md		stack-hash.md

License

Licenses found

ellisdc/logstash-logback-encoder

Folders and files

Latest commit

History

Repository files navigation

Logstash Logback Encoder

Contents:

Including it in your project

Java Version Requirements

Usage

UDP Appender

TCP Appenders

Keep-alive

Multiple Destinations

Reconnection Delay

Write buffer size

Write timeouts

SSL

Async Appenders

Appender Listeners

Encoders / Layouts

LoggingEvent Fields

Standard Fields

MDC fields

Context fields

Caller Info Fields

Custom Fields

Global Custom Fields

Event-specific Custom Fields

AccessEvent Fields

Standard Fields

Header Fields

Customizing Jackson

Data Format

Customizing JSON Factory and Generator

Registering Jackson Modules

Customizing Character Escapes

Masking

Identifying field values to mask by path

Identifying field values to mask by value

Customizing Standard Field Names

Customizing Version

Customizing Timestamp

Customizing Logger Name Length

Customizing Stack Traces

Prefix/Suffix/Separator

Composite Encoder/Layout

Providers for LoggingEvents

Providers for AccessEvents

Nested JSON Provider

Pattern JSON Provider

Omitting fields with empty values

LoggingEvent patterns

AccessEvent patterns

Custom JSON Provider

Status Listeners

Profiling

About

Resources

License

Licenses found

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages