Skip to content

Latest commit

 

History

History
188 lines (109 loc) · 8.36 KB

README.md

File metadata and controls

188 lines (109 loc) · 8.36 KB
Raw
        __                 __
       / /  ___ ________ _/ /  ___  ___ _
      / /__/ _ `/ __/ _ `/ /__/ _ \/ _ `/
     /____/\_,_/_/  \_,_/____/\___/\_, /
                                  /___/

1. What is it?

A Laravel log sender agent.

Laralog is suitable for docker containers and small Laravel or Lumen environments that requires to collect and send logs without install the Logstash and the Java runtime.

Laralog is decoupled from Laravel, so it can read and send the Laravel logs even when Laravel fails.

Laralog can send logs to different services like ElasticSearch and Datadog.

2. Features

  • PHP based solution.
  • Distributed as a single PHAR file.
  • Compatible with Supervisor.
  • Small footprint.
  • Easy to configure.
  • File handler safe (Logs files can be rotated, linked or truncated on fly).
  • Decoupled solution based on Mamuph framework
  • It supports multiple senders.

3. Supported senders

  • ElasticSearch (Default)
  • Datadog

4. Usage examples

  • Read logs sequentially without stop and send logs to a single ES instance using a custom index:

      laralog https://myesinstance.lan:9200 --input=laravel.log --index=myindex

  • Read logs from a external file and logs to a group of servers defined in a file (One instance per line):

      laralog my_es_servers.txt < laravel.log

  • Read logs sequentially without stop and send logs asynchronously using a different timezone:

      laralog https://myinstance.lan:9200 --async --to-timezone=Europe/Madrid

  • Read logs sequentially without stop and send logs to Datadog converting the Laravel log timestamp that uses a different timezone to UTC:

      laralog https://http-intake.logs.datadoghq.eu/v1/input/<DATADOG_API_KEY> --sender=datadog --from-timezone=Europe/Madrid

  • Read logs sequentially without stop and output the new entries to STDOUT:

      laralog https://myinstance.lan:9200 --input=laravel.log -v

  • Read logs sequentially without stop and send logs to Datadog using a different service name and overriding the default hostname:

      laralog https://http-intake.logs.datadoghq.eu/v1/input/<DATADOG_API_KEY> --sender=datadog --index=mylogs --hostname=example.net

5. Parameters

Parameter Description
--input=[laravel.log] Log file path to read sequentially without stop. When this parameter is not used Laralog will read from the STDIN.
--index=[index_name] Document index.
--ignore=[levels] Error levels to ignore separated by comma.
--batch-size=[size] Batch size used by queue. By default the batch size is 10.
--async Send logs asynchronous using the future mode (Faster approach).
--no-check-cert Do not check the SSL certificate when https is used.
--retries=[number] Number of attemps when logs a sent without success. By default 2 are the number of retries when this parameter is not used.
--read-freq=[ms] File read frequency in milliseconds. 500ms (0.5 sec) is the default value.
--from-timezone=[zone] Read logs from a different timezone. See timezone list.
--to-timezone=[zone] Convert logs to a different timezone See timezone list (Only ElasticSearch).
--sender=[sender] Sender to use: elasticsearch (Default) or datadog.
--hostname=[hostname] Override the default hostname.
--smart Serialize JSON context when is possible.
-v Verbose mode that output the new log entries to STDOUT.

6. Senders

6.1 ElasticSearch

6.1.1 Configuration details

Sender name: "elasticsearch" (Default sender).

6.1.2 Limitations and notes

None

6.2 Datadog

6.2.1 Configuration details

Sender name: "datadog" (--sender=datadog)

6.2.2 Limitations and notes

  • Timestamps are always converted to UTC timezone, so the option "--to-timezone" option is ignored when this sender is used. It is not a issue since Datadog can display the log timestamps according to the timezone configured in their settings.
  • Request are always asynchronous, however the "--async" option will improve the request speed because the sender driver will not wait for the response body.
  • The "--no-check-cert" will be ignore. It is because all the Datadog servers should always have a valid certificate when "https" is used (See Datadog API).
  • It's not recommended to use a value higher than 50 for the "--batch-size" because Datadog do not support more than 50 entries per request (See Datadog API).

7. About the mess with timezones

Normally applications and servers use "UTC" as default timezone, however sometimes due operative requirements our apps and mostly their logs are using timestaps formatted for a different timezone.

Laralog will use the configured PHP or OS timezone in order to convert log timestamps (only when "--to-timezone" is used).

In order to deal with timestamps, Laralog provides the following optional parameters:

--from-timezone=[zone] : This parameter specify the Laravel app timezone in other case the default PHP or OS timezone is used.

--to-timezone=[zone] : This parameter specify the target timezone that timestamps are converted (Datadog sender will used always "UTC").

8. About the safe file handler

Laralog support different ways to read logs, for example you can redirect the STDIN or tail a log file, and it's fine to use this way when for example you want to send old logs that has not been collected before, however for log files that are writing continuously the best way is to use the "--input=[laravel.log]" parameter so the safe file handler is used.

Benefits of the safe file handler:

  • Log files are re-opened when they are re-created due a log rotation, re-linked or truncated.
  • Logs are read and sent without stop and on demand.

The safe file handler do not use LibEvent, inotify or another file event notification mechanism so instead it uses a regular file check (See "--read-freq=[ms]" parameter). This makes the log file check process a bit slow but in this way the LibEvent extension is not required.

9. Smart serialization

The option "--smart" will automatically serialize the JSON chunks found in the context and send it as the "parameters" field.

Example:

    [2019-11-24 06:30:56] production.INFO: Job finished {"connection":"database","queue":"default","job":"SendNewsletter"}

The previous log entry contains JSON chunk as context and normally it's send to the Logstash or Datadog as a string, however with the smart serialization option Laralog will try to extract the JSON part and send it as the field "params".

10. How to build Laralog

Laralog is available as a self-executable PHAR file, however if you want to build your own custom PHAR you need to follow the following steps:

  • Download Caveman (The Mamuph Helper Tool)

  • Inside the Laralog directory type:

      caveman build . -x -r

11. Download

If you are lazy and you don't want to build your Laralog version, you are welcome to download the following executable (PHAR file):

Download last release

Signed binaries

From v4 onwards the signed binaries are also available. The signature consist in a public key that should be saved in the same path where the PHAR file is located. The signature is calculated in the whole content of the phar file and avoid to tamper the content.

Note that if you rename the PHAR file like for example from something like "laralog_4.0_unsigned.phar" to "laralog.phar" the public key should be also be renamed from "laralog_4.0_unsighed.phar.pubkey" to "laralog.phar.pubkey".

An unsigned version of the PHAR file is also distributed however their usage is discouraged.

12. Supervisord configuration

Example of a Supervisord configuration file:

[program:laralog-worker]
process_name=%(program_name)s_%(process_num)02d
command=/home/admin/laralog/laralog.phar https://http-intake.logs.datadoghq.eu/v1/input/[KEY] --sender=datadog --hostname=production01 --index=laravel --smart --async --from-timezone=Europe/Copenhagen --input=/home/admin/laravel/storage/logs/laravel.log
autostart=true
autorestart=true
user=admin
numprocs=1
redirect_stderr=true
stdout_logfile=/home/admin/laralog/log/laralog.log

Backers