From 5bf988805d9fdeda0c47ab40c21d7c7fe69c7310 Mon Sep 17 00:00:00 2001 From: Mathieu Martin Date: Fri, 22 Feb 2019 16:07:58 -0500 Subject: [PATCH] Clarify the difference between @timestamp and event.created (#329) --- CHANGELOG.md | 1 + README.md | 4 ++-- code/go/ecs/base.go | 6 ++++-- code/go/ecs/event.go | 18 +++++++++--------- fields.yml | 26 +++++++++++++++----------- schema.json | 4 ++-- schemas/base.yml | 7 +++++-- schemas/event.yml | 21 +++++++++++---------- 8 files changed, 49 insertions(+), 38 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f469b6a3f7..bf51191257 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,7 @@ All notable changes to this project will be documented in this file based on the ### Improvements * Clarified the definition of the host fields #325 * Specify the `object_type` for field `labels`. #331 +* Clarified the difference between `@timestamp` and `event.created`. #329 * Loosen up definition of `geo` field set. Not necessarily geo-ip based, since `geo.name`. #333 ### Deprecated diff --git a/README.md b/README.md index 74da32c6be..ca4900fe50 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ The `base` field set contains all fields which are on the top level. These field | Field | Description | Level | Type | Example | |---|---|---|---|---| -| @timestamp | Date/time when the event originated.
For log events this is the date/time when the event was generated, and not when it was read.
Required field for all events. | core | date | `2016-05-23T08:05:34.853Z` | +| @timestamp | Date/time when the event originated.
This is the date/time extracted from the event, typically representing when the event was generated by the source.
If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.
Required field for all events. | core | date | `2016-05-23T08:05:34.853Z` | | tags | List of keywords used to tag each event. | core | keyword | `["production", "env2"]` | | labels | Custom key/value pairs.
Can be used to add meta information to events. Should not contain nested objects. All values are stored as keyword.
Example: `docker` and `k8s` labels. | core | object | `{'application': 'foo-bar', 'env': 'production'}` | | message | For log events the message field contains the log message, optimized for viewing in a log viewer.
For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event.
If multiple messages exist, they can be combined into one message. | core | text | `Hello World` | @@ -227,7 +227,7 @@ A log is defined as an event containing details of something that happened. Log | event.hash | Hash (perhaps logstash fingerprint) of raw field to be able to demonstrate log integrity. | extended | keyword | `123456789012345678901234567890ABCD` | | event.duration | Duration of the event in nanoseconds.
If event.start and event.end are known this value should be the difference between the end and start time. | core | long | | | event.timezone | This field should be populated when the event's timestamp does not include timezone information already (e.g. default Syslog timestamps). It's optional otherwise.
Acceptable timezone formats are: a canonical ID (e.g. "Europe/Amsterdam"), abbreviated (e.g. "EST") or an HH:mm differential (e.g. "-05:00"). | extended | keyword | | -| event.created | event.created contains the date when the event was created.
This timestamp is distinct from @timestamp in that @timestamp contains the processed timestamp. For logs these two timestamps can be different as the timestamp in the log line and when the event is read for example by Filebeat are not identical. `@timestamp` must contain the timestamp extracted from the log line, event.created when the log line is read. The same could apply to package capturing where @timestamp contains the timestamp extracted from the network package and event.created when the event was created.
In case the two timestamps are identical, @timestamp should be used. | core | date | | +| event.created | event.created contains the date/time when the event was first read by an agent, or by your pipeline.
This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event.
In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source.
In case the two timestamps are identical, @timestamp should be used. | core | date | | | event.start | event.start contains the date when the event started or when the activity was first observed. | extended | date | | | event.end | event.end contains the date when the event ended or when the activity was last observed. | extended | date | | | event.risk_score | Risk score or priority of the event (e.g. security solutions). Use your system's original value here. | core | float | | diff --git a/code/go/ecs/base.go b/code/go/ecs/base.go index 0937a7ed0b..8960635b75 100644 --- a/code/go/ecs/base.go +++ b/code/go/ecs/base.go @@ -27,8 +27,10 @@ import ( // fields are common across all types of events. type Base struct { // Date/time when the event originated. - // For log events this is the date/time when the event was generated, and - // not when it was read. + // This is the date/time extracted from the event, typically representing + // when the event was generated by the source. + // If the event source has no original timestamp, this value is typically + // populated by the first time the event was received by the pipeline. // Required field for all events. Timestamp time.Time `ecs:"@timestamp"` diff --git a/code/go/ecs/event.go b/code/go/ecs/event.go index f8f4eac7ed..56fcebe056 100644 --- a/code/go/ecs/event.go +++ b/code/go/ecs/event.go @@ -108,15 +108,15 @@ type Event struct { // (e.g. "-05:00"). Timezone string `ecs:"timezone"` - // event.created contains the date when the event was created. - // This timestamp is distinct from @timestamp in that @timestamp contains - // the processed timestamp. For logs these two timestamps can be different - // as the timestamp in the log line and when the event is read for example - // by Filebeat are not identical. `@timestamp` must contain the timestamp - // extracted from the log line, event.created when the log line is read. - // The same could apply to package capturing where @timestamp contains the - // timestamp extracted from the network package and event.created when the - // event was created. + // event.created contains the date/time when the event was first read by an + // agent, or by your pipeline. + // This field is distinct from @timestamp in that @timestamp typically + // contain the time extracted from the original event. + // In most situations, these two timestamps will be slightly different. The + // difference can be used to calculate the delay between your source + // generating an event, and the time when your agent first processed it. + // This can be used to monitor your agent's or pipeline's ability to keep + // up with your event source. // In case the two timestamps are identical, @timestamp should be used. Created time.Time `ecs:"created"` diff --git a/fields.yml b/fields.yml index 4503ded5bd..12a6428446 100644 --- a/fields.yml +++ b/fields.yml @@ -94,8 +94,11 @@ description: > Date/time when the event originated. - For log events this is the date/time when the event was generated, and - not when it was read. + This is the date/time extracted from the event, typically representing + when the event was generated by the source. + + If the event source has no original timestamp, this value is typically + populated by the first time the event was received by the pipeline. Required field for all events. @@ -607,16 +610,17 @@ type: date short: Time when the event was first read by an agent or by your pipeline. description: > - event.created contains the date when the event was created. + event.created contains the date/time when the event was first read by an + agent, or by your pipeline. + + This field is distinct from @timestamp in that @timestamp typically contain + the time extracted from the original event. - This timestamp is distinct from @timestamp in that @timestamp contains - the processed timestamp. For logs these two timestamps can be different - as the timestamp in the log line and when the event is read for example - by Filebeat are not identical. `@timestamp` must contain the timestamp - extracted from the log line, event.created when the log line is read. - The same could apply to package capturing where @timestamp contains the - timestamp extracted from the network package and event.created when the - event was created. + In most situations, these two timestamps will be slightly different. + The difference can be used to calculate the delay between your source + generating an event, and the time when your agent first processed it. + This can be used to monitor your agent's or pipeline's ability to + keep up with your event source. In case the two timestamps are identical, @timestamp should be used. diff --git a/schema.json b/schema.json index b278b5dbf3..00d885c8fc 100644 --- a/schema.json +++ b/schema.json @@ -62,7 +62,7 @@ "description": "The `base` field set contains all fields which are on the top level. These fields are common across all types of events.\n", "fields": { "@timestamp": { - "description": "Date/time when the event originated.\nFor log events this is the date/time when the event was generated, and not when it was read.\nRequired field for all events.", + "description": "Date/time when the event originated.\nThis is the date/time extracted from the event, typically representing when the event was generated by the source.\nIf the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.\nRequired field for all events.", "example": "2016-05-23T08:05:34.853Z", "footnote": "", "group": 1, @@ -495,7 +495,7 @@ "type": "keyword" }, "event.created": { - "description": "event.created contains the date when the event was created.\nThis timestamp is distinct from @timestamp in that @timestamp contains the processed timestamp. For logs these two timestamps can be different as the timestamp in the log line and when the event is read for example by Filebeat are not identical. `@timestamp` must contain the timestamp extracted from the log line, event.created when the log line is read. The same could apply to package capturing where @timestamp contains the timestamp extracted from the network package and event.created when the event was created.\nIn case the two timestamps are identical, @timestamp should be used.", + "description": "event.created contains the date/time when the event was first read by an agent, or by your pipeline.\nThis field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event.\nIn most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source.\nIn case the two timestamps are identical, @timestamp should be used.", "example": "", "footnote": "", "group": 2, diff --git a/schemas/base.yml b/schemas/base.yml index c14f33b034..cdea73dc77 100644 --- a/schemas/base.yml +++ b/schemas/base.yml @@ -17,8 +17,11 @@ description: > Date/time when the event originated. - For log events this is the date/time when the event was generated, and - not when it was read. + This is the date/time extracted from the event, typically representing + when the event was generated by the source. + + If the event source has no original timestamp, this value is typically + populated by the first time the event was received by the pipeline. Required field for all events. diff --git a/schemas/event.yml b/schemas/event.yml index ce15bb4baa..42e113017f 100644 --- a/schemas/event.yml +++ b/schemas/event.yml @@ -167,16 +167,17 @@ type: date short: Time when the event was first read by an agent or by your pipeline. description: > - event.created contains the date when the event was created. - - This timestamp is distinct from @timestamp in that @timestamp contains - the processed timestamp. For logs these two timestamps can be different - as the timestamp in the log line and when the event is read for example - by Filebeat are not identical. `@timestamp` must contain the timestamp - extracted from the log line, event.created when the log line is read. - The same could apply to package capturing where @timestamp contains the - timestamp extracted from the network package and event.created when the - event was created. + event.created contains the date/time when the event was first read by an + agent, or by your pipeline. + + This field is distinct from @timestamp in that @timestamp typically contain + the time extracted from the original event. + + In most situations, these two timestamps will be slightly different. + The difference can be used to calculate the delay between your source + generating an event, and the time when your agent first processed it. + This can be used to monitor your agent's or pipeline's ability to + keep up with your event source. In case the two timestamps are identical, @timestamp should be used.