Clarify the difference between @timestamp and event.created (elastic#329

)

CHANGELOG.md

@@ -24,6 +24,7 @@ All notable changes to this project will be documented in this file based on the
 ### Improvements
 * Clarified the definition of the host fields #325
 * Specify the `object_type` for field `labels`. #331
+* Clarified the difference between `@timestamp` and `event.created`. #329
 * Loosen up definition of `geo` field set. Not necessarily geo-ip based, since `geo.name`. #333
 
 ### Deprecated

README.md

@@ -82,7 +82,7 @@ The `base` field set contains all fields which are on the top level. These field
 
 | Field  | Description  | Level  | Type  | Example  |
 |---|---|---|---|---|
-| <a name="@timestamp"></a>@timestamp | Date/time when the event originated.<br/>For log events this is the date/time when the event was generated, and not when it was read.<br/>Required field for all events. | core | date | `2016-05-23T08:05:34.853Z` |
+| <a name="@timestamp"></a>@timestamp | Date/time when the event originated.<br/>This is the date/time extracted from the event, typically representing when the event was generated by the source.<br/>If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.<br/>Required field for all events. | core | date | `2016-05-23T08:05:34.853Z` |
 | <a name="tags"></a>tags | List of keywords used to tag each event. | core | keyword | `["production", "env2"]` |
 | <a name="labels"></a>labels | Custom key/value pairs.<br/>Can be used to add meta information to events. Should not contain nested objects. All values are stored as keyword.<br/>Example: `docker` and `k8s` labels. | core | object | `{'application': 'foo-bar', 'env': 'production'}` |
 | <a name="message"></a>message | For log events the message field contains the log message, optimized for viewing in a log viewer.<br/>For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event.<br/>If multiple messages exist, they can be combined into one message. | core | text | `Hello World` |
@@ -227,7 +227,7 @@ A log is defined as an event containing details of something that happened. Log
 | <a name="event.hash"></a>event.hash | Hash (perhaps logstash fingerprint) of raw field to be able to demonstrate log integrity. | extended | keyword | `123456789012345678901234567890ABCD` |
 | <a name="event.duration"></a>event.duration | Duration of the event in nanoseconds.<br/>If event.start and event.end are known this value should be the difference between the end and start time. | core | long |  |
 | <a name="event.timezone"></a>event.timezone | This field should be populated when the event's timestamp does not include timezone information already (e.g. default Syslog timestamps). It's optional otherwise.<br/>Acceptable timezone formats are: a canonical ID (e.g. "Europe/Amsterdam"), abbreviated (e.g. "EST") or an HH:mm differential (e.g. "-05:00"). | extended | keyword |  |
-| <a name="event.created"></a>event.created | event.created contains the date when the event was created.<br/>This timestamp is distinct from @timestamp in that @timestamp contains the processed timestamp. For logs these two timestamps can be different as the timestamp in the log line and when the event is read for example by Filebeat are not identical. `@timestamp` must contain the timestamp extracted from the log line, event.created when the log line is read. The same could apply to package capturing where @timestamp contains the timestamp extracted from the network package and event.created when the event was created.<br/>In case the two timestamps are identical, @timestamp should be used. | core | date |  |
+| <a name="event.created"></a>event.created | event.created contains the date/time when the event was first read by an agent, or by your pipeline.<br/>This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event.<br/>In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source.<br/>In case the two timestamps are identical, @timestamp should be used. | core | date |  |
 | <a name="event.start"></a>event.start | event.start contains the date when the event started or when the activity was first observed. | extended | date |  |
 | <a name="event.end"></a>event.end | event.end contains the date when the event ended or when the activity was last observed. | extended | date |  |
 | <a name="event.risk_score"></a>event.risk_score | Risk score or priority of the event (e.g. security solutions). Use your system's original value here. | core | float |  |

fields.yml

@@ -94,8 +94,11 @@
           description: >
             Date/time when the event originated.
     
-            For log events this is the date/time when the event was generated, and
-            not when it was read.
+            This is the date/time extracted from the event, typically representing
+            when the event was generated by the source.
+
+            If the event source has no original timestamp, this value is typically
+            populated by the first time the event was received by the pipeline.
 
             Required field for all events.
 
@@ -607,16 +610,17 @@
           type: date
           short: Time when the event was first read by an agent or by your pipeline.
           description: >
-            event.created contains the date when the event was created.
+            event.created contains the date/time when the event was first read by an
+            agent, or by your pipeline.
+    
+            This field is distinct from @timestamp in that @timestamp typically contain
+            the time extracted from the original event.
 
-            This timestamp is distinct from @timestamp in that @timestamp contains
-            the processed timestamp. For logs these two timestamps can be different
-            as the timestamp in the log line and when the event is read for example
-            by Filebeat are not identical. `@timestamp` must contain the timestamp
-            extracted from the log line, event.created when the log line is read.
-            The same could apply to package capturing where @timestamp contains the
-            timestamp extracted from the network package and event.created when the
-            event was created.
+            In most situations, these two timestamps will be slightly different.
+            The difference can be used to calculate the delay between your source
+            generating an event, and the time when your agent first processed it.
+            This can be used to monitor your agent's or pipeline's ability to
+            keep up with your event source.
 
             In case the two timestamps are identical, @timestamp should be used.
 

schema.json

@@ -62,7 +62,7 @@
     "description": "The `base` field set contains all fields which are on the top level. These fields are common across all types of events.\n", 
     "fields": {
       "@timestamp": {
-        "description": "Date/time when the event originated.\nFor log events this is the date/time when the event was generated, and not when it was read.\nRequired field for all events.", 
+        "description": "Date/time when the event originated.\nThis is the date/time extracted from the event, typically representing when the event was generated by the source.\nIf the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.\nRequired field for all events.", 
         "example": "2016-05-23T08:05:34.853Z", 
         "footnote": "", 
         "group": 1, 
@@ -495,7 +495,7 @@
         "type": "keyword"
       }, 
       "event.created": {
-        "description": "event.created contains the date when the event was created.\nThis timestamp is distinct from @timestamp in that @timestamp contains the processed timestamp. For logs these two timestamps can be different as the timestamp in the log line and when the event is read for example by Filebeat are not identical. `@timestamp` must contain the timestamp extracted from the log line, event.created when the log line is read. The same could apply to package capturing where @timestamp contains the timestamp extracted from the network package and event.created when the event was created.\nIn case the two timestamps are identical, @timestamp should be used.", 
+        "description": "event.created contains the date/time when the event was first read by an agent, or by your pipeline.\nThis field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event.\nIn most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source.\nIn case the two timestamps are identical, @timestamp should be used.", 
         "example": "", 
         "footnote": "", 
         "group": 2, 

schemas/base.yml

@@ -17,8 +17,11 @@
       description: >
         Date/time when the event originated.
 
-        For log events this is the date/time when the event was generated, and
-        not when it was read.
+        This is the date/time extracted from the event, typically representing
+        when the event was generated by the source.
+
+        If the event source has no original timestamp, this value is typically
+        populated by the first time the event was received by the pipeline.
 
         Required field for all events.
 

schemas/event.yml

@@ -167,16 +167,17 @@
       type: date
       short: Time when the event was first read by an agent or by your pipeline.
       description: >
-        event.created contains the date when the event was created.
-
-        This timestamp is distinct from @timestamp in that @timestamp contains
-        the processed timestamp. For logs these two timestamps can be different
-        as the timestamp in the log line and when the event is read for example
-        by Filebeat are not identical. `@timestamp` must contain the timestamp
-        extracted from the log line, event.created when the log line is read.
-        The same could apply to package capturing where @timestamp contains the
-        timestamp extracted from the network package and event.created when the
-        event was created.
+        event.created contains the date/time when the event was first read by an
+        agent, or by your pipeline.
+
+        This field is distinct from @timestamp in that @timestamp typically contain
+        the time extracted from the original event.
+
+        In most situations, these two timestamps will be slightly different.
+        The difference can be used to calculate the delay between your source
+        generating an event, and the time when your agent first processed it.
+        This can be used to monitor your agent's or pipeline's ability to
+        keep up with your event source.
 
         In case the two timestamps are identical, @timestamp should be used.