Fix that non index field are shown with keyword in table (elastic#254)

README.md

@@ -207,7 +207,7 @@ The event fields are used for context information about the log or metric event
 | <a name="event.module"></a>event.module | Name of the module this data is coming from.<br/>This information is coming from the modules used in Beats or Logstash. | core | keyword | `mysql` |
 | <a name="event.dataset"></a>event.dataset | Name of the dataset.<br/>The concept of a `dataset` (fileset / metricset) is used in Beats as a subset of modules. It contains the information which is currently stored in metricset.name and metricset.module or fileset.name. | core | keyword | `stats` |
 | <a name="event.severity"></a>event.severity | Severity describes the severity of the event. What the different severity values mean can very different between use cases. It's up to the implementer to make sure severities are consistent across events. | core | long | `7` |
-| <a name="event.original"></a>event.original | Raw text message of entire event. Used to demonstrate log integrity.<br/>This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from `_source`. | core | keyword | `Sep 19 08:26:10 host CEF:0&#124;Security&#124; threatmanager&#124;1.0&#124;100&#124; worm successfully stopped&#124;10&#124;src=10.0.0.1 dst=2.1.2.2spt=1232` |
+| <a name="event.original"></a>event.original | Raw text message of entire event. Used to demonstrate log integrity.<br/>This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from `_source`. | core | (not indexed) | `Sep 19 08:26:10 host CEF:0&#124;Security&#124; threatmanager&#124;1.0&#124;100&#124; worm successfully stopped&#124;10&#124;src=10.0.0.1 dst=2.1.2.2spt=1232` |
 | <a name="event.hash"></a>event.hash | Hash (perhaps logstash fingerprint) of raw field to be able to demonstrate log integrity. | extended | keyword | `123456789012345678901234567890ABCD` |
 | <a name="event.duration"></a>event.duration | Duration of the event in nanoseconds.<br/>If event.start and event.end are known this value should be the difference between the end and start time. | core | long |  |
 | <a name="event.created"></a>event.created | event.created contains the date when the event was created.<br/>This timestamp is distinct from @timestamp in that @timestamp contains the processed timestamp. For logs these two timestamps can be different as the timestamp in the log line and when the event is read for example by Filebeat are not identical. `@timestamp` must contain the timestamp extracted from the log line, event.created when the log line is read. The same could apply to package capturing where @timestamp contains the timestamp extracted from the network package and event.created when the event was created.<br/>In case the two timestamps are identical, @timestamp should be used. | core | date |  |
@@ -310,7 +310,7 @@ Fields which are specific to log events.
 | Field  | Description  | Level  | Type  | Example  |
 |---|---|---|---|---|
 | <a name="log.level"></a>log.level | Log level of the log event.<br/>Some examples are `WARN`, `ERR`, `INFO`. | core | keyword | `ERR` |
-| <a name="log.original"></a>log.original | This is the original log message and contains the full log message before splitting it up in multiple parts.<br/>In contrast to the `message` field which can contain an extracted part of the log message, this field contains the original, full log message. It can have already some modifications applied like encoding or new lines removed to clean up the log message.<br/>This field is not indexed and doc_values are disabled so it can't be queried but the value can be retrieved from `_source`. | core | keyword | `Sep 19 08:26:10 localhost My log` |
+| <a name="log.original"></a>log.original | This is the original log message and contains the full log message before splitting it up in multiple parts.<br/>In contrast to the `message` field which can contain an extracted part of the log message, this field contains the original, full log message. It can have already some modifications applied like encoding or new lines removed to clean up the log message.<br/>This field is not indexed and doc_values are disabled so it can't be queried but the value can be retrieved from `_source`. | core | (not indexed) | `Sep 19 08:26:10 localhost My log` |
 
 
 ## <a name="network"></a> Network fields
@@ -498,7 +498,7 @@ The user_agent fields normally come from a browser request. They often show up i
 
 | Field  | Description  | Level  | Type  | Example  |
 |---|---|---|---|---|
-| <a name="user_agent.original"></a>user_agent.original | Unparsed version of the user_agent. | extended | keyword | `Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1` |
+| <a name="user_agent.original"></a>user_agent.original | Unparsed version of the user_agent. | extended | (not indexed) | `Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1` |
 | <a name="user_agent.name"></a>user_agent.name | Name of the user agent. | extended | keyword | `Safari` |
 | <a name="user_agent.version"></a>user_agent.version | Version of the user agent. | extended | keyword | `12.0` |
 | <a name="user_agent.device.name"></a>user_agent.device.name | Name of the device. | extended | keyword | `iPhone` |

schema.csv

@@ -47,7 +47,7 @@ event.hash,keyword,extended,123456789012345678901234567890ABCD
 event.id,keyword,core,8a4f500d
 event.kind,keyword,extended,state
 event.module,keyword,core,mysql
-event.original,keyword,core,Sep 19 08:26:10 host CEF:0|Security| threatmanager|1.0|100| worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2spt=1232
+event.original,(not indexed),core,Sep 19 08:26:10 host CEF:0|Security| threatmanager|1.0|100| worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2spt=1232
 event.outcome,keyword,extended,success
 event.risk_score,float,core,
 event.risk_score_norm,float,extended,
@@ -91,7 +91,7 @@ http.response.body,keyword,extended,Hello world
 http.response.status_code,long,extended,404
 http.version,keyword,extended,1.1
 log.level,keyword,core,ERR
-log.original,keyword,core,Sep 19 08:26:10 localhost My log
+log.original,(not indexed),core,Sep 19 08:26:10 localhost My log
 network.application,keyword,extended,AIM
 network.bytes,long,core,368
 network.community_id,keyword,extended,1:hO+sN4H+MG5MY/8hIrXPqc4ZQz0=
@@ -165,5 +165,5 @@ user.id,keyword,core,
 user.name,keyword,core,albert
 user_agent.device.name,keyword,extended,iPhone
 user_agent.name,keyword,extended,Safari
-user_agent.original,keyword,extended,"Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1"
+user_agent.original,(not indexed),extended,"Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1"
 user_agent.version,keyword,extended,12.0

scripts/helper.py

@@ -83,6 +83,9 @@ def clean_string_field(field, key):
     else:
         field[key] = ""
 
+    if "index" in field and field["index"] == False:
+        field["type"] = "(not indexed)"
+
 
 def get_markdown_row(field, link, multi_field):
     """Creates a markdown table for the given fields

use-cases/filebeat-apache-access.md

@@ -21,7 +21,7 @@ ECS fields used in Filebeat for the apache module.
 | <a name="http.response.body_sent.bytes"></a>*http.response.body_sent.bytes* | *Http response body bytes sent, currently apache.access.body_sent.bytes* | (use case) | long | `117` |
 | <a name="http.referer"></a>*http.referer* | *Http referrer code, currently apache.access.referrer<br/>NOTE: In the RFC its misspell as referer and has become accepted standard* | (use case) | keyword | `http://elastic.co/` |
 | <a name="user_agent.&ast;"></a>*user_agent.&ast;* | *User agent fields as in schema. Currently under apache.access.user_agent.*<br/>* |  |  |  |
-| [user_agent.original](../README.md#user_agent.original)  | Original user agent. Currently apache.access.agent | extended | keyword | `http://elastic.co/` |
+| [user_agent.original](../README.md#user_agent.original)  | Original user agent. Currently apache.access.agent | extended | (not indexed) | `http://elastic.co/` |
 | <a name="geoip.&ast;"></a>*geoip.&ast;* | *User agent fields as in schema. Currently under apache.access.geoip.*<br/>These are extracted from source.ip<br/>Should they be under source.geoip?<br/>* |  |  |  |
 | <a name="geoip...."></a>*geoip....* | *All geoip fields.* | (use case) | keyword |  |
 

use-cases/web-logs.md

@@ -17,7 +17,7 @@ Using the fields as represented here is not expected to conflict with ECS, but m
 | [http.response.body](../README.md#http.response.body)  | The full http response body. | extended | keyword | `Hello world` |
 | [http.version](../README.md#http.version)  | Http version. | extended | keyword | `1.1` |
 | <a name="user_agent.&ast;"></a>*user_agent.&ast;* | *The user_agent fields normally come from a browser request. They often show up in web service logs coming from the parsed user agent string.<br/>* |  |  |  |
-| [user_agent.original](../README.md#user_agent.original)  | Unparsed version of the user_agent. | extended | keyword | `Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1` |
+| [user_agent.original](../README.md#user_agent.original)  | Unparsed version of the user_agent. | extended | (not indexed) | `Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1` |
 | <a name="user_agent.device"></a>*user_agent.device* | *Name of the physical device.* | (use case) | keyword |  |
 | [user_agent.version](../README.md#user_agent.version)  | Version of the physical device. | extended | keyword | `12.0` |
 | <a name="user_agent.major"></a>*user_agent.major* | *Major version of the user agent.* | (use case) | long |  |