Skip to content

Home

Jump to bottom
doksu edited this page Apr 11, 2017 · 9 revisions

TA-jsontools

JSON Tools Technology Add-On for Splunk

Splunk can export events in JSON via the web interface and when queried via the REST api can return JSON output. It can also parse JSON at index/search-time, but it can't create json at search-time. This app provides a 'mkjson' command that can create a JSON field from a given list or all fields in an event. For examples, please see the "Usage" section below.

OVERVIEW

  • Release notes
  • Support and resources

INSTALLATION AND CONFIGURATION

  • Requirements
  • Installation
  • Configuration

USAGE

  • Multi-value field preservation when exporting to csv
  • Arbitrary fields in KV Store collections and other external lookups
  • Produce _raw from fields for summary indexing
  • Search across all fields, including those not in _raw
  • Integration with third-party systems receiving search output that must be in JSON

OVERVIEW

Release notes

About this release

Version 0.1.* of TA-jsontools is compatible with:

Splunk Enterprise versions 6.3+
Platforms Platform independent
Lookup file changes None
Fixed issues

Version 0.1.2 of TA-jsontools fixes the following issues:

  • Empty fields are no longer output in JSON
Known issues
  • None

Support and resources

Please post questions at https://answers.splunk.com, however this app is provided as is with no warranty, implied or otherwise; please see the LICENSE document for more information. Feedback about possible improvements and good news stories of how this app has helped your organisation are most welcome.

INSTALLATION AND CONFIGURATION

Requirements

Hardware requirements

  • None

Software requirements

To function properly, TA-jsontools requires the following software:

  • Splunk Enterprise 6.3+

Installation

Simply install this app on your search head/s and restart Splunk.

Configuration

No configuration is required.

Usage

If no output field is provided, _raw is used. The mkjson command will include all except hidden fields by default (those that start with an underscore) in the JSON, unless includehidden=true or a list of fields is provided.

... | mkjson [outputfield=<fieldname>] [includehidden=<true|false>] [<fields>]

To simply convert events to JSON:

... | mkjson

Keep in mind that _time is technically a hidden field, so you may like to ... | eval time=_time or ... | convert ctime(_time) AS time before the mkjson command if you don't want to include any other hidden fields.

Below are some example use cases for the 'mkjson' command.

Multi-value field preservation when exporting to csv

Fields with multiple values can be easily preserved when exporting to csv. For example, we can convert a single field to JSON:

... | mkjson outputfield=src src | outputlookup mylookup

This can then be reconstituted with spath:

... | lookup mylookup ... OUTPUT src | spath input=src

Arbitrary fields in KV Store collections and other external lookups

External lookups, such as KV Store collections, cannot take arbitrary fields. This can be problematic in situations where we can't know in advance a full list of fields events might contain that we wish to preserve. In this instance we can convert events to JSON and put that in the lookup:

transforms.conf

[mylookup]
external_type = kvstore
collection = mycollection
fields_list = _key,json

In search:

... | mkjson outputfield=json | outputlookup append=t mylookup

Produce _raw from fields for summary indexing

There are often situations where we don't have an _raw field. When using the collect command to summary index information, it would be desirable to have an _raw field and we can produce it with 'mkjson'.

... | mkjson | collect index=summary sourcetype=my:json

To have the fields extracted, then add a prop:

[my:json]
KV_MODE = json

Search across all fields, including those not in _raw

Sometimes it's desirable to use the search command across all fields, not just _raw. This is especially true in situations where no _raw exists (such as lookups) or field enrichment not present in _raw. Using mkjson is one way of doing this. For example, if we want to search for a string across all the fields in Enterprise Security notables events (most of which are provided by lookups and therefore not in _raw):

`notable` | search NOT `suppression` | mkson | search 10.0.0.1

This is particularly useful in the Incident Review dashboard, where the addition of the mkjson command immediately after the suppression (as seen above) allows us to perform free-form searches across all fields. If you wish to make this change, it's called 'id-main-search' in SA-ThreatIntelligence/appserver/static/js/pages/incident_review.js - N.B. This change will be overridden when ES is upgraded and should be performed at your own risk.

Integration with third-party systems receiving search output that must be in JSON

In some cases we may want to use a vendor TA's search command/alert action/adaptive response to send search results to a third-party system. If that system requires one or more fields to be in JSON format, the mkjson command can be used. For example:

... | mkjson outputfield=ioc src dest protocol | vendor_threatintel ioc

Clone this wiki locally