Skip to content

Home

Jump to bottom
doksu edited this page Mar 13, 2017 · 11 revisions

TA-setops

Set Operations Technology Add-On for Splunk

Discrete mathematics for Splunk is here. Winner of the 2016 Developer Revolution Award, this app provides the ability to evaluate relations, apply set operations (union, intersection, difference, etc.) and a new version of the 'distinctfields' command (https://www.youtube.com/watch?v=Z6w-VG2CpP0) that has significantly better performance and now support for a 'by' field.

N.B. This app replaces the proof-of-concept SetOps app (https://splunkbase.splunk.com/app/3315)

OVERVIEW

  • Release notes
  • Support and resources

INSTALLATION AND CONFIGURATION

  • Requirements
  • Installation
  • Configuration

USAGE

  • setop command
  • distinctfields command

OVERVIEW

Release notes

About this release

Version 1.0.0 of TA-setops is compatible with:

Splunk Enterprise versions 6.3+
Platforms Platform independent
Lookup file changes None
Fixed issues

Version 1.0.0 of TA-setops fixes the following issues:

  • None
Known issues
  • None

Support and resources

Please post questions at https://answers.splunk.com, however this app is provided as is with no warranty, implied or otherwise; please see the LICENSE document for more information. Feedback about possible improvements and good news stories of how this app has helped your organisation are most welcome.

INSTALLATION AND CONFIGURATION

Requirements

Hardware requirements

  • None

Software requirements

To function properly, TA-setops requires the following software:

  • Splunk Enterprise 6.3+

Installation

Simply install this app on your search head/s and restart Splunk.

Configuration

No configuration is required.

Usage

This app uses python's set operators (https://docs.python.org/2/library/sets.html#set-objects). Provided below are a range of examples searches demonstrating their operation. Please copy and paste them to see the output of the commands.

setop command

cardinality
 | stats count | eval a=split("aaa aaa bbb ccc", " ") | fields - count | setop op=cardinality a
operations

unions:

 | stats count | eval a=split("aaa bbb ccc", " ") | eval b=split("bbb aaa ddd", " ") | fields - count | setop op=union a b

intersection:

| stats count | eval a=split("aaa bbb ccc", " ") | eval b=split("bbb aaa ddd", " ") | fields - count | setop op=intersection a b

difference:

| stats count | eval a=split("aaa bbb ccc", " ") | eval b=split("bbb aaa ddd", " ") | fields - count | setop op=difference a b

symmetric difference:

| stats count | eval a=split("aaa bbb ccc", " ") | eval b=split("bbb aaa ddd", " ") | fields - count | setop op=symmetric_difference a b
relations:

equal:

| stats count | eval a=split("aaa bbb ccc", " ") | eval b=split("bbb aaa ccc", " ") | fields - count | setop op=relation a b

partially disjoint:

| stats count | eval a=split("aaa bbb ccc", " ") | eval b=split("bbb aaa ddd", " ") | fields - count | setop op=relation a b

superset:

| stats count | eval a=split("aaa bbb ccc ddd", " ") | eval b=split("bbb aaa ccc", " ") | fields - count | setop op=relation a b

subset:

| stats count | eval a=split("aaa bbb", " ") | eval b=split("bbb aaa ccc", " ") | fields - count | setop op=relation a b

fully disjoint:

| stats count | eval a=split("aaa bbb", " ") | eval b=split("ccc ddd", " ") | fields - count | setop op=relation a b

distinctfields command

It's strongly recommended to put a 'table' or 'fields' command before 'distinctfields' to remove extraneous fields (incl. _raw if not required) to improve performance. This app provides a distinctfields_example lookup which can be used for testing.

Without a 'by' field:

| inputlookup distinctfields_example |  distinctfields field1 field2 field3

With a 'by' field:

| inputlookup distinctfields_example |  distinctfields by=field1 field2 field3
Clone this wiki locally