Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Threat Intel Analytics #1098

Merged
merged 59 commits into from
Jul 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
28591eb
refactored classes (#1037)
jowg-amazon May 21, 2024
4e43caa
ioc match model (#1038)
eirsep May 22, 2024
dbebcb5
Threat Intel Feed Config Model (#1028)
jowg-amazon May 22, 2024
1669953
IOC data model and DTO. (#1029)
AWSHurneyt May 28, 2024
3a04f02
Create TIF Source Config API (#1046)
jowg-amazon May 29, 2024
864815d
Get TIF Source Config API (#1049)
jowg-amazon May 30, 2024
ce15825
renamed source config dao to service and service to management servic…
jowg-amazon Jun 4, 2024
63dd56c
index threat intel monitor rest api (#1057)
eirsep Jun 6, 2024
dcc0e94
Search and delete threat intel monitor api (#1058)
eirsep Jun 6, 2024
b60ef59
TIF Job Scheduler Initial Implementation (#1054)
jowg-amazon Jun 11, 2024
17f7074
Draft of IOC service (#1048)
AWSHurneyt Jun 13, 2024
6766547
Implement Threat Intel Monitor Input and Triggers (#1073)
eirsep Jun 13, 2024
1997575
Search Feeds API and Store/Source Model (#1075)
jowg-amazon Jun 13, 2024
686d317
Delete threat intel source config API (#1066)
jowg-amazon Jun 13, 2024
04687cc
Assets for IOC fetch and index service. (#1077)
AWSHurneyt Jun 17, 2024
d71579c
Implemented ListIOCs API. (#1064)
AWSHurneyt Jun 19, 2024
ee4ae79
Refresh API and adds Update TIF Source Config logic (#1078)
jowg-amazon Jun 19, 2024
af217bc
FetchIOCService update IocStoreConfig with feedConfigId and IOC index…
AWSHurneyt Jun 19, 2024
8a6a389
fix (#1086)
jowg-amazon Jun 19, 2024
c28f84f
Fix ioc store config mappings (#1087)
jowg-amazon Jun 20, 2024
1927009
Change search config api request (#1089)
jowg-amazon Jun 20, 2024
d5ca5f9
Fix mapping and add job parser todo (#1090)
jowg-amazon Jun 21, 2024
04ef1bc
Added "feedName" to IOC model. (#1088)
AWSHurneyt Jun 21, 2024
ee4e52d
ListIOCs API support lists of feedIds, and types. (#1085)
AWSHurneyt Jun 21, 2024
cc6ff7a
Implemented API to test s3 connection. (#1091)
AWSHurneyt Jun 24, 2024
4d5df3a
fix user mappings (#1095)
jowg-amazon Jun 25, 2024
23a6b6d
Logic to delete old iocs and add ioc index rollover (#1094)
jowg-amazon Jun 25, 2024
ad80024
fix threat intel monitor request in indexing flow
eirsep Jun 26, 2024
e47a6ac
add search ioc findings api (#1093)
sbcd90 Jun 26, 2024
ef54c62
Threat intel monitor implementation (#1092)
eirsep Jun 27, 2024
cc7d504
List Ioc Api params change (#1100)
eirsep Jun 27, 2024
1c59b9f
Upload iocs through source config (#1097)
jowg-amazon Jun 27, 2024
c3440ee
adds threat intel alert status update API for Acknowledged and Comple…
eirsep Jun 27, 2024
ae53139
fix search monitor to return dto and not model (#1105)
eirsep Jun 27, 2024
a52b7c1
add guard rail for only one threat intel monitor to exist (#1106)
eirsep Jun 27, 2024
ad35b68
Fix API action names (#1107)
jowg-amazon Jun 28, 2024
c91fe4a
list iocs fix (#1109)
eirsep Jun 28, 2024
f3039d7
Fix validation for source types (#1108)
jowg-amazon Jun 28, 2024
817d319
catch ioc validation (#1110)
jowg-amazon Jun 28, 2024
ce6d9be
fix update threat intel monitor to avoid monitor exists check before …
eirsep Jun 28, 2024
1bbe292
fix merge conflicts
jowg-amazon Jun 28, 2024
b5adadd
Refactored calls to S3. Added support for consuming security analytic…
AWSHurneyt Jun 28, 2024
f6df1b7
add source config validation (#1115)
jowg-amazon Jun 29, 2024
24d8ee6
add feed_name to finding model's ioc detail
eirsep Jun 29, 2024
1523a1d
fix last seen seqeuence number calculation
eirsep Jul 1, 2024
88845f9
fix tif source config Model serde
eirsep Jul 1, 2024
a07e95c
fixed stix ioc serde
eirsep Jul 1, 2024
8d47a48
fix tif source config DTO serde
eirsep Jul 1, 2024
2464f64
apply spotless check
eirsep Jul 1, 2024
7a47ec4
Adjusted IOC type enums. (#1116)
AWSHurneyt Jul 1, 2024
4adc585
Ignore tests that can't run in github CI. (#1118)
AWSHurneyt Jul 1, 2024
3cb21c1
fix ioc ids filtering in get findings api
eirsep Jul 1, 2024
67337bd
fix enum serde in Stix2Ioc
eirsep Jul 1, 2024
d355bf8
fix mapping (#1119)
jowg-amazon Jul 1, 2024
adf1c33
remove ioc finding tests with sleep
eirsep Jul 1, 2024
a81dda0
fix ioc type enum (#1122)
eirsep Jul 2, 2024
50c50f5
Merge branch 'main' into feature/threat_intel
AWSHurneyt Jul 2, 2024
41c3502
fix feed name in finding serde (#1123)
eirsep Jul 2, 2024
5b12588
Fixed test suite. (#1128)
AWSHurneyt Jul 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 31 additions & 9 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,21 @@ buildscript {
opensearch_build = version_tokens[0] + '.0'
plugin_no_snapshot = opensearch_build
opensearch_no_snapshot = opensearch_version.replace("-SNAPSHOT","")
sa_commons_version = '1.0.0'
if (buildVersionQualifier) {
opensearch_build += "-${buildVersionQualifier}"
}
if (isSnapshot) {
opensearch_build += "-SNAPSHOT"

// TODO consider enabling snapshot options once SA commons is published to maven central
// sa_commons_version += "-SNAPSHOT"
}
common_utils_version = System.getProperty("common_utils.version", opensearch_build)
kotlin_version = '1.6.10'
kotlin_version = '1.8.21'

sa_commons_file_name = "security-analytics-commons-${sa_commons_version}.jar"
sa_commons_file_path = "${project.rootDir}/${sa_commons_file_name}"
}

repositories {
Expand Down Expand Up @@ -54,7 +61,7 @@ ext {
noticeFile = rootProject.file('NOTICE')
}

licenseHeaders.enabled = true
licenseHeaders.enabled = false
testingConventions.enabled = false
forbiddenApis.ignoreFailures = true

Expand All @@ -68,7 +75,7 @@ opensearchplugin {
name 'opensearch-security-analytics'
description 'OpenSearch Security Analytics plugin'
classname 'org.opensearch.securityanalytics.SecurityAnalyticsPlugin'
extendedPlugins = ['opensearch-job-scheduler']
extendedPlugins = ['opensearch-job-scheduler', 'opensearch-alerting']
}

javaRestTest {
Expand Down Expand Up @@ -147,27 +154,36 @@ configurations {
resolutionStrategy {
// for spotless transitive dependency CVE
force "org.eclipse.platform:org.eclipse.core.runtime:3.29.0"
force "com.google.guava:guava:32.1.2-jre"
force "com.google.guava:guava:32.1.3-jre"
}
}
}

dependencies {
javaRestTestImplementation project.sourceSets.main.runtimeClasspath
implementation group: 'org.apache.commons', name: 'commons-lang3', version: "${versions.commonslang}"
implementation "org.antlr:antlr4-runtime:4.10.1"
implementation "com.cronutils:cron-utils:9.1.6"
api "org.opensearch:common-utils:${common_utils_version}@jar"
api "org.opensearch.client:opensearch-rest-client:${opensearch_version}"
implementation "org.jetbrains.kotlin:kotlin-stdlib:${kotlin_version}"
compileOnly "org.antlr:antlr4-runtime:4.10.1"
compileOnly "com.cronutils:cron-utils:9.1.7"
compileOnly "org.opensearch:common-utils:${common_utils_version}@jar"
compileOnly "org.opensearch.client:opensearch-rest-client:${opensearch_version}"
compileOnly "org.jetbrains.kotlin:kotlin-stdlib:${kotlin_version}"
compileOnly "org.opensearch:opensearch-job-scheduler-spi:${opensearch_build}"
compileOnly "org.opensearch.alerting:alerting-spi:${opensearch_build}"
implementation "org.apache.commons:commons-csv:1.10.0"
compileOnly "com.google.guava:guava:32.1.3-jre"

// TODO uncomment once SA commons is published to maven central
// api "org.opensearch:security-analytics-commons:${sa_commons_version}@jar"

// TODO remove once SA commons is published to maven central
api files(sa_commons_file_path)

// Needed for integ tests
zipArchive group: 'org.opensearch.plugin', name:'alerting', version: "${opensearch_build}"
zipArchive group: 'org.opensearch.plugin', name:'opensearch-notifications-core', version: "${opensearch_build}"
zipArchive group: 'org.opensearch.plugin', name:'notifications', version: "${opensearch_build}"
zipArchive group: 'org.opensearch.plugin', name:'opensearch-job-scheduler', version: "${opensearch_build}"
testImplementation 'org.junit.jupiter:junit-jupiter-api:5.8.1'
}

// RPM & Debian build
Expand Down Expand Up @@ -360,6 +376,12 @@ afterEvaluate {
into opensearchplugin.name
}

// TODO remove once SA commons is published to maven central
from(project.rootDir) {
include sa_commons_file_name
into opensearchplugin.name
}

user 'root'
permissionGroup 'root'
fileMode 0644
Expand Down
Binary file added security-analytics-commons-1.0.0.jar
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
// Generated from Condition.g4 by ANTLR 4.10.1
// Generated from java-escape by ANTLR 4.11.1
package org.opensearch.securityanalytics.rules.condition;

import org.antlr.v4.runtime.ParserRuleContext;
Expand All @@ -14,6 +10,7 @@
* which can be extended to create a listener which only needs to handle a subset
* of the available methods.
*/
@SuppressWarnings("CheckReturnValue")
public class ConditionBaseListener implements ConditionListener {
/**
* {@inheritDoc}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
// Generated from Condition.g4 by ANTLR 4.10.1
// Generated from java-escape by ANTLR 4.11.1
package org.opensearch.securityanalytics.rules.condition;
import org.antlr.v4.runtime.tree.AbstractParseTreeVisitor;

Expand All @@ -14,6 +10,7 @@
* @param <T> The return type of the visit operation. Use {@link Void} for
* operations with no return type.
*/
@SuppressWarnings("CheckReturnValue")
public class ConditionBaseVisitor<T> extends AbstractParseTreeVisitor<T> implements ConditionVisitor<T> {
/**
* {@inheritDoc}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
// Generated from Condition.g4 by ANTLR 4.10.1
// Generated from java-escape by ANTLR 4.11.1
package org.opensearch.securityanalytics.rules.condition;
import org.antlr.v4.runtime.Lexer;
import org.antlr.v4.runtime.CharStream;
Expand All @@ -13,9 +9,9 @@
import org.antlr.v4.runtime.dfa.DFA;
import org.antlr.v4.runtime.misc.*;

@SuppressWarnings({"all", "warnings", "unchecked", "unused", "cast"})
@SuppressWarnings({"all", "warnings", "unchecked", "unused", "cast", "CheckReturnValue"})
public class ConditionLexer extends Lexer {
static { RuntimeMetaData.checkVersion("4.10.1", RuntimeMetaData.VERSION); }
static { RuntimeMetaData.checkVersion("4.11.1", RuntimeMetaData.VERSION); }

protected static final DFA[] _decisionToDFA;
protected static final PredictionContextCache _sharedContextCache =
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
// Generated from Condition.g4 by ANTLR 4.10.1
// Generated from java-escape by ANTLR 4.11.1
package org.opensearch.securityanalytics.rules.condition;
import org.antlr.v4.runtime.tree.ParseTreeListener;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
// Generated from Condition.g4 by ANTLR 4.10.1
// Generated from java-escape by ANTLR 4.11.1
package org.opensearch.securityanalytics.rules.condition;
import org.antlr.v4.runtime.atn.*;
import org.antlr.v4.runtime.dfa.DFA;
Expand All @@ -13,9 +9,9 @@
import java.util.Iterator;
import java.util.ArrayList;

@SuppressWarnings({"all", "warnings", "unchecked", "unused", "cast"})
@SuppressWarnings({"all", "warnings", "unchecked", "unused", "cast", "CheckReturnValue"})
public class ConditionParser extends Parser {
static { RuntimeMetaData.checkVersion("4.10.1", RuntimeMetaData.VERSION); }
static { RuntimeMetaData.checkVersion("4.11.1", RuntimeMetaData.VERSION); }

protected static final DFA[] _decisionToDFA;
protected static final PredictionContextCache _sharedContextCache =
Expand Down Expand Up @@ -78,7 +74,7 @@
}

@Override
public String getGrammarFileName() { return "Condition.g4"; }
public String getGrammarFileName() { return "java-escape"; }

Check warning on line 77 in src/main/generated/org/opensearch/securityanalytics/rules/condition/ConditionParser.java

View check run for this annotation

Codecov / codecov/patch

src/main/generated/org/opensearch/securityanalytics/rules/condition/ConditionParser.java#L77

Added line #L77 was not covered by tests

@Override
public String[] getRuleNames() { return ruleNames; }
Expand All @@ -94,6 +90,7 @@
_interp = new ParserATNSimulator(this,_ATN,_decisionToDFA,_sharedContextCache);
}

@SuppressWarnings("CheckReturnValue")
public static class StartContext extends ParserRuleContext {
public ExpressionContext expression() {
return getRuleContext(ExpressionContext.class,0);
Expand Down Expand Up @@ -138,6 +135,7 @@
return _localctx;
}

@SuppressWarnings("CheckReturnValue")
public static class ExpressionContext extends ParserRuleContext {
public ExpressionContext(ParserRuleContext parent, int invokingState) {
super(parent, invokingState);
Expand All @@ -149,6 +147,7 @@
super.copyFrom(ctx);
}
}
@SuppressWarnings("CheckReturnValue")
public static class OrExpressionContext extends ExpressionContext {
public ExpressionContext left;
public Token operator;
Expand All @@ -175,6 +174,7 @@
else return visitor.visitChildren(this);
}
}
@SuppressWarnings("CheckReturnValue")
public static class IdentOrSelectExpressionContext extends ExpressionContext {
public TerminalNode SELECTOR() { return getToken(ConditionParser.SELECTOR, 0); }
public TerminalNode IDENTIFIER() { return getToken(ConditionParser.IDENTIFIER, 0); }
Expand All @@ -193,6 +193,7 @@
else return visitor.visitChildren(this);
}
}
@SuppressWarnings("CheckReturnValue")
public static class AndExpressionContext extends ExpressionContext {
public ExpressionContext left;
public Token operator;
Expand All @@ -219,6 +220,7 @@
else return visitor.visitChildren(this);
}
}
@SuppressWarnings("CheckReturnValue")
public static class NotExpressionContext extends ExpressionContext {
public TerminalNode NOT() { return getToken(ConditionParser.NOT, 0); }
public ExpressionContext expression() {
Expand All @@ -239,6 +241,7 @@
else return visitor.visitChildren(this);
}
}
@SuppressWarnings("CheckReturnValue")
public static class ParenExpressionContext extends ExpressionContext {
public ExpressionContext inner;
public TerminalNode LPAREN() { return getToken(ConditionParser.LPAREN, 0); }
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
// Generated from Condition.g4 by ANTLR 4.10.1
// Generated from java-escape by ANTLR 4.11.1
package org.opensearch.securityanalytics.rules.condition;
import org.antlr.v4.runtime.tree.ParseTreeVisitor;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
// Generated from Aggregation.g4 by ANTLR 4.10.1
// Generated from java-escape by ANTLR 4.11.1
package org.opensearch.securityanalytics.rules.condition.aggregation;

import org.antlr.v4.runtime.ParserRuleContext;
Expand All @@ -14,6 +10,7 @@
* which can be extended to create a listener which only needs to handle a subset
* of the available methods.
*/
@SuppressWarnings("CheckReturnValue")
public class AggregationBaseListener implements AggregationListener {
/**
* {@inheritDoc}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
// Generated from Aggregation.g4 by ANTLR 4.10.1
// Generated from java-escape by ANTLR 4.11.1
package org.opensearch.securityanalytics.rules.condition.aggregation;
import org.antlr.v4.runtime.tree.AbstractParseTreeVisitor;

Expand All @@ -14,6 +10,7 @@
* @param <T> The return type of the visit operation. Use {@link Void} for
* operations with no return type.
*/
@SuppressWarnings("CheckReturnValue")
public class AggregationBaseVisitor<T> extends AbstractParseTreeVisitor<T> implements AggregationVisitor<T> {
/**
* {@inheritDoc}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
// Generated from Aggregation.g4 by ANTLR 4.10.1
// Generated from java-escape by ANTLR 4.11.1
package org.opensearch.securityanalytics.rules.condition.aggregation;
import org.antlr.v4.runtime.Lexer;
import org.antlr.v4.runtime.CharStream;
Expand All @@ -13,9 +9,9 @@
import org.antlr.v4.runtime.dfa.DFA;
import org.antlr.v4.runtime.misc.*;

@SuppressWarnings({"all", "warnings", "unchecked", "unused", "cast"})
@SuppressWarnings({"all", "warnings", "unchecked", "unused", "cast", "CheckReturnValue"})
public class AggregationLexer extends Lexer {
static { RuntimeMetaData.checkVersion("4.10.1", RuntimeMetaData.VERSION); }
static { RuntimeMetaData.checkVersion("4.11.1", RuntimeMetaData.VERSION); }

protected static final DFA[] _decisionToDFA;
protected static final PredictionContextCache _sharedContextCache =
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
// Generated from Aggregation.g4 by ANTLR 4.10.1
// Generated from java-escape by ANTLR 4.11.1
package org.opensearch.securityanalytics.rules.condition.aggregation;
import org.antlr.v4.runtime.tree.ParseTreeListener;

Expand Down
Loading
Loading