This component serves as a backend for rpt-viewer. It will connect to the cassandra database via cradle api and expose the data stored in there as REST resources.
http://localhost:8080/messageStreams
- returns a list of message stream names
http://localhost:8080/event/{id}
- returns a single event with the specified id
Event object example:
{
"type": "event",
"eventId": "e21de910-fd30-11ea-8896-d7538a286e60",
"batchId": null,
"isBatched": false,
"eventName": "Send 'OrderMassCancelRequest' message",
"eventType": "sendMessage",
"endTimestamp": {
"nano": 209824000,
"epochSecond": 1600819698
},
"startTimestamp": {
"nano": 209190000,
"epochSecond": 1600819698
},
"parentEventId": "e196df51-fd30-11ea-9da8-ffa990115db7",
"successful": true,
"attachedMessageIds": [],
"body": {} // contains a custom json object
}
http://localhost:8080/events
- returns list of events with the specified ids (at a time you can request no more
eventSearchChunkSize
)
ids
- text, one or more event ids Required
Example:
http://localhost:8082/events/?ids=first_id&ids=second_id
http://localhost:8080/message/{id}
- returns a single message with the specified id
Message object example:
{
"type": "message",
"messageId": "fix01:first:1600854429908302153",
"timestamp": {
"nano": 334000000,
"epochSecond": 1600894596
},
"direction": "IN",
"sessionId": "fix01",
"messageType": "OrderMassCancelReport",
"body": {}, // parsed data
"bodyBase64": "" // base64-encoded binary data
"attachedEventIds": []
}
Filters are formed as follows:
filters={filter name}
- you must register the filter by specifying its name.{filter name}-{parameter}={parameter value}
- each filter parameter
Types of filter parameter
:
value / values
- text, number, boolean, etc. - one / many values to match against a filter.negative
- boolean. - Negates the result of the filter. To retrieve data that does NOT match the specified filter. Defaults tofalse
.conjunct
- boolean. - Used if the filter takes several values. Iftrue
then the element must match ALL givenvalues
. Iffalse
then element must match ANY ofvalues
. Not used in filters with one possible value. Defaults tofalse
.
As example:
/search/sse/events/?startTimestamp=1605872487277&filters=name&filters=type&name-values=Checkpoint&type-values=session&type-negative=true
http://localhost:8080/filters/sse-messages
- get all names of sse message filters
http://localhost:8080/filters/sse-events
- get all names of sse event filters
http://localhost:8080/filters/sse-messages/{filter name}
- get filter info
http://localhost:8080/filters/sse-events/{filter name}
- get filter info
http://localhost:8080/match/event/{id}
- return boolean value. Check that event with the specified id is matched by filter
http://localhost:8080/match/message/{id}
- return boolean value. Check that message with the specified id is matched by filter
{
name: "type", // non-nullable
hint: "matches messages by one of the specified types" // nullable
parameters: [
{
name: "negative", // non-nullable string
type: "boolean", // possible values are "number", "boolean", "string", "string[]"
defaultValue: false, // nullable, should match the type
hint: null // nullable string
},
{
name: "values",
type: "string[]",
defaultValue: null,
hint: "NewOrderSingle, ..."
},
]
}
http://localhost:8080/search/sse/events
- create a sse channel of event metadata that matches the filter. Accepts following query parameters:
-
startTimestamp
- number, unix timestamp in milliseconds - Sets the search starting point. One of the 'startTimestamp' or 'resumeFromId' must not be null -
resumeFromId
- text, last event id. In order to continue the execution of an interrupted sse request, you need to send exactly the same request with an indication of the element ID, from which to resume data transfer. Defaults tonull
. One of the 'startTimestamp' or 'resumeFromId' must not be null -
parentEvent
- text - Will match events with the specified parent element. -
searchDirection
-next
/previous
- Sets the lookup direction. Can be used for pagination. Defaults tonext
. -
resultCountLimit
- number - Sets the maximum amount of events to return. Defaults tonull (unlimited)
. -
endTimestamp
- number, unix timestamp in milliseconds - Sets the timestamp to which the search will be performed, starting withstartTimestamp
. WhensearchDirection
isprevious
,endTimestamp
must be less thenstartTimestamp
. Defaults tonull
(the search is carried out endlessly into the past or the future). -
limitForParent
- number - How many children for each parent do we want to request. Defaultnot limited
. -
metadataOnly
- boolean - Receive only metadata (true
) or entire event (false
) (withoutattachedMessageIds
). Defaulttrue
. -
attachedMessages
- boolean - If themetadataOnly
isfalse
additionally loadattachedMessageIds
. Defaultfalse
.
Event metadata object example (in sse):
{
"type": "eventTreeNode",
"eventId": "e21de910-fd30-11ea-8896-d7538a286e60",
"parentId": "e21de910-gc89-11ea-8345-d7538a286e60",
"eventName": "Send 'OrderMassCancelRequest' message",
"eventType": "sendMessage",
"successful": true,
"startTimestamp": {
"nano": 209190000,
"epochSecond": 1600819698
},
}
attachedMessageId
- Filters the events that are linked to the specified message id. Parameters:values
- text,negative
- boolean,conjunct
- boolean.name
- Will match the events which name contains one of the given substrings. Parameters:values
- text, accepts multiple values, case-insensitive,negative
- boolean,conjunct
- boolean.type
- Will match the events which type contains one of the given substrings. Parameters:values
- text, accepts multiple values, case-insensitive,negative
- boolean,conjunct
- boolean.body
- Will match the events which body contains one of the given substrings. Parameters:values
- text, accepts multiple values, case-insensitive,negative
- boolean,conjunct
- boolean.status
- Will match the events which status equals that specified. Parameters:value
- string, one offailed
orpassed
.negative
- boolean.
http://localhost:8080/search/sse/messages
- create a sse channel of messages that matches the filter. Accepts following query parameters:
-
startTimestamp
- number, unix timestamp in milliseconds - Sets the search starting point. One of the 'startTimestamp' or 'resumeFromId' must not be null -
resumeFromId
- text, last message id. In order to continue the execution of an interrupted sse request, you need to send exactly the same request with an indication of the element ID, from which to resume data transfer. Defaults tonull
. One of the 'startTimestamp' or 'resumeFromId' must not be null -
stream
- text, accepts multiple values - Sets the stream ids to search in. Case-sensitive. Required. -
searchDirection
-next
/previous
- Sets the lookup direction. Can be used for pagination. Defaults tonext
. -
resultCountLimit
- number - Sets the maximum amount of messages to return. Defaults tonull (unlimited)
. -
endTimestamp
- number, unix timestamp in milliseconds - Sets the timestamp to which the search will be performed, starting withstartTimestamp
. WhensearchDirection
isprevious
,endTimestamp
must be less thenstartTimestamp
. Defaults tonull
(the search is carried out endlessly into the past or the future). -
messageId
- text, accepts multiple values - List of message IDs to restore search. If given, it has the highest priority and ignoresstream
(uses streams from ids),startTimestamp
andresumeFromId
. Defaults tonull
-
attachedEvents
- boolean - Iftrue
, additionally loadattachedEventIds
. Defaultfalse
. -
lookupLimitDays
- number - The number of days that will be viewed on the first request to get the one closest to the specified timestamp. Defaultnull
- not limited to the past and up to the present moment to the future.
attachedEventIds
- Filters the messages that are linked to the specified event id. Parameters:values
- text, accepts multiple values,negative
- boolean,conjunct
- boolean.type
- Will match the messages by their full type name. Parameters:values
- text, accepts multiple values, case-insensitive,negative
- boolean,conjunct
- boolean.body
- Will match the messages by their parsed body. Parameters:values
- text, accepts multiple values, case-insensitive,negative
- boolean,conjunct
- boolean.bodyBinary
- Will match the messages by their binary body. Parameters:values
- text, accepts multiple values, case-insensitive,negative
- boolean,conjunct
- boolean.
Elements in channel match the format sse:
event: 'event' / 'message' | 'close' | 'error' | 'keep_alive'
data: 'Event metadata object' / 'message' | 'Empty data' | 'HTTP Error code' | 'Empty data'
id: event / message id | null | null | null
schema component description example (rpt-data-provider.yml):
apiVersion: th2.exactpro.com/v2
kind: Th2CoreBox
metadata:
name: rpt-data-provider
spec:
imageName: ghcr.io/th2-net/th2-rpt-data-provider
imageVersion: 5.7.0 // change this line if you want to use a newer version
type: th2-rpt-data-provider
customConfig:
hostname: localhost
port: 8080
responseTimeout: 60000 // maximum request processing time in milliseconds
eventCacheSize: 1000 // internal event cache size
messageCacheSize: 1000 // internal message cache size
serverCacheTimeout: 60000 // cached event lifetime in milliseconds
ioDispatcherThreadPoolSize: 10 // thread pool size for blocking database calls
codecResponseTimeout: 6000 // if a codec doesn't respond in time, requested message is returned with a 'null' body
checkRequestsAliveDelay: 2000 // response channel check interval in milliseconds
enableCaching: true // enables proxy and client cache (Cache-control response headers)
notModifiedObjectsLifetime: 3600 // max-age in seconds
rarelyModifiedObjects: 500 // max-age in seconds
sseEventSearchStep: 200 // step size in seconds when requesting events
keepAliveTimeout: 5000 // timeout in milliseconds. keep_alive sending frequency
cradleDispatcherPoolSize: 1 // number of threads in the cradle dispatcher
messageExtractorOutputBatchBuffer: 1 // buffer size of message search pipeline
messageConverterOutputBatchBuffer: 1
messageDecoderOutputBatchBuffer: 1
messageUnpackerOutputMessageBuffer: 100
messageFilterOutputMessageBuffer: 100
messageMergerOutputMessageBuffer: 10
messageIdsLookupLimitDays: 7 // lookup limit value for seacing next and previous message ids.
codecPendingBatchLimit: 16 // the total number of messages sent to the codec batches in parallel for all pipelines
codecCallbackThreadPool: 4 // thread pool for parsing messages received from codecs
codecRequestThreadPool: 1 // thread pool for sending message to codecs
grpcWriterMessageBuffer: 10 // buffer before send grpc response
sendEmptyDelay: 100 // frequency of sending empty messages
eventSearchChunkSize: 64 // the size of event chunks during sse search and the maximum size of the batch of messages upon request getEvents
grpcThreadPoolSize: 20 // thread pool size for grpc requests
useStrictMode: false // if true throw an exception when bad messages are received from the codec otherwise return messages with null body and type
serverType: HTTP // provider server type. Allows 'HTTP' and 'GRPC' (case sensetive).
codecUsePinAttributes: true // send raw message to specified codec (true) or send to all codecs (false)
eventSearchTimeOffset: 5000 // sets the offset in milliseconds on search events. (startTimestamp - eventSearchTimeOffset) and (endTimestamp + eventSearchTimeOffset)
searchBySessionGroup: true // if true data-provider uses the session alias to group cache and translates http / gRPC requests by session alias to group th2 storage request
aliasToGroupCacheSize: 1000 // the size of cache for the mapping between session alias and group.
useTransportMode: true // if true data-provider uses th2 transport protocol to interact with thw codecs
pins: // pins are used to communicate with codec components to parse message data
mq:
subscribers:
- name: from_codec
attributes:
- from_codec
- transport-group
- subscribe
linkTo:
- box: codec-fix
pin: out_codec_decode
publishers:
- name: to_codec_fix
attributes:
- to_codec
- transport-group
- publish
filters:
- metadata:
- fieldName: "session_group"
expectedValue: "fix*"
operation: WILDCARD
extendedSettings:
chartCfg:
ref: schema-stable
path: custom-component
service:
enabled: false
nodePort: '31275'
envVariables:
JAVA_TOOL_OPTIONS: >
-XX:+ExitOnOutOfMemoryError
-Ddatastax-java-driver.advanced.connection.init-query-timeout="5000 milliseconds"
-XX:+UseContainerSupport
-XX:MaxRAMPercentage=85
resources:
limits:
memory: 2000Mi
cpu: 600m
requests:
memory: 300Mi
cpu: 50m
- Migrated to native grouped message request:
- The
searchBySessionGroup
option has been removed. Now all requests always query thegrouped_message
table - The
/messageIds
request now searches for the nearest message, regardless of its direction.
- The
- Updated th2 gradle plugin:
0.1.5
(th2-bom:4.9.0)
- Updated cradle api:
5.4.4-dev
- Reduced required memory for executing sse event request with
limitForParent
parameter - Updated th2 gradle plugin:
0.1.3
(th2-bom:4.8.0)
- Fixed the problem data provider can't handle
messageIds
request withmessageId
but withoutstartTimestamp
arguments
- Provided ability to limit
messageIds
request bylookupLimitDays
argument ormessageIdsLookupLimitDays
option - Updated:
- th2 gradle plugin:
0.1.1
- common:
5.14.0-dev
- gradle api:
5.4.2-dev
- common-utils:
2.3.0-dev
- ktor-bom:
2.3.12
- ehcache:
3.10.8
- kotlin-logging:
5.1.4
- th2 gradle plugin:
- Migrate to th2 gradle plugin
0.0.8
- Updated common:
5.12.0-dev
- Updated jaxb-runtime:
2.3.9
- '2.3.9' version has 'EDL 1.0' license instead of 'CDDL GPL 1.1' in the '2.3.1'
- Migrate to th2 gradle plugin
0.0.6
- Updated bom:
4.6.1
- Updated common:
5.11.0-dev
- Updated common-utils:
2.2.3-dev
- Updated cradle api:
5.3.0-dev
- Updated grpc-data-provider:
0.2.0-dev
- Updated cradle api:
5.2.0-dev
- Updated common:
5.8.0-dev
- Updated common-utils:
2.2.2-dev
- Disabled unsupported Cassandra driver metrics
- Fix problem with incorrect events order when requesting search in previous direction
- Enabled Cassandra driver metrics
- Fix problem with accumulating all batches in memory when provider loads messages by group
- Migrated to the cradle version with fixed load pages where
removed
field is null problem. - Updated cradle:
5.1.4-dev
- Updated ktor:
2.3.3
- Updated bom:
4.5.0-dev
- Updated common:
5.4.0-dev
- Updated common-utils:
2.2.0-dev
- Updated kotlin:
1.8.22
- Fixed request groped message on single page [EPOCH, null] problem
- Updated cradle:
5.1.2-dev
- Added optional session alias to group cache to translate cradle queries from
messages
togrouped_messages
tables. User can enable this feature bysearchBySessionGroup
option and set cache size in entries byaliasToGroupCacheSize
option - Added mode to interact with codec by th2 transport protocol
-
gradle
- owasp
8.3.1
- owasp
-
th2
- cradle
5.1.0
- bom
4.4.0
- common
5.3.0
- cradle
- gradle
- gradle-git-properties
2.4.1
(gradle plugin for git metadata genration)
- gradle-git-properties