…ropertys (#4847)

Move the Token Selection Processor in the stand-alone worker.

This reverts commit e2aab1c.

…Event Mediator in FlowWorker (#4832)"

This reverts commit 1e949e7.

Fix update and delete SQL statements within the State Manager
implementation so only the states with the specified key and
version are updated. This prevents race conditions while executing
the optimistic locking and also reduces by half the amount of SQL
statements that needs be executed for every update and delete.

…4858)

This commit provides the ability to override topic configuration, including replica and partition counts, on the topic create CLI commands and via the Helm chart.

The YAML file output by the topic create preview now includes partitions and replicas fields per topic. These default to the global parameters. These are then consumed by the topic create connect command so can be manually modified between preview and use.

The topic create preview and topic create connect commands also both take an --overides/-o option, which is the optional path to a file of the same format as the preview file. This file's contents are merged to override those generated/passed in.

The contents of this file can be provided as YAML under bootstrap.kafka.overrides.

#4827)

Fixes exception handling for minimum platform version submitted using the MGM API to improve the returned error message.

CORE-17431 Store metadata in utxo_transaction_metadata since it is relatively large but only changes occasionally.

From logs locally and in our resiliency tests it's clear sandbox closing and therefore bundle uninstallation can be performed in multiple threads depending on whether purgeExpiryQueue is called from the cache's executor thread via the removal listener, or as part of some other synchronous cache operation which happens in the thread of the client calling it.

The parallelism in SecManagerForkJoinPool is configured to the number of processors. This means we wont just see the members of SandboxGroupContextCacheImpl manipulated across 2 threads, we will see it across many and it could happen concurrently in periods of significant flow activity.

This change makes the sandbox cache single threaded, meaning we only ever see manipulation of SandboxGroupContextCacheImpl on the same thread - confirmed by examining the logs.

The author of Caffeine states here that using the same thread for the executor is quite acceptable, the Cache itself does little work there: https://stackoverflow.com/questions/68101454/caffeine-combining-both-scheduler-and-executor-service

…' in ExpirationProcessor (#4839)

We added a new config value to define the polling frequency for the ExpirationProcessor. Previously, the processor failed to reach a healthy lifecycle state when we did a platform upgrade, because there was no mechanism to push the newly introduced config values to Kafka.
We added a temporary fix which would set the value of the frequencyOfExpirationPoll to a default in case of a missing config exception happens on upgrade.

We now have in place a fix to force initial config reconciliation on platform upgrade scenarios, so this temporary fix should be safe to remove.

Update sandbox level cache to receive virtualNodeContext to fix CPI upgrade failure

…sting endpoint (#4774)

For the RPC Epic the Persistence api will require two endpoints. One endpoint is already merged which relates to ledger-related persistence. This PR will be making a suggested change to the naming of that endpoint for clarity, and will also add the second endpoint required.

When this work is merged we will have the endpoints:
/ledger - for ledger persistence
/persistence - for entity persistence

…flow-changes-revert

…diator. (#4858)"

This reverts commit e43fb77.

…le and align with column changes (#4849)

The utxo_transaction_sources table has been modified. See CORE-17430 Modify the utxo_transaction_sources table by removing unused columns and renaming corda-api#1288. This PR contains the changes to align with those DB changes.
Reference states were not persisted in utxo_transaction_sources until now. Added that persisting logic.

This moves the RPC worker endpoint paths (E.g. `/ledger`) to a common consts file so they can be used to define both client and server config.

1. Do not rely on order of the requests returned.
2. Add more logging when re-registration is actually happening.

…ated state manager db (#4763)

These changes enable an isolated state-manager database to be automatically deployed via the Corda helm chart when db bootstrapping is enabled. 

A state-manager database can be bootstrapped by providing the `bootstrap.db.stateManager` postgres credentials overrides, and a `stateManager.db.host` override, along with credentials in `stateManager.db`. 

If no overrides are provided, the state-manager will be bootstrapped in a schema in the cluster database with cluster credentials. Template functions in _helpers.tpl will fall back to cluster JDBC URL and credentials.

CORE-16203, CORE-17429: Flow changes revert

Remove the attachment API from UTXO ledger transactions.

#4863)

Original scope was: The below conversation shows that if certificate validation fails, it’s unclear which certificate the error refers to. In this case it must refer to the TLS certificate because the session certificate is not used, but the error message should be clearer regardless.

However, we lately merged in a fix where we specify which type of certificate was wrong, so the message should be okay. This change resulted in Internal Server Error 500, which is wrong so modified it to 400.

This PR adds a `RPCClient` which will, in the future, be used by the Multi-source Event Mediator to make calls to Corda workers via RPC. Also included is a simple `HTTPRetryExecutor` which will retry HTTP calls a configurable number of times with incremental backoff and allows for a user-specified set of retryable exceptions.

Add `findUnconsumedStatesByExactType` to `UtxoLedgerService`, allowing developers to retrieve states by type from the vault in pages. This dissuades developers from retrieving every state from the vault in one go and returning them to a flow.

Reuse the vault-named query logic to create a `findUnconsumedStatesByExactType` API that supports paging without too much extra work.

Remove the non-paged version of `findUnconsumedStatesByExactType` as well.

…4709)" (#4868)

This reverts commit d5dc203.

)

Disallow duplicate key entries in the group policy. Before this change, providing duplicate keys in a static network group policy resulted in the last instance of that key being used (ignoring other instances).

Adds support for state metadata to the state and event processor.

Components using the event mediator to process events and states may wish to set metadata alongside the state that is written back to the underlying state storage. This change adds to the StateAndEventProcessor API to facilitate this, as well as migrating existing state and event processors to use this.

---------

Co-authored-by: James Higgs <[email protected]>

…diator in FlowWorker (#4832)

Using Multi-Source Event Mediator instead of State and Event Subscription in FlowWorker.

Polling consumers and committing offsets performed in mediator thread in a synchronous way (without using coroutines)

…saging-defaults.conf

…e` (#4884)

CORE-17429, CORE-16203 - multi-source event mediator

…erInfo (#4855)

After performing a platform upgrade, an MGM’s view of its own MemberInfo is not refreshed, so any lookups performed by such an MGM would continue to show it to be on the older version. This change updates the MembershipGroupReader to decorate the returned MGM MemberInfo (when requested by the MGM) with the active platform version from PlatformInfoProvider. This will be removed on completion of CORE-17811, once support for MGM re-registration is available.

Add support for parallel flows executing within the flow pipeline.

This includes:
- Temporary multi-thread implementation of the state and event pattern.
- Using the `SameThreadExecutor` for flow fibers, which causes the fiber to run on the thread that started the fiber.
- Using a Kryo Pool for Kryo instances, as kryo instances are not thread-safe themselves.

Add `TaskManager` and a `TaskManagerFactory` which is a wrapper around a `ScheduledExecutorService` used for gathering metrics on executing tasks. The built-in Micrometer metrics for executor services are used, along with some additional metrics in the `TaskManager` that are more specific to our use-cases.

---------

Co-authored-by: Conal Smith <[email protected]>

…integration with StateManager (#4874)

Integration of Multi-Source Event Mediator with StateManager.

Our configuration change test causes the lifecycle components within workers to restart. They receive the updated configuration and restart with their new configuration. We had pushed this test to the end of the run to avoid other tests being affected by the restart.

We have identified two issues with this test:
* The order we had defined to push the test to the end had not taken effect
* The code to test the cluster is in a good state did not handle exceptions that naturally happen when a client tries to connect to a server that is restarting; so the test was not waiting for the cluster to reach a good state

To address this we are making the following changes:
* Extract the configuration change test into its own test class
* Remove the method order annotation from the test, we want class order instead
* Increase the `@Order` value to `Int.MAX_VALUE` which is the maximum allowed
   * The default order value is `1073741823`, we previously used `999`, so we were not passing the default
* Change `startRpcFlow` to use `assertWithRetryIgnoringExceptions` which retries on exception

CORE-17661 5.1 Performance integration - Multi-Source Event Mediator integration with StateManager (#4874)

…e Event Mediator (#4875)

Using Multi-Source Event Mediator instead of State and Event Subscription in FlowMapperWorker.

---------

Co-authored-by: James Higgs <[email protected]>

CORE-16203 5.1 Performance integration - FlowMapper using Multi-Source Event Mediator (#4875)

This can happen if we can't query the DB worker for the MGM's group policy.

…rker (#4820)

HTTP RPC server integration into the Crypto processor. Allows Crypto processor to receive HTTP external event requests sent from the flow engine. Client integration will come later.

Description:
used more generic name for cross version testing cluster
added ability to define notary service name when onboarding notaries

CORE-16242, CORE-17822 - Integration with release branch

Adds the state status key to flow mapper state metadata and populates it with the current state.

The metadata key is used by the mapper cleanup logic to ensure that mapper states are removed.

…point (#4889)

Adds session timeout to the metadata stored alongside the checkpoint.

Session timeout is implemented using the state storage lookup mechanism, which requires the metadata to be populated with the session timeout key. The earliest timestamp of session expiry for currently open sessions is the value assigned to this property.

During vNode creation and upgrade.

Adds a route for flow events to the flow worker event mediator.

CORE-17627, CORE-17388, CORE-17882 - integration with release

…PC (#4885)

This PR adds a new implementation of MessagingClient, RPCClient, handling synchronous calls to the Corda workers via RPC/HTTP with configurable retries.

Includes also a small bug-fix for the task manager.

---------

Co-authored-by: Miljenko Brkic <[email protected]>

Adds an integration test for mapper cleanup.

Also addresses a small bug where the key for the cleanup event was not serialized correctly.

CORE-16181 Implementing RPC client, routing external events through RPC (#4885)

…plicatesAndCleanup() (#4899)

Fixed stopping event mediator.

FlowMapperServiceIntegrationTest modified to work with event mediator for flow event processing.

Adjusts the session timeout processor to use an explicit conversion to long.

When going through a cycle of writing metadata to the state store and reading it out again, the original type of any numeric values is lost. This causes a problem as casts between numerics are not allowed. To address this, the underlying numeric value in the metadata is converted to `Number`, and then explicitly converted to a long.

This additionally introduces a more thorough unit test for the flow event mediator factory, which ensures that routes are correct for different event types.

…#4939)

* use KubernetesAgent to handle creation of `label`, `nodeSelector` and
`defaultContainer` in more consistent way across many projects

Add a Kryo serializer for `UtxoSignedLedgerTransaction`.

This fixes a bug where `UtxoSignedLedgerTransaction` was being
serialized with the Kryo serializer for `UtxoSignedTransaction`, due to
it implementing the same interface; however, the serializer can only
correct serialize `UtxoSignedTransaction`s.

Adding a specific serializer means Kryo picks up the correct one to
serialize and deserialize with.

Set SessionStateType.ERROR when creating session state in failure path

So that we will not perform re-registration at all in this case.
Also adding some additional logging.

Update Helm Chart so every worker type can configure its own intance
of the State Manager.

* CORE-17834: Allow empty responses from HTTP RPC
* CORE-17836: Adds the request message key as a custom HTTP header 
Co-authored-by: Viktor Kolomeyko <[email protected]>

An e2e test is flaky and extra logging is required in order to understand what might be causing issues.

…4966)

CORE-17502 - Helm changes to support token selection + state manager

… 404 (#4925)

Stability improvements for the `RPCClient`. We now retry on 4XX and 5XX status codes, and thrown exceptions are wrapped as either `CordaMessageAPIIntermittentException` or `CordaMessageAPIFatalException` so that the FlowEventMediator can handle them appropriately.

…ader providers lifecycle (#4961)

Small fix to ensure the flow engine does not try to run without the membership cache populated and ready.

- Configure logging and tracing

Add a metric to record some information about how long it takes for an RPCClient to receive a response, as well as a distribution of response message size

Problem description
The flow engine currently responds to scheduled task triggers to produce flow timeout events when a session has timed out. These need to be processed to remove the timed out checkpoint and send any cleanup events (session errors and mapper cleanup events) to the rest of the system. The flow status also needs to be updated.

Solution
Add a processor for the flow timeout events.

This PR also includes a checkpoint cleanup handler class, which generates the relevant events for when a flow is terminated with an error. This can be reused in the FlowEventExceptionProcessor, although this has not been integrated in this PR. The objective here is to encapsulate this cleanup logic. At present, there is a bug in the flow engine that means not all of this logic is present everywhere it is needed.

The multi source mediator currently updates the state manager after processing each event in a batch. If there is an error that occurs after this has happened, the consumer will rollback to its last poll position and replay events. The StartFlow event is the only FlowEvent not idempotent. This PR addresses this and will ignore subsequent StartFlowEvents unless the checkpoint is null or we are in a retry state and flow state is null (indicating the StartFlow is what needs to be retried)

The flow worker now calls sandbox code from multiple threads. This appears to cause problems in the sandbox code, so for now put a reentrant lock around all the interface methods of SandboxGroupContextServiceImpl.

The only potentially issue is that there are callbacks out of the sandbox code when eviction happens, and they will hold the lock. However, this appears to be harmless. Without this test, we saw large network test failures in the first few flows, and with this we are running 14,396 flows without issues.

In future we should review all the code used by the flow worker for thread safety.

CORE-17502 - Migrate the flow/token selection RPC from Kafka to the new HTTP RPC Model
CORE-17503 - Migrate the token selection state model from the Kafka state and event pattern to the new State Manage model

New topology Kafka topics are being used (flow.start, flow.session, flow.mapper.start, flow.mapper.session.in, flow.mapper.session.out).

Consumers are polled sequentially and performance impact (related to topics with low traffic) is mitigated by adjusting configuration:
- fetch.max.wait.ms = 20
- pollTimeout = 20ms
- max.poll.records = 100
- linger.ms = 50
- batch.size = 204800
- partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
- group.id=${group}-cooperative

Missing topic permissions are set (CORE-17768 Topology changes - Use new topics corda-api#1306)

---------

Co-authored-by: James Higgs <[email protected]>
Co-authored-by: Conal Smith <[email protected]>
Co-authored-by: Dan Newton <[email protected]>
Co-authored-by: Dries Samyn <[email protected]>
Co-authored-by: Omar Awad <[email protected]>
Co-authored-by: Thiago Viana <[email protected]>
Co-authored-by: David Currie <[email protected]>
Co-authored-by: James Higgs <[email protected]>
Co-authored-by: Ben Millar <[email protected]>

* Remove un-used API function.
* Add suspendCount to fibre cache key.

State manager will update states in a batch using a single query with a CTE.

…e not a fatal one (#4981)

currently we are not handling this exception so it is treated as fatal. We should allow the consumer to call poll() to complete the rebalance

…failing intermittently (#4970)

Description:
There were intermittent failures observed on e2e test branch caused by race condition of registration request persistence on membership side. We were persisting the registration request with different member provided context at 2 places. The member context field is a non-updatable field by default, so it was up to the race condition which version for persisted. However, with the 2nd version re-registration attempts were failing.

Fix:
Persist same context everywhere.
Added null check of serial on vnode upgrade.
Let the persistence update the serial of the request.

This reverts commit c706f7c.

…te concurrency check, we need to clear the available token cache as it could be invalid. (#4987)

…o the outgoing HTTP RPC request header correctly. (#4988)