ERROR: relation "eventuate.saga_instance" does not exist

[kumar-csice]

Stack Trace:
2023-03-09T04:53:23.687+05:30 ERROR 40264 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [INSERT INTO eventuate.saga_instance(saga_type, saga_id, state_name, last_request_id, saga_data_type, saga_data_json, end_state, compensating) VALUES(?, ?, ?, ?, ?, ?, ?, ?)]] with root cause

org.postgresql.util.PSQLException: ERROR: relation "eventuate.saga_instance" does not exist
Position: 13
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:152) ~[postgresql-42.5.4.jar:42.5.4]
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) ~[HikariCP-5.0.1.jar:na]
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) ~[HikariCP-5.0.1.jar:na]
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965) ~[spring-jdbc-6.0.6.jar:6.0.6]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651) ~[spring-jdbc-6.0.6.jar:6.0.6]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) ~[spring-jdbc-6.0.6.jar:6.0.6]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015) ~[spring-jdbc-6.0.6.jar:6.0.6]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1025) ~[spring-jdbc-6.0.6.jar:6.0.6]
at io.eventuate.common.common.spring.jdbc.EventuateSpringJdbcStatementExecutor.update(EventuateSpringJdbcStatementExecutor.java:23) ~[eventuate-common-common-spring-jdbc-0.9.0.RELEASE.jar:na]
at io.eventuate.tram.sagas.orchestration.SagaInstanceRepositoryJdbc.save(SagaInstanceRepositoryJdbc.java:89) ~[eventuate-tram-sagas-orchestration-0.13.0.RELEASE.jar:na]
at io.eventuate.tram.sagas.orchestration.SagaManagerImpl.create(SagaManagerImpl.java:87) ~[eventuate-tram-sagas-orchestration-0.13.0.RELEASE.jar:na]
at io.eventuate.tram.sagas.orchestration.SagaManagerImpl.create(SagaManagerImpl.java:69) ~[eventuate-tram-sagas-orchestration-0.13.0.RELEASE.jar:na]
at io.eventuate.tram.sagas.orchestration.SagaInstanceFactory.create(SagaInstanceFactory.java:21) ~[eventuate-tram-sagas-orchestration-0.13.0.RELEASE.jar:na]
at com.mt.unity.orderservice.saga.OrderSagaService.createOrder(OrderSagaService.java:37) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-6.0.6.jar:6.0.6]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.0.6.jar:6.0.6]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.0.6.jar:6.0.6]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-6.0.6.jar:6.0.6]

Application.yml
spring:
datasource:
driver-class-name: org.postgresql.Driver
username: postgres
password: admin
url: jdbc:postgresql://${DATABASE_HOST:localhost}:5432/${DB_POSTGRES_DATABASE_NAME:orders}
jpa:
properties:
'[hibernate.default_schema]': public note: I used eventuate instead of public but that also doesn't work.
show-sql: true
generate-ddl: true
hibernate:
ddl-auto: update
database: postgresql

eventuate dependencies

eventuatelocal.kafka.bootstrap.servers: ${DOCKER_HOST_IP:localhost}:9092
eventuatelocal.zookeeper.connection.string: ${DOCKER_HOST_IP:localhost}:2181

Main Class:
@SpringBootApplication
@slf4j
@import({
SagaOrchestratorConfiguration.class,
TramMessageProducerJdbcConfiguration.class,
EventuateTramKafkaMessageConsumerConfiguration.class
})
public class OrderServiceApplication {

public static void main(String[] args) {
	SpringApplication.run(OrderServiceApplication.class, args);
}

@Bean
public ChannelMapping channelMapping() {
	return DefaultChannelMapping.builder().build();
}

//@Bean
//public DuplicateMessageDetector duplicateMessageDetector() {
//	return new NoopDuplicateMessageDetector();
//}

   @Bean
public DuplicateMessageDetector duplicateMessageDetector(EventuateSchema eventuateSchema,
        //String currentTimeInMillisecondsSql,
        EventuateJdbcStatementExecutor eventuateJdbcStatementExecutor,
        EventuateTransactionTemplate eventuateTransactionTemplate) {
	return new SqlTableBasedDuplicateMessageDetector(eventuateSchema, null, eventuateJdbcStatementExecutor, eventuateTransactionTemplate);
}

@Bean
public HttpMessageConverters customConverters() {
	HttpMessageConverter<?> additional = new MappingJackson2HttpMessageConverter();
	return new HttpMessageConverters(additional);
}

@Bean
public OncePerRequestFilter logFilter() {
	return new OncePerRequestFilter() {
		@Override
		protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response,
				FilterChain filterChain) throws ServletException, IOException {
			log.info("Path: {}", request.getRequestURI());
			filterChain.doFilter(request, response);
			log.info("Path: {} {}", request.getRequestURI(), response.getStatus());
		}
	};
}

}

Pom.xml

<java.version>17</java.version>
<org.mapstruct.version>1.4.2.Final</org.mapstruct.version>
0.24.0.RELEASE
0.13.0.RELEASE

	<dependency>
		<groupId>io.eventuate.tram.sagas</groupId>
		<artifactId>eventuate-tram-sagas-spring-orchestration-simple-dsl</artifactId>
		<version>${eventuateTramSagasVersion}</version>
	</dependency>

	<!-- Eventuate Tram dependencies -->
	<dependency>
		<groupId>io.eventuate.tram.core</groupId>
		<artifactId>eventuate-tram-spring-jdbc-kafka</artifactId>
		<version>${eventuateTramVersion}</version>
	</dependency>
	<dependency>
		<groupId>io.eventuate.tram.core</groupId>
		<artifactId>eventuate-tram-spring-optimistic-locking</artifactId>
		<version>${eventuateTramVersion}</version>
	</dependency>

[cer]

Some thing is not right with the database. How have you configured the DB? Are you using an Eventuate container image? What's the JDBC connection string?

[kumar-csice]

I haven't used Eventuate Container Image for the Postgres DB instead I have installed it in my local PC or Postgres Container

Order Micro service:
docker run --name order-postgres-container -d --rm -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=admin -e POSTGRES_DB=orders postgres

Customer Micro service:
docker run --name customer-postgres-container -d --rm -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=admin -e POSTGRES_DB=customers postgres

Queries:

1. In my project we are going to have 10 different micro services and each one has its own DB or single DB but with different schema
   • Low memory profile system can have SQL postgres DB
   • High memory profile system can have NoSQL MongoDB or Neo4j DB or Elasticsearch.
   • Medium memory profile system can have both SQL and No SQL DB.
     i.e. 10 different micro services with same business logic but we will switch to different different DB based on the memory profile during build time.
     So how to use your framework for NoSQL and SQL DB with SAGA orchestration?
2. Right now, I have created all the missing eventuate tables and tried to verify the SAGA flow but transactional messaging is not happening. i.e. Txn is still pending. Also Customer-service's CommandHandler is not reading the command from the Kafka Message Broker.
3. Many beans are not created and injecting so I have manually created many @bean in the Main class (Refer my previous post).
   4. What is the STABLE version of eventuate tram saga framework? like CDC, eventuate tram sagac etc...
   5. Can I use your framework in the PROD environment

Please refer below application.yml for JDBC connection string. (Note: already shared these details in my previous post itself)

application.yml
spring:
datasource:
driver-class-name: org.postgresql.Driver
username: postgres
password: admin
url: jdbc:postgresql://${DATABASE_HOST:localhost}:5432/${DB_POSTGRES_DATABASE_NAME:orders}
jpa:
properties:
'[hibernate.default_schema]': public (note: I tried with eventuate instead of public but that also doesn't work.)
show-sql: true
generate-ddl: true
hibernate:
ddl-auto: update
database: postgresql

[cer]

@kumar-csice

• Yes. You can use it in production.
• The latest released version is shown on this page: https://github.com/eventuate-tram/eventuate-tram-sagas/
• The Eventuate CDC only implements the Transaction Outbox pattern for relational databases.
• I'd recommend looking at this example to see which dependencies you need
• Regarding transactional messaging - that's most likely due to the Eventuate CDC Service being misconfigured. What's the status of its container; output of 'docker ps -a'; output of curling its /actuator/health endpoint

[kumar-csice]

Hi @cer,
Thank you for your prompt response.

So this is my understanding

1. We can't use Eventuate framework to implement SAGA with NoSQL DB
2. We can't use Eventuate framework with Single DB per Microservice architecture
3. Eventuate tables will not be created automatically if My Orchestrator Microservices uses different DB like orders than eventuate DB
   It causes me to create a below eventuate tables
   • saga_instance
   • saga_instance_participants
   • saga_lock_table
   • saga_stash_table
   • message

Otherwise I will be getting below issues while creating an order
URL : http://localhost:8082/api/orders
org.postgresql.util.PSQLException: ERROR: relation "eventuate.saga_instance" does not exist

4. The orchestrator should not be a separate service and should be part of the Transaction initiated Micro service. here order microservice.
5. The eventuate will create the same set of eventuate table in all the microservices
6. example has CustomerProxy service to send the command to Message Broker but why we need this ? Messge should be pulled from Message table by CDC service right to ensure atomicity of Transacational Messaging?

So please correct me.

• I am referring to the example. But why eventuate tables are not created automatically.
• Reagarding CDC it says it is unhealthy

cdcservice:
image: eventuateio/eventuate-cdc-service:0.15.0.BUILD-SNAPSHOT
ports:
- "8099:8080"
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/eventuate
SPRING_DATASOURCE_USERNAME: postgres
SPRING_DATASOURCE_PASSWORD: admin
SPRING_DATASOURCE_DRIVER_CLASS_NAME: org.postgresql.Driver
EVENTUATELOCAL_KAFKA_BOOTSTRAP_SERVERS: kafka:29092
EVENTUATELOCAL_ZOOKEEPER_CONNECTION_STRING: zookeeper:2181
EVENTUATELOCAL_CDC_READER_NAME: PostgresWalReader
EVENTUATE_OUTBOX_ID: 1
SPRING_PROFILES_ACTIVE: PostgresWal

Order Micro service:
docker run --name order-postgres-container -d --rm -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=admin -e POSTGRES_DB=orders postgres

application.yml
spring:
datasource:
driver-class-name: org.postgresql.Driver
username: postgres
password: admin
url: jdbc:postgresql://${DATABASE_HOST:localhost}:5432/${DB_POSTGRES_DATABASE_NAME:orders}
jpa:
properties:
'[hibernate.default_schema]': public (note: I tried with eventuate instead of public but that also doesn't work.)
show-sql: true
generate-ddl: true
hibernate:
ddl-auto: update
database: postgresql

[cer]

1. We can't use Eventuate framework to implement SAGA with NoSQL DB

Correct

2. We can't use Eventuate framework with Single DB per Microservice architecture

Yes. You can. You have multiple options:

• services share a logical DB on a single DB server
• services each have their own logical DB on the same DB server - see https://github.com/microservices-patterns/ftgo-application/ - this definitely works with MySQL, AFAIK it works with Postgres.
• DB server per service - see https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders/tree/development and https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders/blob/775a42e67689ffa983526522f700760879a31d16/docker-compose-postgres.yml#L118

The services and the CDC need to be configured appropriately.

3. Eventuate tables will not be created automatically if My Orchestrator Microservices uses different DB like orders than eventuate DB

Correct. However, the unreleased version of Eventuate supports using flyway to create the DB schema

4. The orchestrator should not be a separate service and should be part of the Transaction initiated Micro service. here order microservice.

The orchestrator can either be a services.

5. The eventuate will create the same set of eventuate table in all the microservices

See answer to 2. DB per service supported.

6. example has CustomerProxy service to send the command to Message Broker but why we need this ?
   Messge should be pulled from Message table by CDC service right to ensure atomicity of Transacational Messaging?

The CustomerProxy is building - not sending - the message.
Using a proxy is optional. The send()...build() code that constructs the message could be within the saga.

So please correct me.

• I am referring to the example. But why eventuate tables are not created automatically.

This example uses the not yet released version of Eventuate that supports flyway.

• Reagarding CDC it says it is unhealthy

cdcservice: image: eventuateio/eventuate-cdc-service:0.15.0.BUILD-SNAPSHOT ports: - "8099:8080" environment: SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/eventuate SPRING_DATASOURCE_USERNAME: postgres SPRING_DATASOURCE_PASSWORD: admin SPRING_DATASOURCE_DRIVER_CLASS_NAME: org.postgresql.Driver EVENTUATELOCAL_KAFKA_BOOTSTRAP_SERVERS: kafka:29092 EVENTUATELOCAL_ZOOKEEPER_CONNECTION_STRING: zookeeper:2181 EVENTUATELOCAL_CDC_READER_NAME: PostgresWalReader EVENTUATE_OUTBOX_ID: 1 SPRING_PROFILES_ACTIVE: PostgresWal

This is the simplified configuration of the CDC - here's the multiple DB configuration: https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders/blob/775a42e67689ffa983526522f700760879a31d16/docker-compose-postgres.yml#L118

To debug the CDC problem:

• what's returned by the CDC's /actuator/health endpoint?
• docker logs <cdc_container>

[kumar-csice]

@cer
Thanks for you support.
And please clarify below queries:

1. I am still not clear about eventuate tables. Do I need to create below Eventuate tables in both Customer and Order Micro service DB?

   • saga_instance
   • saga_instance_participants
   • saga_lock_table
   • saga_stash_table
   • message

2. As mentioned below I am getting below error
   ERROR: relation "public.cdc_monitoring" does not exist

   More details:
   As given here https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders/blob/775a42e67689ffa983526522f700760879a31d16/docker-compose-postgres.yml#L118,
   I Just tried to spin up below container services

   1. zookeeper (0.17.0.BUILD-SNAPSHOT)
   2. kafka (0.17.0.BUILD-SNAPSHOT)
   3. customer-service-postgres (0.17.0.BUILD-SNAPSHOT)
   4. order-service-postgres (0.17.0.BUILD-SNAPSHOT)
   5. cdc-service (0.15.0.BUILD-SNAPSHOT)
      And removed order-service, customer-service and api-gateway-service as I have order-service and customer-service in my local as Maven project.

   docker-compose -f docker-compose-postgres-CDC.yml up

could not access database PreparedStatementCallback; bad SQL grammar [update public.cdc_monitoring set last_time = ? where reader_id = ?]; nested exception is org.postgresql.util.PSQLException: ERROR: relation "public.cdc_monitoring" does not exist

[cer]

@cer Thanks for you support. And please clarify below queries:

1. I am still not clear about eventuate tables. Do I need to create below Eventuate tables in both Customer and Order Micro service DB?

   • saga_instance
   • saga_instance_participants
   • saga_lock_table
   • saga_stash_table
   • message

Short answer is question: https://github.com/eventuate-foundation/eventuate-common/tree/master/postgres and https://github.com/eventuate-tram/eventuate-tram-sagas/tree/master/postgres

2. As mentioned below I am getting below error
   ERROR: relation "public.cdc_monitoring" does not exist
   More details:
   As given here https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders/blob/775a42e67689ffa983526522f700760879a31d16/docker-compose-postgres.yml#L118,
   I Just tried to spin up below container services

   1. zookeeper (0.17.0.BUILD-SNAPSHOT)
   2. kafka (0.17.0.BUILD-SNAPSHOT)
   3. customer-service-postgres (0.17.0.BUILD-SNAPSHOT)
   4. order-service-postgres (0.17.0.BUILD-SNAPSHOT)
   5. cdc-service (0.15.0.BUILD-SNAPSHOT)
      And removed order-service, customer-service and api-gateway-service as I have order-service and customer-service in my local as Maven project.

   docker-compose -f docker-compose-postgres-CDC.yml up

could not access database PreparedStatementCallback; bad SQL grammar [update public.cdc_monitoring set last_time = ? where reader_id = ?]; nested exception is org.postgresql.util.PSQLException: ERROR: relation "public.cdc_monitoring" does not exist

You need to create the cdc_monitoring table too.

[kumar-csice]

@cer

Thanks for your reply.

I am finding difficulties with eventuate workflow and its tables creation as I am unable to make SAGA flow working

My Queries:

1. Is there any sequence diagram or workflow which talks about on where(i.e. in which schema) to create eventuate framework tables and who inserts records?
2. How and when messages are pushed and read from Kafka Message Broker?
3. How and when messages are read by CDC and pushed to Kafka Message Broker?
4. Which eventuate tables belongs to which Functional Micro service schemas? public or eventuate schemas?

As per your suggestion now My CDC service is points to 2 different databases.

1. order-service :5433 -> public schema
2. customer-service :5432 -> public schema

But as per the given links there are 5 sql scripts which are all points to eventuate schema
https://github.com/eventuate-foundation/eventuate-common/tree/master/postgres and https://github.com/eventuate-tram/eventuate-tram-sagas/tree/master/postgres

1.initialize-database.sql
2.initialize-database.sql
3.initialize-database.sql
4.initialize-database-json.sql
5.initialize-database-db-id.sql

But I didn't execute all the above script files and just tried to fix the CDC error by creating some of the eventuate tables across 3 DB schema

1. order-service -> Eventuate schema
2. order-service -> public schema
3. customer-service -> public schema

Now CDC is up and running and it is healthy but unfortunately there is no SAGA initiated as still my ORDER is in PENDING state.

[cer]

Hmm.. I see the problem with those scripts - they use Eventuate as the hardwired schema. But the key point is to create the same tables as those scripts.

It's great that the CDC is healthy. Perhaps the Order Service is writing to Eventuate tables in one schema (e.g. eventuate) and the CDC is reading from tabled in a different schema (e.g. public).

I'd would only create the eventuate is one schema in each database server.

For example, I'd take a look at this file: https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders/blob/development/docker-compose-postgres.yml

• Two services each with its own database server
• Each services creates its database schema using flyway using the services JDBC url jdbc:postgresql://order-service-postgres/order_service
• In particular, Eventuate tables are created in 'public' EVENTUATE_DATABASE_SCHEMA: public
• Note the configuration of the CDC: https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders/blob/6526fd92030fa533e8d59f143e24ce67ec97e6eb/docker-compose-postgres.yml#L121-L147

[kumar-csice]

@cer
Thank you for your prompt response!

As you have mentioned I will execute below 5 script files in both order-service and customer-service postgres DB under public schema instead of eventuate schema.
1.initialize-database.sql
2.initialize-database.sql
3.initialize-database.sql
4.initialize-database-json.sql
5.initialize-database-db-id.sql

But where to execute this schema file

1. In order-service Or
2. In customer-service
   sagas/blob/master/postgres/tram-saga-schema.sql)

More queries:

1. How to configure my own topic and partition names in order and customer micro services?

[cer]

But where to execute this schema file
sagas/blob/master/postgres/tram-saga-schema.sql)

Even though some tables are orchestrator specific and some are participant specific, I would execute it in both for both the order and customer services.

[cer]

1. How to configure my own topic and partition names in order and customer micro services?

Please could you clarify the question. Names of which topics?
Partitions don't have names.

[kumar-csice]

@cer
Usually by using spring-cloud stream we will configure our Kafka destination like customer-topic and order-topic in application.yml file .
Is there any way to configure topic names instead of default eventuate framework topic names for publishing and consuming messages?

[cer]

@cer Usually by using spring-cloud stream we will configure our Kafka destination like customer-topic and order-topic in application.yml file . Is there any way to configure topic names instead of default eventuate framework topic names for publishing and consuming messages?

The Customer Service and Order Service have to agree on the name of the customer service's command channel:

• Customer Service
• Order Service

This channel name is defined by you and could be read from a configuration file.

A sagas reply channel name is derived from the saga class name

There is also the concept of ChannelMapping, which by default (if you don't define your own bean) is DefaultChannelMapping

[kumar-csice]

@cer

1. org.postgresql.util.PSQLException: ERROR: relation "eventuate.saga_instance" does not exist
   Position: 13
   As recommended by you I Have created all the eventuate orchestrator and participants tables in both order-service and customer-service's public schema. But after http://localhost:8082/api/orders POST call getting the above error.?
   why it is again expecting eventuate.saga_instance schema as I have already created this table in public schema of both order & customer service DB.

JDBC string:
Order micro service:
datasource:
***
url: jdbc:postgresql://${DATABASE_HOST:localhost}:5433/${DB_POSTGRES_DATABASE_NAME:order_service}
jpa:
properties:
'[hibernate.default_schema]': public

customer micro service:
datasource:
***
url: jdbc:postgresql://${DATABASE_HOST:localhost}:5432/${DB_POSTGRES_DATABASE_NAME:customer_service}
jpa:
properties:
'[hibernate.default_schema]': public

2. List down all possible properties to configure topic names in application.yml like below ?
spring:
cloud:
stream:
bindings:
consumeDeviceTelemetry-in-0:
destination: device-telemetry-kafka-topic
group: device-telemetry-db-group
binder: kafka

As I could see there are 2 other topics were created automatically other than customerService.

1. what is purpose of those?
2. Only __Consumer_offsets partition has some data in it's queue.

[cer]

1. why it is again expecting eventuate.saga_instance schema as I have already created this table in public schema of both order & customer service DB.

I think you are missing: https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders/blob/6526fd92030fa533e8d59f143e24ce67ec97e6eb/docker-compose-postgres.yml#L47

      EVENTUATE_DATABASE_SCHEMA: public

[kumar-csice]

1. why it is again expecting eventuate.saga_instance schema as I have already created this table in public schema of both order & customer service DB.

I think you are missing: https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders/blob/6526fd92030fa533e8d59f143e24ce67ec97e6eb/docker-compose-postgres.yml#L47

      EVENTUATE_DATABASE_SCHEMA: public

Ok.what is the corresponding configuration property
Key for EVENTUATE_DATABASE_SCHEMA in the application.yml file as I have created my own order and customer service in local workspace?

[cer]

Ok.what is the corresponding configuration property
Key for EVENTUATE_DATABASE_SCHEMA in the application.yml file as I have created my own order and customer service in local workspace

eventuate.database.schema

[kumar-csice]

@cer
Thank you for the quick response. I will try and feedback you.
More queries:

1. In which eventuate framework PROD (tentative date) release the eventuate framework tables are created automatically by using some flyway component? I.e
   Orchestrator specific tables are in orchestrator service and participants specific tables are in the corresponding participant micro services?
2. Right now I am using only snapshot versions of zookeeper, Kafka server and CDC service. What is the current PROD version and when is the next PROD release date?
3. Why customer_credit_reservation table is created for SAGA instead it can use same customer table for updating the credit limit right? Because while querying my customer table always my credit is same irrespective of the orders
4. For my actual micro service implementation i have to use different tables(no order and customer tables) so how to avoid creating tables like customer_credit_reservation as i don't want additional tables created for domain entity in the functional microservice?
5. Why we need version column ( optimistic locking) instead we can use modified date column right? Because we use audit columns for all of our micro services. So why we need to create additional version column.
   Audit columns are Created by, created on, modified by, modified on
6. Why business entity tables have some logic or eventuate specific logic ? It results in tightly coupled entity or aggregate object.

[cer]

@cer Thank you for the quick response. I will try and feedback you. More queries:

1. In which eventuate framework PROD (tentative date) release the eventuate framework tables are created automatically by using some flyway component? I.e

The next version, which should be released in the next couple of weeks.

Orchestrator specific tables are in orchestrator service and participants specific tables are in the corresponding participant micro services?

The split into common, orchestrator and participant tables might not happen until later.

2. Right now I am using only snapshot versions of zookeeper, Kafka server and CDC service. What is the current PROD version and when is the next PROD release date?

Take a look at https://github.com/eventuate-platform/eventuate-platform-dependencies
That's the platform release.

If you then look in gradle.properties for the latest release you can see the released versions of various components

3. Why customer_credit_reservation table is created for SAGA instead it can use same customer table for updating the credit limit right? Because while querying my customer table always my credit is same irrespective of the orders

That table is created by the application. Not the saga implementation.

4. For my actual micro service implementation i have to use different tables(no order and customer tables) so how to avoid creating tables like customer_credit_reservation as i don't want additional tables created for domain entity in the functional microservice?

Those tables are created by the application. That's entirely up to you.

5. Why we need version column ( optimistic locking) instead we can use modified date column right? Because we use audit columns for all of our micro services. So why we need to create additional version column.
   Audit columns are Created by, created on, modified by, modified on

That's application level.

6. Why business entity tables have some logic or eventuate specific logic ? It results in tightly coupled entity or aggregate object.

Which business entity tables are you referring to?

[kumar-csice]

@cer

Sorry for some of my dumb question as I clearly want to understand your framework so that I can use it in my application.

Queries:
3. Why customer_credit_reservation table is created for SAGA instead it can use same customer table for updating the credit limit right? Because while querying my customer table always my credit is same irrespective of the orders
That table is created by the application. Not the saga implementation.
I could see only below entity class and not sure/clear how this customer_credit_reservation table is created and configured in Customer.java class
@entity
@table(name = "Customer")
5. Version Column:
Yes. It is part of my order and customer entity class to support optimistic locking to avoid dirty read and lost updates. So instead of version column can we use modified date as this will updated on each txn and we will influence this to avoid dirty read and lost updates.

7. Why business entity tables have some logic or eventuate specific logic ? It results in tightly coupled entity or aggregate object.
   Which business entity tables are you referring to?
   Order and Customer entity. Usually, it is a class which contains fields and corresponding getter and setter methods and JPA annotations but here we have business logic too like below
   public void reserveCredit(Long orderId, Money orderTotal) {
   if (availableCredit().isGreaterThanOrEqual(orderTotal)) {
   creditReservations.put(orderId, orderTotal);
   } else
   throw new CustomerCreditLimitExceededException();
   }

[cer]

3. Why customer_credit_reservation table is created for SAGA instead it can use same customer table for updating the credit limit right? Because while querying my customer table always my credit is same irrespective of the orders
   That table is created by the application. Not the saga implementation.
   I could see only below entity class and not sure/clear how this customer_credit_reservation table is created and configured in Customer.java class
   @entity
   @table(name = "Customer")

JPA generated by https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders/blob/6526fd92030fa533e8d59f143e24ce67ec97e6eb/customer-service/customer-service-domain/src/main/java/io/eventuate/examples/tram/sagas/ordersandcustomers/customers/domain/Customer.java#L22-L23

[cer]

7. Order and Customer entity. Usually, it is a class which contains fields and corresponding getter and setter methods and JPA annotations but here we have business logic too like below

Actually, what you are describing is often an anti-pattern: https://martinfowler.com/bliki/AnemicDomainModel.html

Quite often a better approach is to have a Rich domain model (https://martinfowler.com/eaaCatalog/domainModel.html) where entities have data and behavior.

[kumar-csice]

@cer
2. Right now I am using only snapshot versions of zookeeper, Kafka server and CDC service. What is the current PROD version and when is the next PROD release date?
Take a look at https://github.com/eventuate-platform/eventuate-platform-dependencies
That's the platform release.

Currently I am using

1. image: eventuateio/eventuate-cdc-service:0.15.0.BUILD-SNAPSHOT
2. image: eventuateio/eventuate-kafka:0.17.0.BUILD-SNAPSHOT

In https://github.com/eventuate-platform/eventuate-platform-dependencies/blob/master/gradle.properties#L13 it is eventuateCdcVersion=0.15.0.BUILD-SNAPSHOT

You mean ****BUILD-SNAPSHOT is PROD version?

[cer]

In https://github.com/eventuate-platform/eventuate-platform-dependencies/blob/master/gradle.properties#L13 it is eventuateCdcVersion=0.15.0.BUILD-SNAPSHOT

You mean ****BUILD-SNAPSHOT is PROD version?

No. You need to look at a release of the platform. e..g for example: https://github.com/eventuate-platform/eventuate-platform-dependencies/blob/2022.2.RELEASE/gradle.properties

[kumar-csice]

In https://github.com/eventuate-platform/eventuate-platform-dependencies/blob/master/gradle.properties#L13 it is eventuateCdcVersion=0.15.0.BUILD-SNAPSHOT
You mean ****BUILD-SNAPSHOT is PROD version?

No. You need to look at a release of the platform. e..g for example: https://github.com/eventuate-platform/eventuate-platform-dependencies/blob/2022.2.RELEASE/gradle.properties

Thank you 👍

[kumar-csice]

@cer

5. Version Column:
   Yes. It is part of my order and customer entity class to support optimistic locking to avoid dirty read and lost updates. So instead of version column can we use modified date as this will be updated on each Txn and we will use this field to avoid dirty read and lost updates. So that we can avoid adding Version field/column.
   Can we modify/customize this approach?

[kumar-csice]

@cer

1. I have used
   image: postgres:latest
   instead of
   image: eventuateio/eventuate-vanilla-postgres:0.17.0.BUILD-SNAPSHOT
   and getting below error while executing below queries

SELECT * FROM pg_create_logical_replication_slot('eventuate_slot', 'wal2json');
SELECT * FROM pg_create_logical_replication_slot('eventuate_slot2', 'wal2json');

ERROR:  logical decoding requires wal_level >= logical
SQL state: 55000

2. what is the PROD verison of image: eventuateio/eventuate-vanilla-postgres:0.17.0.BUILD-SNAPSHOT ?

[cer]

ERROR: logical decoding requires wal_level >= logical
SQL state: 55000

I'm guessing that WAL is not installed. See https://github.com/eventuate-foundation/eventuate-common/blob/master/postgres/Dockerfile-vanilla

[cer]

2. what is the PROD verison of image: eventuateio/eventuate-vanilla-postgres:0.17.0.BUILD-SNAPSHOT ?

This is new in the forthcoming release.

[kumar-csice]

ERROR: logical decoding requires wal_level >= logical
SQL state: 55000

I'm guessing that WAL is not installed. See https://github.com/eventuate-foundation/eventuate-common/blob/master/postgres/Dockerfile-vanilla

@cer

1. Let me enable WAL configuration and try. But with postgres image it is disabled by default, right?
2. how secure this eventuate framework is? Does it follow any security standards and any encryption mechanism?
3. Is there any orchstration GUI tool/service for the monitoring purpose?

[kumar-csice]

@cer

1. Let me enable WAL configuration and try. But with postgres image it is disabled by default, right?
2. how secure this eventuate framework is? Does it follow any security standards and any encryption mechanism?
3. Is there any orchestration GUI tool/service for the monitoring this eventuate orchestration flow ?

[cer]

• Let me enable WAL configuration and try. But with postgres image it is disabled by default, right?

• Let me enable WAL configuration and try. But with postgres image it is disabled by default, right?

Strictly speaking https://packages.ubuntu.com/focal/postgresql-12-wal2json is a PostgreSQL logical decoding JSON output plugin. See https://www.postgresql.org/docs/current/logicaldecoding.html

[cer]

3. Is there any orchestration GUI tool/service for the monitoring this eventuate orchestration flow ?

Not currently.

[cer]

• how secure this eventuate framework is? Does it follow any security standards and any encryption mechanism?

Not explicitly. It inserts into the DB; publishes to a broker; and subscribes to a broker. It leverages the security capabilities of the DB and the broker.

Did you have any specific requirements in mind?

[kumar-csice]

@cer
how secure this eventuate framework is? Does it follow any security standards and any encryption mechanism?
Not explicitly. It inserts into the DB; publishes to a broker; and subscribes to a broker. It leverages the security capabilities of the DB and the broker.

Did you have any specific requirements in mind?

Not specifically but framework like AXON provides security features and data protection and also we want to fix issues found by SCA (Software Composition Analyses tool)
https://developer.axoniq.io/axon-data-protection/overview
https://docs.axoniq.io/reference-guide/axon-server/security

[kumar-csice]

@cer

Let me enable WAL configuration and try. But with postgres image it is disabled by default, right?
Strictly speaking https://packages.ubuntu.com/focal/postgresql-12-wal2json is a PostgreSQL logical decoding JSON output plugin. See https://www.postgresql.org/docs/current/logicaldecoding.html

Is there any docker image and helm chart for PostgreSQL with WAL enabled as I currently we are using official postgres image from the docker hub and I dont see any environmental variable for the WAL configuration ?

https://hub.docker.com/_/postgres/

[cer]

Is there any docker image and helm chart for PostgreSQL with WAL enabled as I currently we are using official postgres image from the docker hub and I dont see any environmental variable for the WAL configuration ?

I'm not aware of any. It's a simple matter to build your own image (or use the Eventuate one).

[cer]

Not specifically but framework like AXON provides security features and data protection and also we want to fix issues found by SCA (Software Composition Analyses tool)
https://developer.axoniq.io/axon-data-protection/overview

The Axxon GDPR module is targeted when you have an immutable event store for Event Sourcing. The Eventuate Tram (Saga) framework is not an immutable event store for Event Sourcing. e.g. Kafka retains events for a finite period of time.

Having said that, is this something that you think you need?

https://docs.axoniq.io/reference-guide/axon-server/security

Client's community with the Axxon server so it has specific security needs.
In contrast, the Eventuate CDC connects to DBs and publishes to Kafka. It does not have an API for clients.

I'm happy to discuss concrete requirements, however.