add another node to e2e test setup

.env

@@ -17,10 +17,14 @@ POSTGRES_HOST_PORT=5432
 REDIS_CONTAINER_PORT=6379
 REDIS_HOST_PORT=6379
 
+KAVA_CONTAINER_TAG=v0.24.0
 KAVA_CONTAINER_EVM_RPC_PORT=8545
-KAVA_HOST_EVM_RPC_PORT=8545
 KAVA_CONTAINER_COSMOS_RPC_PORT=26657
+
+KAVA_HOST_EVM_RPC_PORT=8545
 KAVA_HOST_COSMOS_RPC_PORT=26657
+KAVA_PRUNING_HOST_EVM_RPC_PORT=8555
+KAVA_PRUNING_HOST_COSMOS_RPC_PORT=26667
 
 PROXY_CONTAINER_PORT=7777
 PROXY_CONTAINER_EVM_RPC_DATA_PORT=7778
@@ -34,13 +38,13 @@ TEST_PROXY_SERVICE_EVM_RPC_HOSTNAME=localhost:7777
 TEST_PROXY_SERVICE_EVM_RPC_DATA_URL=http://localhost:7778
 TEST_PROXY_BACKEND_EVM_RPC_HOST_URL=http://localhost:8545
 TEST_DATABASE_ENDPOINT_URL=localhost:5432
-TEST_PROXY_BACKEND_HOST_URL_MAP=localhost:7777>http://kava:8545,localhost:7778>http://kava:8545
+TEST_PROXY_BACKEND_HOST_URL_MAP=localhost:7777>http://kava-validator:8545,localhost:7778>http://kavapruning:8545
 # What level of logging to use for service objects constructed during
 # unit tests
 TEST_SERVICE_LOG_LEVEL=ERROR
 # endpoint the proxy service should use for querying
 # evm blockchain information related to proxied requests
-TEST_EVM_QUERY_SERVICE_URL=http://kava:8545
+TEST_EVM_QUERY_SERVICE_URL=http://kava-validator:8545
 
 ##### Kava Node Config
 
@@ -52,7 +56,7 @@ LOG_LEVEL=TRACE
 HTTP_READ_TIMEOUT_SECONDS=30
 HTTP_WRITE_TIMEOUT_SECONDS=60
 # Address of the origin server to proxy all requests to
-PROXY_BACKEND_HOST_URL_MAP=localhost:7777>http://kava:8545,localhost:7778>http://kava:8545
+PROXY_BACKEND_HOST_URL_MAP=localhost:7777>http://kava-validator:8545,localhost:7778>http://kavapruning:8545
 # Configuration for the servcie to connect to it's database
 DATABASE_NAME=postgres
 DATABASE_ENDPOINT_URL=postgres:5432
@@ -72,7 +76,7 @@ DATABASE_QUERY_LOGGING_ENABLED=true
 METRIC_COMPACTION_ROUTINE_INTERVAL_SECONDS=5
 # endpoint the proxy service should use for querying
 # evm blockchain information related to proxied requests
-EVM_QUERY_SERVICE_URL=http://kava:8545
+EVM_QUERY_SERVICE_URL=http://kava-validator:8545
 # Whether the proxy service should attempt to track and store metrics
 # related to proxied requests
 METRIC_COLLECTION_ENABLED=true

.gitignore

@@ -21,3 +21,7 @@ cover.html
 
 # ignore editor files
 .vscode/
+
+# ignore e2e test validator files
+docker/shared/genesis.json
+docker/shared/VALIDATOR_NODE_ID

DEVELOPMENT.md

@@ -96,6 +96,8 @@ The e2e tests won't pass if the proxy service and it's dependencies aren't fully
 make ready e2e-test
 ```
 
+For details on the local E2E setup, see [the `docker` directory](./docker/README.md).
+
 ## Test Coverage Report
 
 The test commands `make test`, `make unit-test`, and `make e2e-test` generate a `cover.out` raw test coverage report. The coverage can be converted into a user-friendly webpage:

Makefile

@@ -68,6 +68,8 @@ up:
 .PHONY: down
 # stop the service and it's dependencies
 down:
+	rm docker/shared/genesis.json
+	rm docker/shared/VALIDATOR_NODE_ID
 	docker compose down
 
 .PHONY: restart
@@ -78,6 +80,8 @@ restart:
 .PHONY: reset
 # wipe state and restart the service and all it's dependencies
 reset: lint
+	rm docker/shared/genesis.json || exit 0
+	rm docker/shared/VALIDATOR_NODE_ID || exit 0
 	docker compose up -d --build --remove-orphans --renew-anon-volumes --force-recreate
 
 .PHONY: refresh

docker-compose.yml

@@ -8,6 +8,7 @@ services:
       - "${POSTGRES_HOST_PORT}:${POSTGRES_CONTAINER_PORT}"
     expose:
       - "${POSTGRES_CONTAINER_PORT}"
+
   # run redis for proxy service to cache responses
   redis:
     image: 'bitnami/redis:latest'
@@ -16,13 +17,15 @@ services:
       - "${REDIS_HOST_PORT}:${REDIS_CONTAINER_PORT}"
     expose:
       - "${REDIS_CONTAINER_PORT}"
+
   # run single validator kava node to provide a local kava network for development and testing of the proxy service
-  kava:
-    image: kava/kava:latest
-    entrypoint: /docker/kava/kava-entrypoint.sh
+  kava-validator:
+    image: kava/kava:${KAVA_CONTAINER_TAG}
+    entrypoint: /docker/kava/kava-validator-entrypoint.sh
     env_file: .env
     volumes:
-      - ./docker/kava:/docker/kava
+      - ./docker/kava-validator:/docker/kava
+      - ./docker/shared:/docker/shared
     ports:
       - "${KAVA_HOST_COSMOS_RPC_PORT}:${KAVA_CONTAINER_COSMOS_RPC_PORT}"
       - "${KAVA_HOST_EVM_RPC_PORT}:${KAVA_CONTAINER_EVM_RPC_PORT}"
@@ -31,6 +34,23 @@ services:
     expose:
       - "${KAVA_CONTAINER_COSMOS_RPC_PORT}"
       - "${KAVA_CONTAINER_EVM_RPC_PORT}"
+
+  # peer node with api running validator's network
+  kavapruning:
+    image: kava/kava:${KAVA_CONTAINER_TAG}
+    entrypoint: /docker/shared/kava-entrypoint.sh
+    env_file: .env
+    volumes:
+      - ./docker/shared:/docker/shared
+    ports:
+      - "${KAVA_PRUNING_HOST_COSMOS_RPC_PORT}:${KAVA_CONTAINER_COSMOS_RPC_PORT}"
+      - "${KAVA_PRUNING_HOST_EVM_RPC_PORT}:${KAVA_CONTAINER_EVM_RPC_PORT}"
+    # expose ports for other services to be able to connect to within
+    # the default docker-compose network
+    expose:
+      - "${KAVA_CONTAINER_COSMOS_RPC_PORT}"
+      - "${KAVA_CONTAINER_EVM_RPC_PORT}"
+
   # run proxy service to observe, route, and scale requests to kava api endpoints
   proxy:
     build:

docker/README.md

@@ -0,0 +1,75 @@
+# e2e test setup
+
+For the most accurate reflection of real-world use cases, this repo includes an end-to-end test
+setup that includes a kava node running with multiple validators.
+
+This directory contains the configuration and startup files necessary for that setup.
+
+From the repo root, the following make commands work with these files:
+* `make up` - starts the service. if it has never been started before, setup will be performed
+* `make down` - stops & destroys the nodes & network
+* `make reset` - destroys & recreates the nodes & network. setup will be performed
+* `make ready` - blocks the process until the network is producing blocks
+
+`make e2e-test` and `make test` (which runs unit & e2e tests) both rely on the network being running and ready.
+
+The following does not affect the network:
+* `make refresh` - destroys and recreates only the proxy service. useful for picking up new proxy env variables.
+
+
+## how it works
+
+There is a network running with a single validator. The config and entrypoint for this node is in [kava-validator](./kava-validator/).
+
+The `shared/` directory is shared between all nodes and is used to share necessary details from the validator to the other nodes in the network.
+
+The validator has an [entrypoint](./kava-validator/kava-validator-entrypoint.sh) that does the following on first startup:
+* `init`s a new kava home directory
+* creates a gentx and initializes the network genesis file
+* writes its node id to `shared/VALIDATOR_NODE_ID` so peers can connect to it
+* copies the genesis file to `shared/genesis` so other peers can use it to connect
+* starts the network
+
+Meanwhile, any peer node in the network (configured in [docker-compose.yml](../docker-compose.yml)) has an [entrypoint](./shared/kava-entrypoint.sh)
+that does the following:
+* `init`s a new kava home directory
+* waits for the validator to share the genesis file
+* copies over the validator
+* reads the validator's node id from `shared/VALIDATOR_NODE_ID`
+* starts the network with the validator as a peer
+
+## add more nodes
+
+We'll want more shards in the future!! This setup supports this. To add another api-enabled node to the network:
+1. Add a configuration in the [docker-compose.yml](../docker-compose.yml)
+```yml
+  # peer node with api running validator's network
+  nodename:
+    image: kava/kava:${KAVA_CONTAINER_TAG}
+    entrypoint: /docker/shared/kava-entrypoint.sh
+    env_file: .env
+    volumes:
+      - ./docker/shared:/docker/shared
+    # expose ports for other services to be able to connect to within
+    # the default docker-compose network
+    expose:
+      - "${KAVA_CONTAINER_COSMOS_RPC_PORT}"
+      - "${KAVA_CONTAINER_EVM_RPC_PORT}"
+    # optional: bind host ports to access outside docker network
+    ports:
+      - "${EXPOSED_RPC_PORT}:${KAVA_CONTAINER_COSMOS_RPC_PORT}"
+      - "${EXPOSED_EVM_JSON_RPC_PORT}:${KAVA_CONTAINER_EVM_RPC_PORT}"
+```
+Note that `nodename` should be replaced with whatever you call your new node.
+
+The `ports` bindings are only necessary if you want to directly query the node from outside the docker network.
+If so, replace and create new env variables for
+* `EXPOSED_RPC_PORT` - the host (outside docker network) port for the rpc api
+* `EXPOSED_EVM_JSON_RPC_PORT` - the host (outside docker network) port for the evm json rpc api
+
+2. Add the new node to the proxy backend host map config: `localhost:${PROXY_PORT_FOR_NEW_HOST}>http://nodename:8545`
+
+3. Make sure the proxy port routes to the proxy service. Configure this in the docker-compose `ports`
+   of `proxy`: `- "${PROXY_PORT_FOR_NEW_HOST}:${PROXY_CONTAINER_PORT}"`
+
+4. Run `make reset` and the new node should be running & connected to the network.