-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' of https://github.com/TNO/knowledge-engine into…
… dependabot/maven/jersey2-version-3.1.3
- Loading branch information
Showing
33 changed files
with
1,186 additions
and
294 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time | ||
| # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven | ||
|
|
||
| # This workflow uses actions that are not certified by GitHub. | ||
| # They are provided by a third-party and are governed by | ||
| # separate terms of service, privacy policy, and support | ||
| # documentation. | ||
|
|
||
| name: Java CI with Maven | ||
|
|
||
| on: | ||
| push: | ||
| branches: [ "master" ] | ||
| pull_request: | ||
| branches: [ "master" ] | ||
|
|
||
| env: | ||
| MAVEN_OPTS: -Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true | ||
| MAVEN_CLI_OPTS: --batch-mode --errors --fail-at-end --show-version | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest | ||
| container: maven:3.8.7-eclipse-temurin-17-alpine | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Compile test source code with Maven | ||
| run: mvn $MAVEN_CLI_OPTS test-compile | ||
| - name: Run the Maven verify phase | ||
| run: mvn $MAVEN_CLI_OPTS verify |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| .python-version |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| FROM ghcr.io/tno/knowledge-engine/smart-connector:1.2.3 | ||
|
|
||
| RUN apt update -y | ||
| RUN apt-get install iptables sudo -y |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| services: | ||
| # This is the knowledge directory, facilitating discovery between different | ||
| # runtimes. It exposes its service over port 8282. | ||
| knowledge-directory: | ||
| image: ghcr.io/tno/knowledge-engine/knowledge-directory:1.2.3 | ||
|
|
||
| # These services are seperate Knowledge Engine runtime, which can host | ||
| # multiple smart connectors. Note that the REST API port is a DIFFERENT port | ||
| # number than the ones configured below. It is still the default 8280. | ||
| runtime-1: | ||
| build: . | ||
| #to allow configuring iptables | ||
| cap_add: | ||
| - NET_ADMIN | ||
| environment: | ||
| KE_RUNTIME_PORT: 8081 # The port that the KE uses to listen for inter-KE-runtime communication. | ||
| KE_RUNTIME_EXPOSED_URL: http://runtime-1:8081 # The URL where the runtime is available for inter-runtime communication from the outside. | ||
| KD_URL: http://knowledge-directory:8282 | ||
| ENABLE_REASONER: false | ||
| JAVA_TOOL_OPTIONS: "-Djdk.httpclient.keepalive.timeout=1" | ||
| runtime-2: | ||
| build: . | ||
| cap_add: | ||
| - NET_ADMIN | ||
| environment: | ||
| KE_RUNTIME_PORT: 8081 | ||
| KE_RUNTIME_EXPOSED_URL: http://runtime-2:8081 | ||
| KD_URL: http://knowledge-directory:8282 | ||
| ENABLE_REASONER: false | ||
| JAVA_TOOL_OPTIONS: "-Djdk.httpclient.keepalive.timeout=1" | ||
| runtime-3: | ||
| build: . | ||
| cap_add: | ||
| - NET_ADMIN | ||
| environment: | ||
| KE_RUNTIME_PORT: 8081 | ||
| KE_RUNTIME_EXPOSED_URL: http://runtime-3:8081 | ||
| KD_URL: http://knowledge-directory:8282 | ||
| ENABLE_REASONER: false | ||
| JAVA_TOOL_OPTIONS: "-Djdk.httpclient.keepalive.timeout=1" | ||
|
|
||
|
|
||
| # These Knowledge Bases use the different runtimes, and exchange data with eachother. | ||
| kb1: | ||
| build: ../common/asking_kb | ||
| environment: | ||
| KE_URL: http://runtime-1:8280/rest | ||
| KB_ID: http://example.org/kb1 | ||
| PREFIXES: | | ||
| { | ||
| "ex": "http://example.org/" | ||
| } | ||
| GRAPH_PATTERN: | | ||
| ?a ex:relatedTo ?b . | ||
| kb2: | ||
| build: ../common/answering_kb | ||
| environment: | ||
| KE_URL: http://runtime-2:8280/rest | ||
| KB_ID: http://example.org/kb2 | ||
| PREFIXES: | | ||
| { | ||
| "ex": "http://example.org/" | ||
| } | ||
| GRAPH_PATTERN: | | ||
| ?a ex:relatedTo ?b . | ||
| KB_DATA: | | ||
| [ | ||
| { | ||
| "a": "<http://example.org/Math>", | ||
| "b": "<http://example.org/Science>" | ||
| }, | ||
| { | ||
| "a": "<http://example.org/Books>", | ||
| "b": "<http://example.org/Magazines>" | ||
| } | ||
| ] | ||
| kb3: | ||
| build: ../common/answering_kb | ||
| environment: | ||
| KE_URL: http://runtime-3:8280/rest | ||
| KB_ID: http://example.org/kb3 | ||
| PREFIXES: | | ||
| { | ||
| "ex": "http://example.org/" | ||
| } | ||
| GRAPH_PATTERN: | | ||
| ?a ex:relatedTo ?b . | ||
| KB_DATA: | | ||
| [ | ||
| { | ||
| "a": "<http://example.org/Music>", | ||
| "b": "<http://example.org/Songs>" | ||
| }, | ||
| { | ||
| "a": "<http://example.org/Red>", | ||
| "b": "<http://example.org/Blue>" | ||
| } | ||
| ] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| ## Knowledge Engine's distributed mode test | ||
| This docker compose project is used to test the Knowledge Engine's behavior in distributed mode when something exceptional happens (i.e. divergence from the happy flow). For example, one participant in the Knowledge Network configured its KER incorrectly and therefore it can reach out, but no one can contact the KER from the outside (via the Inter-KER protocol). Under such circumstances, we want the Knowledge Engine to keep functioning and behave as normal as possible. | ||
|
|
||
| To test this, we setup a distributed KER environment with 3 KER+KB combis that exchange data. We have `runtime-1+kb1`, `runtime-2+kb2` and `runtime-3+kb3`. By using the `iptables` tool for `runtime-3` we can simulate a misconfigured KER and test how the other Knowledge Engines behave. Use the following instructions to simulate the misconfigured KER. | ||
|
|
||
| Start the docker compose project: `docker compose up -d` | ||
|
|
||
| Retrieve the internal IP address of the KB3 (because it needs to always be able to contact it `runtime-3` we need its IP to make an exception in `iptables`). This is not really necessary if we use the hostname `kb3` of knowledge base 3 like we do below, but if you use an IP address there you should use the commands below to retrieve this IP. It changes everytime you restart the docker compose project. | ||
|
|
||
| ``` | ||
| > docker compose exec kb3 sh | ||
| > hostname -i | ||
| ``` | ||
|
|
||
| Make sure runtime-3 is configured to switch between being reachable to being unreachable. First open a shell for runtime-3. | ||
|
|
||
| ``` | ||
| docker compose exec runtime-3 bash | ||
| ``` | ||
|
|
||
| Configure `iptables-legacy` to allow the following packets to go through when we block incoming traffic: | ||
|
|
||
| ``` | ||
| iptables-legacy -A INPUT -i lo -j ACCEPT | ||
| iptables-legacy -A INPUT -p tcp -s kb3 -j ACCEPT | ||
| iptables-legacy -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT | ||
| ``` | ||
|
|
||
| You can quickly test from inside a container whether another container that contains a KER is reachable from there using the following command: `wget -qO- http://runtime-3:8081/runtimedetails` | ||
|
|
||
| For example, if `iptables-legacy` is active and blocking all input traffic, you should no longer be able to receive JSON if you go to `runtime-2` and execute a `wget` to `runtime-3`, but you should be able to do the same from `runtime-3` to `runtime-2`. | ||
|
|
||
| You can quickly test from inside a container whether another container that contains a KER is reachable from there using the following command: `wget -qO- http://runtime-3:8081/runtimedetails`. You should receive some JSON that looks like: | ||
|
|
||
| ``` | ||
| {"runtimeId":"http://runtime-3:8081","smartConnectorIds":["http://example.org/kb3"]} | ||
| ``` | ||
|
|
||
| Now, keep an eye on the log file with `docker compose logs -f` and use the following `iptables-legacy` commands to switch between unreachable and reachable. | ||
|
|
||
| ``` | ||
| iptables-legacy -P INPUT DROP | ||
| #runtime-3 is now unreachable for other KERs, but can still reach the Knowledge Directory (KD) and other KERs. | ||
| iptables-legacy -P INPUT ACCEPT | ||
| #runtime-3 is now reachable again for other KERs and can also reach the KD and other KERs. | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.