Merge pull request #481 from TNO/463-when-an-other-ker-is-unavailable…

…-the-current-ker-should-not-log-a-lot-of-completionexceptions-caused-by-contacting-it-anyway

463 when an other ker is unavailable the current ker should not log a lot of completionexceptions caused by contacting it anyway

README.md

@@ -257,3 +257,13 @@ nohup java -cp "smart-connector-rest-dist-1.2.3.jar:dependency/*" eu.knowledge.e
 
 ### Starting the Knowledge Engine in distributed mode
 The Knowledge Engine can also start in distributed mode, where it connects with a remote knowledge directory and where different instances of the Knowledge Engine (each instance hosting one or more smart connectors) can communicate with each other. More information about starting the Knowledge Engine in distributed mode can be found in the [documentation](docs/04_distributed_mode.md).
+
+### Additional configuration environment variables
+
+*Increasing the wait time for other KBs to respond*
+
+By default, a Smart Connector waits `10` seconds max for a reply from another Smart Connector when sending an ASK/POST message. This time is configurable via the `KE_KB_WAIT_TIMEOUT` environment variable and setting it to `0` means the Smart Connector will wait indefinitely (this can be useful when dealing with Human KBs).
+
+*Increasing the HTTP timeouts*
+
+By default, a KER waits `5` seconds max for a HTTP response from another KER when sending a message via the inter-KER protocol. The time is configurable via the `KE_HTTP_TIMEOUT` environment variable.

admin-ui/pom.xml

@@ -115,15 +115,22 @@
 		<dependency>
 			<groupId>org.junit.jupiter</groupId>
 			<artifactId>junit-jupiter-api</artifactId>
-			<version>5.10.1</version>
+			<version>5.10.2</version>
 			<scope>test</scope>
 		</dependency>
 		<dependency>
 			<groupId>org.junit.jupiter</groupId>
 			<artifactId>junit-jupiter-engine</artifactId>
-			<version>5.7.0</version>
+			<version>5.10.2</version>
 			<scope>test</scope>
 		</dependency>
+		<dependency>
+			<groupId>org.junit.jupiter</groupId>
+			<artifactId>junit-jupiter</artifactId>
+			<version>5.10.2</version>
+			<scope>test</scope>
+		</dependency>
+
 		<dependency>
 			<groupId>eu.knowledge.engine</groupId>
 			<artifactId>smart-connector-rest-server</artifactId>
@@ -200,6 +207,7 @@
 				<configuration>
 					<shadedArtifactAttached>true</shadedArtifactAttached>
 					<shadedClassifierName>with-dependencies</shadedClassifierName>
+					<createDependencyReducedPom>false</createDependencyReducedPom>
 					<transformers>
 						<transformer
 							implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">

examples/unreachable-runtimes/Dockerfile

@@ -1,4 +1,4 @@
-FROM ghcr.io/tno/knowledge-engine/smart-connector:1.2.4
+FROM ghcr.io/tno/knowledge-engine/smart-connector:1.2.4-SNAPSHOT
 
 USER root
 

examples/unreachable-runtimes/readme.md

@@ -1,7 +1,7 @@
 ## Knowledge Engine's distributed mode test
 This docker compose project is used to test the Knowledge Engine's behavior in distributed mode when something exceptional happens (i.e. divergence from the happy flow). For example, one participant in the Knowledge Network configured its KER incorrectly and therefore it can reach out, but no one can contact the KER from the outside (via the Inter-KER protocol). Under such circumstances, we want the Knowledge Engine to keep functioning and behave as normal as possible.
 
-To test this, we setup a distributed KER environment with 3 KER+KB combis that exchange data. We have `runtime-1+kb1`, `runtime-2+kb2` and `runtime-3+kb3`. By using the `iptables` tool for `runtime-3` we can simulate a misconfigured KER and test how the other Knowledge Engines behave. Use the following instructions to simulate the misconfigured KER.
+To test this, we setup a distributed KER environment with 3 KER+KB combis that exchange data. We have `runtime-1+kb1`, `runtime-2+kb2` and `runtime-3+kb3`. By using the `iptables` tool for `runtime-3` we can simulate a misconfigured KER and test how the other Knowledge Engines behave. Use the following instructions to simulate the misconfigured KER. In the future we might want to use [Awall](https://github.com/alpinelinux/awall) instead of `iptables`.
 
 Start the docker compose project: `docker compose up -d`
 
@@ -15,7 +15,7 @@ Retrieve the internal IP address of the KB3 (because it needs to always be able
 Make sure runtime-3 is configured to switch between being reachable to being unreachable. First open a shell for runtime-3.
 
 ```
-docker compose exec runtime-3 bash
+docker compose exec runtime-3 sh
 ```
 
 Configure `iptables-legacy` to allow the following packets to go through when we block incoming traffic:
@@ -44,4 +44,10 @@ iptables-legacy -P INPUT DROP
 
 iptables-legacy -P INPUT ACCEPT
 #runtime-3 is now reachable again for other KERs and can also reach the KD and other KERs.
+```
+
+Another scenario that you can check is when other KERs can access runtime-3, but it cannot send back a response to runtime-1. To do this, use the following filewall rule:
+
+```
+iptables-legacy -A OUTPUT -p tcp -d runtime-1 -m state --state NEW -j DROP
 ```

knowledge-directory/pom.xml

@@ -44,9 +44,21 @@
 			<scope>compile</scope>
 		</dependency>
 		<dependency>
-			<groupId>junit</groupId>
-			<artifactId>junit</artifactId>
-			<version>${junit-version}</version>
+			<groupId>org.junit.jupiter</groupId>
+			<artifactId>junit-jupiter-api</artifactId>
+			<version>5.10.2</version>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>org.junit.jupiter</groupId>
+			<artifactId>junit-jupiter-engine</artifactId>
+			<version>5.10.2</version>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>org.junit.jupiter</groupId>
+			<artifactId>junit-jupiter</artifactId>
+			<version>5.10.2</version>
 			<scope>test</scope>
 		</dependency>
 		<dependency>
@@ -229,6 +241,7 @@
 				<configuration>
 					<shadedArtifactAttached>true</shadedArtifactAttached>
 					<shadedClassifierName>with-dependencies</shadedClassifierName>
+					<createDependencyReducedPom>false</createDependencyReducedPom>
 					<transformers>
 						<transformer
 							implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">

reasoner/pom.xml

@@ -19,15 +19,22 @@
 		<dependency>
 			<groupId>org.junit.jupiter</groupId>
 			<artifactId>junit-jupiter-api</artifactId>
-			<version>5.10.1</version>
+			<version>5.10.2</version>
 			<scope>test</scope>
 		</dependency>
 		<dependency>
 			<groupId>org.junit.jupiter</groupId>
 			<artifactId>junit-jupiter-engine</artifactId>
-			<version>5.7.0</version>
+			<version>5.10.2</version>
 			<scope>test</scope>
 		</dependency>
+		<dependency>
+			<groupId>org.junit.jupiter</groupId>
+			<artifactId>junit-jupiter</artifactId>
+			<version>5.10.2</version>
+			<scope>test</scope>
+		</dependency>
+
 		<!-- logging -->
 		<dependency>
 			<groupId>org.slf4j</groupId>

smart-connector-rest-dist/pom.xml

@@ -25,13 +25,19 @@
 		<dependency>
 			<groupId>org.junit.jupiter</groupId>
 			<artifactId>junit-jupiter-api</artifactId>
-			<version>5.10.1</version>
+			<version>5.10.2</version>
 			<scope>test</scope>
 		</dependency>
 		<dependency>
 			<groupId>org.junit.jupiter</groupId>
 			<artifactId>junit-jupiter-engine</artifactId>
-			<version>5.7.0</version>
+			<version>5.10.2</version>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>org.junit.jupiter</groupId>
+			<artifactId>junit-jupiter</artifactId>
+			<version>5.10.2</version>
 			<scope>test</scope>
 		</dependency>
 	</dependencies>
@@ -64,6 +70,7 @@
 				<configuration>
 					<shadedArtifactAttached>true</shadedArtifactAttached>
 					<shadedClassifierName>with-dependencies</shadedClassifierName>
+					<createDependencyReducedPom>false</createDependencyReducedPom>
 					<transformers>
 						<transformer
 							implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">

smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/RestServerHelper.java

@@ -11,14 +11,14 @@
 public class RestServerHelper {
 	private static final Logger LOG = LoggerFactory.getLogger(RestServerHelper.class);
 	private static int WAIT_BEFORE_NEXT_POLL = 300;
-	
+
 	private Thread thread;
 
 	public void start(int port) {
 		var r = new Runnable() {
 			@Override
 			public void run() {
-				RestServer.main(new String[] {String.format("%d", port)});
+				RestServer.main(new String[] { String.format("%d", port) });
 			}
 		};
 		this.thread = new Thread(r);
@@ -44,10 +44,10 @@ public void cleanUp() {
 	}
 
 	private static boolean portAvailable(int port) {
-    try (Socket ignored = new Socket("localhost", port)) {
-        return false;
-    } catch (IOException ignored) {
-        return true;
-    }
+		try (Socket ignored = new Socket("localhost", port)) {
+			return false;
+		} catch (IOException ignored) {
+			return true;
+		}
 	}
 }

smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestRegisterKnowledgeInteraction.java

@@ -6,6 +6,7 @@
 import java.net.URL;
 import java.util.Map;
 
+import org.apache.jena.atlas.logging.Log;
 import org.junit.jupiter.api.AfterAll;
 import org.junit.jupiter.api.BeforeAll;
 import org.junit.jupiter.api.Disabled;
@@ -30,34 +31,19 @@ public void setUpServer() {
 	public void testRegisterKi() throws IOException {
 		URL url = new URL("http://localhost:" + PORT + "/rest");
 
-		HttpTester registerKb = new HttpTester(new URL(url + "/sc"), "POST", "{\"knowledgeBaseId\": \"http://example.com/kb\", \"knowledgeBaseName\": \"KB\", \"knowledgeBaseDescription\": \"KB\"}", Map.of(
-			"Content-Type", "application/json",
-			"Accept", "*/*"
-		));
+		HttpTester registerKb = new HttpTester(new URL(url + "/sc"), "POST",
+				"{\"knowledgeBaseId\": \"http://example.com/kb\", \"knowledgeBaseName\": \"KB\", \"knowledgeBaseDescription\": \"KB\"}",
+				Map.of("Content-Type", "application/json", "Accept", "*/*"));
 		registerKb.expectStatus(200);
 
-		HttpTester registerKiWithName = new HttpTester(
-			new URL(url + "/sc/ki"),
-			"POST",
-			"{\"knowledgeInteractionType\": \"AskKnowledgeInteraction\", \"knowledgeInteractionName\": \"some-name\", \"graphPattern\": \"?a ?b ?c.\"}",
-			Map.of(
-				"Knowledge-Base-Id", "http://example.com/kb",
-				"Content-Type", "application/json",
-				"Accept", "*/*"
-			)
-		);
+		HttpTester registerKiWithName = new HttpTester(new URL(url + "/sc/ki"), "POST",
+				"{\"knowledgeInteractionType\": \"AskKnowledgeInteraction\", \"knowledgeInteractionName\": \"some-name\", \"graphPattern\": \"?a ?b ?c.\"}",
+				Map.of("Knowledge-Base-Id", "http://example.com/kb", "Content-Type", "application/json", "Accept",
+						"*/*"));
 		registerKiWithName.expectStatus(200);
 
-		HttpTester getKiWithName = new HttpTester(
-			new URL(url + "/sc/ki"),
-			"GET",
-			null,
-			Map.of(
-				"Knowledge-Base-Id", "http://example.com/kb",
-				"Content-Type", "application/json",
-				"Accept", "*/*"
-			)
-		);
+		HttpTester getKiWithName = new HttpTester(new URL(url + "/sc/ki"), "GET", null, Map.of("Knowledge-Base-Id",
+				"http://example.com/kb", "Content-Type", "application/json", "Accept", "*/*"));
 		var body = getKiWithName.getBody();
 		assertTrue(body.contains("\"http://example.com/kb/interaction/some-name\""));
 	}

smart-connector-rest-server/pom.xml

@@ -120,13 +120,19 @@
 		<dependency>
 			<groupId>org.junit.jupiter</groupId>
 			<artifactId>junit-jupiter-api</artifactId>
-			<version>5.10.1</version>
+			<version>5.10.2</version>
 			<scope>test</scope>
 		</dependency>
 		<dependency>
 			<groupId>org.junit.jupiter</groupId>
 			<artifactId>junit-jupiter-engine</artifactId>
-			<version>5.7.0</version>
+			<version>5.10.2</version>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>org.junit.jupiter</groupId>
+			<artifactId>junit-jupiter</artifactId>
+			<version>5.10.2</version>
 			<scope>test</scope>
 		</dependency>
 

smart-connector-rest-server/src/main/java/eu/knowledge/engine/rest/api/impl/SmartConnectorLifeCycleApiServiceImpl.java

@@ -145,7 +145,9 @@ public void scPost(@Parameter(description = "", required = true) @NotNull @Valid
 
 		final boolean reasonerEnabled = smartConnector.getReasonerEnabled() == null ? false
 				: smartConnector.getReasonerEnabled();
-
+
+		LOG.info("Creating smart connector with ID {}.", kbId);
+
 		// Tell the manager to create a KB, store it, and have it set up a SC etc.
 		this.manager.createKB(new SmartConnector().knowledgeBaseId(kbId.toString()).knowledgeBaseName(kbName)
 				.knowledgeBaseDescription(kbDescription).leaseRenewalTime(smartConnector.getLeaseRenewalTime())
@@ -154,8 +156,6 @@ public void scPost(@Parameter(description = "", required = true) @NotNull @Valid
 					asyncResponse.resume(Response.ok().build());
 				});
 
-		LOG.info("Creating smart connector with ID {}.", kbId);
-
 		return;
 	}
 

smart-connector/pom.xml

@@ -32,13 +32,19 @@
 		<dependency>
 			<groupId>org.junit.jupiter</groupId>
 			<artifactId>junit-jupiter-api</artifactId>
-			<version>5.10.1</version>
+			<version>5.10.2</version>
 			<scope>test</scope>
 		</dependency>
 		<dependency>
 			<groupId>org.junit.jupiter</groupId>
 			<artifactId>junit-jupiter-engine</artifactId>
-			<version>5.7.0</version>
+			<version>5.10.2</version>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>org.junit.jupiter</groupId>
+			<artifactId>junit-jupiter</artifactId>
+			<version>5.10.2</version>
 			<scope>test</scope>
 		</dependency>