Looking for the easiest way to start Scorpio? This is it.
curl https://raw.githubusercontent.com/ScorpioBroker/ScorpioBroker/development/docker-compose-aaio.yml sudo docker-compose -f docker-compose-aaio.yml up
The easiest way to start Scorpio is to use docker-compose. We provide 2 main docker-compose files which rely on dockerhub. docker-compose-aaio.yml and docker-compose-dist.yml. You can use this files directly as they are to start Scorpio When you want to run Scorpio in the distributed variant exchange the yml file in the command above.
AAIO here stands for almost all in one. In this variant the core components of Scorpio and the Spring Cloud components are started within one container. Additional containers are only Kafka and Postgres. For testing and small to medium size deployments this is most likely what you want to use.
In this variant each Scorpio component is started in a different container. This makes it highly flexible and allows you to replace individual components or to start new instances of some core components.
There are multiple ways to enter environment variables into docker. We will not got through all of them but only through the docker-compose files. However the scorpio relevant parts apply to all these variants. Configuration of Scorpio is done via the Spring Cloud configuration system. For a complete overview of the used parameters and the default values have a look at the application.yml for the AllInOneRunner here, https://github.com/ScorpioBroker/ScorpioBroker/blob/development/AllInOneRunner/src/main/resources/application-aaio.yml. To provide a new setting you can provide those via an environment entry in the docker-compose file. The variable we want to set is called spring_args. Since we only want to set this option for the Scorpio container we make it a sub part of the Scorpio Container entry like this
scorpio: image: scorpiobroker/scorpio:scorpio-aaio_1.0.0 ports: - "9090:9090" depends_on: - kafka - postgres environment: spring_args: --maxLimit=1000
With this we would set the maximum limit for a query reply to 1000 instead of the default 500.
Some docker containers can be quite noisy and you don't want all of that output. The easy solution is to add this
logging: driver: none
in the docker-compose file to respective container config. E.g. to make Kafka quite.
kafka: image: wurstmeister/kafka hostname: kafka ports: - "9092" environment: KAFKA_ADVERTISED_HOST_NAME: kafka KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_PORT: 9092 KAFKA_LOG_RETENTION_MS: 10000 KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 5000 volumes: - /var/run/docker.sock:/var/run/docker.sock depends_on: - zookeeper logging: driver: none
Scorpio uses the Spring Cloud/Boot configuration system. This is done via the application.yml files in the corresponding folders. The AllInOneRunner has a complete set of all available configuration options in them.
Those can be overwriten via the command line or in the docker case as described above.
Config Option | Description | Default Value |
---|---|---|
atcontext.url | the url to be used for the internal context server | http://localhost:9090/ngsi-ld/contextes/ |
bootstrap.servers | the host and port of the internal kafka | kafka:9092 (default used for docker) |
broker.id a | unique id for the broker. needed for federation | Broker1 |
broker.parent. location.url | url for the parent broker in a federation setup | SELF (meaning no federation) |
broker. geoCoverage | GeoJSON description of the coverage. used for registration in a federation setup. | empty |
defaultLimit | The default limit for a query if no limit is provided | 50 |
maxLimit | The maximum number of results in a query | 500 |
reader.datasource .password | If you change the postgres setup here you set the password | ngb |
reader.datasource .url | JDBC URL to postgres | jdbc:postgresql://postgres:5432/ngb?ApplicationName=ngb_storagemanager_reader |
reader.datasource .username | username for the postgres db | ngb |
writer.datasource .password | If you change the postgres setup here you set the password | ngb |
writer.datasource .url | JDBC URL to postgres | jdbc:postgresql://postgres:5432/ngb?ApplicationName=ngb_storagemanager_writer |
writer.datasource .username | username for the postgres db | ngb |
spring.datasource .password | Same as above but used by flyway for database migration | ngb |
spring.datasource .url | Same as above but used by flyway for database migration | jdbc:postgresql://postgres:5432/ngb?ApplicationName=ngb_storagemanager_writer |
spring.datasource .username | Same as above but used by flyway for database migration | ngb |
Scorpio is developed in Java using SpringCloud as microservice framework
and Apache Maven as build tool. Some of the tests require a running
Apache Kafka messagebus (further instruction are in the Setup chapter).
If you want to skip those tests you can run
mvn clean package -DskipTests
to just build the individual
microservices.
Further down this document you will get exact build commands/arguments for the different flavors. This part will give you an overview on how the different arguments work.
There currently three available Maven build profiles
If you provide no -P argument Maven will produce individual jar files for the microservices and the AllInOneRunner with each "full" microservice packaged (this will result in ca. 500 MB size for the AllInOneRunner)
This will trigger the Maven to build docker containers for each microservice.
This will trigger the Maven to build one docker container, containing the AllInOneRunner and the spring cloud components (eureka, configserver and gateway)
These arguments are provided via -D in the command line.
Generally recommended if you want to speed up the build or you don't have a kafka instance running, which is required by some of the tests.
This is a special argument for the Scorpio build. This argument will disable springs repacking for the individual microservices and will allow for a smaller AllInOneRunner jar file. This argument shoulnd ONLY be used in combination with the docker-aaio profile.
Spring supports also profiles which can be activated when launching a jar file. Currently there 3 profiles actively used in Scorpio. The default profiles assume the default setup to be a individual microservices. The exception is the AllInOneRunner which as default assumes to be running in the docker-aaio setup.
Currently you should be able to run everything with a default profile except the gateway in combination with the AllInOneRunner. In order to use these two together you need to start the gateway with the aaio spring profile. This can be done by attaching this to your start command -Dspring.profiles.active=aaio.
Additonally some components have a dev profile available which is purely meant for development purposes and should only be used for such.
Scorpio requires two components to be installed.
Please download the Postgres DB and the Postgis extension and follow the instructions on the websites to set them up.
Scorpio has been tested and developed with Postgres 10.
The default username and password which Scorpio uses is "ngb". If you want to use a different username or password you need to provide them as parameter when starting the StorageManager and the RegistryManager.
e.g.
java -jar Storage/StorageManager/target/StorageManager-<VERSIONNUMBER>-SNAPSHOT.jar --reader.datasource.username=funkyusername --reader.datasource.password=funkypassword
OR
java -jar Registry/RegistryManager/target/RegistryManager-<VERSIONNUMBER>-SNAPSHOT.jar --spring.datasource.username=funkyusername --spring.datasource.password=funkypassword
Don't forget to create the corresponding user ("ngb" or the different username you chose) in postgres. It will be used by the SpringCloud services for database connection. While in terminal, log in to the psql console as postgres user:
sudo -u postgres psql
Then create a database "ngb":
postgres=# create database ngb;
Create a user "ngb" and make him a superuser:
postgres=# create user ngb with encrypted password 'ngb';
postgres=# alter user ngb with superuser;
Grant privileges on database:
postgres=# grant all privileges on database ngb to ngb;
Also create an own database/schema for the Postgis extension:
postgres=# CREATE DATABASE gisdb;
postgres=# \connect gisdb;
postgres=# CREATE SCHEMA postgis;
postgres=# ALTER DATABASE gisdb SET search_path=public, postgis, contrib;
postgres=# \connect gisdb;
postgres=# CREATE EXTENSION postgis SCHEMA postgis;
Scorpio uses Apache Kafka for the communication between the microservices.
Scorpio has been tested and developed with Kafka version 2.12-2.1.0
Please download Apache Kafka and follow the instructions on the website.
In order to start kafka you need to start two components: Start zookeeper with
<kafkafolder>/bin/[Windows]/zookeeper-server-start.[bat|sh] <kafkafolder>/config/zookeeper.properties
Start kafkaserver with
<kafkafolder>/bin/[Windows]/kafka-server-start.[bat|sh] <kafkafolder>/config/server.properties
For more details please visit the Kafka website.
The current maven build supports two types of docker container generations from the build using maven profiles to trigger it.
The first profile is called 'docker' and can be called like this
sudo mvn clean package -DskipTests -Pdocker
this will generate individual docker containers for each micro service.
The corresponding docker-compose file is docker-compose-dist.yml
The second profile is called 'docker-aaio' (for almost all in one). This will generate one single docker container for all components the broker except the kafka message bus and the postgres database.
To get the aaio version run the maven build like this
sudo mvn clean package -DskipTests -DskipDefault -Pdocker-aaio
The corresponding docker-compose file is docker-compose-aaio.yml
To start the docker container please use the corresponding docker-compose files. I.e.
sudo docker-composer -f docker-compose-aaio.yml up
to stop the container properly execute
sudo docker-composer -f docker-compose-aaio.yml down
The Kafka docker container requires you to provide the environment
variable KAFKA_ADVERTISED_HOST_NAME
. This has to be changed in the
docker-compose files to match your docker host IP. You can use
127.0.0.1
however this will disallow you to run Kafka in a cluster
mode.
For further details please refer to https://hub.docker.com/r/wurstmeister/kafka
If you want to have the build of the jars separated from the docker build you need to provide certain VARS to docker. The following list shows all the vars and their intended value if you run docker build from the root dir
BUILD_DIR_ACS = Core/AtContextServer
BUILD_DIR_SCS = SpringCloudModules/config-server
BUILD_DIR_SES = SpringCloudModules/eureka
BUILD_DIR_SGW = SpringCloudModules/gateway
BUILD_DIR_HMG = History/HistoryManager
BUILD_DIR_QMG = Core/QueryManager
BUILD_DIR_RMG = Registry/RegistryManager
BUILD_DIR_EMG = Core/EntityManager
BUILD_DIR_STRMG = Storage/StorageManager
BUILD_DIR_SUBMG = Core/SubscriptionManager
JAR_FILE_BUILD_ACS = AtContextServer-${project.version}.jar
JAR_FILE_BUILD_SCS = config-server-${project.version}.jar
JAR_FILE_BUILD_SES = eureka-server-${project.version}.jar
JAR_FILE_BUILD_SGW = gateway-${project.version}.jar
JAR_FILE_BUILD_HMG = HistoryManager-${project.version}.jar
JAR_FILE_BUILD_QMG = QueryManager-${project.version}.jar
JAR_FILE_BUILD_RMG = RegistryManager-${project.version}.jar
JAR_FILE_BUILD_EMG = EntityManager-${project.version}.jar
JAR_FILE_BUILD_STRMG = StorageManager-${project.version}.jar
JAR_FILE_BUILD_SUBMG = SubscriptionManager-${project.version}.jar
JAR_FILE_RUN_ACS = AtContextServer.jar
JAR_FILE_RUN_SCS = config-server.jar
JAR_FILE_RUN_SES = eureka-server.jar
JAR_FILE_RUN_SGW = gateway.jar
JAR_FILE_RUN_HMG = HistoryManager.jar
JAR_FILE_RUN_QMG = QueryManager.jar
JAR_FILE_RUN_RMG = RegistryManager.jar
JAR_FILE_RUN_EMG = EntityManager.jar
JAR_FILE_RUN_STRMG = StorageManager.jar
JAR_FILE_RUN_SUBMG = SubscriptionManager.jar
After the build start the individual components as normal Jar files.
Start the SpringCloud services by running
java -jar SpringCloudModules/eureka/target/eureka-server-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar SpringCloudModules/gateway/target/gateway-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar SpringCloudModules/config-server/target/config-server-<VERSIONNUMBER>-SNAPSHOT.jar
Start the broker components
java -jar Storage/StorageManager/target/StorageManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar Core/QueryManager/target/QueryManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar Registry/RegistryManager/target/RegistryManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar Core/EntityManager/target/EntityManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar History/HistoryManager/target/HistoryManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar Core/SubscriptionManager/target/SubscriptionManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar Core/AtContextServer/target/AtContextServer-<VERSIONNUMBER>-SNAPSHOT.jar
All configurable options are present in application.properties files. In
order to change those you have two options. Either change the properties
before the build or you can override configs by add
--<OPTION_NAME>=<OPTION_VALUE)
e.g.
java -jar Storage/StorageManager/target/StorageManager-<VERSIONNUMBER>-SNAPSHOT.jar --reader.datasource.username=funkyusername --reader.datasource.password=funkypassword`
You can enable cors support in the gateway by providing these configuration options - gateway.enablecors - default is False. Set to true for general enabling - gateway.enablecors.allowall - default is False. Set to true to enable CORS from all origins, allow all headers and all methods. Not secure but still very often used. - gateway.enablecors.allowedorigin - A comma separated list of allowed origins - gateway.enablecors.allowedheader - A comma separated list of allowed headers - gateway.enablecors.allowedmethods - A comma separated list of allowed methods - gateway.enablecors.allowallmethods - default is False. Set to true to allow all methods. If set to true it will override the allowmethods entry
When starting the eureka-server you may facing the
java.lang.TypeNotPresentException: Type javax.xml.bind.JAXBContext not
present exception. It's very likely that you are running Java 11 on
your machine then. Starting from Java 9 package javax.xml.bind
has
been marked deprecated and was finally completely removed in Java 11.
In order to fix this issue and get eureka-server running you need to
manually add below JAXB Maven dependencies to
ScorpioBroker/SpringCloudModules/eureka/pom.xml
before starting:
...
<dependencies>
...
<dependency>
<groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-core</artifactId>
<version>2.3.0.1</version>
</dependency>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.3.1</version>
</dependency>
<dependency>
<groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-impl</artifactId>
<version>2.3.1</version>
</dependency>
...
</dependencies>
...
This should be fixed now using conditional dependencies.