diff --git a/README.md b/README.md index 3b16146..e2b925a 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,120 @@ -[![Build Status](https://img.shields.io/jenkins/build?jobUrl=https%3A%2F%2Fjenkins.iudx.io%2Fjob%2Fiudx%2520gis-interface%2520%28master%29%2520pipeline%2F)](https://jenkins.iudx.io/job/iudx%20gis-interface%20(master)%20pipeline/lastBuild/) -[![Jenkins Coverage](https://img.shields.io/jenkins/coverage/jacoco?jobUrl=https%3A%2F%2Fjenkins.iudx.io%2Fjob%2Fiudx%2520gis-interface%2520%28master%29%2520pipeline%2F)](https://jenkins.iudx.io/job/iudx%20gis-interface%20(master)%20pipeline/lastBuild/jacoco/) -[![Unit Tests](https://img.shields.io/jenkins/tests?jobUrl=https%3A%2F%2Fjenkins.iudx.io%2Fjob%2Fiudx%2520gis-interface%2520%28master%29%2520pipeline%2F&label=unit%20tests)](https://jenkins.iudx.io/job/iudx%20gis-interface%20(master)%20pipeline/lastBuild/testReport/) -[![Security Tests](https://img.shields.io/jenkins/build?jobUrl=https%3A%2F%2Fjenkins.iudx.io%2Fjob%2Fiudx%2520gis-interface%2520%28master%29%2520pipeline%2F&label=security%20tests)](https://jenkins.iudx.io/job/iudx%20gis-interface%20(master)%20pipeline/lastBuild/zap/) -[![Integration Tests](https://img.shields.io/jenkins/build?jobUrl=https%3A%2F%2Fjenkins.iudx.io%2Fjob%2Fiudx%2520gis-interface%2520%28master%29%2520pipeline%2F&label=integration%20tests)](https://jenkins.iudx.io/job/iudx%20gis-interface%20(master)%20pipeline/Integration_20Test_20Report/) - -![IUDX](./readme/iudx.png) +![IUDX](./docs/iudx.png) # iudx-gis-interface The GIS interface is a resource server which defines the interactions between the consumers interested in GIS data and the upstream GIS Servers. - -## Architecture diagram +The gis interface is [IUDXs](https://iudx.org.in) data discovery, data publication. +It allows admins to publish their gis related info in accordance to the IUDX vocabulary annotated meta-data document. +The consumers can access data from the gis interface using HTTPs.

- +

+## Features + +- Provides data access from available resources using standard APIs. +- Integration with authorization server (token introspection) to serve private data as per the access control policies set by the provider. +- Allows admin to publish, edit or delete gis related info as per IUDX data descriptor. +- Secure data access over TLS. +- Scalable, service mesh architecture based implementation using open source components: Vert.X API framework, PostgreSQL for database and RabbitMQ for data broker. +- Hazelcast and Zookeeper based cluster management and service discovery. + +## API Docs +The api docs can be found [here](https://gis.iudx.org.in/apis). + +## Prerequisites +### External Dependencies Installation + +The Gis-interface Server connects with various external dependencies namely +- PostgreSQL +- ImmuDB +- RabbitMQ + +Find the installations of the above along with the configurations to modify the database url, port and associated credentials in the appropriate sections +[here](SETUP.md) + +## Get Started + +### Docker based +1. Install docker and docker-compose +2. Clone this repo +3. Build the images + ` ./docker/build.sh` +4. Modify the `docker-compose.yml` file to map the config file you just created +5. Start the server in production (prod) or development (dev) mode using docker-compose + ` docker-compose up prod ` + + +### Maven based +1. Install java 13 and maven +2. Use the maven exec plugin based starter to start the server + `mvn clean compile exec:java@gis-server` + +### JAR based +1. Install java 11 and maven +2. Set Environment variables +``` +export RS_URL=https:// +export LOG_LEVEL=INFO +``` +3. Use maven to package the application as a JAR + `mvn clean package -Dmaven.test.skip=true` +4. 2 JAR files would be generated in the `target/` directory + - `iudx.gis.interface-cluster-0.0.1-SNAPSHOT-fat.jar` - clustered vert.x containing micrometer metrics + - `iudx.gis.interface-dev-0.0.1-SNAPSHOT-fat.jar` - non-clustered vert.x and does not contain micrometer metrics + +#### Running the clustered JAR + +**Note**: The clustered JAR requires Zookeeper to be installed. Refer [here](https://zookeeper.apache.org/doc/r3.3.3/zookeeperStarted.html) to learn more about how to set up Zookeeper. Additionally, the `zookeepers` key in the config being used needs to be updated with the IP address/domain of the system running Zookeeper. + +The JAR requires 3 runtime arguments when running: + +* --config/-c : path to the config file +* --hostname/-i : the hostname for clustering +* --modules/-m : comma separated list of module names to deploy + +e.g. `java -jar target/iudx.gis.interface-cluster-0.0.1-SNAPSHOT-fat.jar --host $(hostname) -c configs/config.json -m iudx.gis.interface.database.archives.DatabaseVerticle,iudx.gis.interface.authenticator.AuthenticationVerticle +,iudx.gis.interface.metering.MeteringVerticle,iudx.gis.interface.database.postgres.PostgresVerticle` + +Use the `--help/-h` argument for more information. You may additionally append an `RS_JAVA_OPTS` environment variable containing any Java options to pass to the application. + +e.g. +``` +$ export RS_JAVA_OPTS="-Xmx4096m" +$ java $RS_JAVA_OPTS -jar target/iudx.gis.interface-cluster-0.0.1-SNAPSHOT-fat.jar ... +``` + +#### Running the non-clustered JAR +The JAR requires 1 runtime argument when running: + +* --config/-c : path to the config file + +e.g. `java -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4j2LogDelegateFactory -jar target/iudx.gis.interface-dev-0.0.1-SNAPSHOT-fat.jar -c configs/config.json` + +Use the `--help/-h` argument for more information. You may additionally append an `RS_JAVA_OPTS` environment variable containing any Java options to pass to the application. + +e.g. +``` +$ export RS_JAVA_OPTS="-Xmx1024m" +$ java $RS_JAVA_OPTS -jar target/iudx.gis.interface-dev-0.0.1-SNAPSHOT-fat.jar ... +``` + +### Testing + +### Unit tests +1. Run the server through either docker, maven or redeployer +2. Run the unit tests and generate a surefire report + `mvn clean test-compile surefire:test surefire-report:report` +3. Reports are stored in `./target/` + + +## Contributing +We follow Git Merge based workflow +1. Fork this repo +2. Create a new feature branch in your fork. Multiple features must have a hyphen separated name, or refer to a milestone name as mentioned in Github -> Projects +3. Commit to your fork and raise a Pull Request with upstream + +## License +[MIT](./LICENSE.txt) diff --git a/SETUP.md b/SETUP.md new file mode 100644 index 0000000..c427fbb --- /dev/null +++ b/SETUP.md @@ -0,0 +1,231 @@ +SETUP GUIDE +---- + +This document contains the installation and configuration processes +of the external modules of each Verticle in IUDX Gis Interface. + +

+ +

+ +The Gis Interface connects with various external dependencies namely +- `PostgreSQL` : used to store and query data related to + - Token Invalidation + - Fetching data +- `ImmuDB` : used to store metering information +- `RabbitMQ` : used to receive token invalidation info + + + +The Gis Interface also connects with various DX dependencies namely +- Authorization Server : used to download the certificate for token decoding +- Catalogue Server : used to download the list of resources, access policies and query types supported on a resource. + +---- +## Setting up RabbitMQ for IUDX Gis Interface +- Refer to the docker files available [here](https://github.com/datakaveri/iudx-deployment/blob/master/Docker-Swarm-deployment/single-node/databroker) to setup RMQ. + + +In order to connect to the appropriate RabbitMQ instance, required information such as dataBrokerIP, dataBrokerPort etc. should be updated in the DataBrokerVerticle module available in [config-example.json](configs/config-example.json). + +**DataBrokerVerticle** +``` +{ + id": "iudx.gis.server.databroker.DataBrokerVerticle", + "verticleInstances": , + "dataBrokerIP": "localhost", + "dataBrokerPort": , + "dataBrokerVhost": , + "dataBrokerUserName": , + "dataBrokerPassword": , + "dataBrokerManagementPort": , + "connectionTimeout": , + "requestedHeartbeat": , + "handshakeTimeout": , + "requestedChannelMax": , + "networkRecoveryInterval": , + "automaticRecoveryEnabled": "true" +} +``` +--- + +## Setting up PostgreSQL for IUDX Gis Interface +- Refer to the docker files available [here](https://github.com/datakaveri/iudx-deployment/blob/master/Docker-Swarm-deployment/single-node/postgres) to setup PostgreSQL + +**Note** : PostgresQL database should be configured with a RBAC user having CRUD privileges + +In order to connect to the appropriate Postgres database, required information such as databaseIP, databasePort etc. should be updated in the PostgresVerticle module available in [config-example.json](configs/config-example.json). + +**PostgresVerticle** +``` +{ + "id": "iudx.gis.server.database.postgres.PostgresVerticle", + "verticleInstances": , + "databaseIp": "localhost", + "databasePort": , + "databaseName": , + "databaseUserName": , + "databasePassword": , + "poolSize": +} +``` +**DatabaseVerticle** +``` +{ + "id": "iudx.gis.server.database.DatabaseVerticle", + "verticleInstances": , + "databaseIP": "localhost", + "databasePort": , + "databaseName": , + "databaseUserName": , + "databasePassword": , + "dbClientPoolSize": +} +``` + + +#### Schemas for PostgreSQL tables in IUDX Gis Interface +1. **Token Invalidation Table Schema** +``` +CREATE TABLE IF NOT EXISTS revoked_tokens +( + _id uuid NOT NULL, + expiry timestamp with time zone NOT NULL, + created_at timestamp without time zone NOT NULL, + modified_at timestamp without time zone NOT NULL, + CONSTRAINT revoke_tokens_pk PRIMARY KEY (_id) +); +``` + +2. **Gis Table Schema** +``` +CREATE TABLE IF NOT EXISTS gis +( + iudx_resource_id character varying NOT NULL, + url varchar NOT NULL, + isOpen BOOLEAN NOT NULL, + port integer NOT NULL, + created_at timestamp without time zone NOT NULL, + modified_at timestamp without time zone NOT NULL, + username varchar, + password varchar, + tokenurl character varying, + CONSTRAINT gis_pk PRIMARY KEY (iudx_resource_id) +); +``` +---- + +## Setting up ImmuDB for IUDX Gis Interface +- Refer to the docker files available [here](https://github.com/datakaveri/iudx-deployment/blob/master/Docker-Swarm-deployment/single-node/immudb) to setup ImmuDB. + +In order to connect to the appropriate ImmuDB database, required information such as meteringDatabaseIP, meteringDatabasePort etc. should be updated in the MeteringVerticle module available in [config-example.json](configs/config-example.json). + +**MeteringVerticle** + +``` +{ + "id": "iudx.gis.server.metering.MeteringVerticle", + "verticleInstances": , + "meteringDatabaseIP": "localhost", + "meteringDatabasePort": , + "meteringDatabaseName": , + "meteringDatabaseUserName": , + "meteringDatabasePassword": , + "meteringPoolSize": +} +``` + +**Metering Table Schema** +``` +CREATE TABLE IF NOT EXISTS gisauditingtable +( + id uuid NOT NULL, + api varchar NOT NULL, + userid varchar NOT NULL, + epochtime integer NOT NULL, + resourceid varchar NOT NULL, + isotime timestamp with timezone NOT NULL, + providerid varchar NOT NULL, + CONSTRAINT metering_pk PRIMARY KEY (id) +); +``` + +---- + +## Setting up RabbitMQ for IUDX Gis Interface +- Refer to the docker files available [here](https://github.com/datakaveri/iudx-deployment/blob/master/Docker-Swarm-deployment/single-node/databroker) to setup RMQ. + + +In order to connect to the appropriate RabbitMQ instance, required information such as dataBrokerIP, dataBrokerPort etc. should be updated in the DataBrokerVerticle module available in [config-example.json](configs/config-example.json). + +**DataBrokerVerticle** +``` +{ + id": "iudx.gis.server.databroker.DataBrokerVerticle", + "verticleInstances": , + "dataBrokerIP": "localhost", + "dataBrokerPort": , + "dataBrokerVhost": , + "dataBrokerUserName": , + "dataBrokerPassword": , + "dataBrokerManagementPort": , + "connectionTimeout": , + "requestedHeartbeat": , + "handshakeTimeout": , + "requestedChannelMax": , + "networkRecoveryInterval": , + "automaticRecoveryEnabled": +} +``` + +---- +## Connecting with DX Catalogue Interface + +In order to connect to the DX catalogue server, required information such as catServerHost, catServerPort etc. should be updated in the AuthenticationVerticle and ApiServerVerticle modules availabe in [config-example.json](configs/config-example.json). + +**AuthenticationVerticle** +``` +{ + "id": "iudx.gis.server.authenticator.AuthenticationVerticle", + "verticleInstances": , + "host": , + "authServerHost": , + "catServerHost": , + "catServerPort": , + "serverMode": , + "jwtIgnoreExpiry": +} +``` + +**ApiServerVerticle** +``` +{ + "id": "iudx.gis.server.apiserver.ApiServerVerticle", + "ssl": , + "keystore": , + "keystorePassword": , + "verticleInstances": , + "catServerHost": , + "catServerPort": , + "production" : , + "port" : +} +``` + +## Connecting with DX Authorization Server + +In order to connect to the DX authentication server, required information such as authServerHost should be updated in the AuthenticationVerticle module availabe in [config-example.json](configs/config-example.json). +``` +{ + "id": "iudx.gis.server.authenticator.AuthenticationVerticle", + "verticleInstances": , + "host": , + "authServerHost": , + "catServerHost": , + "catServerPort": , + "serverMode": , + "jwtIgnoreExpiry": +} +``` diff --git a/docs/apidoc.html b/docs/apidoc.html new file mode 100644 index 0000000..b760d5b --- /dev/null +++ b/docs/apidoc.html @@ -0,0 +1,24 @@ + + + + IUDX Resource Server API Docs + + + + + + + + + + + + + diff --git a/docs/gis_server_overview.png b/docs/gis_server_overview.png new file mode 100644 index 0000000..e30cfdb Binary files /dev/null and b/docs/gis_server_overview.png differ diff --git a/docs/iudx.png b/docs/iudx.png new file mode 100644 index 0000000..e2d9eb3 Binary files /dev/null and b/docs/iudx.png differ diff --git a/docs/openapi.yaml b/docs/openapi.yaml new file mode 100644 index 0000000..44dbd8d --- /dev/null +++ b/docs/openapi.yaml @@ -0,0 +1,458 @@ +openapi: 3.0.2 + +schemes: + - "http" + - "https" + +produces: + - "application/json" + - "text/plain" + +consumes: + - "application/json" + - "text/plain" + +info: + title: "IUDX Resource Server APIs" + version: 3.0.0 + description: | + # Introduction + + The GIS interface is a resource server which defines the interactions between the consumers interested in GIS data and the upstream GIS Servers. + It allows data *admins* to publish gis related info as per the IUDX . + It enables *Consumers* of datasources to search and query for data using HTTPs APIs. + + # API HTTP Responses + Apart from the response codes specified in each API, the API server may respond with certain 4xx and 5xx error codes which are related to common API Gateway behaviours. + The application should address them accordingly. + + + # Tokens + All the *API* calls to the gis interface should now contain a valid IUDX auth token presented in the query header. To search (or) access a *Secure / Closed* resource, a consumer must get a token for the *Secure / Closed* resource along with the associated constraints from the IUDX Auth Server. For publication of a resource, a consumer must get a token along with the associated constraints from the IUDX Auth Server. + + contact: + name: For support contact IUDX Team at + email: support@iudx.org.in + +tags: + - name: Latest Search + x-displayName: "Latest URL" + description: | + A entity search is used by IUDX Data *Consumers* for querying a resource using a valid IUDX `id`.
+ + - name: Admin API + x-displayName: "Admin Apis" + description: | + This API allows an admin to post, put or delete gis related info as per IUDX . + + +paths: + + /ngsi-ld/v1/entities/{id}: + get: + tags: + - Latest Search + summary: Latest URL + operationId: latest-entities + parameters: + - name: token + in: header + description: A valid IUDX Auth token to process the request.
This field is mandatory for accessing a resource .
If not provided with a valid token, user will be responded with 401 Unauthorized . + required: true + schema: + type: string + maxLength: 512 + pattern: '^[a-zA-Z0-9\/\@\.]*$' + + - name: id + in: path + description: A valid IUDX ID of the resource. + required: true + schema: + type: string + maxLength: 512 + + responses: + 200: + description: 'Successful query' + content: + application/json: + schema: + $ref: '#/components/schemas/entityresponse' + 204: + description: 'Empty Response' + + 400: + content: + application/json: + schema: + "$ref": "#/components/schemas/standardBadRequestErrorResponse" + description: 'Bad query' + + 401: + content: + application/json: + schema: + "$ref": "#/components/schemas/standardInvalidTokenErrorResponse" + description: 'Unauthorized' + + 404: + content: + application/json: + schema: + "$ref": "#/components/schemas/standardResourceNotFoundErrorResponse" + description: 'Not Found' + + + description: | + This API is used to get the data of a resource.
It uses the IUDX ID a.k.a `id` to query the resource server.
To get data a valid IUDX Auth token is mandatory.
The `/entities` end point followed by `id` as a path parameter is used to get the latest data. + + deprecated: false + + + /admin/gis/serverInfo: + post: + tags: + - Admin API + summary: Publish GIS information + operationId: registergisinforequest + parameters: + - name: token + in: header + description: 'IUDX auth token for accessing API.' + required: true + style: simple + schema: + type: string + maxLength: 512 + pattern: '^[a-zA-Z0-9\/\@\.]*$' + + requestBody: + description: '' + content: + application/json: + schema: + $ref: '#/components/schemas/registergisinforequest' + required: true + responses: + 200: + description: 'Successful query' + content: + application/json: + schema: + $ref: '#/components/schemas/registerInformation' + + 400: + content: + application/json: + schema: + "$ref": "#/components/schemas/postInfoBadRequestErrorResponse" + description: 'Bad query' + + 401: + content: + application/json: + schema: + "$ref": "#/components/schemas/standardInvalidTokenErrorResponse" + description: 'Unauthorized' + + description: | + Post Admin API allows an admin to publish the GIS info into the database. Based on the input provided, the API validates the access control of the user with IUDX Auth server to allow or deny access to the API. + deprecated: false + + put: + tags: + - Admin API + summary: Update GIS information + operationId: editGisInfoRequest + parameters: + - name: token + in: header + description: 'IUDX auth token for accessing API.' + required: true + style: simple + schema: + type: string + maxLength: 512 + pattern: '^[a-zA-Z0-9\/\@\.]*$' + + requestBody: + description: '' + content: + application/json: + schema: + $ref: '#/components/schemas/editgisinforequest' + required: true + responses: + 200: + content: + application/json: + schema: + $ref: '#/components/schemas/registerInformation' + description: 'Successful query' + + 400: + content: + application/json: + schema: + "$ref": "#/components/schemas/postInfoBadRequestErrorResponse" + description: 'Bad query' + + 404: + content: + application/json: + schema: + "$ref": "#/components/schemas/putInfoBadRequestErrorResponse" + description: 'Unauthorized' + + + description: | + Put Admin API allows an admin to update the GIS info present. Based on the input provided, the API validates the access control of the user with IUDX Auth server to allow or deny access to the API. + + /admin/gis/serverInfo/{id}: + delete: + tags: + - Admin API + summary: Delete GIS information + operationId: deleteInfo + parameters: + - name: token + in: header + description: 'IUDX auth token for accessing API.' + required: true + style: simple + schema: + type: string + maxLength: 512 + pattern: '^[a-zA-Z0-9\/\@\.]*$' + + - name: id + in: path + description: 'id of the Gis information.' + required: true + style: simple + schema: + type: string + maxLength: 512 + responses: + 200: + description: 'Successful query' + content: + application/json: + schema: + $ref: '#/components/schemas/registerInformation' + + 400: + content: + application/json: + schema: + "$ref": "#/components/schemas/deleteInfoBadRequestErrorResponse" + description: 'Bad query' + + 404: + content: + application/json: + schema: + "$ref": "#/components/schemas/putInfoBadRequestErrorResponse" + description: 'Resource not found' + + description: | + Delete admin API allows an admin to delete Gis information. Based on the input provided, the API validates the access control of the user with IUDX Auth server to allow or deny access to the API. + + +components: + schemas: + + registergisinforequest: + title: registergisinforequest + required: + - entities + type: object + properties: + entities: + type: array + items: + type: string + example: + id: + 'iisc.ac.in/89a36273d77dac4cf38114fca1bbe64392547f86/rs.iudx.io/pune-env-flood/FWR059' + server: URL + server-port: 1234 + isSecure: false + + editgisinforequest: + title: editgisinforequest + required: + - entities + type: object + properties: + entities: + type: array + items: + type: string + example: + id: + 'iisc.ac.in/89a36273d77dac4cf38114fca1bbe64392547f86/rs.iudx.io/pune-env-flood/FWR059' + server: URL + server-port: 1234 + isSecure: true + + entityresponse: + title: entityresponse + type: object + properties: + type: + type: string + title: + type: string + results: + type: object + properties: + id: + type: string + observationDateTime: + type: string + trip_id: + type: string + route_id: + type: string + limit: + type: string + offset: + type: string + totalHits: + type: string + + example: + type: urn:dx:rs:success + title: Successful Operation + results: [ + { + "URL": "https://map.varanasismartcity.gov.in/varanasismartcity" + } + ] + + + registerInformation: + title: registerInformation + required: + - username + - apiKey + - id + - URL + - port + - vHost + type: object + properties: + type: + type: string + title: + type: string + results: + type: object + properties: + username: + type: string + apiKey: + type: string + id: + type: string + URL: + type: string + port: + type: string + vHost: + type: string + example: + type: urn:dx:rs:success + title: Successful Operation + detail: Success + + + + + + standardBadRequestErrorResponse: + title: standardBadRequestErrorResponse + properties: + type: + type: string + title: + type: string + detail: + type: string + example: + type: urn:dx:rs:badRequest + title: Bad Request Data + detail: Empty response | Missing/Invalid parameters + + standardInvalidTokenErrorResponse: + title: standardInvalidTokenErrorResponse + properties: + type: + type: string + title: + type: string + detail: + type: string + example: + type: urn:dx:rs:invalidAuthorizationToken + title: Not Authorized + detail: Not Authorized + + standardResourceNotFoundErrorResponse: + title: standardResourceNotFoundErrorResponse + properties: + type: + type: string + title: + type: string + detail: + type: string + example: + type: urn:dx:rs:resourceNotFound + title: Not Found + detail: Resource Not Found + + + postInfoBadRequestErrorResponse: + title: postInfoBadRequestErrorResponse + properties: + type: + type: string + title: + type: string + detail: + type: string + example: + type: urn:dx:gis:invalidPayloadFormat + title: Bad Request + detail: provided string should respect pattern ^[a-zA-Z0-9.]{4,100}/{1}[a-zA-Z0-9.]{4,100}/{1}[a-zA-Z.]{4,100}/{1}[a-zA-Z-_.]{4,100}/{1}[a-zA-Z0-9-_.]{4,100}$ + + putInfoBadRequestErrorResponse: + title: putInfoBadRequestErrorResponse + properties: + type: + type: string + title: + type: string + detail: + type: string + example: + type: urn:dx:gis:invalidPayloadFormat + title: Bad Request + detail: Update operation failed because passed resource ID does not exist in DB + + deleteInfoBadRequestErrorResponse: + title: deleteInfoBadRequestErrorResponse + properties: + type: + type: string + title: + type: string + detail: + type: string + example: + type: urn:dx:gis:invalidPayloadFormat + title: Bad Request + detail: "Validation error : Invalid id" \ No newline at end of file