Support to cloud deployment

[robsgnao]

ETF Improvement Proposal (EIP)

Background and Motivation:

Current JRC Hosting environment does not offer suffient capacity to satisfy a high demand while running ETF in production, lacking the possibility to scale-up and scale down resources to cope with variable workload patterns, as well as lacking capacity to upload and store large volumes od data (e.g. GML files) to support data validation process.

Proposed change

Current container solution may be used as a starting point, extending ETF functionality to support horizonthal scaling:

• Validation results should be persisted either on DB or S3, outside the container
• Hashes should be generated ouside each container (e.g. at DBlevel) level to avoid collisions
• Move from embedded BaseX to Relational or NoSQL available on AWS offering for data which shall be persisted (not for data which can be re-created)
• If needed (still need to check the code), de-couple the persistency layer from the business logic, to handle differentiations in the storage solutions

Alternatives

Adapt current container solution to provide maximum vertical scalability, e.g. by moving to an Enterprise Java Application Server.

Funding

JRC will be ready to fund, in part or maybe in full, within its current development contract.

Additional information

• Develop these changes as “configurable items” in the code, supporting use of ETF-Webapp in both Cloud and Traditional environments (current layout)
• Shorten as much as possible the time needed to deploy a new container, in order to respond to peaks with the shortest waiting time for users

[cportele]

@jonherrmann: TC to review options for cloud-native storage options for test objects, reports, attachments, etc. and report back to SG.

@robsgnao: Provide a presentation with additional information.

See minutes of the 3rd SG meeting.

[robsgnao]

2018-08-07_ETF_Cloud_Hosting_and_Deployment.pdf
Added presentation from Bilbomatica, Guadaltel and Geograma: possible step-wiseapproach to ETF Cloud hosting.

[jonherrmann]

• Validation results should be persisted either on DB or S3, outside the container
• Move from embedded BaseX to Relational or NoSQL available on AWS offering for data which shall be persisted (not for data which can be re-created)

A new S3 based database could be realized which could replace the BaseX data storage in AWS deployments. The BaseX based storage automatically resolves the references when querying a single item and converts the output to an appropriate JSON, XML format or Java object. In the S3 DB, these formats could already be stored during the initialization in corresponding S3 objects. The references to other objects could be directly stored into the S3 object properties. Full HTML Test reports could be stored as well.

• Hashes should be generated ouside each container (e.g. at DBlevel) level to avoid collisions

UUIDs could also be generated by each instance, by concatenating a timestamp plus a random part. The hashed MAC-Address could be included as well.

• If needed (still need to check the code), de-couple the persistency layer from the business logic, to handle differentiations in the storage solutions

The database contains only functionalities that it needs to perform its tasks as an XML database. For the sake of simplicity, the reports have also been integrated into the database. If an S3 database were implemented, this would be a building block that would move from the database into its own module. The coupling is quite loose and the module could be replaced by another one.

This EIP could be implemented.

[jenriquesoriano]

To do: specify phases and experiments for the cloud deployment

[carlospzurita]

The phases and experiments for this EIP can be these:

1. Deploy on a single EC2 instance and ensure ETF works properly
2. Develop the module to decouple the storage from the service, which will act as an interface.
3. Once given an S3 instance, connect this interface to AWS for retrieving and persisting files
4. Carry out stress tests to check if one instance of the service, with the external storage, is sufficient for running the service
5. If needed add capabilities on the ETF cloud deployment to escalate and create instances on demand.
6. Develop the necessary changes to match instances and persistence

[MarcoMinghini]

After the above discussion on cloud deployment, we progressed with a different AWS-based architecture, so this EIP is no longer relevant. Unless there are different views, we propose to close it.