quickstart

Schemata - Quickstart

Table of Content (ToC)

• References
• Initialization
  • Java and Maven
  • Data contracts - Schema Quickstart
  • Additional utilities
• Generate Schemata JAR artifact
• Schemata utilities to validate and document the data contracts

Created by gh-md-toc

References

• GitHub - Data Engineering helpers - Data contracts (this project)

Initialization

Java and Maven

• If not already done so, install Java (JDK 17) and Maven. The easiest is to use SDKMan
  • To install SDKMan (behind some corporate firewalls, one may need to setup a proxy with http and https Shell environment variables):

$ curl -s "https://get.sdkman.io" | bash

• Proceed as requested by SDKMan (check that the SDKMan initialization is done properly within the Shell init scripts)

• In order to check the currently installed versions of Java, Maven and so on, use sdk list tool, for instance:
  • Check the installed versions of Java:

$ sdk list java

• Check the installed versions of Maven:

$ sdk list maven

• Install the Amazon Coretto JDK17 (or whatever else JDK you prefer, but it has to be some JDK with version 17):

$ sdk install 17.0.6-amzn

• Install Maven:

$ sdk install 3.9.1

Data contracts - Schema Quickstart

• If not already done so, clone the Data contracts Git repository

• Go into the Data contracts Schemata quickstart directory:

$ pushd ~/dev/infra/data-contracts/schemata/quickstart

Additional utilities

• JQ comes handy to parse JSON structures. It is packages on most of the systems (MacOS, Linux)

Generate Schemata JAR artifact

As of April 2023, Schemata does not release its JAR artifact on public repositories such as Maven Central. Hence, one has to clone the Schemata Git repository and to generate the JAR artifact locally.

• If not already done so, clone the Schemata Git repository:

$ mkdir -p ~/dev/infra && \
  git clone [email protected]:ananthdurai/schemata.git ~/dev/infra/schemata

• Go into the Schemata directory:

  pushd ~/dev/infra/schemata

• Compile and package (generate the JAR artifact for) Schemata:

$ make build-all

• Check that the JAR artifacts have been correctly generated:

$ ls -lFh target/*.jar
-rw-r--r--  1 user  staff 112K Apr 13 16:27 target/original-schemata-1.0.jar
-rw-r--r--  1 user  staff 7.8M Apr 13 16:27 target/schemata-1.0-sources.jar
-rw-r--r--  1 user  staff  20M Apr 13 16:27 target/schemata-1.0.jar

• Leave the Schemata directory:

$ popd

• Go into the Data contracts Schemata quickstart directory:

$ pushd ~/dev/infra/data-contracts/schemata/quickstart

• Copy the just generated Schemata JAR artifact:

$ cp ~/dev/infra/schemata/target/schemata-1.0.jar target/

• Leave the Data contracts Schemata quickstart directory:

$ popd

Schemata utilities to validate and document the data contracts

• Go into the Data contracts Schemata quickstart directory:

$ pushd ~/dev/infra/data-contracts/schemata/quickstart

• Generate the data contract descriptors (they are needed by the Shell scripts and need to be refreshed/re-generated whenever the data contracts change):

$ make all

• Check that the data contract descriptors have been correctly generated:

$ ls -lFh *.desc
-rw-r--r--  1 user  staff 111K Apr 13 17:01 model.desc
$ strings model.desc | head -3
activity.proto

org.examplesB&
org.protocol.schemaB

• Validate the model:

$ ./validate.sh
...
Schema validation success. No error to report

• Score the model:

$ ./score.sh org.examples.User
...

Schemata score for org.examples.User : 0.19

• Document the model
  • Raw output:

$ ./document.sh

• Parsed with JQ:

$ ./document.sh |grep -v "^[0-9]\{2\}"|jq
[
  {
    "name": "org.examples.Category",
    "description": "This is the description of the Category table",
    ...
  }
]