-
Notifications
You must be signed in to change notification settings - Fork 11
tX Development Architecture
Document Status: [Draft|Proposal|Accepted]
This document explains the layout of the translationConvertor (tX) conversion platform and how the components of the system should interact with one another.
If you just want to use the tX API, see tX API Example Usage
Keep reading if you want to contribute to tX.
tX is intended to be a conversion tool for the content in the Door43 Platform. The goal is to support several different input formats, output formats, and resource types.
Development goals are:
- Keep the system modular, in order to:
- Encourage others to contribute and make it simple to do so
- Contain development, testing, and deployment to each individual component
- Constrain feature, bugfixes, and security issues to a smaller codebase for each component
- Continuous Deployment, which means
- Automated testing is required
- Continuous integration is required
- Checks and balances on our process
All code for tX is run by AWS Lambda. Developers use Apex, Travis CI, and Coveralls. This is a "serverless" API.
Modules may be written in any language supported by AWS Lambda (including some that are available via "shimming"). As of July, 2016, this list includes:
- Java (v8)
- Python (v2.7)
- Node.js (v0.10 or v4.3)
- Go lang (any version)
Modules MUST all present an API endpoint that the other components of the system can use. Modules MAY present API endpoints that the public can use.
Every part of tX is broken into components referred to as tX modules
. Each tX module has one or more functions that it provides to the overall system. The list of tX modules is given here, with a full description in its respective heading below.
- tX Management Module
- tX Webhook Module
- tX Conversion Modules, including:
- tx-md2pdf - Converts Markdown to PDF
- tx-md2docx - Converts Markdown to DOCX
- tx-md2html - Converts Markdown to HTML
- tx-usfm2pdf - Converts USFM to PDF
- tx-usfm2docx - Converts USFM to DOCX
- tx-usfm2html - Converts USFM to HTML
The tX Management Module provides access to three functions:
- Maintains the registry for all modules in tX
- Authorization for clients
- Job queue management and rendered file presentation (main public API endpoint)
The tX Webhook Module is a convenience function for our Gogs repositories. This module does the following:
- Accepts the default webhook notification from
git.door43.org
- Gets the data from the repository
- Identifies the Resource Type
- Formats the request
- Interacts with the appropriate HTML conversion module
- Uploads the resulting HTML to the appropriate S3 bucket
Each conversion module accepts a specific type of text format as its input and the module returns a specific type of output document. For example, there is a md2pdf
module that converts Markdown text into a rendered PDF. The conversion modules also require that you specify the resource type, which affects the formatting of the output document.
All requests to modules must be valid JSON and valid JSON will be returned.
There are currently two accepted input format types:
- Markdown -
md
- Unified Standard Format Markers -
usfm
For each type of input format, the following output formats are supported:
- PDF -
pdf
- DOCX -
docx
- HTML -
html
Each of these resource types affects the expected input and the rendered output of the text. The recognized resource types are:
- Open Bible Stories -
obs
- Scripture/Bible -
bible
- translationNotes -
tn
- translationWords -
tw
- translationQuestions -
tq
- translationAcademy -
ta
Conversion modules specify a list of options
that they accept to help format the output document. Every conversion module MUST support these options:
-
"language": "en"
- Defaults toen
if not provided, MUST be a valid IETF code, may affect font used -
"css": "http://some.url/your_custom_css"
- A CSS file that you provide. You can override or extend any of the CSS in the templates with your own values.
Conversion modules MAY support these options:
-
"columns": [1, 2, 3, 4]
- Not available forobs
input -
"page_size": ["A4", "A5", "Letter", "Statement"]
- Not available for HTML output "line_spacing": "100%"
-
"toc_levels": [1, 2, 3, 4, ...]
- To specify how many heading levels you want to appear in your TOC. -
"page_margins": { "top": ".5in","right": ".5in","bottom": ".5in","left": ".5in" }
- If you want to override the default page margins for PDF or DOCX output.
Each module is initially deployed to AWS Lambda via the apex
command. After this, Travis CI is configured to manage continuous deployment of the module (see docs at https://docs.travis-ci.com/user/deployment/lambda).
Continuous deployment of the module should be setup such that:
- the
master
branch is deployed toproduction
whenever it is updated - the
develop
branch is deployed tostaging
whenever it is updated
The deployment process looks like this:
- Code in progress lives in a feature-named branch until the developer is happy and automated tests pass.
- Code is peer-reviewed, then
- Merged into
develop
until automated testing passes and it integrates correctly instaging
. - Merged into
master
which triggers the auto-deployment
Every module (except tx-manager) MUST register itself with tx-manager
. A module MUST provide the following information to tx-manager
:
- Public endpoints
- Private endpoints
A conversion module MUST also provide:
- Input format types accepted
- Output format types accepted
- Resource types accepted
- Conversion options accepted
Example registration for md2pdf
:
Request
POST https://api.door43.org/tx/module
{
"name": "md2pdf",
"resource_types": [ "obs", "bible" ],
"input_format": [ "md" ]
"output_format": [ "pdf" ]
"options": [ "language", "css", "line_spacing" ],
"links": [
{
"href": "/md2pdf",
"rel": "list",
"method": "GET"
},
{
"href": "/md2pdf",
"rel": "create",
"method": "POST"
},
]
}
Response:
201 Created
{
"name": "md2pdf",
"resource_types": [ "obs", "bible" ],
"input_format": [ "md" ]
"output_format": [ "pdf" ]
"options": [ "language", "css", "line_spacing" ],
"links": [
{
"href": "/md2pdf",
"rel": "list",
"method": "GET"
},
{
"href": "/md2pdf",
"rel": "create",
"method": "POST"
},
]
}