Topcoder TaaS API

Tech Stack

Local Setup

Requirements

Steps to run locally

📦 Install npm dependencies
```
npm install
```
⚙ Local config
1. In the taas-apis root directory create .env file with the next environment variables. Values for Auth0 config should be shared with you on the forum.
```
# Auth0 config
AUTH0_URL=
AUTH0_AUDIENCE=
AUTH0_AUDIENCE_UBAHN=
AUTH0_CLIENT_ID=
AUTH0_CLIENT_SECRET=

# Locally deployed services (via docker-compose)
ES_HOST=dockerhost:9200
DATABASE_URL=postgres://postgres:postgres@dockerhost:5432/postgres
BUSAPI_URL=http://dockerhost:8002/v5
```
  - Values from this file would be automatically used by many npm commands.
  - ⚠️ Never commit this file or its copy to the repository!
2. Set dockerhost to point the IP address of Docker. Docker IP address depends on your system. For example if docker is run on IP 127.0.0.1 add a the next line to your /etc/hosts file:
```
127.0.0.1       dockerhost
```
  Alternatively, you may update .env file and replace dockerhost with your docker IP address.

🚢 Start docker-compose with services which are required to start Topcoder TaaS API locally

npm run services:up

Wait until all containers are fully started. As a good indicator, wait until taas-es-processor successfully started by viewing its logs:

npm run services:logs -- -f taas-es-processor

Click to see a good logs example

first it would be waiting for kafka-client to create all the required topics and exit, you would see:

tc-taas-es-processor  | Waiting for kafka-client to exit....

after that, taas-es-processor would be started itself. Make sure it successfully connected to Kafka, you should see 9 lines with text Subscribed to taas.:

tc-taas-es-processor  | 2021-01-22T14:27:48.971Z DEBUG no-kafka-client Subscribed to taas.jobcandidate.create:0 offset 0 leader kafka:9093
tc-taas-es-processor  | 2021-01-22T14:27:48.972Z DEBUG no-kafka-client Subscribed to taas.job.create:0 offset 0 leader kafka:9093
tc-taas-es-processor  | 2021-01-22T14:27:48.972Z DEBUG no-kafka-client Subscribed to taas.resourcebooking.delete:0 offset 0 leader kafka:9093
tc-taas-es-processor  | 2021-01-22T14:27:48.973Z DEBUG no-kafka-client Subscribed to taas.jobcandidate.delete:0 offset 0 leader kafka:9093
tc-taas-es-processor  | 2021-01-22T14:27:48.974Z DEBUG no-kafka-client Subscribed to taas.jobcandidate.update:0 offset 0 leader kafka:9093
tc-taas-es-processor  | 2021-01-22T14:27:48.975Z DEBUG no-kafka-client Subscribed to taas.resourcebooking.create:0 offset 0 leader kafka:9093
tc-taas-es-processor  | 2021-01-22T14:27:48.976Z DEBUG no-kafka-client Subscribed to taas.job.delete:0 offset 0 leader kafka:9093
tc-taas-es-processor  | 2021-01-22T14:27:48.977Z DEBUG no-kafka-client Subscribed to taas.job.update:0 offset 0 leader kafka:9093
tc-taas-es-processor  | 2021-01-22T14:27:48.978Z DEBUG no-kafka-client Subscribed to taas.resourcebooking.update:0 offset 0 leader kafka:9093
tc-taas-es-processor  | 2021-01-22T14:27:48.979Z DEBUG no-kafka-client Subscribed to taas.workperiod.create:0 offset 0 leader kafka:9093
tc-taas-es-processor  | 2021-01-22T14:27:48.980Z DEBUG no-kafka-client Subscribed to taas.workperiod.update:0 offset 0 leader kafka:9093
tc-taas-es-processor  | 2021-01-22T14:27:48.981Z DEBUG no-kafka-client Subscribed to taas.workperiod.delete:0 offset 0 leader kafka:9093

If you want to learn more about docker-compose configuration

see more details here

This docker-compose file starts the next services:

Service	Name	Port
PostgreSQL	postgres	5432
Elasticsearch	elasticsearch	9200
Zookeeper	zookeeper	2181
Kafka	kafka	9092
tc-bus-api	tc-bus-api	8002
taas-es-processor	taas-es-processor	5000

as many of the Topcoder services in this docker-compose require Auth0 configuration for M2M calls, our docker-compose file passes environment variables AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_URL, AUTH0_AUDIENCE, AUTH0_PROXY_SERVER_URL to its containers. docker-compose takes them from .env file if provided.
docker-compose automatically would create Kafka topics which are used by taas-es-processor listed in local/kafka-client/topics.txt.
To view the logs from any container inside docker-compose use the following command, replacing SERVICE_NAME with the corresponding value under the Name column in the above table:
```
npm run services:log -- -f SERVICE_NAME
```
If you want to modify the code of any of the services which are run inside this docker-compose file, you can stop such service inside docker-compose by command docker-compose -f local/docker-compose.yml stop <SERVICE_NAME> and run the service separately, following its README file.

NOTE: In production these dependencies / services are hosted & managed outside Topcoder TaaS API.

♻ Init DB, ES
```
npm run local:init
```
This command will do 3 things:
- create Database tables (drop if exists)
- create Elasticsearch indexes (drop if exists)
- import demo data to Database and index it to ElasticSearch (clears any existent data if exist)
🚀 Start Topcoder TaaS API
```
npm run dev
```
Runs the Topcoder TaaS API using nodemon, so it would be restarted after any of the files is updated. The Topcoder TaaS API will be served on http://localhost:3000.

NPM Commands

Command	Description
`npm run lint`	Check for for lint errors.
`npm run lint:fix`	Check for for lint errors and fix error automatically when possible.
`npm run build`	Build source code for production run into `dist` folder.
`npm run start`	Start app in the production mode from prebuilt `dist` folder.
`npm run dev`	Start app in the development mode using `nodemon`.
`npm run test`	Run tests.
`npm run init-db`	Initializes Database.
`npm run create-index`	Create Elasticsearch indexes. Use `-- --force` flag to skip confirmation
`npm run delete-index`	Delete Elasticsearch indexes. Use `-- --force` flag to skip confirmation
`npm run data:import <filePath>`	Imports data into ES and db from filePath (`./data/demo-data.json` is used as default). Use `-- --force` flag to skip confirmation
`npm run data:export <filePath>`	Exports data from ES and db into filePath (`./data/demo-data.json` is used as default). Use `-- --force` flag to skip confirmation
`npm run index:all`	Indexes all data from db into ES. Use `-- --force` flag to skip confirmation
`npm run index:jobs <jobId>`	Indexes job data from db into ES, if jobId is not given all data is indexed. Use `-- --force` flag to skip confirmation
`npm run index:job-candidates <jobCandidateId>`	Indexes job candidate data from db into ES, if jobCandidateId is not given all data is indexed. Use `-- --force` flag to skip confirmation
`npm run index:resource-bookings <resourceBookingsId>`	Indexes resource bookings data from db into ES, if resourceBookingsId is not given all data is indexed. Use `-- --force` flag to skip confirmation
`npm run index:work-periods <workPeriodId>`	Indexes work periods data from db into ES, if workPeriodId is not given all data is indexed. Use `-- --force` flag to skip confirmation
`npm run services:up`	Start services via docker-compose for local development.
`npm run services:down`	Stop services via docker-compose for local development.
`npm run services:logs -- -f <service_name>`	View logs of some service inside docker-compose.
`npm run local:init`	Recreate Database and Elasticsearch indexes and populate demo data for local development (removes any existent data).
`npm run local:reset`	Recreate Database and Elasticsearch indexes (removes any existent data).
`npm run cov`	Code Coverage Report.
`npm run migrate`	Run any migration files which haven't run yet.
`npm run migrate:undo`	Revert most recent migration.

Import and Export data

📤 Export data

To export data to the default file data/demo-data.json, run:

npm run data:export

If you want to export data to another file, run:

npm run data:export -- --file path/to-file.json

List of models that will be exported are defined in scripts/data/exportData.js.

📥 Import data

⚠️ This command would clear any existent data in DB and ES before importing.

During importing, data would be first imported to the database, and after from the database it would be indexed to the Elasticsearch index.

To import data from the default file data/demo-data.json, run:

npm run data:import

If you want to import data from another file, run:

npm run data:import -- --file path/to-file.json

List of models that will be imported are defined in scripts/data/importData.js.

Kafka commands

If you've used docker-compose with the file local/docker-compose.yml during local setup to spawn kafka & zookeeper, you can use the following commands to manipulate kafka topics and messages: (Replace TOPIC_NAME with the name of the desired topic)

Create Topic

docker exec tc-taas-kafka /opt/kafka/bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic TOPIC_NAME

List Topics

docker exec tc-taas-kafka /opt/kafka/bin/kafka-topics.sh --list --zookeeper zookeeper:2181

Watch Topic

docker exec tc-taas-kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TOPIC_NAME

Post Message to Topic (from stdin)

docker exec -it tc-taas-kafka /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TOPIC_NAME

Enter or copy/paste the message into the console after starting this command.

DB Migration

npm run migrate: run any migration files which haven't run yet.
npm run migrate:undo: revert most recent migration.

Configuration for migration is at ./config/config.json.

The following parameters can be set in the config file or via env variables:

username: set via env DB_USERNAME; datebase username
password: set via env DB_PASSWORD; datebase password
database: set via env DB_NAME; datebase name
host: set via env DB_HOST; datebase host name

Testing

Run npm run test to execute unit tests
Run npm run cov to execute unit tests and generate coverage report.

📋 Code Guidelines

General Requirements

Split code into reusable methods where applicable.
Lint should pass.

Documentation and Utils

When we add, update or delete models and/or endpoints we have to make sure that we keep documentation and utility scripts up to date.

Swagger
Postman
ES Mapping
- Update mapping definitions for ElasticSearch indexes inside both repositories taas-apis and taas-es-processor.
Reindex
- NPM command index:all should re-index data in all ES indexes.
- There should be an individual NPM command index:* which would re-index data only in one ES index.
Import/Export
- NPM commands data:import and data:export should support importing/exporting data from/to all the models.
Create/Delete Index
- NPM commands create-index and delete-index should support creating/deleting all the indexes.
DB Migration
- If there are any updates in DB schemas, create a DB migration script inside migrations folder which would make any necessary updates to the DB schema.
- Test, that when we migrate DB from the previous state using npm run migrate, we get exactly the same DB schema as if we create DB from scratch using command npm run init-db force.

Name		Name	Last commit message	Last commit date
Latest commit History 274 Commits
.circleci		.circleci
config		config
data		data
docker		docker
docs		docs
local		local
migrations		migrations
scripts		scripts
src		src
test/unit		test/unit
.dockerignore		.dockerignore
.gitignore		.gitignore
.nvmrc		.nvmrc
README.md		README.md
app-constants.js		app-constants.js
app-routes.js		app-routes.js
app.js		app.js
build.sh		build.sh
docker-compose.yml		docker-compose.yml
package-lock.json		package-lock.json
package.json		package.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Topcoder TaaS API

Tech Stack

Local Setup

Requirements

Steps to run locally

NPM Commands

Import and Export data

📤 Export data

📥 Import data

Kafka commands

Create Topic

List Topics

Watch Topic

Post Message to Topic (from stdin)

DB Migration

Testing

📋 Code Guidelines

General Requirements

Documentation and Utils

About

Releases

Packages

Languages

fikzzzy/taas-apis

Folders and files

Latest commit

History

Repository files navigation

Topcoder TaaS API

Tech Stack

Local Setup

Requirements

Steps to run locally

NPM Commands

Import and Export data

📤 Export data

📥 Import data

Kafka commands

Create Topic

List Topics

Watch Topic

Post Message to Topic (from stdin)

DB Migration

Testing

📋 Code Guidelines

General Requirements

Documentation and Utils

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages