Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes by create-pull-request action #91

Merged
merged 2 commits into from
Dec 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/dictionary/wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -284,3 +284,10 @@ bigint
dbschema
erDiagram
jsonb
ACCESSIONROUTING
Bigpicture
INGESTROUTING
MAPPINGROUTING
bigpicture
stableIDs
syncapi
6 changes: 4 additions & 2 deletions docs/services/sda.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Repository:
[neicnordic/sensitive-data-archive](https://github.com/neicnordic/sensitive-data-archive)

`sda` repository consists of a suite of services which are part of [NeIC Sensitive Data Archive](https://neic-sda.readthedocs.io/en/latest/) and implements the components required for data submission.
It can be used as part of a [Federated EGA](https://ega-archive.org/federated) or as an isolated Sensitive Data Archive.
It can be used as part of a [Federated EGA](https://ega-archive.org/federated) or as a stand-alone (isolated) Sensitive Data Archive.
`sda` was built with support for both S3 and POSIX storage.

The SDA submission pipeline has four main steps:
Expand All @@ -15,7 +15,9 @@ The SDA submission pipeline has four main steps:
3. [Finalize](finalize.md) associates a stable accessionID with each archive file and backups the file.
4. [Mapper](mapper.md) maps file accessionIDs to a datasetID.

There are also three additional support services:
There are also additional support services:

1. [Intercept](intercept.md) relays messages from `CentralEGA` to the system.
2. [s3inbox](s3inbox.md) proxies uploads to the an S3 compatible storage backend.
3. [sync]() project.
4. [syncapi]() project for mirroring data between two installations of SDA.
69 changes: 69 additions & 0 deletions docs/services/syncapi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# sync-api

The sync-api service is used in the [Bigpicture](https://bigpicture.eu/) project.

## Service Description

The sync service facilitates replication of data and metadata between the nodes in the consortium.

When enabled the service will perform the following tasks:

1. Upon receiving a POST request with JSON data to the `/dataset` route.
1. Parse the JSON blob and validate it against the `file-sync` schema.
2. Build and send messages to start ingestion of files.
3. Build and send messages to assign stableIDs to files.
4. Build and send messages to map files to a dataset.

## Configuration

There are a number of options that can be set for the sync service.
These settings can be set by mounting a yaml-file at `/config.yaml` with settings.

ex.

```yaml
log:
level: "debug"
format: "json"
```

They may also be set using environment variables like:

```bash
export LOG_LEVEL="debug"
export LOG_FORMAT="json"
```

### Service settings

- `SYNC_API_PASSWORD`: password for the API user
- `SYNC_API_USER`: User that will be allowed to send POST requests to the API

### RabbitMQ broker settings

These settings control how sync connects to the RabbitMQ message broker.

- `BROKER_HOST`: hostname of the rabbitmq server
- `BROKER_PORT`: rabbitmq broker port (commonly `5671` with TLS and `5672` without)
- `BROKER_EXCHANGE`: exchange to send messages to
- `BROKER_USER`: username to connect to rabbitmq
- `BROKER_PASSWORD`: password to connect to rabbitmq
- `BROKER_PREFETCHCOUNT`: Number of messages to pull from the message server at the time (default to 2)

The default routing keys for sending ingestion, accession and mapping messages can be overridden by setting the following values:

- `SYNC_API_ACCESSIONROUTING`
- `SYNC_API_INGESTROUTING`
- `SYNC_API_MAPPINGROUTING`

### Logging settings

- `LOG_FORMAT` can be set to “json” to get logs in json format. All other values result in text logging
- `LOG_LEVEL` can be set to one of the following, in increasing order of severity:
- `trace`
- `debug`
- `info`
- `warn` (or `warning`)
- `error`
- `fatal`
- `panic`