Skip to content

Commit

Permalink
remove data loading functions. update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
TangoYankee committed Aug 13, 2024
1 parent da911a6 commit efab79d
Show file tree
Hide file tree
Showing 5 changed files with 23 additions and 96 deletions.
36 changes: 21 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ docker compose up
If you need to install docker compose, follow [these instructions](https://docs.docker.com/compose/install/).

### Configure PostGIS
Enable geospatial features by installing the postgis extension
Enable geospatial features by activating the postgis extension

```sh
npm run pg:configure
Expand All @@ -39,13 +39,6 @@ Structure the database tables with drizzle
npm run drizzle:migrate
```

### Data loading
Load the data into the tables. Under the hood, this uses the Postgres COPY command.

```sh
npm run pg:copy
```

## Local development
The OpenAPI documentation and the API implementation complement each other.
The OpenAPI documentation is written first, defining expectations for endpoints.
Expand Down Expand Up @@ -78,13 +71,6 @@ Generate these resources with:
npm run kubb:generate
```

The console may include the output below. It can be safely ignored.
```console
✖ Something went wrong

Error: The "paths[0]" argument must be of type string. Received undefined
```

Finally, to run the api locally:
```sh
npm run dev
Expand All @@ -104,6 +90,26 @@ If you make changes to any database schema in `src/schema`, it will be necessary
npm run drizzle:generate
```

### Data loading
This API manages its own database schemas (using the drizzle steps listed above). However, the data itself is loaded into the database through a separate ETL repository. The ETL repository is `NYCPlanning/ae-data-flow`.
For local development, the ETL communicates with the Zoning API through a shared docker network. The Zoning API docker compose creates this network, which the ETL repository then looks for and joins when it starts.

#### Troubleshooting the docker network

The docker network should be created automatically when bringing the Zoning API `up` and the ETL should also join it automatically. However, there are two common scenarios that may cause issues.

The ETL cannot find the network:
- Check the configuration of the `compose.yaml` in this repository (NYCPlanning/ae-zoning-api). It should be configured to have a network called `data`.
- Configure the compose to create a `data` network, if it does not already.
- If it does already configure this network, attempt to rebuild the container with `docker compose up --build -d`

The ETL cannot find the Zoning Database on the network:
- In this scenario, the network exists from a previous build. However, the Zoning API is failing to join it when it starts.
- First confirm the network exists.
- Run `docker network ls`
- Confirm there is a network called `ae-zoning-api_data`
- Connect the Zoning API database container to the data network by running `docker network connect ae-zoning-api_data ae-zoning-api-db-1`

## Production builds
Running a production version of the site is a two step process.
First, generate production versions of the OpenAPI documentation and API.
Expand Down
8 changes: 0 additions & 8 deletions db/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,3 @@ RUN apt update
RUN apt install -y postgresql-15-postgis-3

WORKDIR /var/lib/postgresql

ARG STORAGE_URL
ADD --chown=postgres $STORAGE_URL/tables/borough.csv borough.csv
ADD --chown=postgres $STORAGE_URL/tables/land_use.csv land_use.csv
ADD --chown=postgres $STORAGE_URL/tables/tax_lot.csv tax_lot.csv
ADD --chown=postgres $STORAGE_URL/tables/zoning_district.csv zoning_district.csv
ADD --chown=postgres $STORAGE_URL/tables/zoning_district_class.csv zoning_district_class.csv
ADD --chown=postgres $STORAGE_URL/tables/zoning_district_zoning_district_class.csv zoning_district_zoning_district_class.csv
72 changes: 0 additions & 72 deletions db/query/copy.sql

This file was deleted.

1 change: 0 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
"start:prod": "node dist/main",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"pg:configure": "ts-node ./db/query.ts configure",
"pg:copy": "ts-node ./db/query.ts copy",
"drizzle:migrate": "drizzle-kit migrate",
"drizzle:generate": "drizzle-kit generate",
"redoc:build": "redocly build-docs openapi/openapi.yaml --output openapi/index.html",
Expand Down
2 changes: 2 additions & 0 deletions sample.env
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,5 @@ DATABASE_NAME=zoning
DATABASE_PORT=8010
DATABASE_HOST=localhost
DATABASE_ENV=development

STORAGE_URL=

0 comments on commit efab79d

Please sign in to comment.