Skip to content

Commit

Permalink
Merge pull request geoserver#417 from groldan/dev_compose_folder
Browse files Browse the repository at this point in the history 
Move development docker compose files to ./compose/*yml and improve documentation
  • Loading branch information
@groldan
groldan authored Jan 28, 2024
2 parents b6edaf1 + fe11544 commit 9a20aff
Show file tree
Hide file tree
Showing 17 changed files with 647 additions and 531 deletions.
179 changes: 117 additions & 62 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,6 @@ This project is an opinionated effort to split *GeoServer*'s geospatial services

As such, it builds on top of existing *GeoServer* software components, adapting and/or extending them in an attempt to achieve functional decomposition by business capability; which roughly means each OWS service, the Web UI, the REST API, and probably other components such as the *Catalog and Configuration subsystem*, become self-contained, individually deployable and scalable micro-services.

## Quick start

You can easily deploy and test locally GeoServer Cloud after cloning this repository on your computer by running the following command:

```bash
docker-compose -f docker-compose.yml -f docker-compose-shared_datadir.yml up -d
```

This will start a GeoServer Cloud stack using a file based backend as catalog system.

Browse to [http://localhost:9090/geoserver/cloud](http://localhost:9090/geoserver/cloud) to access the GeoServer UI.


## Architecture

The following diagram depicts the system's general architecture.
Expand Down Expand Up @@ -78,14 +65,41 @@ OAuth is available by using the geOrchestra Gateway in replacement of the GeoSer

*GeoServer Cloud* licensed under the [GPLv2](LICENSE.txt).

## Distribution and deployment

Docker images for all the services are available on DockerHub, under the [GeoServer Cloud organization](https://hub.docker.com/u/geoservercloud/).

You can find production-suitable deployment files for docker-compose and podman under the [docs/deploy](docs/deploy) folder.

Also, a base Helm chart and examples for Kubernetes is available at the [camptocamp/helm-geoserver-cloud](https://github.com/camptocamp/helm-geoserver-cloud) Github repository.

## Contributing

Please read [the contribution guidelines](CONTRIBUTING.md) before contributing pull requests to the GeoServer Cloud project.

Follow the [developer's guide](docs/develop/index.md) to know more about the project's technical details.

## Status

`v1.5.2` released against GeoServer `2.24.1`.

Read the [changelog](https://github.com/geoserver/geoserver-cloud/releases/) for more information.

## Bugs

*GeoServer Cloud*'s issue tracking is at this [GitHub](https://github.com/geoserver/geoserver-cloud/issues) repository.

## Roadmap

Follow the development progress on these [GitHub Kanban boards](https://github.com/geoserver/geoserver-cloud/projects)

## Building

Requirements:

* Java >= 17 JDK
* [Maven](https://maven.apache.org/) >= `3.6.3`
* [Docker](https://docs.docker.com/engine/install/) version >= `19.03.3`
* [docker-compose](https://docs.docker.com/compose/) version >= `1.26.2`
* A recent [Docker](https://docs.docker.com/engine/install/) version with the [Compose](https://docs.docker.com/compose/) plugin.

The simple `make` command from the project root directory will build, test, and install all the project artifacts, and build the GeoServer-Cloud Docker images. So for a full build just run:

Expand All @@ -105,16 +119,6 @@ and run tests with
make test
```

### Custom upstream GeoServer version

*GeoServer Cloud* depends on a custom GeoServer branch, `gscloud/gs_version/integration`, which contains patches to upstream GeoServer that have not yet been integrated into the mainstream `main` branch.

Additionally, this branch changes the artifact versions (e.g. from `2.23-SNAPSHOT` to `2.23.0-CLOUD`), to avoid confusing maven if you also work with vanilla GeoServer, and to avoid your IDE downloading the latest `2.23-SNAPSHOT` artifacts from the OsGeo maven repository, overriding your local maven repository ones, and having confusing compilation errors that would require re-building the branch we need.

The `gscloud/gs_version/integration` branch is checked out as a submodule on the (camptocamp/geoserver-cloud-geoserver)[https://github.com/camptocamp/geoserver-cloud-geoserver] repository, which publishes the custom geoserver maven artifacts to the Github maven package registry.

The root pom adds this additional maven repository, so no further action is required for the geoserver-cloud build to use those dependencies.

### Build the docker images

As mentioned above, a `make` with no arguments will build everything.
Expand All @@ -125,6 +129,11 @@ But to build only the docker images, run:
make build-image
```

This runs the `build-base-images`, `build-image-infrastructure`, and `build-image-geoserver` targets,
which you can also run individually during development depending on your needs. Usually,
you'd run `make build-image-geoserver` to speed up the process when made a change and want
to test the geoserver containers, without having to rebuild the base and infra images.

### Targeted builds

*GeoServer Cloud*-specific modules source code is under the `src/` directory.
Expand All @@ -143,74 +152,120 @@ $ cd src/
$ ../mvnw clean install
```

### Development runs
### Note on custom upstream GeoServer version

To run the development docker composition using a shared data directory.
*GeoServer Cloud* depends on a custom GeoServer branch, `gscloud/gs_version/integration`, which contains patches to upstream GeoServer that have not yet been integrated into the mainstream `main` branch.

GeoServer-Cloud can start from an empty directory.
Additionally, this branch changes the artifact versions (e.g. from `2.23-SNAPSHOT` to `2.23.0-CLOUD`), to avoid confusing maven if you also work with vanilla GeoServer, and to avoid your IDE downloading the latest `2.23-SNAPSHOT` artifacts from the OsGeo maven repository, overriding your local maven repository ones, and having confusing compilation errors that would require re-building the branch we need.

Edit your ```.env``` file to set the ```GS_USER``` variable to your user id and group,
then do:
The `gscloud/gs_version/integration` branch is checked out as a submodule on the (camptocamp/geoserver-cloud-geoserver)[https://github.com/camptocamp/geoserver-cloud-geoserver] repository, which publishes the custom geoserver maven artifacts to the Github maven package registry.

```bash
$ mkdir docker-compose_datadir
$ alias dcd="docker-compose -f docker-compose.yml -f docker-compose-shared_datadir.yml"
$ dcd up -d
```
The root pom adds this additional maven repository, so no further action is required for the geoserver-cloud build to use those dependencies.

## Development runs

The `./compose` folder contains docker-compose files intended only for **development**.

For instructions on running GeoServer Cloud in your environment, follow the [Quick Start](https://geoserver.org/geoserver-cloud/#quick-start) guide on the user guide.

### Run as non-root

Verify the services are running with `dcd ps`.
First thing first, edit the `.env` file to set the `GS_USER` variable to the user and group ids
the applications should run as.

Healthchecks use `curl` hitting the `http:localhost:8081/actuator/health`.
Usually the GID and UID of your user, such as:

```
echo `id -g`:`id -u`
1000:1000
```

Healthchecks use `curl` hitting the `http://localhost:8081/actuator/health` spring-boot actuator endpoint, which
also provides Kubernetes liveness and readiness probes at `/actuator/health/liveness` and `/actuator/health/readiness`
respectively.

The services run on the `8080` port, and are exposed using different host ports. The spring-boot-actuator is set up at port `8081`.

The `gateway-service` proxies requests from the `9090` local port:

```
$ curl "http://localhost:9090/geoserver/cloud/ows?request=getcapabilities&service={WMS,WFS,WCS}"
$ curl -u admin:geoserver "http://localhost:9090/geoserver/cloud/rest/workspaces.json"
### Choose your Catalog and Configuration back-end

You need to run `compose.yml` and pick one compose override file for a given GeoServer Catalog
and Configuration back-end.

#### DataDirectory Catalog back-end

The `datadir` spring boot profile enables the traditional "data directory" catalog back-end,
with all GeoServer containers sharing the same directory. On a k8s deployment you would need a
`ReadWriteMany` persistent volume.

GeoServer-Cloud can start from an empty data directory.

The `catalog-datadir.yml` docker compose override enables the `datadir` profile and
initializes a volume with the default GeoServer release data directory.

Run with:

```bash
$ alias dcd="docker compose -f compose.yml -f catalog-datadir.yml"
$ dcd up -d
```

Browse to [http://localhost:9090/geoserver/cloud/](http://localhost:9090/geoserver/cloud/)
#### PostgreSQL Catalog back-end

> Note the `/geoserver/cloud` context path is set up in the `gateway-service`'s externalized
> configuration, and enforced through the `GEOSERVER_BASE_PATH` in `docker-compose.yml`.
> You can change it to whatever you want. The default [config/gateway-service.yml](config/gateway-service.yml)
> configuration file does not set up a context path at all, and hence GeoServer will
> be available at the root URL.
The `pgconfig` spring boot profile enables the PostgreSQL catalog back-end.

To test GeoServer Cloud with different catalog backends, change the alias to replace the `docker-compose-shared_datadir.yml` file by one of the following:
- Use `docker-compose-jdbcconfig.yml` for a JDBC based catalog backend (uses jdbc config and jdbc store modules)
- Use `docker-compose-pgconfig.yml` for a JNDI based catalog backend
This is the preferred Catalog back-end for production deployments,
and requires a PostgreSQL 15.0+ database

You can also run `docker-compose -f docker-compose.yml -f docker-compose-discovery-ha.yml` to run extra discovery service instances for HA.
The `catalog-pgconfig.yml` docker compose override enables the `pgconfig` profile and
sets up a PostgreSQL container named `pgconfigdb`.

## Deployment
> On a production deployment, it is expected that the database is a provided service
and not part of the GeoServer Cloud deployment.

Docker images for all the services are available on DockerHub, under the GeoServer Cloud organization (https://hub.docker.com/u/geoservercloud/).
Run with:

You can find production-suitable deployment files for docker-compose and podman under the [docs/deploy](docs/deploy) folder.
```bash
$ alias dcp="docker compose -f compose.yml -f catalog-pgconfig.yml"
$ dcp up -d
```

Also, a ready-to-use Helm chart for Kubernetes is available at the [camptocamp/helm-geoserver-cloud](https://github.com/camptocamp/helm-geoserver-cloud) Github repository.
**PGBouncer**:

## Contributing
Given the `pgconfig` catalog back-end will set up a database connection pool on each container,
when scaling out you might run out of available connections in the Postgres server. A good way
to avoid that and make better use of resources is to use a connection pooling service, such
as [pgbouncer](https://www.pgbouncer.org/).

Please read [the contribution guidelines](CONTRIBUTING.md) before contributing pull requests to the GeoServer Cloud project.
Use the `catalog-pgconfig.yml` in combination with the `pgbouncer.yml` docker compose override. `pgbouncer.yml`
will override the three database containers with separate pgbouncer instances for each:

Follow the [developer's guide](docs/develop/index.md) to know more about the project's technical details.
* `pgconfigdb` becomes a `pgbouncer` container pointing to the `pgconfigdb_pg` container.
* `acldb` becomes a `pgbouncer` container pointing to the `acldb_pg` container, and holds the [GeoServer ACL](https://github.com/geoserver/geoserver-acl) database
* `postgis` becomes a `pgbouncer` container pointing to the `postgis_pg` container.

## Status
> The `postgis` is container used to host sample data, it is not required but useful during development.
`v1.6.0` released against GeoServer `2.24.2`.

Read the [changelog](https://github.com/geoserver/geoserver-cloud/releases/tag/v1.6.0) for more information.

## Bugs
#### Access GeoServer

*GeoServer Cloud*'s issue tracking is at this [GitHub](https://github.com/geoserver/geoserver-cloud/issues) repository.
Verify the services are running with `dcd ps` or `dcp ps` as appropriate.

## Roadmap
```
$ curl "http://localhost:9090/geoserver/cloud/ows?request=getcapabilities&service={WMS,WFS,WCS,WPS}"
$ curl -u admin:geoserver "http://localhost:9090/geoserver/cloud/rest/workspaces.json"
```

Follow the development progress on these [GitHub Kanban boards](https://github.com/geoserver/geoserver-cloud/projects)
Browse to [http://localhost:9090/geoserver/cloud/](http://localhost:9090/geoserver/cloud/)

> Note the `/geoserver/cloud` context path is set up in the `gateway-service`'s externalized
> configuration, and enforced through the `GEOSERVER_BASE_PATH` in `compose.yml`.
> You can change it to whatever you want. The default [config/gateway-service.yml](config/gateway-service.yml)
> configuration file does not set up a context path at all, and hence GeoServer will
> be available at the root URL.

4 changes: 2 additions & 2 deletions .env → compose/.env
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# docker-compose default environment variable values
COMPOSE_PROJECT_NAME=gscloud
COMPOSE_PROJECT_NAME=gscloud_dev
TAG=1.7-SNAPSHOT
ACL_TAG=2.0-SNAPSHOT
GS_USER="1000:1000"

BASE_PATH=/geoserver/cloud

GEOSERVER_DEFAULT_PROFILES="acl"
GEOSERVER_DEFAULT_PROFILES="default,debug"
ACL_DEFAULT_PROFILES="acl"

#GEOSERVER_DEFAULT_PROFILES="acl,acl_debug"
Expand Down
15 changes: 8 additions & 7 deletions .env.yjp.example → compose/.env.yjp.example
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,12 @@

YJP_OPTS=-agentpath:/usr/local/YourKit-JavaProfiler-2023.9/bin/linux-x86-64/libyjpagent.so=broker_url=https://broker.yourkit.com/<broker-id>/,broker_token=<broker-token>

WFS_JAVA_OPTS=$YJP_OPTS,sessionname=wfs-service
WMS_JAVA_OPTS=$YJP_OPTS,sessionname=wms-service
WCS_JAVA_OPTS=$YJP_OPTS,sessionname=wcs-service
WPS_JAVA_OPTS=$YJP_OPTS,sessionname=wps-service
REST_JAVA_OPTS=$YJP_OPTS,sessionname=rest-service
WEBUI_JAVA_OPTS=$YJP_OPTS,sessionname=webui-service
GWC_JAVA_OPTS=$YJP_OPTS,sessionname=gwc-service
JAVA_OPTS=-XX:MaxRAMPercentage=80

WFS_JAVA_OPTS=$JAVA_OPTS $YJP_OPTS,sessionname=wfs-service
WMS_JAVA_OPTS=$JAVA_OPTS $YJP_OPTS,sessionname=wms-service
WCS_JAVA_OPTS=$JAVA_OPTS $YJP_OPTS,sessionname=wcs-service
WPS_JAVA_OPTS=$JAVA_OPTS $YJP_OPTS,sessionname=wps-service
REST_JAVA_OPTS=$JAVA_OPTS $YJP_OPTS,sessionname=rest-service
WEBUI_JAVA_OPTS=$JAVA_OPTS $YJP_OPTS,sessionname=webui-service
GWC_JAVA_OPTS=$JAVA_OPTS $YJP_OPTS,sessionname=gwc-service
Binary file added compose/catalog-datadir.tgz
View file
Binary file not shown.
85 changes: 85 additions & 0 deletions compose/catalog-datadir.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
version: "3.8"

#
# Configures all geoserver services to use a shared data directory as catalog backend.
# Run with `docker compose -f compose.yml -f catalog-datadir.yml up -d`
# Note the default data directory location is /opt/app/data_directory
# To set it to a different path, change the mount point and add the following env variable: GEOSERVER_DATA_DIR: </path/to/data_directory>

volumes:
datadir:
# driver_opts:
# type: none
# o: bind
# device: $PWD/docker-compose_datadir

services:
init-datadir:
image: alpine:3.18.4
user: ${GS_USER}
command: sh -c "cd /opt/app/data_directory; if [ ! -f global.xml ]; then tar xfvz /tmp/datadir.tgz; fi"
volumes:
- datadir:/opt/app/data_directory
- ./catalog-datadir.tgz:/tmp/datadir.tgz
wfs:
environment:
SPRING_PROFILES_ACTIVE: "${GEOSERVER_DEFAULT_PROFILES},datadir"
depends_on:
init-datadir:
condition: service_completed_successfully
volumes:
- datadir:/opt/app/data_directory

wms:
environment:
SPRING_PROFILES_ACTIVE: "${GEOSERVER_DEFAULT_PROFILES},datadir"
depends_on:
init-datadir:
condition: service_completed_successfully
volumes:
- datadir:/opt/app/data_directory

wcs:
environment:
SPRING_PROFILES_ACTIVE: "${GEOSERVER_DEFAULT_PROFILES},datadir"
depends_on:
init-datadir:
condition: service_completed_successfully
volumes:
- datadir:/opt/app/data_directory

wps:
environment:
SPRING_PROFILES_ACTIVE: "${GEOSERVER_DEFAULT_PROFILES},datadir"
depends_on:
init-datadir:
condition: service_completed_successfully
volumes:
- datadir:/opt/app/data_directory

rest:
environment:
SPRING_PROFILES_ACTIVE: "${GEOSERVER_DEFAULT_PROFILES},datadir"
depends_on:
init-datadir:
condition: service_completed_successfully
volumes:
- datadir:/opt/app/data_directory

webui:
environment:
SPRING_PROFILES_ACTIVE: "${GEOSERVER_DEFAULT_PROFILES},datadir"
depends_on:
init-datadir:
condition: service_completed_successfully
volumes:
- datadir:/opt/app/data_directory

gwc:
environment:
SPRING_PROFILES_ACTIVE: "${GEOSERVER_DEFAULT_PROFILES},datadir"
depends_on:
init-datadir:
condition: service_completed_successfully
volumes:
- datadir:/opt/app/data_directory
4 changes: 1 addition & 3 deletions docker-compose-jdbcconfig.yml → compose/catalog-jdbcconfig.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ volumes:

#
# Configures all geoserver services to use the postgresql database server with jdbcconfig as catalog backend.
# Run with `docker-compose --compatibility -f docker-compose.yml -f docker-compose-jdbcconfig.yml up -d`
# Run with `docker compose -f compose.yml -f catalog-jdbcconfig.yml up -d`
#

services:
Expand All @@ -18,8 +18,6 @@ services:
POSTGRES_PASSWORD: "${JDBCCONFIG_PASSWORD}"
ports:
- 54321:5432
networks:
- gs-cloud-network
volumes:
- postgresql_config_data:/var/lib/postgresql/data
deploy:
Expand Down
Loading

0 comments on commit 9a20aff

Please sign in to comment.