diff --git a/.github/workflows/lint-codebase.yaml b/.github/workflows/lint-codebase.yaml index f1bc968..bea5bd2 100644 --- a/.github/workflows/lint-codebase.yaml +++ b/.github/workflows/lint-codebase.yaml @@ -22,4 +22,4 @@ jobs: VALIDATE_ALL_CODEBASE: false DEFAULT_BRANCH: master GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - FILTER_REGEX_INCLUDE: ".*README.md" + FILTER_REGEX_EXCLUDE: '.*doc/.*|.*github/.*' diff --git a/README.md b/README.md index c329d0b..3acddfe 100644 --- a/README.md +++ b/README.md @@ -1,248 +1,19 @@ - -# Table of Contents +# CARTO Self-hosted for Docker Compose -- [CARTO Self Hosted [Docker]](#carto-self-hosted-docker) - - [Installation](#installation) - - [Prerequisites](#prerequisites) - - [Deployment Customizations](#deployment-customizations) - - [Installation Steps](#installation-steps) - - [Post-installation Checks](#post-installation-checks) - - [Update](#update) - - [Uninstall](#uninstall) - - [Migrate to Kubernetes](#migrate-to-kubernetes) - - [Preconditions](#preconditions) - - [Steps](#steps) +![header](./img/header-docker.png) -# CARTO Self Hosted [Docker] +This repository contains the Docker Compose configuration for CARTO self-hosted used to deploy CARTO in Docker environments. -This repository contains the necessary files for deploying a CARTO Self Hosted installation in your own cloud infrastructure using `docker-compose`. +For a Kubernetes flavor of CARTO, please refer to -To be able to run CARTO Self Hosted you need to have a license. [Contact CARTO](https://carto.com/request-live-demo/) to get one. +Documentation and instructions can be found at -If you are looking for another installation method, CARTO Self Hosted is provided in two flavors: +## Prerequisites -- [Kubernetes Helm](https://github.com/CartoDB/carto-selfhosted-helm) -- [Docker compose](https://github.com/CartoDB/carto-selfhosted) +CARTO self-hosted requires technical knowledge, if you don't have the experience or you just want to try CARTO, we offer a cloud managed solution where you can start a trial in a few clicks, please refer to . -## Installation +CARTO self-hosted also requires an enterprise license. Please [contact our team](https://carto.com/request-live-demo) if you're interested. -> :warning: CARTO provides an out-of-the-box installation that is **not production ready**. In order to make your CARTO installation production ready take a look at the [customization section](customizations/README.md). +## Release notes -### Prerequisites - -You will need a Linux machine with at least: - -- Ubuntu 18.04 or above -- 60 GB disk -- 2 CPUs (x86) -- 8 GB memory -- Docker version 20.10 or above -- Docker compose version 1.29 or above -- A TLS certificate for the domain/subdomain (if not provided a self-signed one will be generated) -- Configuration and license files received from CARTO -- Internet HTTP/HTTPS access from the machine to the [whitelisted domains list](proxy/config/whitelisted_domains.md) - -> Note that you should additionally allow access to any datawarehouse endpoint configured. - -### Deployment Customizations - -Please, read the available [customization](customizations/README.md) options. - -### Installation Steps - -1. Log into the machine where you are going to deploy CARTO. - -2. Clone this repository: - - ```bash - git clone https://github.com/CartoDB/carto-selfhosted.git - cd carto-selfhosted - ``` - -3. After contacting CARTO, you should have received two files, copy them in the current directory (`carto-selfhosted`): - - `customer.env` - - `key.json` - -4. Configure your CARTO Self Hosted domain by updating the env var `SELFHOSTED_DOMAIN` value, which defaults to `carto3-onprem.lan`. In order to access your CARTO Self Hosted you should modify your `/etc/hosts` file to point `localhost` to this domain: - - ```bash - sudo vi /etc/hosts - ``` - - ```bash - # Carto Self Hosted - 127.0.0.1 carto3-onprem.lan - ``` - -5. Configure your deployment. Please, read the available [customization options](customizations/README.md). - -6. Checkout to the [latest stable release](https://github.com/CartoDB/carto-selfhosted/releases): - ```bash - git checkout tags/2022.12.14 - ``` - -7. Run the `install.sh` script to generate the `.env` file out of the `customer.env` file: - - ```bash - bash install.sh - ``` - -8. Bring up the environment: - - ```bash - docker-compose up -d - ``` - -9. Open a browser and go to `https://carto3-onprem.lan` (or the custom domain configured). - -### Post-installation Checks - -In order to verify CARTO Self Hosted was correctly installed and it's functional, we recommend performing the following checks: - -1. Check all the containers are up and running: - - ```bash - docker-compose ps - ``` - - > All containers should be in state `Up`, except for `workspace-migrations` which state should be `Exit 0`, meaning the database migrations finished correctly. - -2. Sign in to your Self Hosted, create a user and a new organization. - -3. Go to the `Connections` page, in the left-hand menu, create a new connection to one of the available providers. - -4. Go to the `Data Explorer` page, click on the `Upload` button right next to the `Connections` panel. Import a dataset from a local file. - -5. Go back to the `Maps` page, and create a new map. - -6. In this new map, add a new layer from a table using the connection created in step 3. - -7. Create a new layer from a SQL Query to the same table. You can use a simple query like: - - ```bash - SELECT * FROM LIMIT 100; - ``` - -8. Create a new layer from the dataset imported in step 4. - -9. Make the map public, copy the sharing URL and open it in a new incognito window. - -10. Go back to the `Maps` page, and verify your map appears there and the map thumbnail represents the latest changes you made to the map. - -## Update - -To update you CARTO Self Hosted to the newest version you will need run the following commands: - -1. Go to the CARTO installation directory: - - ```bash - cd carto-selfhosted - ``` - -2. Pull last changes from the origin repo: - - ```bash - git pull - ``` - -3. Save a backup copy of your current `customer.env`: - - ```bash - mv customer.env customer.env.bak - ``` - -4. Download the latest customer package (containing `customer.env` and `key.json` files) using the tool described [here](tools/README.md#download-customer-package-tool). Then unzip the file. - -5. Copy the new customer.env file in the installation directory: - - ```bash - cp /new_file_location/customer.env . - ``` - -6. Open side by side `customer.env` and `customer.env.bak` and apply the customizations from `customer.env.bak` in the new `customer.env` - -7. Generate the `.env` file - - ```bash - bash install.sh - ``` - -8. Recreate the containers - - ```bash - docker-compose up -d - ``` - -## Uninstall - -You can just stop the Self Hosted services (including removing any volumes from the system) and delete the `carto-selfhosted` directory. - -> :warning: In case you are running a local Postgres database (which is not recommended for Production environments), take into account that removing the docker volumes will delete the database information and your CARTO Self Hosted users information with it. - -```bash -docker-compose down -V -``` - -## Migrate to Kubernetes - -To migrate your CARTO Self Hosted from Docker Compose installation to -[Kubernetes/Helm](https://github.com/CartoDB/carto-selfhosted-helm) you need to follow these steps: - -> :warning: Migration incurs in downtime. To minimize it, reduce the DNS TTL before starting the process - -### Preconditions - -- You have a running Self Hosted deployed with Docker Compose i.e using a Google Compute engine instance. -- You have configured external databases (Redis and PostgreSQL). -- You have a K8s cluster to deploy the new self hosted and credentials to deploy. -- You have received a new customer package from CARTO (with files for Kubernetes and for Docker). If you do not have them, please contact Support. - -### Steps - -1. [Update](#update) Docker installation to the latest release with the customer package received. - -2. Allow network connectivity from k8s nodes to your pre-existing databases. [i.e (Cloud SQL connection notes](https://github.com/CartoDB/carto-selfhosted/README.md#cloud-sql-connection-configuration)). - -3. Create a `customizations.yaml` following [this instructions](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations). Keep the same external database connection settings you are using in CARTO for Docker. [Postgres](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations#configure-external-postgres) and [Redis](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations#configure-external-redis). - - > :warning: NOTE: Do not trust the default values and fill all variables related to database connections, example: - - ```yaml - externalPostgresql: - host: "" - adminUser: postgres - adminPassword: - password: - database: workspace - user: workspace_admin - sslEnabled: true - internalPostgresql: - enabled: false - - internalRedis: - # Disable the internal Redis - enabled: false - externalRedis: - host: "yourRedisHost" - port: "6379" - password: " - tlsEnabled: false - ``` - - > Read also the instructions on how to [expose the Kubernetes](https://github.com/CartoDB/carto-selfhosted-helm/blob/main/customizations/README.md#access-to-carto-from-outside-the-cluster) installation to outside the cluster. - -4. Create a `customizations.yaml` following [these instructions](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations). Keep the same external database connection settings you are using in CARTO for Docker. [Postgres](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations#configure-external-postgres) and [Redis](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations#configure-external-redis). - -5. Shut down you CARTO for Docker installation: `docker-compose down`. - - > Once you execute this, the service is down. - -6. Deploy to your cluster. Follow the [installation steps](https://github.com/CartoDB/carto-selfhosted-helm#installation). - -7. Check pods are running and stable with `kubectl get pods <-n your_namespace>`. - -8. Change DNS records to point to the new service (`helm install` will point how to get the IP or DNS), it will take some time to propagate. - -9. Test your CARTO Self Hosted for Kubernetes installation. Service is restored. - -> If for whatever reason the installation did not go as planned. You can bring back the docker installation and point back your DNS to it. +Release notes are available at diff --git a/customizations/README.md b/customizations/README.md deleted file mode 100644 index c9c690a..0000000 --- a/customizations/README.md +++ /dev/null @@ -1,726 +0,0 @@ - - -# Table of Contents - -- [Customizations](#customizations) - - [Production Ready](#production-ready) - - [Custom Service Account](#custom-service-account) - - [How to apply the configurations](#how-to-apply-the-configurations) - - [Available configurations](#available-configurations) - - [Self Hosted domain](#self-hosted-domain) - - [Custom SSL certificate](#custom-ssl-certificate) - - [External database](#external-database) - - [Configure SSL](#configure-ssl) - - [Azure PostgreSQL](#azure-postgresql) - - [Troubleshooting](#troubleshooting) - - [External Redis](#external-redis) - - [Configure TLS](#configure-tls) - - [External proxy](#external-proxy) - - [Important notes](#important-notes) - - [Configuration](#configuration) - - [Proxy HTTP](#proxy-http) - - [Proxy HTTPS](#proxy-https) - - [Supported datawarehouses](#supported-datawarehouses) - - [Custom buckets](#custom-buckets) - - [Pre-requisites](#pre-requisites) - - [Google Cloud Storage](#google-cloud-storage) - - [AWS S3](#aws-s3) - - [Azure Blob Storage](#azure-blob-storage) - - [Enable BigQuery OAuth connections](#enable-bigquery-oauth-connections) - - [External Data warehouse tuning](#external-data-warehouse-tuning) - - [Google Maps](#google-maps) - - [Redshift imports](#redshift-imports) - - [Enabling-Disabling TrackJS](#enabling-disabling-trackjs) - -# Customizations - -## Production Ready - -The default Docker compose configuration provided by Carto works out-of-the-box, but it's **not production ready**. -There are a few things to configure in order to make the Carto installation production ready. - -Mandatory: - -- [Self Hosted Domain](#self-hosted-domain) - -Recommended: - -- [Custom SSL Certificate](#custom-ssl-certificate) -- [External Database](#external-database) - -Optional: - -- [External Redis](#external-redis) -- [Google Maps](#google-maps) -- [Custom Buckets](#custom-buckets) -- [Enable BigQuery OAuth connections](#enable-bigquery-oauth-connections) - -## Custom Service Account - -CARTO deploys a dedicated infrastructure for every self hosted installation, including a Service Account key that is required to use some of the services deployed. - -If you prefer using your own GCP Service Account, please do the following prior to the Self Hosted installation: - -1. Create a dedicated Service Account for the CARTO Self Hosted. - -2. Contact CARTO support team and provide them the service account email. - -## How to apply the configurations - -Make your changes to the `customer.env` file before starting the installation steps. - -> :warning: Anytime you edit the `customer.env` file to change the CARTO configuration you will need to apply it to your installation: -> -> 1. Run the `install.sh` script to update the `.env` file used by Docker Compose. -> -> `bash install.sh` -> -> 2. Refresh the installation configuration. -> -> `docker-compose down && docker-compose up -d` - -## Available configurations - -Here you will find the supported custom configurations you can apply to your CARTO Self Hosted deployment. - -### Self Hosted domain - -Configure your CARTO Self Hosted domain by updating the env var `SELFHOSTED_DOMAIN` value, which defaults to `carto3-onprem.lan`. - -#### Custom SSL certificate - -By default CARTO Self Hosted will generate and use a self-signed certificate if you don't provide it with your own certificate. - -**Prerequisites** - -- A `.crt` file with your custom domain x509 certificate. -- A `.key` file with your custom domain private key. - -**Configuration** - -1. Create a `certs` folder in the current directory (`carto-selfhosted`). - -2. Copy your `.crt` and `.key` files in the `certs` folders (the files must be directly accesible from the server, i.e.: not protected with password and with the proper permissions). - -3. Modify the following vars in the `customer.env` file: - - ```diff - - # ROUTER_SSL_AUTOGENERATE= <1 to enable | 0 to disable> - - # ROUTER_SSL_CERTIFICATE_PATH=/etc/nginx/ssl/.crt - - # ROUTER_SSL_CERTIFICATE_KEY_PATH=/etc/nginx/ssl/.key - + ROUTER_SSL_AUTOGENERATE=0 - + ROUTER_SSL_CERTIFICATE_PATH=/etc/nginx/ssl/.crt - + ROUTER_SSL_CERTIFICATE_KEY_PATH=/etc/nginx/ssl/.key - ``` - - > Remember to replace the `` value above with the correct file name. - -### External database - -CARTO Self Hosted comes with an embedded PostgreSQL database that is not recommended for production installations, we recommend to use your own PostgreSQL database that lives outside the Docker ecosystem. This database is for internal data of the CARTO Self Hosted. - -Here are some Terraform examples of databases created in different providers: - -- [GCP Cloud SQL](../examples/terraform/gcp/postgresql.tf). -- [AWS RDS](../examples/terraform/aws/postgresql-rds.tf). -- [Azure Database](../examples/terraform/azure/postgresql.tf). - -**Prerequisites** - -- PostgreSQL 11 or above - -**Configuration** - -Open with an editor the `customer.env` file and modify the following variables: - -1. Comment the local Postgres configuration: - - ```diff - # Configuration for using a local postgres, instead of an external one (comment when external postgres) - - LOCAL_POSTGRES_SCALE=1 - - WORKSPACE_POSTGRES_HOST=workspace-postgres - - WORKSPACE_POSTGRES_PORT=5432 - - WORKSPACE_POSTGRES_USER=workspace_admin - - WORKSPACE_POSTGRES_PASSWORD= - - WORKSPACE_POSTGRES_DB=workspace - - WORKSPACE_POSTGRES_SSL_ENABLED=false - - WORKSPACE_POSTGRES_SSL_MODE=disable - - POSTGRES_ADMIN_USER=postgres - - POSTGRES_ADMIN_PASSWORD= - + # LOCAL_POSTGRES_SCALE=1 - + # WORKSPACE_POSTGRES_HOST=workspace-postgres - + # WORKSPACE_POSTGRES_PORT=5432 - + # WORKSPACE_POSTGRES_USER=workspace_admin - + # WORKSPACE_POSTGRES_PASSWORD= - + # WORKSPACE_POSTGRES_DB=workspace - + # WORKSPACE_POSTGRES_SSL_ENABLED=false - + # WORKSPACE_POSTGRES_SSL_MODE=disable - + # POSTGRES_ADMIN_USER=postgres - + # POSTGRES_ADMIN_PASSWORD= - + # In case your Postgres doesn't contain the default postgres database - + # POSTGRES_ADMIN_DB=postgres - ``` - -2. Uncomment the external Postgres configuration: - - ```diff - # Your custom configuration for an external postgres database (comment when local postgres) - - # LOCAL_POSTGRES_SCALE=0 - - # WORKSPACE_POSTGRES_HOST= - - # WORKSPACE_POSTGRES_PORT= - - # WORKSPACE_POSTGRES_USER=workspace_admin - - # WORKSPACE_POSTGRES_PASSWORD= - - # WORKSPACE_POSTGRES_DB=workspace - - # WORKSPACE_POSTGRES_SSL_ENABLED=true - - # WORKSPACE_POSTGRES_SSL_MODE=require - # Only applies if Postgres SSL certificate is self signed, read the documentation - # WORKSPACE_POSTGRES_SSL_CA=/usr/src/certs/postgresql-ssl-ca.crt - - # POSTGRES_ADMIN_USER= - - # POSTGRES_ADMIN_PASSWORD= - + LOCAL_POSTGRES_SCALE=0 - + WORKSPACE_POSTGRES_HOST= - + WORKSPACE_POSTGRES_PORT= - + WORKSPACE_POSTGRES_USER=workspace_admin - + WORKSPACE_POSTGRES_PASSWORD= - + WORKSPACE_POSTGRES_SSL_ENABLED=true - + WORKSPACE_POSTGRES_SSL_MODE=require - + WORKSPACE_POSTGRES_DB=workspace - + POSTGRES_ADMIN_USER= - + POSTGRES_ADMIN_PASSWORD= - # In case your Postgres doesn't contain the default postgres database - # POSTGRES_ADMIN_DB=postgres - ``` - -3. Replace the `` placeholders with the right values. - -#### Configure SSL - -By default CARTO Self Hosted will try to connect to your external PostgreSQL using SSL. - -```bash -WORKSPACE_POSTGRES_SSL_ENABLED=true -WORKSPACE_POSTGRES_SSL_MODE=require -``` - -> :warning: In case the SSL certificate is selfsigned or from a custom CA, you will need to configure the `WORKSPACE_POSTGRES_SSL_CA` variable. -> -> 1. Copy you CA `.crt` file inside `certs` folder. Rename the CA `.crt` file to `postgresql-ssl-ca.crt`. -> -> 2. Uncomment the `WORKSPACE_POSTGRES_SSL_CA` env var in the `customer.env` file: -> -> ```diff -> # Only applies if Postgres SSL certificate is selfsigned, read the documentation -> - # WORKSPACE_POSTGRES_SSL_CA=/usr/src/certs/postgresql-ssl-ca.crt -> + WORKSPACE_POSTGRES_SSL_CA=/usr/src/certs/postgresql-ssl-ca.crt -> ``` - -To connect to your external Postgresql without SSL, you'll need to configure `WORKSPACE_POSTGRES_SSL` variables accordingly: - -```diff -- WORKSPACE_POSTGRES_SSL_ENABLED=true -- WORKSPACE_POSTGRES_SSL_MODE=require -+ WORKSPACE_POSTGRES_SSL_ENABLED=false -+ WORKSPACE_POSTGRES_SSL_MODE=disable -``` - -#### Azure PostgreSQL - -In case you are connection to an Azure hosted Postgres you will need to uncomment the `WORKSPACE_POSTGRES_INTERNAL_USER` and `POSTGRES_LOGIN_USER` env vars where: - -- `WORKSPACE_POSTGRES_INTERNAL_USER` - same value as `WORKSPACE_POSTGRES_USER` but without the `@db-name` prefix. -- `POSTGRES_LOGIN_USER` - same value as `POSTGRES_ADMIN_USER` but without the `@db-name` prefix. - -#### Troubleshooting - -If you are connecting with public or private ip to a Google Cloud SQL in your self hosted, you need to add to the instance configuration external static (for public) or internal static IPs ranges as Authorized networks. If you have the resource terraformed you can add the networks with this way (take as a guide): - -```hcl -resource "random_id" "db_name_suffix" { - byte_length = 4 -} - -locals { - onprem = ["192.168.1.2", "192.168.2.3"] -} - -resource "google_sql_database_instance" "postgres" { - name = "postgres-instance-${random_id.db_name_suffix.hex}" - database_version = "POSTGRES_11" - - settings { - tier = "db-f1-micro" - - ip_configuration { - - dynamic "authorized_networks" { - for_each = local.onprem - iterator = onprem - - content { - name = "onprem-${onprem.key}" - value = onprem.value - } - } - } - } -} -``` - -Or in the web console: - -Captura de pantalla 2022-04-05 a las 11 11 11 - -### External Redis - -CARTO comes with an embedded Redis that is not recommended for production installations, we recommend to use your own Redis that lives outside the Docker ecosystem. - -Here are some Terraform examples of Redis instances created in different providers: - -- [GCP Redis](../examples/terraform/gcp/redis.tf). -- [AWS Redis](../examples/terraform/aws/redis.tf). -- [Azure Redis](../examples/terraform/azure/redis.tf). - -**Prerequisites** - -- Redis 6 or above. - -**Configuration** - -1. Comment the local Redis configuration: - - ```diff - # Configuration for using a local redis, instead of a external one (comment when external redis) - - LOCAL_REDIS_SCALE=1 - - REDIS_HOST=redis - - REDIS_PORT=6379 - - REDIS_TLS_ENABLED=false - + # LOCAL_REDIS_SCALE=1 - + # REDIS_HOST=redis - + # REDIS_PORT=6379 - + # REDIS_TLS_ENABLED=false - ``` - -2. Uncomment the external Redis configuration: - - ```diff - # Your custom configuration for a external redis (comment when local redis) - - # LOCAL_REDIS_SCALE=0 - - # REDIS_HOST= - - # REDIS_PORT= - - # REDIS_PASSWORD= - - # REDIS_TLS_ENABLED=true - # Only applies if Redis TLS certificate it's self signed, read the documentation - # REDIS_TLS_CA= - + LOCAL_REDIS_SCALE=0 - + REDIS_HOST= - + REDIS_PORT= - + REDIS_PASSWORD= - + REDIS_TLS_ENABLED=true - ``` - -3. Replace the `` placeholders with the right values. - -#### Configure TLS - -By default CARTO will try to connect to your Redis without TLS, in case you want to connect via TLS ,you can configure it via `REDIS_TLS_ENABLED` env vars in the `customer.env`file - -```bash -REDIS_TLS_ENABLED=true -``` - -> :warning: In case you are connection to a Redis where the TLS certificate is self signed or from a custom CA you will need to configure the `REDIS_TLS_CA` variable - -1. Copy you CA `.crt` file inside `certs` folder. Rename the CA `.crt` file to `redis-tls-ca.crt`. - -2. Uncomment the `REDIS_TLS_CA` env var in the `customer.env` file. - - ```diff - # Only applies if Redis TLS certificate it's selfsigned, read the documentation - - # REDIS_TLS_CA=/usr/src/certs/redis-tls-ca.crt - + REDIS_TLS_CA=/usr/src/certs/redis-tls-ca.crt - ``` - -### External proxy - -#### Important notes - -:warning: Please consider the following important notes regarding the proxy configuration: - -- CARTO self-hosted does not install any proxy component, instead it supports connecting to an existing proxy software deployed by the customer. - -- CARTO Self-hosted supports both **HTTP** and **HTTPS** proxies. - -- At the moment, password authentication is not supported for the proxy connection. - -- [Importing data](https://docs.carto.com/carto-user-manual/data-explorer/importing-data) using an **HTTPS Proxy configured with a certificate signed by a Custom CA** currently has some limitations. Please, contact CARTO Support for this use case. - - :information_source: Please check [Proxy HTTPS](#proxy-https) to understand the difference between a **custom CA** and a **well known CA**. - -#### Configuration - -CARTO self-hosted provides support for operating behind an HTTP or HTTPS proxy. The proxy acts as a gateway, enabling CARTO self-hosted components to establish connections with essential external services like Google APIs, Mapbox, and others. - -A comprehensive list of domains that must be whitelisted by the proxy for the proper functioning of CARTO self-hosted can be found [here](../proxy/config/whitelisted_domains.md). The list includes domains for the essential core services of CARTO self-hosted, as well as additional optional domains that should be enabled to access specific features. - -In order to enable this feature, set the following environment variables (:warning: both uppercase and lowercase variables) in your `.env` file, depending on the protocol your proxy uses. - -##### Proxy HTTP - -- `HTTP_PROXY` (mandatory): Proxy connection string, consisting of `http://:`. -- `HTTPS_PROXY` (mandatory): Same as `HTTP_PROXY`. -- `GRPC_PROXY` (mandatory): Same as `HTTP_PROXY`. -- `NO_PROXY` (optional): Comma-separated list of domains to exclude from proxying. - -Example: - -```bash -HTTP_PROXY="http://my-proxy:3128" -http_proxy="http://my-proxy:3128" -HTTPS_PROXY="http://my-proxy:3128" -https_proxy="http://my-proxy:3128" -GRPC_PROXY="http://my-proxy:3128" -grpc_proxy="http://my-proxy:3128" -NO_PROXY="localhost,mega.io,dropbox.com,filestack.com" -no_proxy="localhost,mega.io,dropbox.com,filestack.com" -``` - -##### Proxy HTTPS - -> :warning: Currently, using a Snowflake connection with a Proxy HTTPS is not supported. - -- `HTTP_PROXY` (mandatory): Proxy connection string, consisting of `https://:`. -- `HTTPS_PROXY` (mandatory): Same as `HTTP_PROXY`. -- `NO_PROXY` (optional): Comma-separated list of domains to exclude from proxying. -- `NODE_EXTRA_CA_CERTS` (optional): Path to the proxy CA certificate. - - :information_source: Please read carefully the [important notes](#important-notes) to understand the current limitations with **custom CAs**. - - :information_source: If the proxy certificate is signed by a **custom CA**, such CA must be included here. - - :information_source: If the proxy certificate is signed by a **well known CA**, there is no need to add it here. **Well known CAs** are usually part of the [ca-certificates package](https://askubuntu.com/questions/857476/what-is-the-use-purpose-of-the-ca-certificates-package) -- `NODE_TLS_REJECT_UNAUTHORIZED` (optional): Specify if CARTO Self-hosted should check if the proxy certificate is valid (`1`) or not (`0`). - - :information_source: For instance, **self signed certificates** validation must be skipped. - -Example: - -```bash -HTTP_PROXY="https://my-proxy:3129" -http_proxy="https://my-proxy:3129" -HTTPS_PROXY="https://my-proxy:3129" -https_proxy="https://my-proxy:3129" -NO_PROXY="mega.io,dropbox.com,filestack.com" -no_proxy="mega.io,dropbox.com,filestack.com" -NODE_EXTRA_CA_CERTS=/opt/carto/certs/proxy-ca.crt -NODE_TLS_REJECT_UNAUTHORIZED=0 -``` - -#### Supported datawarehouses - -Note that while certain data warehouses can be configured to work with the proxy, **there are others that will inherently bypass it**. Therefore, if you have a restrictive network policy in place, you will need to explicitly allow this egress non-proxied traffic. - -| Datawarehouse | Proxy HTTP | Proxy HTTPS | Automatic proxy bypass \*\* | -| ------------- | ---------- | ----------- | --------------------------- | -| BigQuery | Yes | Yes | N/A | -| Snowflake | Yes | No | No \*\*\* | -| Databricks | No | No | Yes | -| Postgres | No | No | Yes | -| Redshift | No | No | Yes | - -> :warning: \*\* There's no need to include the non supported datawarehouses in the `NO_PROXY` environment variable list. CARTO self-hosted components will automatically attempt a direct connection to those datawarehouses, with the exception of **HTTPS Proxy + Snowflake**. - -> :warning: \*\*\* If an HTTPS proxy is required in your deployment and you are a Snowflake Warehouse user, you need to explicitly exclude snowflake traffic using the configuration below: - -```bash -NO_PROXY=".snowflakecomputing.com" ## Check your Snowflake warehouse URL -no_proxy=".snowflakecomputing.com" ## Check your Snowflake warehouse URL -``` - -### Custom buckets - -For every CARTO Self Hosted installation, we create GCS buckets on our side as part of the required infrastructure for importing data, map thumbnails and customization assets (custom logos and markers) and other internal data. - -You can create and use your own storage buckets in any of the following supported storage providers: - -- Google Cloud Storage. [Terraform code example](../examples/terraform/gcp/storage.tf). -- AWS S3. [Terraform code example](../examples/terraform/aws/storage.tf). -- Azure Storage. [Terraform code example](../examples/terraform/azure/storage.tf). - -> :warning: You can only set one provider at a time. - -#### Pre-requisites - -1. Create 3 buckets in your preferred Cloud provider: - - - Import Bucket - - Client Bucket - - Thumbnails Bucket. - - > There're no name constraints - - > :warning: Map thumbnails storage objects (.png files) can be configured to be `public` (default) or `private`. In order to change this, set `WORKSPACE_THUMBNAILS_PUBLIC="false"`. For the default configuration to work, the bucket must allow public objects/blobs. Some features, such as branding and custom markers, won't work unless the bucket is public. However, there's a workaround to avoid making the whole bucket public, which requires allowing public objects, allowing ACLs (or non-uniform permissions) and disabling server-side encryption. - -2. CORS configuration: Thumbnails and Client buckets require having the following CORS headers configured. - - - Allowed origins: `*` - - Allowed methods: `GET`, `PUT`, `POST` - - Allowed headers (common): `Content-Type`, `Content-MD5`, `Content-Disposition`, `Cache-Control` - - GCS (extra): `x-goog-content-length-range`, `x-goog-meta-filename` - - Azure (extra): `Access-Control-Request-Headers`, `X-MS-Blob-Type` - - Max age: `3600` - - > CORS is configured at bucket level in GCS and S3, and at storage account level in Azure. - - > How do I setup CORS configuration? Check the provider docs: [GCS](https://cloud.google.com/storage/docs/configuring-cors), [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html), [Azure Storage](https://docs.microsoft.com/en-us/rest/api/storageservices/cross-origin-resource-sharing--cors--support-for-the-azure-storage-services#enabling-cors-for-azure-storage). - -3. Generate credentials with Read/Write permissions to access those buckets, our supported authentication methods are: - - GCS: Service Account Key - - AWS: Access Key ID and Secret Access Key - - Azure: Access Key - -#### Google Cloud Storage - -In order to use Google Cloud Storage custom buckets you need to: - -1. Create the buckets. - - > :warning: If you enable `Prevent public access` in the bucket properties, then set `appConfigValues.workspaceThumbnailsPublic` and `appConfigValues.workspaceImportsPublic` to `false`. - -2. Configure the required [CORS settings](#pre-requisites). - -3. Create a [custom Service account](#custom-service-account). - -4. Grant this service account with the following role (in addition to the buckets access): `roles/iam.serviceAccountTokenCreator`. - - > :warning: We don't recommend grating this role at project IAM level, but instead at the Service Account permissions level (IAM > Service Accounts > `your_service_account` > Permissions). - -5. Set the following variables in your customer.env file: - - ```bash - # Thumbnails bucket - WORKSPACE_THUMBNAILS_PROVIDER='gcp' - WORKSPACE_THUMBNAILS_PUBLIC= - WORKSPACE_THUMBNAILS_BUCKET= - WORKSPACE_THUMBNAILS_KEYFILENAME= - WORKSPACE_THUMBNAILS_PROJECTID= - - # Client bucket - WORKSPACE_IMPORTS_PROVIDER='gcp' - WORKSPACE_IMPORTS_PUBLIC= - WORKSPACE_IMPORTS_BUCKET= - WORKSPACE_IMPORTS_KEYFILENAME= - WORKSPACE_IMPORTS_PROJECTID= - - # Import bucket - IMPORT_PROVIDER='gcp' - IMPORT_BUCKET= - IMPORT_KEYFILENAME= - IMPORT_PROJECTID= - ``` - - > If `_KEYFILENAME` is not defined env `GOOGLE_APPLICATION_CREDENTIALS` is used as default value. When the selfhosted service account is setup in a Compute Engine instance as the default service account, there's no need to set any of these, as the containers will inherit the instance default credentials. - - > If `_PROJECTID` is not defined env `GOOGLE_CLOUD_PROJECT` is used as default value. - -#### AWS S3 - -In order to use AWS S3 custom buckets you need to: - -1. Create the buckets. - - > :warning: If you enable `Block public access` in the bucket properties, then set `WORKSPACE_THUMBNAILS_PUBLIC` and `WORKSPACE_IMPORTS_PUBLIC` to `false`. - -2. Configure the required [CORS settings](#pre-requisites). - -3. Create an IAM user and generate a programmatic key ID and secret. If server-side encryption is enabled, the user must be granted with permissions over the KMS key used. - -4. Grant this user with read/write access permissions over the buckets. - -5. Set the following variables in your customer.env file: - - ```bash - # Thumbnails bucket - WORKSPACE_THUMBNAILS_PROVIDER='s3' - WORKSPACE_THUMBNAILS_PUBLIC= - WORKSPACE_THUMBNAILS_BUCKET= - WORKSPACE_THUMBNAILS_ACCESSKEYID= - WORKSPACE_THUMBNAILS_SECRETACCESSKEY= - WORKSPACE_THUMBNAILS_REGION= - - # Client bucket - WORKSPACE_IMPORTS_PROVIDER='s3' - WORKSPACE_IMPORTS_PUBLIC= - WORKSPACE_IMPORTS_BUCKET= - WORKSPACE_IMPORTS_ACCESSKEYID= - WORKSPACE_IMPORTS_SECRETACCESSKEY= - WORKSPACE_IMPORTS_REGION= - - # Import bucket - IMPORT_PROVIDER='s3' - IMPORT_BUCKET= - IMPORT_ACCESSKEYID= - IMPORT_SECRETACCESSKEY= - IMPORT_REGION= - ``` - -#### Azure Blob Storage - -In order to use Azure Storage buckets (aka containers) you need to: - -1. Create an storage account if you don't have one already. - -2. Configure the required [CORS settings](#pre-requisites). - -3. Create the storage buckets. - - > :warning: If you set the `Public Access Mode` to `private` in the bucket properties, then set the `WORKSPACE_THUMBNAILS_PUBLIC` and `WORKSPACE_IMPORTS_PUBLIC` environment variables to `false`. - -4. Generate an Access Key, from the storage account's Security properties. - -5. Set the following variables in your customer.env file: - - ```bash - # Thumbnails bucket - WORKSPACE_THUMBNAILS_PROVIDER='azure-blob' - WORKSPACE_THUMBNAILS_PUBLIC= - WORKSPACE_THUMBNAILS_BUCKET= - WORKSPACE_THUMBNAILS_STORAGE_ACCOUNT= - WORKSPACE_THUMBNAILS_STORAGE_ACCESSKEY= - - # Client bucket - WORKSPACE_IMPORTS_PROVIDER='azure-blob' - WORKSPACE_IMPORTS_PUBLIC= - WORKSPACE_IMPORTS_BUCKET= - WORKSPACE_IMPORTS_STORAGE_ACCOUNT= - WORKSPACE_IMPORTS_STORAGE_ACCESSKEY= - - # Import bucket - IMPORT_PROVIDER='azure-blob' - IMPORT_BUCKET= - IMPORT_STORAGE_ACCOUNT= - IMPORT_STORAGE_ACCESSKEY= - ``` - -### Enable BigQuery OAuth connections - -This feature allows users to create a BigQuery connection using `Sign in with Google` instead of providing a service account key. - -> :warning: Connections created with OAuth cannot be shared with other organization users. - -1. Create an OAuth consent screen inside the desired GCP project. - - - Introduce an app name and a user support email. - - Add an authorized domain (the one used in your email). - - Add another email as dev contact info (it can be the same). - - Add the following scopes: `./auth/userinfo.email`, `./auth/userinfo.profile` & `./auth/bigquery`. - -2. Create an OAuth credentials. - - - Type: Web application. - - Authorized JavaScript origins: `https://`. - - Authorized redirect URIs: `https:///connections/bigquery/oauth`. - - Download the credentials file. - -3. In your self hosted's customer.env file, set the following vars with the values from the credentials file: - - ```bash - REACT_APP_BIGQUERY_OAUTH=true - BIGQUERY_OAUTH2_CLIENT_ID= - BIGQUERY_OAUTH2_CLIENT_SECRET= - ``` - -### External Data warehouse tuning - -CARTO Self Hosted connects to your data warehouse to perform the analysis with your data. When connecting it with Postgres -or with Redshift it is important to understand and configure the connection pool. - -Each node will have a connection pool controlled by the environment variables `MAPS_API_V3_POSTGRES_POOL_SIZE` and -`MAPS_API_V3_REDSHIFT_POOL_SIZE`. The pool is per connection created from CARTO Self Hosted. If each user creates a different -connection, each one will have its own pool. The maximum connections can be calculated with the following formula: - -```javascript -max_connections = pool_size * number_connections * number_nodes; -``` - -### Google Maps - -In order to enable Google Maps basemaps inside CARTO Self Hosted, you need to own a Google Maps API key enabled for the Maps JavaScript API and set it via `GOOGLE_MAPS_API_KEY` in your customer.env file. - -### Redshift imports - -> :warning: This is currently a feature flag and it's disabled by default. Please, contact support if you are interested on using it. - -CARTO selfhosted supports importing data to a Redshift cluster or serverless. Follow these instructions to setup your Redshift integration: - -> :warning: This requires access to an AWS account and an existing accessible Redshift endpoint. - -1. Create an AWS IAM user with programmatic access. Take note of the user's arn, key ID and key secret. - -2. Create an AWS S3 Bucket: - - - ACLs should be allowed. - - If server-side encryption is enabled, the user must be granted with permissions over the KMS key used. - -3. Create an AWS IAM role with the following settings: - - 1. Trusted entity type: `Custom trust policy`. - 2. Custom trust policy: Make sure to replace ``. - - ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": ["sts:AssumeRole", "sts:TagSession"] - } - ] - } - ``` - - 3. Add permissions: Create a new permissions policy, replacing ``. - - ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": "s3:ListBucket", - "Resource": "arn:aws:s3:::" - }, - { - "Effect": "Allow", - "Action": "s3:*Object", - "Resource": "arn:aws:s3:::/*" - } - ] - } - ``` - -4. Add the following lines to your `customer.env` file: - -```bash -IMPORT_AWS_ACCESS_KEY_ID= -IMPORT_AWS_SECRET_ACCESS_KEY= -IMPORT_AWS_ROLE_ARN= -``` - -5. Perform a `docker-compose up -d` before continuing with the following steps. - -6. Log into your CARTO selfhosted, go to `Data Explorer > Connections > Add new connection` and create a new Redshift connection. - -7. Then go to `Settings > Advanced > Integrations > Redshift > New`, introduce your S3 Bucket name and region and copy the policy generated. - -8. From the AWS console, go to your `S3 > Bucket > Permissions > Bucket policy` and paste the policy obtained in the previous step in the policy editor. - -9. Go back to the CARTO Selfhosted (Redshift integration page) and check the `I have already added the S3 bucket policy` box and click on the `Validate and save button`. - -10. Go to `Data Exporer > Import data > Redshift connection` and you should be able to import a local dataset to Redshift. - -### Enabling-Disabling TrackJS - -- TrackJS is enabled by default in the www components, but you can disable it with these variables in your `customer.env` file: - - ```bash - REACT_APP_ACCOUNTS_WWW_ENABLE_TRACKJS=false - REACT_APP_WORKSPACE_WWW_ENABLE_TRACKJS=false - ``` diff --git a/doc/README.md b/doc/README.md new file mode 100644 index 0000000..0f30a86 --- /dev/null +++ b/doc/README.md @@ -0,0 +1,4 @@ +# Documentation + +> **Important** +> The following documentation is deprecated and will eventually be migrated to [docs.carto.com](https://docs.carto.com/carto-self-hosted) diff --git a/doc/customizations-configure-an-external-proxy.md b/doc/customizations-configure-an-external-proxy.md new file mode 100644 index 0000000..f5e33bc --- /dev/null +++ b/doc/customizations-configure-an-external-proxy.md @@ -0,0 +1,90 @@ +### External proxy + +#### Important notes + +:warning: Please consider the following important notes regarding the proxy configuration: + +- CARTO self-hosted does not install any proxy component, instead it supports connecting to an existing proxy software deployed by the customer. + +- CARTO Self-hosted supports both **HTTP** and **HTTPS** proxies. + +- At the moment, password authentication is not supported for the proxy connection. + +- [Importing data](https://docs.carto.com/carto-user-manual/data-explorer/importing-data) using an **HTTPS Proxy configured with a certificate signed by a Custom CA** currently has some limitations. Please, contact CARTO Support for this use case. + - :information_source: Please check [Proxy HTTPS](#proxy-https) to understand the difference between a **custom CA** and a **well known CA**. + +#### Configuration + +CARTO self-hosted provides support for operating behind an HTTP or HTTPS proxy. The proxy acts as a gateway, enabling CARTO self-hosted components to establish connections with essential external services like Google APIs, Mapbox, and others. + +A comprehensive list of domains that must be whitelisted by the proxy for the proper functioning of CARTO self-hosted can be found [here](../proxy/config/whitelisted_domains.md). The list includes domains for the essential core services of CARTO self-hosted, as well as additional optional domains that should be enabled to access specific features. + +In order to enable this feature, set the following environment variables (:warning: both uppercase and lowercase variables) in your `.env` file, depending on the protocol your proxy uses. + +##### Proxy HTTP + +- `HTTP_PROXY` (mandatory): Proxy connection string, consisting of `http://:`. +- `HTTPS_PROXY` (mandatory): Same as `HTTP_PROXY`. +- `GRPC_PROXY` (mandatory): Same as `HTTP_PROXY`. +- `NO_PROXY` (optional): Comma-separated list of domains to exclude from proxying. + +Example: + +```bash +HTTP_PROXY="http://my-proxy:3128" +http_proxy="http://my-proxy:3128" +HTTPS_PROXY="http://my-proxy:3128" +https_proxy="http://my-proxy:3128" +GRPC_PROXY="http://my-proxy:3128" +grpc_proxy="http://my-proxy:3128" +NO_PROXY="localhost,mega.io,dropbox.com,filestack.com" +no_proxy="localhost,mega.io,dropbox.com,filestack.com" +``` + +##### Proxy HTTPS + +> :warning: Currently, using a Snowflake connection with a Proxy HTTPS is not supported. + +- `HTTP_PROXY` (mandatory): Proxy connection string, consisting of `https://:`. +- `HTTPS_PROXY` (mandatory): Same as `HTTP_PROXY`. +- `NO_PROXY` (optional): Comma-separated list of domains to exclude from proxying. +- `NODE_EXTRA_CA_CERTS` (optional): Path to the proxy CA certificate. + - :information_source: Please read carefully the [important notes](#important-notes) to understand the current limitations with **custom CAs**. + - :information_source: If the proxy certificate is signed by a **custom CA**, such CA must be included here. + - :information_source: If the proxy certificate is signed by a **well known CA**, there is no need to add it here. **Well known CAs** are usually part of the [ca-certificates package](https://askubuntu.com/questions/857476/what-is-the-use-purpose-of-the-ca-certificates-package) +- `NODE_TLS_REJECT_UNAUTHORIZED` (optional): Specify if CARTO Self-hosted should check if the proxy certificate is valid (`1`) or not (`0`). + - :information_source: For instance, **self signed certificates** validation must be skipped. + +Example: + +```bash +HTTP_PROXY="https://my-proxy:3129" +http_proxy="https://my-proxy:3129" +HTTPS_PROXY="https://my-proxy:3129" +https_proxy="https://my-proxy:3129" +NO_PROXY="mega.io,dropbox.com,filestack.com" +no_proxy="mega.io,dropbox.com,filestack.com" +NODE_EXTRA_CA_CERTS=/opt/carto/certs/proxy-ca.crt +NODE_TLS_REJECT_UNAUTHORIZED=0 +``` + +#### Supported datawarehouses + +Note that while certain data warehouses can be configured to work with the proxy, **there are others that will inherently bypass it**. Therefore, if you have a restrictive network policy in place, you will need to explicitly allow this egress non-proxied traffic. + +| Datawarehouse | Proxy HTTP | Proxy HTTPS | Automatic proxy bypass \*\* | +| ------------- | ---------- | ----------- | --------------------------- | +| BigQuery | Yes | Yes | N/A | +| Snowflake | Yes | No | No \*\*\* | +| Databricks | No | No | Yes | +| Postgres | No | No | Yes | +| Redshift | No | No | Yes | + +> :warning: \*\* There's no need to include the non supported datawarehouses in the `NO_PROXY` environment variable list. CARTO self-hosted components will automatically attempt a direct connection to those datawarehouses, with the exception of **HTTPS Proxy + Snowflake**. + +> :warning: \*\*\* If an HTTPS proxy is required in your deployment and you are a Snowflake Warehouse user, you need to explicitly exclude snowflake traffic using the configuration below: + +```bash +NO_PROXY=".snowflakecomputing.com" ## Check your Snowflake warehouse URL +no_proxy=".snowflakecomputing.com" ## Check your Snowflake warehouse URL +``` \ No newline at end of file diff --git a/doc/customizations-configure-custom-ssl-certificate.md b/doc/customizations-configure-custom-ssl-certificate.md new file mode 100644 index 0000000..95d8495 --- /dev/null +++ b/doc/customizations-configure-custom-ssl-certificate.md @@ -0,0 +1,27 @@ +#### Custom SSL certificate + +By default CARTO Self Hosted will generate and use a self-signed certificate if you don't provide it with your own certificate. + +**Prerequisites** + +- A `.crt` file with your custom domain x509 certificate. +- A `.key` file with your custom domain private key. + +**Configuration** + +1. Create a `certs` folder in the current directory (`carto-selfhosted`). + +2. Copy your `.crt` and `.key` files in the `certs` folders (the files must be directly accesible from the server, i.e.: not protected with password and with the proper permissions). + +3. Modify the following vars in the `customer.env` file: + + ```diff + - # ROUTER_SSL_AUTOGENERATE= <1 to enable | 0 to disable> + - # ROUTER_SSL_CERTIFICATE_PATH=/etc/nginx/ssl/.crt + - # ROUTER_SSL_CERTIFICATE_KEY_PATH=/etc/nginx/ssl/.key + + ROUTER_SSL_AUTOGENERATE=0 + + ROUTER_SSL_CERTIFICATE_PATH=/etc/nginx/ssl/.crt + + ROUTER_SSL_CERTIFICATE_KEY_PATH=/etc/nginx/ssl/.key + ``` + + > Remember to replace the `` value above with the correct file name. \ No newline at end of file diff --git a/doc/customizations-custom-service-account.md b/doc/customizations-custom-service-account.md new file mode 100644 index 0000000..fe13480 --- /dev/null +++ b/doc/customizations-custom-service-account.md @@ -0,0 +1,9 @@ +## Custom Service Account + +CARTO deploys a dedicated infrastructure for every self hosted installation, including a Service Account key that is required to use some of the services deployed. + +If you prefer using your own GCP Service Account, please do the following prior to the Self Hosted installation: + +1. Create a dedicated Service Account for the CARTO Self Hosted. + +2. Contact CARTO support team and provide them the service account email. \ No newline at end of file diff --git a/doc/customizations-enable-bigquery-oauth-connections.md b/doc/customizations-enable-bigquery-oauth-connections.md new file mode 100644 index 0000000..ab25136 --- /dev/null +++ b/doc/customizations-enable-bigquery-oauth-connections.md @@ -0,0 +1,27 @@ +### Enable BigQuery OAuth connections + +This feature allows users to create a BigQuery connection using `Sign in with Google` instead of providing a service account key. + +> :warning: Connections created with OAuth cannot be shared with other organization users. + +1. Create an OAuth consent screen inside the desired GCP project. + + - Introduce an app name and a user support email. + - Add an authorized domain (the one used in your email). + - Add another email as dev contact info (it can be the same). + - Add the following scopes: `./auth/userinfo.email`, `./auth/userinfo.profile` & `./auth/bigquery`. + +2. Create an OAuth credentials. + + - Type: Web application. + - Authorized JavaScript origins: `https://`. + - Authorized redirect URIs: `https:///connections/bigquery/oauth`. + - Download the credentials file. + +3. In your self hosted's customer.env file, set the following vars with the values from the credentials file: + + ```bash + REACT_APP_BIGQUERY_OAUTH=true + BIGQUERY_OAUTH2_CLIENT_ID= + BIGQUERY_OAUTH2_CLIENT_SECRET= + ``` \ No newline at end of file diff --git a/doc/customizations-enable-disable-trackjs.md b/doc/customizations-enable-disable-trackjs.md new file mode 100644 index 0000000..9c188a7 --- /dev/null +++ b/doc/customizations-enable-disable-trackjs.md @@ -0,0 +1,8 @@ +### Enabling-Disabling TrackJS + +- TrackJS is enabled by default in the www components, but you can disable it with these variables in your `customer.env` file: + + ```bash + REACT_APP_ACCOUNTS_WWW_ENABLE_TRACKJS=false + REACT_APP_WORKSPACE_WWW_ENABLE_TRACKJS=false + ``` \ No newline at end of file diff --git a/doc/customizations-external-datawarehouse-tunning.md b/doc/customizations-external-datawarehouse-tunning.md new file mode 100644 index 0000000..c8f9bf3 --- /dev/null +++ b/doc/customizations-external-datawarehouse-tunning.md @@ -0,0 +1,12 @@ +### External Data warehouse tuning + +CARTO Self Hosted connects to your data warehouse to perform the analysis with your data. When connecting it with Postgres +or with Redshift it is important to understand and configure the connection pool. + +Each node will have a connection pool controlled by the environment variables `MAPS_API_V3_POSTGRES_POOL_SIZE` and +`MAPS_API_V3_REDSHIFT_POOL_SIZE`. The pool is per connection created from CARTO Self Hosted. If each user creates a different +connection, each one will have its own pool. The maximum connections can be calculated with the following formula: + +```javascript +max_connections = pool_size * number_connections * number_nodes; +``` diff --git a/doc/customizations-google-maps.md b/doc/customizations-google-maps.md new file mode 100644 index 0000000..5fb9cf3 --- /dev/null +++ b/doc/customizations-google-maps.md @@ -0,0 +1,3 @@ +### Google Maps + +In order to enable Google Maps basemaps inside CARTO Self Hosted, you need to own a Google Maps API key enabled for the Maps JavaScript API and set it via `GOOGLE_MAPS_API_KEY` in your customer.env file. diff --git a/doc/customizations-how-to-apply-the-configurations.md b/doc/customizations-how-to-apply-the-configurations.md new file mode 100644 index 0000000..b2b7999 --- /dev/null +++ b/doc/customizations-how-to-apply-the-configurations.md @@ -0,0 +1,13 @@ +## How to apply the configurations + +Make your changes to the `customer.env` file before starting the installation steps. + +> :warning: Anytime you edit the `customer.env` file to change the CARTO configuration you will need to apply it to your installation: +> +> 1. Run the `install.sh` script to update the `.env` file used by Docker Compose. +> +> `bash install.sh` +> +> 2. Refresh the installation configuration. +> +> `docker-compose down && docker-compose up -d` \ No newline at end of file diff --git a/doc/customizations-redshift-imports.md b/doc/customizations-redshift-imports.md new file mode 100644 index 0000000..acf7412 --- /dev/null +++ b/doc/customizations-redshift-imports.md @@ -0,0 +1,74 @@ +### Redshift imports + +> :warning: This is currently a feature flag and it's disabled by default. Please, contact support if you are interested on using it. + +CARTO selfhosted supports importing data to a Redshift cluster or serverless. Follow these instructions to setup your Redshift integration: + +> :warning: This requires access to an AWS account and an existing accessible Redshift endpoint. + +1. Create an AWS IAM user with programmatic access. Take note of the user's arn, key ID and key secret. + +2. Create an AWS S3 Bucket: + + - ACLs should be allowed. + - If server-side encryption is enabled, the user must be granted with permissions over the KMS key used. + +3. Create an AWS IAM role with the following settings: + + 1. Trusted entity type: `Custom trust policy`. + 2. Custom trust policy: Make sure to replace ``. + + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": "" + }, + "Action": ["sts:AssumeRole", "sts:TagSession"] + } + ] + } + ``` + + 3. Add permissions: Create a new permissions policy, replacing ``. + + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "s3:ListBucket", + "Resource": "arn:aws:s3:::" + }, + { + "Effect": "Allow", + "Action": "s3:*Object", + "Resource": "arn:aws:s3:::/*" + } + ] + } + ``` + +4. Add the following lines to your `customer.env` file: + +```bash +IMPORT_AWS_ACCESS_KEY_ID= +IMPORT_AWS_SECRET_ACCESS_KEY= +IMPORT_AWS_ROLE_ARN= +``` + +5. Perform a `docker-compose up -d` before continuing with the following steps. + +6. Log into your CARTO selfhosted, go to `Data Explorer > Connections > Add new connection` and create a new Redshift connection. + +7. Then go to `Settings > Advanced > Integrations > Redshift > New`, introduce your S3 Bucket name and region and copy the policy generated. + +8. From the AWS console, go to your `S3 > Bucket > Permissions > Bucket policy` and paste the policy obtained in the previous step in the policy editor. + +9. Go back to the CARTO Selfhosted (Redshift integration page) and check the `I have already added the S3 bucket policy` box and click on the `Validate and save button`. + +10. Go to `Data Exporer > Import data > Redshift connection` and you should be able to import a local dataset to Redshift. \ No newline at end of file diff --git a/doc/root-migrate-to-kubernetes.md b/doc/root-migrate-to-kubernetes.md new file mode 100644 index 0000000..70e253d --- /dev/null +++ b/doc/root-migrate-to-kubernetes.md @@ -0,0 +1,63 @@ +## Migrate to Kubernetes + +To migrate your CARTO Self Hosted from Docker Compose installation to +[Kubernetes/Helm](https://github.com/CartoDB/carto-selfhosted-helm) you need to follow these steps: + +> :warning: Migration incurs in downtime. To minimize it, reduce the DNS TTL before starting the process + +### Preconditions + +- You have a running Self Hosted deployed with Docker Compose i.e using a Google Compute engine instance. +- You have configured external databases (Redis and PostgreSQL). +- You have a K8s cluster to deploy the new self hosted and credentials to deploy. +- You have received a new customer package from CARTO (with files for Kubernetes and for Docker). If you do not have them, please contact Support. + +### Steps + +1. [Update](#update) Docker installation to the latest release with the customer package received. + +2. Allow network connectivity from k8s nodes to your pre-existing databases. [i.e (Cloud SQL connection notes](https://github.com/CartoDB/carto-selfhosted/README.md#cloud-sql-connection-configuration)). + +3. Create a `customizations.yaml` following [this instructions](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations). Keep the same external database connection settings you are using in CARTO for Docker. [Postgres](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations#configure-external-postgres) and [Redis](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations#configure-external-redis). + + > :warning: NOTE: Do not trust the default values and fill all variables related to database connections, example: + + ```yaml + externalPostgresql: + host: "" + adminUser: postgres + adminPassword: + password: + database: workspace + user: workspace_admin + sslEnabled: true + internalPostgresql: + enabled: false + + internalRedis: + # Disable the internal Redis + enabled: false + externalRedis: + host: "yourRedisHost" + port: "6379" + password: " + tlsEnabled: false + ``` + + > Read also the instructions on how to [expose the Kubernetes](https://github.com/CartoDB/carto-selfhosted-helm/blob/main/customizations/README.md#access-to-carto-from-outside-the-cluster) installation to outside the cluster. + +4. Create a `customizations.yaml` following [these instructions](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations). Keep the same external database connection settings you are using in CARTO for Docker. [Postgres](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations#configure-external-postgres) and [Redis](https://github.com/CartoDB/carto-selfhosted-helm/tree/main/customizations#configure-external-redis). + +5. Shut down you CARTO for Docker installation: `docker-compose down`. + + > Once you execute this, the service is down. + +6. Deploy to your cluster. Follow the [installation steps](https://github.com/CartoDB/carto-selfhosted-helm#installation). + +7. Check pods are running and stable with `kubectl get pods <-n your_namespace>`. + +8. Change DNS records to point to the new service (`helm install` will point how to get the IP or DNS), it will take some time to propagate. + +9. Test your CARTO Self Hosted for Kubernetes installation. Service is restored. + +> If for whatever reason the installation did not go as planned. You can bring back the docker installation and point back your DNS to it. \ No newline at end of file diff --git a/examples/terraform/aws/eks-iam-roles.tf b/examples/terraform/aws/eks-iam-roles.tf deleted file mode 100644 index 16bb41f..0000000 --- a/examples/terraform/aws/eks-iam-roles.tf +++ /dev/null @@ -1,129 +0,0 @@ -data "aws_caller_identity" "current" {} -data "aws_partition" "current" {} -data "tls_certificate" "cluster" { - url = module.eks.cluster_oidc_issuer_url -} - -locals { - role_to_user_map = { - EksAdmin = "admin", - EksDeveloper = "developer" - } - - role_map_users = [ - for role_name, user in local.role_to_user_map : { - rolearn = "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:role/${role_name}" - username = user - groups = (user == "admin") ? ["system:masters"] : ["none"] - } - ] -} - -resource "aws_iam_role" "admin" { - name = "EksAdmin" - max_session_duration = 43200 - - assume_role_policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Action = "sts:AssumeRole" - Effect = "Allow" - Sid = "" - Principal = { - AWS = var.assume_admin_role - } - }, - ] - }) - - inline_policy { - name = "eks_admin_policy" - - policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Action = ["eks:*"] - Effect = "Allow" - Resource = "*" - }, - ] - }) - } -} - -resource "aws_iam_role" "developer" { - name = "EksDeveloper" - max_session_duration = 43200 - - assume_role_policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Action = "sts:AssumeRole" - Effect = "Allow" - Sid = "" - Principal = { - AWS = var.assume_developer_role - } - }, - ] - }) - - inline_policy { - name = "eks_developer_policy" - - policy = jsonencode({ - Version = "2012-10-17" - Statement = [ - { - Action = ["eks:DescribeCluster"] - Effect = "Allow" - Resource = "*" - }, - ] - }) - } -} - -# create the IAM OIDC provider for the cluster -resource "aws_iam_openid_connect_provider" "eks-cluster" { - client_id_list = ["sts.amazonaws.com"] - thumbprint_list = [data.tls_certificate.cluster.certificates[0].sha1_fingerprint] - url = module.eks.cluster_oidc_issuer_url -} - -#resource "aws_iam_role" "eks-service-account-role" { -# name = "workload_sa" -# -# assume_role_policy = jsonencode({ -# Version = "2012-10-17" -# Statement = [ -# { -# Action = ["sts:AssumeRoleWithWebIdentity"] -# Effect = "Allow" -# Sid = "" -# Principal = { -# Federated = aws_iam_openid_connect_provider.eks-cluster.arn -# } -# }, -# ] -# }) -# -# inline_policy { -# name = "eks_service_account_policy" -# -# policy = jsonencode({ -# Version = "2012-10-17" -# Statement = [ -# { -# Action = ["s3:GetBucket", "s3:GetObject", "s3:PutObject"] -# Effect = "Allow" -# Resource = "*" -# }, -# ] -# }) -# } -#} - diff --git a/examples/terraform/aws/eks.tf b/examples/terraform/aws/eks.tf deleted file mode 100644 index a652b65..0000000 --- a/examples/terraform/aws/eks.tf +++ /dev/null @@ -1,50 +0,0 @@ -locals { - cluster_name = "${var.cluster_name}-${random_string.suffix.result}" -} - -resource "random_string" "suffix" { - length = 4 - special = false -} - -module "eks" { - source = "terraform-aws-modules/eks/aws" - version = "17.24.0" - cluster_name = local.cluster_name - cluster_version = var.cluster_version - subnets = module.vpc.private_subnets - map_roles = local.role_map_users - - vpc_id = module.vpc.vpc_id - - workers_group_defaults = { - root_volume_type = "gp2" - } - - worker_groups = [ - { - name = "worker-group-1" - instance_type = var.node_group_default_instance_type - additional_security_group_ids = [aws_security_group.worker_group_mgmt_one.id] - asg_desired_capacity = var.node_group_desired_capacity - asg_max_size = var.node_group_max_capacity - asg_min_size = var.node_group_min_capacity - }, - { - name = "worker-group-2" - instance_type = var.node_group_default_instance_type - additional_security_group_ids = [aws_security_group.worker_group_mgmt_two.id] - asg_desired_capacity = var.node_group_desired_capacity - asg_max_size = var.node_group_max_capacity - asg_min_size = var.node_group_min_capacity - }, - ] -} - -data "aws_eks_cluster" "cluster" { - name = module.eks.cluster_id -} - -data "aws_eks_cluster_auth" "cluster" { - name = module.eks.cluster_id -} \ No newline at end of file diff --git a/examples/terraform/aws/outputs.tf b/examples/terraform/aws/outputs.tf deleted file mode 100644 index ce79421..0000000 --- a/examples/terraform/aws/outputs.tf +++ /dev/null @@ -1,34 +0,0 @@ -output "cluster_id" { - description = "EKS cluster ID." - value = module.eks.cluster_id -} - -output "cluster_endpoint" { - description = "Endpoint for EKS control plane." - value = module.eks.cluster_endpoint -} - -output "cluster_security_group_id" { - description = "Security group ids attached to the cluster control plane." - value = module.eks.cluster_security_group_id -} - -output "kubectl_config" { - description = "kubectl config as generated by the module." - value = module.eks.kubeconfig -} - -output "config_map_aws_auth" { - description = "A kubernetes configuration to authenticate to this EKS cluster." - value = module.eks.config_map_aws_auth -} - -output "region" { - description = "AWS region" - value = var.region -} - -output "cluster_name" { - description = "Kubernetes Cluster Name" - value = local.cluster_name -} \ No newline at end of file diff --git a/examples/terraform/aws/postgresql-rds.tf b/examples/terraform/aws/postgresql-rds.tf deleted file mode 100644 index f6c308c..0000000 --- a/examples/terraform/aws/postgresql-rds.tf +++ /dev/null @@ -1,43 +0,0 @@ -##################################################################################### -# Terraform Examples: -# These are pieces of code added as configuration examples for guidance, -# therefore they may require additional resources and variable or local declarations. -##################################################################################### - -locals { - name = "carto-postgresql" - region = "us-east-1" -} - -# RDS PostgreSQL instance, using the official RDS module -module "db_default" { - source = "terraform-aws-modules/rds/aws" - version = "4.2.0" - identifier = "${local.name}-default" - - create_db_option_group = false - create_db_parameter_group = false - - # All available versions: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts - engine = "postgres" - engine_version = "13.5" - family = "postgres13" # DB parameter group - major_engine_version = "13" # DB option group - instance_class = "db.t4g.large" - - allocated_storage = 20 - - # NOTE: Do NOT use 'user' as the value for 'username' as it throws: - # "Error creating DB Instance: InvalidParameterValue: MasterUsername - # user cannot be used as it is a reserved word used by the engine" - db_name = "postgres" - username = "postgres" - port = 5432 - - db_subnet_group_name = module.vpc.database_subnet_group - vpc_security_group_ids = [module.postgresql_security_group.security_group_id] - - maintenance_window = "Mon:00:00-Mon:03:00" - backup_window = "03:00-06:00" - backup_retention_period = 0 -} diff --git a/examples/terraform/aws/redis.tf b/examples/terraform/aws/redis.tf deleted file mode 100644 index 674624e..0000000 --- a/examples/terraform/aws/redis.tf +++ /dev/null @@ -1,27 +0,0 @@ -##################################################################################### -# Terraform Examples: -# These are pieces of code added as configuration examples for guidance, -# therefore they may require additional resources and variable or local declarations. -##################################################################################### - -locals { - # Instance name - redis_instance_name = "${var.redis_name}-${random_integer.random_redis.id}" -} - -# Name suffix -resource "random_integer" "random_redis" { - min = 1000 - max = 9999 -} - -# Redis instance -resource "aws_elasticache_cluster" "example" { - cluster_id = local.redis_instance_name - engine = "redis" - node_type = "cache.m4.large" - num_cache_nodes = 1 - parameter_group_name = "default.redis6.x" - engine_version = "6.0" - port = 6379 -} diff --git a/examples/terraform/aws/security-groups.tf b/examples/terraform/aws/security-groups.tf deleted file mode 100644 index 66b8735..0000000 --- a/examples/terraform/aws/security-groups.tf +++ /dev/null @@ -1,50 +0,0 @@ - -resource "aws_security_group" "worker_group_mgmt_one" { - description = "Security group for workers management" - name_prefix = "worker_group_mgmt_one" - vpc_id = module.vpc.vpc_id - - ingress { - from_port = 22 - to_port = 22 - protocol = "tcp" - - cidr_blocks = [ - "10.0.0.0/8", - ] - } -} - -resource "aws_security_group" "worker_group_mgmt_two" { - description = "Security group for workers management" - name_prefix = "worker_group_mgmt_two" - vpc_id = module.vpc.vpc_id - - ingress { - from_port = 22 - to_port = 22 - protocol = "tcp" - - cidr_blocks = [ - "192.168.0.0/16", - ] - } -} - -resource "aws_security_group" "all_worker_mgmt" { - description = "Security group for workers management" - name_prefix = "all_worker_management" - vpc_id = module.vpc.vpc_id - - ingress { - from_port = 22 - to_port = 22 - protocol = "tcp" - - cidr_blocks = [ - "10.0.0.0/8", - "172.16.0.0/12", - "192.168.0.0/16", - ] - } -} \ No newline at end of file diff --git a/examples/terraform/aws/settings.tf b/examples/terraform/aws/settings.tf deleted file mode 100644 index 65fa641..0000000 --- a/examples/terraform/aws/settings.tf +++ /dev/null @@ -1,44 +0,0 @@ -terraform { - required_version = "~> 1.0" - - required_providers { - aws = { - source = "hashicorp/aws" - version = ">= 3.72" - } - null = { - source = "hashicorp/null" - version = ">= 3.1" - } - local = { - source = "hashicorp/local" - version = "2.1.0" - } - random = { - source = "hashicorp/random" - version = "3.1.0" - } - kubernetes = { - source = "hashicorp/kubernetes" - version = ">= 2.0.1" - } - } - - backend "gcs" {} - -} - -provider "aws" {} - -# Kubernetes provider -# https://learn.hashicorp.com/terraform/kubernetes/provision-eks-cluster#optional-configure-terraform-kubernetes-provider -# To learn how to schedule deployments and services using the provider, go here: https://learn.hashicorp.com/terraform/kubernetes/deploy-nginx-kubernetes - -# The Kubernetes provider is included in this file so the EKS module can complete successfully. Otherwise, it throws an error when creating `kubernetes_config_map.aws_auth`. -# You should **not** schedule deployments and services in this workspace. This keeps workspaces modular (one for provision EKS, another for scheduling Kubernetes resources) as per best practices. - -provider "kubernetes" { - host = data.aws_eks_cluster.cluster.endpoint - token = data.aws_eks_cluster_auth.cluster.token - cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data) -} diff --git a/examples/terraform/aws/storage.tf b/examples/terraform/aws/storage.tf deleted file mode 100644 index 4cc9d92..0000000 --- a/examples/terraform/aws/storage.tf +++ /dev/null @@ -1,59 +0,0 @@ -##################################################################################### -# Terraform Examples: -# These are pieces of code added as configuration examples for guidance, -# therefore they may require additional resources and variable or local declarations. -##################################################################################### - -locals { - # List of storage buckets to create - storage_buckets = [ - "mycarto-import-s3-bucket", - "mycarto-client-s3-bucket", - "mycarto-thumbnails-s3-bucket", - ] -} - -# S3 Buckets -resource "aws_s3_bucket" "default" { - for_each = toset(local.storage_buckets) - bucket = each.value - acl = "private" - - cors_rule { - allowed_origins = ["*"] - allowed_methods = ["GET", "PUT", "POST"] - allowed_headers = [ - "Content-Type", - "Content-MD5", - "Content-Disposition", - "Cache-Control" - ] - } - versioning { - enabled = false - } - server_side_encryption_configuration { - rule { - apply_server_side_encryption_by_default { - kms_master_key_id = aws_kms_key.default.arn - sse_algorithm = "aws:kms" - } - } - } -} - -# Block public access setting -resource "aws_s3_bucket_public_access_block" "default" { - for_each = aws_s3_bucket.default - bucket = each.value.id - block_public_acls = true - block_public_policy = true - ignore_public_acls = true - restrict_public_buckets = true -} - -#KMS key for data encryption -resource "aws_kms_key" "default" { - description = "Default" - enable_key_rotation = true -} diff --git a/examples/terraform/aws/terraform.tfvars b/examples/terraform/aws/terraform.tfvars deleted file mode 100644 index a6fab15..0000000 --- a/examples/terraform/aws/terraform.tfvars +++ /dev/null @@ -1,15 +0,0 @@ -additional_tags = {} - -# EKS -cluster_name = "eks" -cluster_version = "1.20" -node_group_default_instance_type = "m5.large" -node_group_desired_capacity = 1 -node_group_min_capacity = 1 -node_group_max_capacity = 3 - -# arn thag can assume the eks developer role -assume_developer_role = [] - -# arn thag can assume the eks admin role -assume_admin_role = [] diff --git a/examples/terraform/aws/variables.tf b/examples/terraform/aws/variables.tf deleted file mode 100644 index 0ee6415..0000000 --- a/examples/terraform/aws/variables.tf +++ /dev/null @@ -1,59 +0,0 @@ -variable "tags" { - description = "Tags assigned to all the resources" - type = map(string) - default = { - Product = "Carto" - } -} - -variable "region" { - description = "AWS region" - type = string - default = "us-east-1" -} - -variable "cluster_name" { - description = "Name of the EKS cluster" - type = string -} - -variable "cluster_version" { - description = "Kubernetes version to use for the EKS cluster" - type = string -} - -variable "node_group_default_instance_type" { - description = "Default EC2 instance type for the node group" - type = string - default = "m5.large" -} - -variable "node_group_desired_capacity" { - description = "The desired number of EC2 instances in the node group" - type = string - default = "1" -} - -variable "node_group_min_capacity" { - description = "The minimum number of EC2 instances in the node group at a given time" - type = string - default = "1" -} - -variable "node_group_max_capacity" { - description = "The maximum number of EC2 instances in the node group at a given time. Used when auto scaling is enabled" - type = string - default = "3" -} - -variable "assume_developer_role" { - description = "A list of ARN's of users/roles that can assume the cluster_developer role" - type = list(string) - default = [""] -} - -variable "assume_admin_role" { - description = "A list of ARN's of users/roles that can assume the cluster_admin role" - type = list(string) - default = [""] -} \ No newline at end of file diff --git a/examples/terraform/aws/vpc.tf b/examples/terraform/aws/vpc.tf deleted file mode 100644 index bae1a3c..0000000 --- a/examples/terraform/aws/vpc.tf +++ /dev/null @@ -1,55 +0,0 @@ -data "aws_availability_zones" "available" {} - -module "vpc" { - source = "terraform-aws-modules/vpc/aws" - version = "3.2.0" - - name = "vpc-${local.cluster_name}" - cidr = "10.0.0.0/16" - azs = data.aws_availability_zones.available.names - private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"] - public_subnets = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"] - database_subnets = ["10.0.7.0/24", "10.0.8.0/24", "10.0.9.0/24"] - - enable_nat_gateway = true - single_nat_gateway = true - enable_dns_hostnames = true - create_database_subnet_group = true - create_database_subnet_route_table = true - - tags = { - "kubernetes.io/cluster/${local.cluster_name}" = "shared" - } - - public_subnet_tags = { - "kubernetes.io/cluster/${local.cluster_name}" = "shared" - "kubernetes.io/role/elb" = "1" - } - - private_subnet_tags = { - "kubernetes.io/cluster/${local.cluster_name}" = "shared" - "kubernetes.io/role/internal-elb" = "1" - } -} - -module "postgresql_security_group" { - source = "terraform-aws-modules/security-group/aws" - version = "~> 4.0" - - name = "postgresql-sg" - description = "Complete PostgreSQL example security group" - vpc_id = module.vpc.vpc_id - - # ingress - ingress_with_cidr_blocks = [ - { - from_port = 5432 - to_port = 5432 - protocol = "tcp" - description = "PostgreSQL access from within VPC" - cidr_blocks = module.vpc.vpc_cidr_block - }, - ] - - tags = local.tags -} diff --git a/examples/terraform/azure/aks.tf b/examples/terraform/azure/aks.tf deleted file mode 100644 index c0fe8f9..0000000 --- a/examples/terraform/azure/aks.tf +++ /dev/null @@ -1,90 +0,0 @@ -# https://docs.microsoft.com/en-us/azure/developer/terraform/create-k8s-cluster-with-tf-and-aks - -locals { - cluster_name = "aks-${random_integer.aks_suffix.result}" -} - -resource "random_integer" "aks_suffix" { - min = 1000 - max = 9999 -} - -# They are enabled but they way tfsec expect it it's deprecated -# tfsec:ignore:azure-container-logging -# tfsec:ignore:azure-container-use-rbac-permissions -resource "azurerm_kubernetes_cluster" "default" { - name = local.cluster_name - location = azurerm_resource_group.default.location - resource_group_name = azurerm_resource_group.default.name - dns_prefix = local.cluster_name - - kubernetes_version = "1.21.9" - - # Performance - ## Worker nodes - default_node_pool { - name = "agentpool" - vm_size = "Standard_B2s" - enable_auto_scaling = true - min_count = 1 - max_count = 5 - } - - # Security - public_network_access_enabled = true - api_server_authorized_ip_ranges = ["0.0.0.0/0"] # ! World-wide access - role_based_access_control_enabled = true - - identity { - type = "SystemAssigned" - } - - # Allow connecting to Kubernetes nodes via SSH - # linux_profile { - # admin_username = "ubuntu" - # ssh_key { - # key_data = file(var.ssh_public_key) - # } - # } - - # Networking - # Changing this forces a new resource to be created. - network_profile { - load_balancer_sku = "Standard" - network_plugin = "kubenet" - network_policy = "calico" - } - - # Logging and Monitoring - oms_agent { - log_analytics_workspace_id = azurerm_log_analytics_workspace.default.id - } - - # tags = { - # Environment = "Development" - # } -} - -# Logging & Monitoring - -# refer https://azure.microsoft.com/pricing/details/monitor/ for log analytics pricing -resource "azurerm_log_analytics_workspace" "default" { - # The WorkSpace name has to be unique across the whole of azure, not just the current subscription/tenant. - name = "CartoDefaultLogAnalyticsWorkspace" - location = azurerm_resource_group.default.location - resource_group_name = azurerm_resource_group.default.name - sku = "PerGB2018" -} - -resource "azurerm_log_analytics_solution" "default" { - solution_name = "ContainerInsights" - location = azurerm_resource_group.default.location - resource_group_name = azurerm_resource_group.default.name - workspace_resource_id = azurerm_log_analytics_workspace.default.id - workspace_name = azurerm_log_analytics_workspace.default.name - - plan { - publisher = "Microsoft" - product = "OMSGallery/ContainerInsights" - } -} diff --git a/examples/terraform/azure/outputs.tf b/examples/terraform/azure/outputs.tf deleted file mode 100644 index aa75d7c..0000000 --- a/examples/terraform/azure/outputs.tf +++ /dev/null @@ -1,46 +0,0 @@ -# AKS - -output "kube_config" { - description = "AKS cluster kubeconfig for kubectl" - value = azurerm_kubernetes_cluster.default.kube_config_raw - sensitive = true -} - -# Redis - -output "redis_access_key" { - description = "Redis access key" - value = azurerm_redis_cache.default.primary_access_key - sensitive = true -} - -output "redis_host" { - description = "Redis host" - value = azurerm_redis_cache.default.hostname -} - -# Postgresql - -output "postgres_host" { - description = "Postgresql FQDN" - value = azurerm_postgresql_server.default.fqdn -} - -output "postgres_admin_user" { - description = "Postgresql admin username" - value = azurerm_postgresql_server.default.administrator_login - sensitive = true -} - -output "postgres_admin_password" { - description = "Postgresql admin password" - value = azurerm_postgresql_server.default.administrator_login_password - sensitive = true -} - -# Storage -output "storage_account_primary_access_key" { - description = "Storage Account Primary Access Key" - value = azurerm_storage_account.default.primary_access_key - sensitive = true -} diff --git a/examples/terraform/azure/postgresql.tf b/examples/terraform/azure/postgresql.tf deleted file mode 100644 index 221626c..0000000 --- a/examples/terraform/azure/postgresql.tf +++ /dev/null @@ -1,80 +0,0 @@ -##################################################################################### -# Terraform Examples: -# These are pieces of code added as configuration examples for guidance, -# therefore they may require additional resources and variable or local declarations. -##################################################################################### - -locals { - postgresql_name = "postgresql-${random_integer.postgres_suffix.result}" - postgres_admin_user = "postgres_admin_${random_integer.postgres_admin_user.result}" -} - -# Name suffix -resource "random_integer" "postgres_suffix" { - min = 1000 - max = 9999 -} - -# Database instance -resource "azurerm_postgresql_server" "default" { - name = local.postgresql_name - location = azurerm_resource_group.default.location - resource_group_name = azurerm_resource_group.default.name - - # Security - administrator_login = local.postgres_admin_user - administrator_login_password = random_password.postgres_admin_password.result - - # Version - version = "11" - - # Performance - sku_name = "B_Gen5_1" # Basic - storage_mb = 10240 # 10 GB - - # Backups - backup_retention_days = 7 - geo_redundant_backup_enabled = false - auto_grow_enabled = true - - # Networking - public_network_access_enabled = true - ssl_enforcement_enabled = true - ssl_minimal_tls_version_enforced = "TLS1_2" -} - -# Database configuration -resource "azurerm_postgresql_configuration" "default" { - for_each = toset([ - "connection_throttling", - "log_checkpoints", - "log_connections" - ]) - name = each.value - resource_group_name = azurerm_resource_group.default.name - server_name = azurerm_postgresql_server.default.name - value = "on" -} - -# Firewall -resource "azurerm_postgresql_firewall_rule" "default" { - name = "AllowAll" - resource_group_name = azurerm_resource_group.default.name - server_name = azurerm_postgresql_server.default.name - # Warning: The instance will be publicly accessible - start_ip_address = "0.0.0.0" - end_ip_address = "255.255.255.255" -} - -# Postgres user -resource "random_integer" "postgres_admin_user" { - min = 1000 - max = 9999 -} - -# Postgres user's password -resource "random_password" "postgres_admin_password" { - length = 64 - special = true - override_special = "!#$%&*()-_=+[]" -} diff --git a/examples/terraform/azure/redis.tf b/examples/terraform/azure/redis.tf deleted file mode 100644 index 29e1ef5..0000000 --- a/examples/terraform/azure/redis.tf +++ /dev/null @@ -1,46 +0,0 @@ -##################################################################################### -# Terraform Examples: -# These are pieces of code added as configuration examples for guidance, -# therefore they may require additional resources and variable or local declarations. -##################################################################################### - -locals { - # Instance name - redis_instance_name = "${var.redis_name}-${random_integer.random_redis.id}" -} - -# Name suffix -resource "random_integer" "random_redis" { - min = 1000 - max = 9999 -} - -# Redis instance -resource "azurerm_redis_cache" "default" { - # name needs to be globally unique - name = local.redis_instance_name - location = azurerm_resource_group.default.location - resource_group_name = azurerm_resource_group.default.name - - redis_version = 6 - - # Performance - capacity = 0 - family = "C" # Basic/Satandard - sku_name = "Basic" - - # Networking - public_network_access_enabled = true - enable_non_ssl_port = false - minimum_tls_version = "1.2" -} - -# Firewall -resource "azurerm_redis_firewall_rule" "default" { - name = "AllowAll" - resource_group_name = azurerm_resource_group.default.name - redis_cache_name = azurerm_redis_cache.default.name - # Warning: The instance will be publicly accessible - start_ip = "0.0.0.0" - end_ip = "255.255.255.255" -} diff --git a/examples/terraform/azure/storage.tf b/examples/terraform/azure/storage.tf deleted file mode 100644 index 2beb3e3..0000000 --- a/examples/terraform/azure/storage.tf +++ /dev/null @@ -1,60 +0,0 @@ -locals { - # We remove everything that it's not a letter or a number - resource_group_name_parsed = replace(var.resource_group_name, "/[^a-z0-9]/", "") -} - -# FIXME: For rp -# tfsec:ignore:azure-storage-queue-services-logging-enabled -resource "azurerm_storage_account" "default" { - # The name must be unique across all existing storage account names in Azure. - # It must be 3 to 24 characters long, and can contain only lowercase letters - # and numbers. - name = local.resource_group_name_parsed - resource_group_name = azurerm_resource_group.default.name - location = azurerm_resource_group.default.location - - # Performance - account_kind = "StorageV2" - account_tier = "Standard" - account_replication_type = "LRS" # For production ready use GRS or higher - - # Networking - allow_blob_public_access = true - min_tls_version = "TLS1_2" # Older versions are not secure anymore - - # Security - blob_properties { - cors_rule { - allowed_origins = ["*"] - allowed_methods = ["GET", "PUT", "POST"] - allowed_headers = [ - "Access-Control-Request-Headers", - "Cache-Control", - "Content-Disposition", - "Content-MD5", - "Content-Type", - "X-MS-Blob-Type" - ] - exposed_headers = ["*"] - max_age_in_seconds = 3600 - } - } -} - -locals { - # List of storage containers to create. - storage_container = [ - - ] -} - -resource "azurerm_storage_container" "default" { - for_each = toset(local.storage_container) - # This name may only contain lowercase letters, numbers, and hyphens, and must - # begin with a letter or a number. Each hyphen must be preceded and followed - # by a non-hyphen character. The name must also be between 3 and 63 characters - # long. - name = each.value - storage_account_name = azurerm_storage_account.default.name - container_access_type = "private" -} diff --git a/examples/terraform/gcp/gke-autopilot.tf b/examples/terraform/gcp/gke-autopilot.tf deleted file mode 100644 index ca782fb..0000000 --- a/examples/terraform/gcp/gke-autopilot.tf +++ /dev/null @@ -1,87 +0,0 @@ -# Please see the Autopilot documentation -# https://github.com/CartoDB/carto-selfhosted-helm/blob/main/doc/gke/gke-autopilot.md - -# VPC -resource "google_compute_network" "gke_autopilot_network" { - name = "gke-autopilot-network" - project = local.project_id - auto_create_subnetworks = false -} - -# Subnet -resource "google_compute_subnetwork" "gke_autopilot_subnet" { - name = "gke-autopilot-subnet" - project = local.project_id - ip_cidr_range = "10.5.0.0/16" - region = var.region - network = google_compute_network.gke_autopilot_network.id -} - -# GKE Autopilot private cluster -resource "google_container_cluster" "default" { - name = "gke-autopilot" - project = local.project_id - location = var.region - - # Private clusters use nodes that do not have external IP addresses. - # This means that clients on the internet cannot connect to the IP addresses of the nodes. - private_cluster_config { - enable_private_nodes = true - enable_private_endpoint = true - # Control plane nodes are not accessible globally - master_global_access_config { - enabled = false - } - } - - release_channel { - channel = "STABLE" - } - - network = google_compute_network.gke_autopilot_network.name - subnetwork = google_compute_subnetwork.gke_autopilot_subnet.name - - ip_allocation_policy { - # There settings are permanent and they cannot be changed once the cluster is deployed - # Cluster default pod address range. All pods in the cluster are assigned an IP address from this range. Enter a range (in CIDR notation) within a network range, a mask, or leave this field blank to use a default range. - # We recommend at least /21 mask for pods - cluster_ipv4_cidr_block = "/21" - # Service address range. Cluster services will be assigned an IP address from this IP address range. Enter a range (in CIDR notation) within a network range, a mask, or leave this field blank to use a default range. - # We recommend at least /24 mask for services - services_ipv4_cidr_block = "/24" - } - - # At this point, these are the only IP addresses that have access to the control plane: - # - The primary range for the subnet: google_compute_subnetwork.gke_autopilot_subnet - # - The secondary range for the pods: google_container_cluster.default.ip_allocation_policy.cluster_ipv4_cidr_block - # If you need to allow external networks to access Kubernetes master through HTTPS, please see: - # https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_cluster#master_authorized_networks_config - - # Enabling Autopilot for this cluster - enable_autopilot = true -} - -# ServiceAccount to be using in workload identity -resource "google_service_account" "workload_identity_sa" { - project = local.project_id - account_id = "workload-identity-iam-sa" - display_name = "A service account to be used by GKE Workload Identity" -} - -# Binding between IAM SA and Kubernetes SA -resource "google_service_account_iam_binding" "gke_iam_binding" { - service_account_id = google_service_account.workload_identity_sa.name - role = "roles/iam.workloadIdentityUser" - - members = [ - # "serviceAccount:.svc.id.goog[/-common-backend]" - "serviceAccount:${local.project_id}.svc.id.goog[carto/carto-common-backend]", - ] -} - -# This role enables impersonation of service accounts to create OAuth2 access tokens, sign blobs, or sign JWTs -resource "google_service_account_iam_member" "workload_identity_sa_sign_urls" { - service_account_id = google_service_account.workload_identity_sa.name - role = "roles/iam.serviceAccountTokenCreator" - member = "serviceAccount:${google_service_account.workload_identity_sa.email}" -} diff --git a/examples/terraform/gcp/gke.tf b/examples/terraform/gcp/gke.tf deleted file mode 100644 index a05bb3b..0000000 --- a/examples/terraform/gcp/gke.tf +++ /dev/null @@ -1,60 +0,0 @@ - -locals { - cluster_type = "nodepool" - cluster_name = "${var.gke_cluster_name}-${random_integer.suffix.result}" -} - -resource "random_integer" "suffix" { - min = 1000 - max = 9999 -} - -data "google_client_config" "default" {} - -provider "kubernetes" { - host = "https://${module.gke.endpoint}" - token = data.google_client_config.default.access_token - cluster_ca_certificate = base64decode(module.gke.ca_certificate) -} - -# tflint-ignore: terraform_module_version -module "gke" { - source = "terraform-google-modules/kubernetes-engine/google" - project_id = local.project_id - name = local.cluster_name - region = var.region - zones = var.zones - network = google_compute_network.carto_selfhosted_network.name - subnetwork = google_compute_subnetwork.carto_selfhosted_subnet.name - ip_range_pods = var.ip_range_pods_name - ip_range_services = var.ip_range_services_name - create_service_account = true - remove_default_node_pool = true - disable_legacy_metadata_endpoints = false - default_max_pods_per_node = 16 - node_pools = [ - { - name = "pool-01" - machine_type = var.node_pool_instance_type - node_locations = "${var.region}-b" - autoscaling = true - min_count = 1 - max_count = 5 - disk_size_gb = 30 - disk_type = "pd-standard" - auto_upgrade = false - }, - { - name = "pool-02" - machine_type = var.node_pool_instance_type - node_locations = "${var.region}-d" - autoscaling = true - min_count = 1 - max_count = 5 - disk_size_gb = 30 - disk_type = "pd-standard" - auto_upgrade = false - }, - ] - -} diff --git a/examples/terraform/gcp/main.tf b/examples/terraform/gcp/main.tf deleted file mode 100644 index 3ec10cf..0000000 --- a/examples/terraform/gcp/main.tf +++ /dev/null @@ -1,32 +0,0 @@ -locals { - # project_id = - - # Postgresql - postgresql_availability_type = var.production_mode ? "REGIONAL" : "ZONAL" - postgreql_maintenance_window = var.production_mode ? { - day = 1 - hour = 5 - update_track = "stable" - } : { - day = 5 - hour = 7 - update_track = "canary" - } - postgreql_backup_configuration = var.enable_create_internal_sql_backups ? { - enabled = true - pitr_enabled = true - } : { - enabled = false - pitr_enabled = false - } - postgresql_deletion_protection = var.postgresql_deletion_protection != null ? var.postgresql_deletion_protection : var.production_mode - - # Redis - redis_maintenance_window = var.production_mode ? { - day = "MONDAY" - hour = 5 - } : { - day = "FRIDAY" - hour = 7 - } -} diff --git a/examples/terraform/gcp/postgresql.tf b/examples/terraform/gcp/postgresql.tf deleted file mode 100644 index 76468e0..0000000 --- a/examples/terraform/gcp/postgresql.tf +++ /dev/null @@ -1,118 +0,0 @@ -##################################################################################### -# Terraform Examples: -# These are pieces of code added as configuration examples for guidance, -# therefore they may require additional resources and variable or local declarations. -##################################################################################### - -# Cloud SQL instance -resource "google_sql_database_instance" "default" { - name = var.postgresl_name - project = local.project_id - database_version = var.postgresql_version - deletion_protection = local.postgresql_deletion_protection - region = var.region - settings { - disk_autoresize = var.postgresql_disk_autoresize - disk_size = var.postgresql_disk_size_gb - disk_type = var.production_mode ? "PD_SSD" : "PD_HDD" - tier = var.postgresql_tier - availability_type = local.postgresql_availability_type - - user_labels = { - "owner" = "product" - } - - dynamic "database_flags" { - for_each = { - log_checkpoints = "on" - log_connections = "on" - log_disconnections = "on" - log_lock_waits = "on" - log_temp_files = "0" - } - iterator = flag - - content { - name = flag.key - value = flag.value - } - } - - ip_configuration { - # Necessary to connect via Unix sockets - # https://cloud.google.com/sql/docs/mysql/connect-run#connecting_to - ipv4_enabled = true - private_network = google_compute_network.carto_selfhosted_network.id - require_ssl = false - } - - location_preference { - zone = var.zone - } - - maintenance_window { - day = local.postgreql_maintenance_window.day - hour = local.postgreql_maintenance_window.hour - update_track = local.postgreql_maintenance_window.update_track - } - - backup_configuration { - enabled = local.postgreql_backup_configuration.enabled - point_in_time_recovery_enabled = local.postgreql_backup_configuration.pitr_enabled - backup_retention_settings { - retained_backups = 30 - } - } - - insights_config { - query_insights_enabled = true - query_string_length = 1024 - record_application_tags = false - record_client_address = true - } - } - - lifecycle { - create_before_destroy = true - } -} - -# Credentials - -## Postgres Admin User - -resource "google_sql_user" "postgres_admin_user" { - name = "postgres" - project = local.project_id - instance = google_sql_database_instance.default.name - type = "BUILT_IN" - password = random_password.postgres-admin-user-password.result - lifecycle { - ignore_changes = [ - type - ] - } -} - -## Postgres Admin Password - -resource "random_password" "postgres-admin-user-password" { - length = 16 - special = false - upper = true - lower = true - number = true -} - -resource "google_secret_manager_secret" "postgres_admin_user_password_secret" { - secret_id = "postgres-admin-password-${google_sql_database_instance.default.name}" - project = local.project_id - replication { - automatic = true - } -} - -resource "google_secret_manager_secret_version" "cloudrun_admin_user_password_secret_version" { - secret = google_secret_manager_secret.postgres_admin_user_password_secret.id - secret_data = random_password.postgres-admin-user-password.result -} diff --git a/examples/terraform/gcp/private-service-access.tf b/examples/terraform/gcp/private-service-access.tf deleted file mode 100644 index f4f29d8..0000000 --- a/examples/terraform/gcp/private-service-access.tf +++ /dev/null @@ -1,14 +0,0 @@ -resource "google_compute_global_address" "service_range" { - name = "address" - project = local.project_id - purpose = "VPC_PEERING" - address_type = "INTERNAL" - prefix_length = 16 - network = google_compute_network.carto_selfhosted_network.name -} - -resource "google_service_networking_connection" "private_service_connection" { - network = google_compute_network.carto_selfhosted_network.id - service = "servicenetworking.googleapis.com" - reserved_peering_ranges = [google_compute_global_address.service_range.name] -} diff --git a/examples/terraform/gcp/redis.tf b/examples/terraform/gcp/redis.tf deleted file mode 100644 index 048857a..0000000 --- a/examples/terraform/gcp/redis.tf +++ /dev/null @@ -1,55 +0,0 @@ -##################################################################################### -# Terraform Examples: -# These are pieces of code added as configuration examples for guidance, -# therefore they may require additional resources and variable or local declarations. -##################################################################################### - -locals { - # Instance name - redis_instance_name = "${var.redis_name}-${random_integer.random_redis.id}" -} - -# Name suffix -resource "random_integer" "random_redis" { - min = 1000 - max = 9999 -} - -# Redis instance -resource "google_redis_instance" "default" { - name = local.redis_instance_name - project = local.project_id - region = var.region - location_id = var.zone - memory_size_gb = var.redis_memory_size_gb - auth_enabled = true - tier = var.redis_tier - redis_version = var.redis_version - connect_mode = "PRIVATE_SERVICE_ACCESS" - authorized_network = google_compute_network.carto_selfhosted_network.id - depends_on = [google_service_networking_connection.private_service_connection] - - maintenance_policy { - weekly_maintenance_window { - day = local.redis_maintenance_window.day - start_time { - hours = local.redis_maintenance_window.hour - } - } - } -} - -# Credentials stored in Google Secret Manager - -resource "google_secret_manager_secret" "redis_password" { - secret_id = "redis-auth-string-${google_redis_instance.default.name}" - project = local.project_id - replication { - automatic = true - } -} - -resource "google_secret_manager_secret_version" "redis_password" { - secret = google_secret_manager_secret.redis_password.id - secret_data = google_redis_instance.default.auth_string -} diff --git a/examples/terraform/gcp/settings.tf b/examples/terraform/gcp/settings.tf deleted file mode 100644 index 49512df..0000000 --- a/examples/terraform/gcp/settings.tf +++ /dev/null @@ -1,18 +0,0 @@ -terraform { - required_providers { - google = { - source = "hashicorp/google" - version = "~> 4.0" - } - kubernetes = { - source = "hashicorp/kubernetes" - } - random = { - source = "hashicorp/random" - version = ">= 2.2" - } - } - required_version = "~> 1.0" - - backend "gcs" {} -} \ No newline at end of file diff --git a/examples/terraform/gcp/storage.tf b/examples/terraform/gcp/storage.tf deleted file mode 100644 index 5ade563..0000000 --- a/examples/terraform/gcp/storage.tf +++ /dev/null @@ -1,120 +0,0 @@ -locals { - bucket_client_name = "${local.project_id}-client-storage" - bucket_thumbnails_name = "${local.project_id}-thumbnails-storage" - bucket_import_name = "${local.project_id}-import-storage" - carto_service_account_id = "carto-selfhosted-serv-account" -} - -## GCS - -# Client storage bucket -resource "google_storage_bucket" "client_storage" { - name = local.bucket_client_name - project = local.project_id - location = var.region - - uniform_bucket_level_access = true - - cors { - origin = ["*"] - method = ["GET", "PUT", "POST", ] - response_header = [ - "Content-Type", - "Content-MD5", - "Content-Disposition", - "Cache-Control", - "x-goog-content-length-range", - "x-goog-meta-filename" - ] - max_age_seconds = 3600 - } - - lifecycle_rule { - condition { - age = 30 - } - action { - type = "Delete" - } - } -} - -resource "google_storage_bucket_iam_binding" "bucket_client_storage_workspace_api" { - bucket = local.bucket_client_name - role = "roles/storage.admin" - members = ["serviceAccount:${google_service_account.carto_selfhosted_service_account.email}"] -} - -# Thumbnails storage bucket -resource "google_storage_bucket" "thumbnails_storage" { - name = local.bucket_thumbnails_name - project = local.project_id - location = var.region - - uniform_bucket_level_access = true - - cors { - origin = ["*"] - method = ["GET", "PUT", "POST", ] - response_header = [ - "Content-Type", - "Content-MD5", - "Content-Disposition", - "Cache-Control", - "x-goog-content-length-range", - "x-goog-meta-filename" - ] - max_age_seconds = 3600 - } - - versioning { - enabled = true - } -} - -resource "google_storage_bucket_iam_binding" "bucket_thumbnails_storage_workspace_api" { - bucket = local.bucket_thumbnails_name - role = "roles/storage.admin" - members = ["serviceAccount:${google_service_account.carto_selfhosted_service_account.email}"] -} - -# Import storage bucket -resource "google_storage_bucket" "import_storage" { - name = local.bucket_import_name - project = local.project_id - location = var.region - - uniform_bucket_level_access = true - - lifecycle_rule { - condition { - age = 30 - } - action { - type = "Delete" - } - } -} - -resource "google_storage_bucket_iam_binding" "import_storage_import" { - bucket = local.bucket_import_name - role = "roles/storage.admin" - members = ["serviceAccount:${google_service_account.carto_selfhosted_service_account.email}"] -} - - -## IAM - -# Service account for the self hosted -resource "google_service_account" "carto_selfhosted_service_account" { - project = local.project_id - account_id = local.carto_service_account_id - display_name = "Carto Self Hosted Service Account" -} - -# Allows Carto self hosted service account to create signedUrls -resource "google_service_account_iam_member" "carto_selfhosted_service_account_token_creator" { - service_account_id = google_service_account.carto_selfhosted_service_account.id - role = "roles/iam.serviceAccountTokenCreator" - member = "serviceAccount:${google_service_account.carto_selfhosted_service_account.email}" -} diff --git a/examples/terraform/gcp/terraform.tfvars b/examples/terraform/gcp/terraform.tfvars deleted file mode 100644 index f504aa2..0000000 --- a/examples/terraform/gcp/terraform.tfvars +++ /dev/null @@ -1,26 +0,0 @@ - -# GKE -gke_cluster_name = "gke-default" -region = "europe-west1" -zones = ["europe-west1-b", "europe-west1-c", "europe-west1-d"] -node_pool_instance_type = "e2-standard-8" -ip_range_pods_name = "pod-ranges" -ip_range_services_name = "services-range" - -activate_apis_custom = [ - "container.googleapis.com", - "secretmanager.googleapis.com", -] - -# Postgresql -postgresl_name = "carto-selfhosted-postgres" -postgresql_version = "POSTGRES_13" -enable_create_internal_sql_backups = true -postgresql_tier = "db-custom-1-3840" - -# Redis -redis_name = "carto-selfhosted-redis" - -# common -production_mode = false -zone = "europe-west1-b" diff --git a/examples/terraform/gcp/variables.tf b/examples/terraform/gcp/variables.tf deleted file mode 100644 index c22df16..0000000 --- a/examples/terraform/gcp/variables.tf +++ /dev/null @@ -1,112 +0,0 @@ -# GKE - -variable "gke_cluster_name" { - type = string - description = "Name of the EKS cluster" -} - -variable "region" { - type = string - description = "GCP region" -} - -variable "zones" { - type = list(string) - description = "The zone to host the cluster in (required if is a zonal cluster)" -} - -variable "node_pool_instance_type" { - type = string - description = "Node pool machine types to deploy pods in gke cluster" -} - -variable "ip_range_pods_name" { - type = string - description = "IP range subnet name for pods" -} - -variable "ip_range_services_name" { - type = string - description = "IP range subnet name for services" -} - -# common - -variable "production_mode" { - description = "If production_mode is enabled we enable backup, PITR and HA" - type = bool -} - -variable "zone" { - description = "Gcloud project zone" - type = string -} - -# redis - -variable "redis_name" { - type = string - description = "Name of the Redis instance" -} - -variable "redis_memory_size_gb" { - type = number - description = "Redis memory size" - default = 1 -} - -variable "redis_tier" { - type = string - description = "Redis tier. If we are going to really use in production, we must use `var.production_mode`" - default = "BASIC" -} - -variable "redis_version" { - type = string - description = "Redis version to use.\nhttps://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/redis_instance#redis_version" - default = "REDIS_6_X" -} - -# Postgres - -variable "postgresl_name" { - type = string - description = "Name of the postgresql instance" -} - -variable "postgresql_version" { - type = string - description = "Version of postgres to use.\nhttps://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/sql_database_instance#database_version" - default = "POSTGRES_13" -} - -variable "postgresql_deletion_protection" { - type = bool - description = "Enable the deletion_protection for the database please. By default, it's the same as `production_mode` variable" - default = null -} - -variable "postgresql_disk_autoresize" { - type = bool - description = "Enable postgres autoresize" - default = true -} - -variable "postgresql_disk_size_gb" { - type = number - description = "Default postgres disk_size. Keep in mind that the value could be auto-increased using `postgresql_disk_autoresize` variable" - default = 10 - -} - -variable "postgresql_tier" { - description = "Postgres machine type to use" - type = string -} - -## Backups - -variable "enable_create_internal_sql_backups" { - description = "Indicate if create internal db backups managed by cloud-sql" - type = bool -} diff --git a/examples/terraform/gcp/vpc.tf b/examples/terraform/gcp/vpc.tf deleted file mode 100644 index eae54b0..0000000 --- a/examples/terraform/gcp/vpc.tf +++ /dev/null @@ -1,23 +0,0 @@ -#tfsec:ignore:google-compute-enable-vpc-flow-logs -resource "google_compute_subnetwork" "carto_selfhosted_subnet" { - name = "carto-selfhosted-subnet" - project = local.project_id - ip_cidr_range = "10.2.0.0/16" - region = var.region - network = google_compute_network.carto_selfhosted_network.id - secondary_ip_range { - range_name = var.ip_range_services_name - ip_cidr_range = "192.168.1.0/24" - } - - secondary_ip_range { - range_name = var.ip_range_pods_name - ip_cidr_range = "192.168.64.0/22" - } -} - -resource "google_compute_network" "carto_selfhosted_network" { - name = "carto-selfhosted-network" - project = local.project_id - auto_create_subnetworks = false -} diff --git a/img/header-docker.png b/img/header-docker.png new file mode 100644 index 0000000..bf06127 Binary files /dev/null and b/img/header-docker.png differ diff --git a/proxy/config/whitelisted_domains.md b/proxy/config/whitelisted_domains.md deleted file mode 100644 index 631c6fa..0000000 --- a/proxy/config/whitelisted_domains.md +++ /dev/null @@ -1,133 +0,0 @@ -# Whitelisted domains - -In case you are setting up some firewall to control the outgoing connections from CARTO Self Hosted, the following -domains needs to be accepted: - -
-Full whitelisted domain list - -``` -## Global -auth.carto.com -bigquery.googleapis.com -cloudresourcemanager.googleapis.com -gcr.io -iamcredentials.googleapis.com -logging.googleapis.com -pubsub.googleapis.com -storage.googleapis.com -tools.google.com -www.googleapis.com -clientstream.launchdarkly.com -events.launchdarkly.com -stream.launchdarkly.com - -## Datawarehouses -.snowflakecomputing.com - -## Mapbox geocoding -api.mapbox.com - -## Tomtom geocoding -api.tomtom.com - -## Here geocoding -isoline.router.hereapi.com - -## Google geocoding and basemaps -maps.googleapis.com - -## Custom external dabases -sqladmin.googleapis.com - -## AWS S3 buckets -.amazonaws.com - -## Azure storage buckets -.blob.core.windows.net - -## Bigquery Oauth connections -oauth2.googleapis.com -``` - -
- -## General setup - -| URL | -|---| -| auth.carto.com | -| bigquery.googleapis.com | -| cloudresourcemanager.googleapis.com | -| gcr.io | -| iamcredentials.googleapis.com | -| logging.googleapis.com | -| pubsub.googleapis.com | -| storage.googleapis.com | -| tools.google.com | -| www.googleapis.com | -| clientstream.launchdarkly.com | -| events.launchdarkly.com | -| stream.launchdarkly.com | - - -## Datawarehouses - -### Snowflake - -| URL | -|---| -| *.snowflakecomputing.com | - -## Custom Geocoding configurations - -### Tomtom geocoding -| URL | -|---| -| api.tomtom.com | - -### Mapbox geocoding - -| URL | -|---| -| api.mapbox.com | - -### Here geocoding and isolines - -| URL | -|---| -| isoline.router.hereapi.com | - -### Google Maps geocoding and basemaps - -| URL | -|---| -| maps.googleapis.com | - -## Custom external dabases - -### Google Cloud SQL - -| URL | -|---| -| sqladmin.googleapis.com | - -## Custom Buckets - -### AWS S3 buckets - -| URL | -|---| -|*.amazonaws.com (or your full bucket URLs) | - -### Azure Blob Storage - -| URL | -|---| -| *.blob.core.windows.net (or your full bucket URLs) | - -## BigQuery Oauth connections - -| URL | -|---| -| oauth2.googleapis.com |