Skip to content

Commit

Permalink
Got it working in customer environment (#14)
Browse files Browse the repository at this point in the history
* Refactoring docker compose files for easier execution and customer PoC usage of repo-converter
* Moved running process check to beginning of loop to skip earlier
* Better handling and checking of child processes
* Deleting abandoned lockfiles
* Starting on batch sizes (not finished)
* Removing psutils from docker image build to speed up build time
* Updating README to match recent changes
* Updating cloud-agent to latest
* Showing more output for LOG_LEVEL=INFO
  • Loading branch information
marcleblanc2 authored Feb 20, 2024
1 parent 05518a8 commit c9a859a
Show file tree
Hide file tree
Showing 18 changed files with 621 additions and 222 deletions.
37 changes: 21 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Running src serve-git and the agent together on the same Docker network allows t
Docker compose also allows for easier upgrades, troubleshooting, monitoring, logging, flexibility of hosting, etc. than running the binaries directly on the OS.

## Setup - Sourcegraph Staff Only
1. Add the needed entries to the sourcegraphConnect targetGroups list in the Cloud instance's config.yaml, get your PR approved and merged
1. Add the needed entries to the sourcegraphConnect targetGroups list in the Cloud instance's config.yaml, and get your PR approved and merged
```yaml
- dnsName: src-serve-git-ubuntu.local
listeningAddress: 100.100.100.0
Expand All @@ -22,20 +22,25 @@ Docker compose also allows for easier upgrades, troubleshooting, monitoring, log
ports:
- 443
```
2. Run the Reload frontend GitHub Action, as this seems to be needed for the frontend pods to start using tunnel connections
2. Run the "Reload Instance for srcconnect config change" GitHub Action, as many containers need to be restarted to pick up tunnel connection config changes
3. Clone this repo to a customer's bridge VM, install Docker and Docker's Compose plugin
4. Copy the `config.yaml` and `service-account-key.json` files using the instructions on the instance's Cloud Ops dashboard
- Paste them into `./config/cloud-agent-config.yaml` and `./config/cloud-agent-service-account-key.json`
5. Modify the `./config/cloud-agent-config.yaml` file
- `serviceAccountKeyFile: /sourcegraph/cloud-agent-service-account-key.json` so that the Go binary inside the agent container finds this file in the path that's mapped via the docker-compose.yaml files
- Only include the `- dialAddress` entries that this cloud agent instance can reach, remove the others, so the Cloud instance doesn't try using this agent instance for code hosts it can't reach
- Use extra caution when pasting the config.yaml in Windows, as it may use Windows' line endings or extra spaces, which breaks YAML, as a whitespace-dependent format
6. Clone the customer's repos into the `repos-to-serve` directory at the root of this repo on the bridge VM, or update the volume mount path for the src-serve-git service in the docker-compose.yaml file
7. Run `docker compose up -d`
8. Add a Code Host config to the customer's Cloud instance
- Type: src serve-git
- URL: "http://src-serve-git-ubuntu.local:443"
- or
- URL: "http://src-serve-git-wsl.local:443"
- Note that the :443 port may be required, as this seems to default to port 443, even when used with http://
9. Use the repo-converter to convert SVN, TFVC, or Git repos, to Git format, which will store them in the `repos-to-serve` directory, or use any other means to get the repos into the directory
- Paste them into `./config/cloud-agent-config.yaml` and `./config/cloud-agent-service-account-key.json`
5. Modify the contents of the `./config/cloud-agent-config.yaml` file:
- `serviceAccountKeyFile: /sourcegraph/cloud-agent-service-account-key.json` so that the Go binary inside the agent container finds this file in the path that's mapped via the docker-compose.yaml files
- Only include the `- dialAddress` entries that this cloud agent instance can reach, remove the others, so the Cloud instance doesn't try using this agent instance for code hosts it can't reach
- Use extra caution when pasting the config.yaml in Windows, as it may use Windows' line endings or extra spaces, which breaks YAML, as a whitespace-dependent format
6. Run `docker compose up -d`
7. Add a Code Host config to the customer's Cloud instance
- Type: src serve-git
- `"url": "http://src-serve-git-ubuntu.local:443",`
- or
- `"url": "http://src-serve-git-wsl.local:443",`
- Note the port 443, even when used with http://
8. Use the repo-converter to convert SVN, TFVC, or Git repos, to Git format, which will store them in the `src-serve-root` directory, or use any other means to get the repos into the directory
- There are docker-compose.yaml and override files in a few different directories in this repo, separated by use case, so that each use case only needs to run `docker compose up -d` in one directory, and not fuss around with `-f` paths.
- The only difference between the docker-compose-override.yaml files in host-ubuntu vs host-wsl is the src-serve-git container's name, which is how we get a separate `dnsName` for each.
- If you're using the repo-converter:
- If you're using the pre-built images, `cd ./repo-converter && docker compose up -d`
- If you're building the Docker images, `cd ./repo-converter/build && docker compose up -d --build`
- Either of these will start all 3 containers: cloud-agent, src-serve-git, and the repo-converter
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ services:

cloud-agent:
container_name: cloud-agent
image: index.docker.io/sourcegraph/src-tunnel-agent:2024-02-05-22-28-333320ee
image: index.docker.io/sourcegraph/src-tunnel-agent:2024-02-15-04-02-110c2ea9
volumes:
- ../config/cloud-agent-service-account-key.json:/sourcegraph/cloud-agent-service-account-key.json:ro
- ../config/cloud-agent-config.yaml:/sourcegraph/cloud-agent-config.yaml:ro
Expand All @@ -15,7 +15,7 @@ services:

src-serve-git:
# Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
container_name: src-serve-git-wsl.local
container_name: src-serve-git-ubuntu.local
image: index.docker.io/sourcegraph/src-cli:latest
volumes:
- ../src-serve-root/:/sourcegraph/src-serve-root:ro
Expand Down
5 changes: 5 additions & 0 deletions host-ubuntu/docker-compose-override.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
services:

src-serve-git:
# Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
container_name: src-serve-git-ubuntu.local
31 changes: 4 additions & 27 deletions host-ubuntu/docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -1,27 +1,4 @@
version: '2.4'

services:

cloud-agent:
container_name: cloud-agent
image: index.docker.io/sourcegraph/src-tunnel-agent:2024-02-05-22-28-333320ee
volumes:
- ../config/cloud-agent-service-account-key.json:/sourcegraph/cloud-agent-service-account-key.json:ro
- ../config/cloud-agent-config.yaml:/sourcegraph/cloud-agent-config.yaml:ro
command: ["-config=/sourcegraph/cloud-agent-config.yaml"]
restart: always
networks:
- sourcegraph

src-serve-git:
# Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
container_name: src-serve-git-ubuntu.local
image: index.docker.io/sourcegraph/src-cli:latest
volumes:
- ../src-serve-root/:/sourcegraph/src-serve-root:ro
command: "serve-git -addr :443 /sourcegraph/src-serve-root"
networks:
- sourcegraph

networks:
sourcegraph:
include:
- path:
- ../config/docker-compose-common-services.yaml
- docker-compose-override.yaml
5 changes: 5 additions & 0 deletions host-wsl/docker-compose-override.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
services:

src-serve-git:
# Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
container_name: src-serve-git-wsl.local
4 changes: 4 additions & 0 deletions host-wsl/docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
include:
- path:
- ../config/docker-compose-common-services.yaml
- docker-compose-override.yaml
5 changes: 3 additions & 2 deletions host-wsl2-ubuntu/wsl2-startup.sh → host-wsl/wsl2-startup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,20 @@
log_file="./log"
git_exit_status=""
docker_compose_exit_status=""
repo_build_path="/sourcegraph/implementation-bridges/repo-converter/build"

echo "Starting $0 $@" >> $log_file

# Git pull latest commits to main
if ! git pull
if ! git -C $repo_build_path pull
then
git_exit_status=$?
echo "git pull failed, exit code $git_exit_status" >> $log_file
exit $git_exit_status
fi

# Start docker compose services
if ! docker compose up -d
if ! docker compose -f $repo_build_path/docker-compose.yaml up -d --build
then
docker_compose_exit_status=$?
echo "docker compose up failed, exit code $docker_compose_exit_status" >> $log_file
Expand Down
5 changes: 3 additions & 2 deletions repo-converter/build/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,9 @@ RUN apt-get update && \
apt-get upgrade -y && \
apt-get install --no-install-recommends -y \
cron \
gcc \
git \
git-svn \
python3 \
python3-dev \
python3-pip \
python3-wheel \
subversion \
Expand All @@ -33,6 +31,9 @@ RUN apt-get update && \
systemctl enable cron && \
systemctl start cron

# python3-dev \
# gcc \

# Download dependencies as a separate step to take advantage of Docker's caching
# Leverage a cache mount to /root/.cache/pip to speed up subsequent builds
# Leverage a bind mount to requirements.txt to avoid having to copy them into
Expand Down
6 changes: 6 additions & 0 deletions repo-converter/build/docker-compose-override.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
services:

src-serve-git:
# Uses a valid hostname as container_name, to trick the cloud agent and code host config into finding this container on the Docker network
container_name: src-serve-git-wsl.local
command: "-v serve-git -addr :443 /sourcegraph/src-serve-root"
11 changes: 9 additions & 2 deletions repo-converter/build/docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -1,15 +1,22 @@
version: '2.4'

include:
- path:
- ../../config/docker-compose-common-services.yaml
- docker-compose-override.yaml

services:

repo-converter:
container_name: repo-converter
build:
context: .
image: sourcegraph/implementation-bridge-repo-converter:build
restart: always
volumes:
- ../../config/repos-to-convert.yaml:/sourcegraph/repos-to-convert.yaml:ro
- ../../config/toprc:/root/.config/procps/toprc
- ../../src-serve-root/:/sourcegraph/src-serve-root
environment:
- BRIDGE_REPO_CONVERTER_INTERVAL_SECONDS=10
- LOG_LEVEL=DEBUG # DEBUG INFO WARNING ERROR CRITICAL # Default is INFO
- BRIDGE_REPO_CONVERTER_INTERVAL_SECONDS=60
- LOG_LEVEL=DEBUG # DEBUG INFO WARNING ERROR CRITICAL # Default is INFO
14 changes: 14 additions & 0 deletions repo-converter/build/pull-build-start.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/bin/bash

# To be used in a cronjob to always pull and build the latest commit to the current branch
# every 10 minutes
# so that the running container is only 10 minutes behind the latest commit in the branch

# crontab -e
# */10 * * * * sudo bash /sourcegraph/implementation-bridges/repo-converter/build/pull-build-start.sh >> /sourcegraph/implementation-bridges/repo-converter/build/pull-build-start.log 2>&1

repo_converter_build_path="/sourcegraph/implementation-bridges/repo-converter/build"

git -C $repo_converter_build_path pull

docker compose -f $repo_converter_build_path/docker-compose.yaml up -d --build
1 change: 0 additions & 1 deletion repo-converter/build/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
psutil
PyYAML
Loading

0 comments on commit c9a859a

Please sign in to comment.