Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial draft of linux instructions and removal of internal instructions #391

Merged
merged 3 commits into from
Jul 17, 2024
Merged

Initial draft of linux instructions and removal of internal instructions #391

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7459442
Initial draft of linux instructions and removal of internal instructions
carsoncall Jun 21, 2024
8b91eb5
add changes from google doc
carsoncall Jun 27, 2024
e1cbbfd
readme: move issues to github good first issues, move database instru…
devxpy Jul 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Procfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,4 @@ dashboard: poetry run streamlit run Home.py --server.port 8501 --server.headless

celery: poetry run celery -A celeryapp worker -P threads -c 16 -l DEBUG

ui: cd ../gooey-ui/; PORT=3000 npm run dev
ui: cd ../gooey-gui/; PORT=3000 npm run dev
245 changes: 90 additions & 155 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,20 @@
## Setup
## What is Gooey Server?
nikochiko marked this conversation as resolved.
Show resolved Hide resolved
Gooey.AI is a low-code AI recipe platform and Gooey Server is our core repo.
It allows users to discover, customize, and deploy AI "recipes" using the best of private and open source AI,
all using a single API with a single auth token.
Recipes are workflows that incorporate various models to accomplish a task; they are designed to be highly customizable and shareable.

### Who is this for and why would I want to use it?
For the vast majority of developers, we DO NOT recommend running or forking Gooey Server; use our APIs or client SDK instead.
The repo is intended only for developers that want to run and deploy their own server cluster or run Gooey locally for development purposes.

Specifically, this repo may be for you if:
- You want to create a new recipe (instead of changing the parameters on an existing one)
- You want to add an AI model that we currently don’t support.
- You are an enterprise who has specific requirements regarding data practices, such as using specific cloud providers.
- You want to add some other functionality that we don’t support.

* Install [pyenv](https://github.com/pyenv/pyenv) & install the same python version as in our [Dockerfile](Dockerfile)
* Install [poetry](https://python-poetry.org/docs/)
* Clone the github repo to gooey-server (and make sure that's the folder name)
* Create & activate a virtualenv (e.g. `poetry shell`)
* Run `poetry install --with dev`
* Install [redis](https://redis.io/docs/getting-started/installation/install-redis-on-mac-os/), [rabbitmq](https://www.rabbitmq.com/install-homebrew.html), and [postgresql](https://formulae.brew.sh/formula/postgresql@15) (e.g. `brew install redis rabbitmq postgresql@15`)
* Enable background services for `redis`, `rabbitmq`, and `postgresql` (e.g. with `brew services start redis` and similar for `rabbitmq` and `postgresql`)
* Use `sqlcreate` helper to create a user and database for gooey:
* `./manage.py sqlcreate | psql postgres`
* make sure you are able to access the database with `psql -W -U gooey gooey` (and when prompted for password, entering `gooey`)
* Create an `.env` file from `.env.example` (Read [12factor.net/config](https://12factor.net/config))
* Run `./manage.py migrate`
* Install the zbar library (`brew install zbar`)
## Setup

### Create a google cloud / firebase account

Expand All @@ -35,19 +37,70 @@
5. Create and Download a JSON Key for this service account and save it to the project root as `serviceAccountKey.json`.
6. Add your project & bucket name to `.env`

* Run tests to see if everything is working fine:
```
./scripts/run-tests.sh
```
(If you run into issues with the number of open files, you can remove the limit with `ulimit -n unlimited`)

### MacOS

* Install [pyenv](https://github.com/pyenv/pyenv) & install the same python version as in our [Dockerfile](Dockerfile)
* Install [poetry](https://python-poetry.org/docs/)
* Clone the github repo to `gooey-server` (and make sure that's the folder name)
* Create & activate a virtualenv (e.g. `poetry shell`)
* Run `poetry install --with dev`
* Install [redis](https://redis.io/docs/getting-started/installation/install-redis-on-mac-os/), [rabbitmq](https://www.rabbitmq.com/install-homebrew.html), and [postgresql](https://formulae.brew.sh/formula/postgresql@15) (e.g. `brew install redis rabbitmq postgresql@15`)
* Enable background services for `redis`, `rabbitmq`, and `postgresql` (e.g. with `brew services start redis` and similar for `rabbitmq` and `postgresql`)
* Use `sqlcreate` helper to create a user and database for gooey:
* `./manage.py sqlcreate | psql postgres`
* make sure you are able to access the database with `psql -W -U gooey gooey` (and when prompted for password, entering `gooey`)
* Create an `.env` file from `.env.example` (Read [12factor.net/config](https://12factor.net/config))
* Run `./manage.py migrate`
* Install the zbar library (`brew install zbar`)
* (optional) Install imagemagick - Needed for HEIC image support - https://docs.wand-py.org/en/0.5.7/guide/install.html
```
brew install freetype imagemagick
export MAGICK_HOME=/opt/homebrew
```

### Linux
* Install [pyenv](https://github.com/pyenv/pyenv) & install the same python version as in our [Dockerfile](Dockerfile) (currently Python 3.10)
- `curl https://pyenv.run | bash`
* Install [poetry](https://python-poetry.org/docs/)
- This is likely available in your distro's package repos.
* Clone this repository:
* Create and activate a virtualenv using `poetry shell`
* Install dependencies using `poetry install --with dev`
- Note: you may have to remove `package-mode=false` on line 7 of `pyproject.toml`
* Install redis, rabbitmq-server, and postgresql 15 using your distro's package manager.
* Enable these services as background services using `sudo systemctl enable --now redis rabbitmq-server postgresql`
* Configure Postgres to ensure that password authentication is enabled for the gooey user
- open the pg_hba.conf file in a text editor. On Linux, by default, it is usually located either at ```/etc/postgresql/<version>/main/``` or ```/var/lib/pgsql/<version>/data/```
- add/edit the file so that there are lines at the bottom that looks like this:
```
local all gooey md5
host all gooey md5
```
- restart postgresql using ```sudo systemctl restart postgresql```
* Use the manage.py script to set up the Postgres database:
- To create the user and database for gooey: `./manage.py sqlcreate | sudo -u postgres psql postgres `
- Test your setup to ensure that `gooey-server` can access the database by running `psql -W -U gooey gooey` and supplying "gooey" as the password
* Create a .env file from `.env.example`
* Install the zbar library using your distro's package manager.

### Frontend

Clone [gooey-gui](https://github.com/GooeyAI/gooey-gui) repo, in the same directory as `gooey-server` and follow the setup steps.

## Running Tests

```
./scripts/run-tests.sh
```

(If you run into issues with the number of open files, you can remove the limit with `ulimit -n unlimited`)

## Run

You can start all required processes in one command with Honcho:
_Note: The `gooey-server` project is not currently set up to be run without support from Gooey. This software requires access to a Google Cloud instance as well as business data loaded in the database. If you are interested in running this software totally independently, reach out to [email protected] to communicate with our enterprise team._

```shell
$ poetry run honcho start
```
### Services

The processes that it starts are defined in [`Procfile`](Procfile).
Currently they are these:
Expand All @@ -61,16 +114,26 @@ Currently they are these:
| UI | `3000` |
| Vespa | `8085` |

### Honcho

You can start all required processes in one command with Honcho:

```shell
poetry run honcho start
```
This will spin up the API server at `http://localhost:8080`. To view the autogenerated API documentation, navigate to `http://localhost:8080/docs`

This default startup assumes that Redis, RabbitMQ, and PostgreSQL are installed and running
as background services on ports 6379, 5672, and 5432 respectively.
It also assumes that the gooey-ui repo can be found at `../gooey-ui/` (adjacent to where the
gooey-server repo sits). You can open the Procfile and comment this out if you don't need
as background services on ports `6379`, `5672`, and `5432` respectively.

The gooey-gui repo should be cloned at `../gooey-gui/`
(adjacent to where the`gooey-server` repo sits). You can open the Procfile and comment this out if you don't need
to run it.

**Note:** the Celery worker must be manually restarted on code changes. You
can do this by stopping and starting Honcho.

## To run vespa (used for vector search)
### Vespa (used for vector search)

You need to install OrbStack or Docker Desktop for this to work.

Expand All @@ -91,135 +154,7 @@ docker run \
./manage.py runscript setup_vespa_db
```

## To connect to our GPU cluster

* Connect to k8s cluster -
```bash
gcloud container clusters get-credentials cluster-5 --zone us-central1-a
```

* Port-forward the rabbitmq and redis services -
```bash
kubectl port-forward rabbitmq-1-rabbitmq-0 15674:15672 5674:5672 & kubectl port-forward redis-ha-1-server-0 6374:6379
```

* Add the following to `.env` file -
```
GPU_CELERY_BROKER_URL="amqp://rabbit:<password>@localhost:5674"
GPU_CELERY_RESULT_BACKEND="redis://:<password>@localhost:6374"
```

### Install imagemagick

Needed for HEIC image support - https://docs.wand-py.org/en/0.5.7/guide/install.html

```
brew install freetype imagemagick
export MAGICK_HOME=/opt/homebrew
```

### Code Formatting

Use black - https://pypi.org/project/black

**Recommended**: Black IDE integration Guide: [Pycharm](https://black.readthedocs.io/en/stable/integrations/editors.html#pycharm-intellij-idea)


## Running test Whatsapp bot

We use the following facebook app for testing -
```
gooey.ai (dev)
App ID: 228027632918921
```

Create a [meta developer account](https://developers.facebook.com/docs/development/register/) & send admin your **facebook ID** to add you to the test app [here](https://developers.facebook.com/apps/228027632918921/roles/roles/?business_id=549319917267066)

1. start ngrok

```
ngrok http 8080
```

2. set env var `FB_WEBHOOK_TOKEN = asdf1234`


3. Open [WhatsApp Configuration](https://developers.facebook.com/apps/228027632918921/whatsapp-business/wa-settings/?business_id=549319917267066), set the Callback URL and Verify Token
<img width="500" alt="image" src="https://github.com/GooeyAI/gooey-server/assets/19492893/95bb3a87-ae4f-4f6b-a04e-583ee51b85de">

4. Open [WhatsApp API Setup](https://developers.facebook.com/apps/228027632918921/whatsapp-business/wa-dev-console/?business_id=549319917267066), send yourself a message from the test number.
<img width="500" alt="image" src="https://github.com/GooeyAI/gooey-server/assets/19492893/f9417723-77c0-4be5-9814-778662215d9c">

5. Copy the temporary access token there and set env var `WHATSAPP_ACCESS_TOKEN = XXXX`


**(Optional) Use the test script to send yourself messages**

```bash
python manage.py runscript test_wa_msg_send --script-args 104696745926402 +918764022384
```
Replace `+918764022384` with your number and `104696745926402` with the test number ID

## Dangerous postgres commands

### backup & restore postgres db

**on server**
```bash
# select a running container
cid=$(docker ps | grep gooey-api-prod | cut -d " " -f 1 | head -1)
# give it a nice name
fname=gooey_db_$(date +"%Y-%m-%d_%I-%M-%S_%p").dump
# exec the script to create the fixture
docker exec -it $cid pg_dump --dbname $PGDATABASE --format c -f "$fname"
# copy the fixture outside container
docker cp $cid:/app/$fname .
# print the absolute path
echo $PWD/$fname
```

**on local**
```bash
# reset the database
./manage.py reset_db -c
# create the database with an empty template
createdb -T template0 $PGDATABASE
# restore the database
pg_restore --no-privileges --no-owner -d $PGDATABASE $fname
```

### create & load fixtures

**on server**
```bash
# select a running container
cid=$(docker ps | grep gooey-api-prod | cut -d " " -f 1 | head -1)
# exec the script to create the fixture
docker exec -it $cid poetry run ./manage.py runscript create_fixture
# upload the fixture
docker exec -it $cid poetry run ./manage.py runscript upload_fixture
```

To load the fixture on local db -

```bash
# reset the database
./manage.py reset_db -c
# create the database
./manage.py sqlcreate | psql postgres
# run migrations
./manage.py migrate
# load the fixture
./manage.py loaddata fixture.json
# create a superuser to access admin
./manage.py createsuperuser
```

### copy one postgres db to another

**on server**
```bash
./manage.py reset_db
createdb -T template0 $PGDATABASE
pg_dump $SOURCE_DATABASE | psql -q $PGDATABASE
```