-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
readme: move issues to github good first issues, move database instru…
…ctions to github discussions, update gooey-gui instructions
- Loading branch information
Showing
2 changed files
with
63 additions
and
105 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,39 +1,20 @@ | ||
| ## What is Gooey Server? | ||
| Gooey.AI is a low-code AI recipe platform and Gooey Server is our core repo. It allows users to discover, customize, and deploy AI "recipes" using the best of private and open source AI, all using a single API with a single auth token. Recipes are workflows that incorporate various models to accomplish a task; they are designed to be highly customizable and shareable. | ||
| Gooey.AI is a low-code AI recipe platform and Gooey Server is our core repo. | ||
| It allows users to discover, customize, and deploy AI "recipes" using the best of private and open source AI, | ||
| all using a single API with a single auth token. | ||
| Recipes are workflows that incorporate various models to accomplish a task; they are designed to be highly customizable and shareable. | ||
|
|
||
| ### Who is this for and why would I want to use it? | ||
| For the vast majority of developers, we DO NOT recommend running or forking Gooey Server; use our APIs or client SDK instead. The repo is intended only for developers that want to run and deploy their own server cluster or run Gooey locally for development purposes. Specifically, this repo may be for you if: | ||
| You want to create a new recipe (instead of changing the parameters on an existing one) | ||
| You want to add an AI model that we currently don’t support. | ||
| You are an enterprise who has specific requirements regarding data practices, such as using specific cloud providers. | ||
| You want to add some other functionality that we don’t support. | ||
| For the vast majority of developers, we DO NOT recommend running or forking Gooey Server; use our APIs or client SDK instead. | ||
| The repo is intended only for developers that want to run and deploy their own server cluster or run Gooey locally for development purposes. | ||
|
|
||
| ### Prerequisites | ||
| Google JSON key - only for auth / storage. | ||
| Updated DB fixture | ||
| OS: Mac or Linux. Likely works on other *nix, but this is untested. | ||
| Specifically, this repo may be for you if: | ||
| - You want to create a new recipe (instead of changing the parameters on an existing one) | ||
| - You want to add an AI model that we currently don’t support. | ||
| - You are an enterprise who has specific requirements regarding data practices, such as using specific cloud providers. | ||
| - You want to add some other functionality that we don’t support. | ||
|
|
||
| ### Issues that contributors could work on: | ||
| Firebase / Supabase migration. | ||
| Provide a simpler workaround for localhost auth. | ||
| Provide an open source alternative to Google auth for use in VPC. | ||
| Provide a way to do storage locally. | ||
|
|
||
| ## Setup (Mac) | ||
|
|
||
| * Install [pyenv](https://github.com/pyenv/pyenv) & install the same python version as in our [Dockerfile](Dockerfile) | ||
| * Install [poetry](https://python-poetry.org/docs/) | ||
| * Clone the github repo to gooey-server (and make sure that's the folder name) | ||
| * Create & activate a virtualenv (e.g. `poetry shell`) | ||
| * Run `poetry install --with dev` | ||
| * Install [redis](https://redis.io/docs/getting-started/installation/install-redis-on-mac-os/), [rabbitmq](https://www.rabbitmq.com/install-homebrew.html), and [postgresql](https://formulae.brew.sh/formula/postgresql@15) (e.g. `brew install redis rabbitmq postgresql@15`) | ||
| * Enable background services for `redis`, `rabbitmq`, and `postgresql` (e.g. with `brew services start redis` and similar for `rabbitmq` and `postgresql`) | ||
| * Use `sqlcreate` helper to create a user and database for gooey: | ||
| * `./manage.py sqlcreate | psql postgres` | ||
| * make sure you are able to access the database with `psql -W -U gooey gooey` (and when prompted for password, entering `gooey`) | ||
| * Create an `.env` file from `.env.example` (Read [12factor.net/config](https://12factor.net/config)) | ||
| * Run `./manage.py migrate` | ||
| * Install the zbar library (`brew install zbar`) | ||
| ## Setup | ||
|
|
||
| ### Create a google cloud / firebase account | ||
|
|
||
|
|
@@ -56,21 +37,34 @@ Provide a way to do storage locally. | |
| 5. Create and Download a JSON Key for this service account and save it to the project root as `serviceAccountKey.json`. | ||
| 6. Add your project & bucket name to `.env` | ||
|
|
||
| * Run tests to see if everything is working fine: | ||
| ``` | ||
| ./scripts/run-tests.sh | ||
| ``` | ||
| (If you run into issues with the number of open files, you can remove the limit with `ulimit -n unlimited`) | ||
|
|
||
| ## Setup (Linux) | ||
| ### MacOS | ||
|
|
||
| * Install [pyenv](https://github.com/pyenv/pyenv) & install the same python version as in our [Dockerfile](Dockerfile) | ||
| * Install [poetry](https://python-poetry.org/docs/) | ||
| * Clone the github repo to `gooey-server` (and make sure that's the folder name) | ||
| * Create & activate a virtualenv (e.g. `poetry shell`) | ||
| * Run `poetry install --with dev` | ||
| * Install [redis](https://redis.io/docs/getting-started/installation/install-redis-on-mac-os/), [rabbitmq](https://www.rabbitmq.com/install-homebrew.html), and [postgresql](https://formulae.brew.sh/formula/postgresql@15) (e.g. `brew install redis rabbitmq postgresql@15`) | ||
| * Enable background services for `redis`, `rabbitmq`, and `postgresql` (e.g. with `brew services start redis` and similar for `rabbitmq` and `postgresql`) | ||
| * Use `sqlcreate` helper to create a user and database for gooey: | ||
| * `./manage.py sqlcreate | psql postgres` | ||
| * make sure you are able to access the database with `psql -W -U gooey gooey` (and when prompted for password, entering `gooey`) | ||
| * Create an `.env` file from `.env.example` (Read [12factor.net/config](https://12factor.net/config)) | ||
| * Run `./manage.py migrate` | ||
| * Install the zbar library (`brew install zbar`) | ||
| * (optional) Install imagemagick - Needed for HEIC image support - https://docs.wand-py.org/en/0.5.7/guide/install.html | ||
| ``` | ||
| brew install freetype imagemagick | ||
| export MAGICK_HOME=/opt/homebrew | ||
| ``` | ||
|
|
||
| ### Linux | ||
| * Install [pyenv](https://github.com/pyenv/pyenv) & install the same python version as in our [Dockerfile](Dockerfile) (currently Python 3.10) | ||
| - `curl https://pyenv.run | bash` | ||
| * Install [poetry](https://python-poetry.org/docs/) | ||
| - This is likely available in your distro's package repos. | ||
| * Clone the gooey-server repository: | ||
| - `git clone https://github.com/GooeyAI/gooey-server.git` | ||
| * If you want to use the web application frontend, you must clone that repo as well, in the same directory as gooey-server: | ||
| - `git clone https://github.com/GooeyAI/gooey-ui` | ||
| * Clone this repository: | ||
| * Create and activate a virtualenv using `poetry shell` | ||
| * Install dependencies using `poetry install --with dev` | ||
| - Note: you may have to remove `package-mode=false` on line 7 of `pyproject.toml` | ||
|
|
@@ -86,22 +80,27 @@ Provide a way to do storage locally. | |
| - restart postgresql using ```sudo systemctl restart postgresql``` | ||
| * Use the manage.py script to set up the Postgres database: | ||
| - To create the user and database for gooey: `./manage.py sqlcreate | sudo -u postgres psql postgres ` | ||
| - Test your setup to ensure that gooey-server can access the database by running `psql -W -U gooey gooey` and supplying "gooey" as the password | ||
| - Test your setup to ensure that `gooey-server` can access the database by running `psql -W -U gooey gooey` and supplying "gooey" as the password | ||
| * Create a .env file from `.env.example` | ||
| * Install the zbar library using your distro's package manager. | ||
| ## Run | ||
| ### Frontend | ||
| Note: The gooey-server project is not currently set up to be run without support from Gooey. This software requires access to a Google Cloud instance as well as business data loaded in the database. If you are interested in running this software totally independently, reach out to support@gooey.ai to communicate with our enterprise team. | ||
| Clone [gooey-gui](https://github.com/GooeyAI/gooey-gui) repo, in the same directory as `gooey-server` and follow the setup steps. | ||
| You can start all required processes in one command with Honcho: | ||
| ## Running Tests | ||
| ```shell | ||
| $ poetry run honcho start | ||
| ``` | ||
| ./scripts/run-tests.sh | ||
| ``` | ||
| This will spin up the API server at `http://localhost:8080`. To view the autogenerated API documentation, navigate to `http://localhost:8080/docs` | ||
| If you installed the gooey-ui server, you can navigate to 'http://localhost:3000' to access the web application. | ||
| (If you run into issues with the number of open files, you can remove the limit with `ulimit -n unlimited`) | ||
| ## Run | ||
| _Note: The `gooey-server` project is not currently set up to be run without support from Gooey. This software requires access to a Google Cloud instance as well as business data loaded in the database. If you are interested in running this software totally independently, reach out to [email protected] to communicate with our enterprise team._ | ||
| ### Services | ||
| The processes that it starts are defined in [`Procfile`](Procfile). | ||
| Currently they are these: | ||
|
|
@@ -115,16 +114,26 @@ Currently they are these: | |
| | UI | `3000` | | ||
| | Vespa | `8085` | | ||
| ### Honcho | ||
| You can start all required processes in one command with Honcho: | ||
| ```shell | ||
| poetry run honcho start | ||
| ``` | ||
| This will spin up the API server at `http://localhost:8080`. To view the autogenerated API documentation, navigate to `http://localhost:8080/docs` | ||
|
|
||
| This default startup assumes that Redis, RabbitMQ, and PostgreSQL are installed and running | ||
| as background services on ports 6379, 5672, and 5432 respectively. | ||
| It also assumes that the gooey-ui repo can be found at `../gooey-ui/` (adjacent to where the | ||
| gooey-server repo sits). You can open the Procfile and comment this out if you don't need | ||
| as background services on ports `6379`, `5672`, and `5432` respectively. | ||
|
|
||
| The gooey-gui repo should be cloned at `../gooey-gui/` | ||
| (adjacent to where the`gooey-server` repo sits). You can open the Procfile and comment this out if you don't need | ||
| to run it. | ||
|
|
||
| **Note:** the Celery worker must be manually restarted on code changes. You | ||
| can do this by stopping and starting Honcho. | ||
|
|
||
| ## To run vespa (used for vector search) | ||
| ### Vespa (used for vector search) | ||
|
|
||
| You need to install OrbStack or Docker Desktop for this to work. | ||
|
|
||
|
|
@@ -145,58 +154,7 @@ docker run \ | |
| ./manage.py runscript setup_vespa_db | ||
| ``` | ||
|
|
||
| ### Install imagemagick | ||
|
|
||
| Needed for HEIC image support - https://docs.wand-py.org/en/0.5.7/guide/install.html | ||
|
|
||
| ``` | ||
| brew install freetype imagemagick | ||
| export MAGICK_HOME=/opt/homebrew | ||
| ``` | ||
|
|
||
| ### Code Formatting | ||
|
|
||
| Use black - https://pypi.org/project/black | ||
|
|
||
| **Recommended**: Black IDE integration Guide: [Pycharm](https://black.readthedocs.io/en/stable/integrations/editors.html#pycharm-intellij-idea) | ||
|
|
||
|
|
||
| ### backup & restore postgres db | ||
|
|
||
| ```bash | ||
| # reset the database | ||
| ./manage.py reset_db -c | ||
| # create the database with an empty template | ||
| createdb -T template0 $PGDATABASE | ||
| # restore the database | ||
| pg_restore --no-privileges --no-owner -d $PGDATABASE $fname | ||
| ``` | ||
|
|
||
| ### create & load fixtures | ||
|
|
||
| ```bash | ||
| ./scripts/run-tests.sh | ||
| ``` | ||
|
|
||
| To load the fixture on local db - | ||
|
|
||
| ```bash | ||
| # reset the database | ||
| ./manage.py reset_db -c | ||
| # create the database | ||
| ./manage.py sqlcreate | psql postgres | ||
| # run migrations | ||
| ./manage.py migrate | ||
| # load the fixture | ||
| ./manage.py loaddata fixture.json | ||
| # create a superuser to access admin | ||
| ./manage.py createsuperuser | ||
| ``` | ||
|
|
||
| ### copy one postgres db to another | ||
|
|
||
| ```bash | ||
| ./manage.py reset_db | ||
| createdb -T template0 $PGDATABASE | ||
| pg_dump $SOURCE_DATABASE | psql -q $PGDATABASE | ||
| ``` | ||