Merge branch 'homanp:main' into landing-page

.docker/README.md

@@ -1,6 +1,61 @@
 # Run Superagent locally with Docker and docker-compose
 
-## Setup environment variables first
+## Getting Started
+
+In the `.docker` folder there are several docker-compose files available.
+
+After you have carefully set all the values in both the root `.env`, `ui/.env` and `.docker/docker.env` files, you can spin everything up in docker:, see [Setup environment variables](#setup-environment-variables)
+
+- `docker-compose.yml` - Main file, starts the Superagent API, a PostgresDB and PGAdmin to administer the DB, fine for most use cases (combine with other dockerfiles using the `-f` flag)
+
+  ``` bash
+  docker-compose --env-file docker.env up --build -d
+  ```
+
+- `docker-compose.ui.yml` - Starts the Superagent UI, run it with the API and DB by passing both files in with `-f`
+
+  ``` bash
+  docker-compose --env-file docker.env -f docker-compose.yml -f docker-compose.ui.yml up --build -d
+  ```
+
+- `docker-compose.api.yml` - Can be used to independently start Superagent API
+
+  ``` bash
+  docker-compose --env-file docker.env -f docker-compose.api.yml up --build -d
+  ```
+
+- `docker-compose.pgdb.yml` - Can be used to independently start the PostgresDB with PGAdmin
+
+  ``` bash
+  docker-compose --env-file docker.env -f docker-compose.pgdb.yml up --build -d
+  ```
+
+- `docker-compose.minio.yml` - Can be used to start a [Minio](https://github.com/minio/minio) server. Minio is a locally hostable swap in replacement for AWS S3, it has a UI and an AWS S3 compatible API, great for local development. See the section [Setting up Minio](#setting-up-minio-as-a-local-replacement-for-aws-s3) for details on additional setup needed
+
+  ``` bash
+  # Start it with everything else
+  docker-compose --env-file docker.env -f docker-compose.yml -f docker-compose.ui.yml -f docker-compose.minio.yml up --build -d
+  ```
+
+Stop everything with:
+
+``` bash
+docker-compose --env-file docker.env down
+
+# Additionally remove orphaned containers if there are any
+docker-compose --env-file docker.env down --remove-orphans
+
+# Delete eveything including volumes images, this will result in total data loss
+docker-compose --env-file docker.env down --rmi all -v
+
+# Remove all unused networks
+docker network prune
+
+# To remove all stopped containers, unused networks, dangling images, and build cache, you can use:
+docker system prune
+```
+
+## Setup environment variables
 
 1. Create and fill in the API and UI environment variables. Check the `README` for both the API and the UI for details.
 
@@ -19,17 +74,17 @@
    cp .docker/docker.env.example .docker/docker.env
    ```
 
-3. Set the NETWORK_HOST mode suitable for your setup:
+3. Set the NETWORK_MODE mode suitable for your setup:
 
-   ``` bash
-   # Default docker network, same as not having set NETWORK_HOST in docker-compose files. 
-   NETWORK_HOST=superagent_default
+   You can create a custom network to use with `docker network create superagent_default` this will allow all the services to communicate with each other even if you start them seperately.
 
-   # Simulate running the docker resources on localhost
-   NETWORK_HOST=host
+   ``` bash
+   # Create and use a docker network.
+   # Create the network first with `docker network create superagent_default`
+   NETWORK_MODE=superagent_default
 
-   # Custom network, for some cloud deployments
-   NETWORK_HOST=mycustom_network_name
+   # Or Simulate running the docker resources on localhost
+   NETWORK_MODE=host
    ```
 
    If running `superagent_default` mode, you may need to add the docker host names for the services to your `hosts` file on Windows, then use these values in place of `localhost` or `127.0.0.1` in your `.env` files. This is especially needed when running the UI in docker as the browser will not be able to access the docker host, while the UI server code can.
@@ -39,13 +94,24 @@
    - Add the following lines at the bottom
 
    ``` bash
-   127.0.0.1		superagent-api
-   127.0.0.1		superagent-ui
-   127.0.0.1		pgdb
-   127.0.0.1		pgadmin
+   127.0.0.1  superagent-api
+   127.0.0.1  superagent-ui
+   127.0.0.1  pgdb
+   127.0.0.1  pgadmin
+   127.0.0.1  minio
    ```
 
-   You can then access any of these resources, running in docker, using these host names in your `.env` files and in the browser. e.g `http://superagent-ui:3000`, `http://superagent-api:8080`, `http://pgadmin:5050`
+   You can then access any of these resources, running in docker, using these host names in your `.env` files and in the browser. e.g
+
+   Superagent UI: `http://superagent-ui:3000`
+
+   Superagent API: `http://superagent-api:8080`
+
+   PGAdmin: `http://pgadmin:5050`
+
+   Minio Console (web ui): `http://minio:9090`
+
+   Minio S3 Compatible API: `http://minio:9000`
 
 4. In order to connect to the db running in docker, with the example values from `docker.env`, set the following in the root `.env`. Set the `DATABASE_URL` and `DATABASE_MIGRATION_URL` to the same value and comment out, or remove the `DATABASE_SHADOW_URL`
 
@@ -59,7 +125,7 @@
     # DATABASE_SHADOW_URL= # Mandatory for Neon DB
 
     # 'host' network mode
-    DATABASE_URL=postgresql://admin:password@locahost:5432/superagent
+    DATABASE_URL=postgresql://admin:password@localhost:5432/superagent
     DATABASE_MIGRATION_URL=postgresql://admin:password@localhost:5432/superagent
     # DATABASE_SHADOW_URL= # Mandatory for Neon DB
     ```
@@ -76,69 +142,70 @@
     NEXT_PUBLIC_SUPERAGENT_API_URL="http://127.0.0.1:8080/api/v1"
     ```
 
-## Run Superagent with docker-compose
-
-After you have carefully set all the values in both the root `.env`, `ui/.env` and `.docker/docker.env` files, you can spin everything up in docker:
-
-### Run API and a Postgres DB together
+## Viewing logs in the Docker containers
 
-The default `docker-compose.yml` file will run the API and a Postgres DB (including the PGAdmin tool), meaning you can run the UI either locally with npm or in docker seperately.
+Each container is named so you can easily view the logs using docker on the CMD line
 
 ``` bash
-cd .docker
-docker-compose --env-file docker.env up --build -d
-
-# OR: Optionally also start the UI in docker too.
-docker-compose --env-file docker.env -f docker-compose.yml -f docker-compose.ui.yml up --build -d
+# UI logs
+docker logs superagent-ui -f
 
-# you can spin everything down with
-docker compose --env-file docker.env down
+# API logs
+docker logs superagent-api -f
 
-# It's often best to make sure the last run is shut down everytime to avoid issues, combine down and up commands:
-docker compose --env-file docker.env down && docker-compose --env-file docker.env up --build -d
+# DB logs
+docker logs pgdb -f
 
-```
+## Adminer logs
+docker logs pgadmin -f
 
-### Run any single part individually or in combination with docker-compose
+## Minio (S3) Logs
+docker logs minio -f
 
-You can run just the bits you need in docker, meaning you can run the UI locally with npm, or the API with venv/conda locally to develop, while only running dependencies in docker for example.
+```
 
-Or if you are developing only the UI for example, you could just run both the API and DB in docker, while running the UI with npm locally, any combination is fine.
+## Setting up Minio as a local replacement for AWS S3
 
-``` bash
-cd .docker
+You will need to setup a few things to use Minio in place of AWS S3 locally. This is a very basic setup suitable for local use. See the the [Minio](https://github.com/minio/minio) github repo for other options if you want to deploy this to production.
 
-# Run just the Postgress DB 
-docker-compose --env-file docker.env -f docker-compose.pgdb.yml up --build -d
+1. Update `.docker/docker.env` with the user and pass for Minio
+   This is used both for logging into the Minio web ui, the user is used as the S Key, the pass as the S3 secret. (you can set up other users to use as key and secret in the Minio web ui)
 
-# Run just the UI
-docker-compose --env-file docker.env -f docker-compose.ui.yml up --build -d
+   ``` env
+   MINIO_ROOT_USER:root
+   MINIO_ROOT_PASSWORD:password
+   ```
 
-# Run just the API
-docker-compose --env-file docker.env -f docker-compose.api.yml up --build -d
+2. Update `ui/.env` with the setting for Minio in place of AWS S3, the region can be any, it's not used by minio, but will be needed by the AWS SDK.
 
-# Run the DB and the UI together
-docker-compose --env-file docker.env -f docker-compose.pgdb.yml -f docker-compose.ui.yml up --build -d
+   ``` env
+   NEXT_PUBLIC_AWS_S3_BUCKET=superagent
+   NEXT_PUBLIC_AWS_S3_REGION=eu-west-1
+   NEXT_PUBLIC_AMAZON_S3_ACCESS_KEY_ID=root
+   NEXT_PUBLIC_AMAZON_S3_SECRET_ACCESS_KEY=password
+   NEXT_PUBLIC_AWS_OVERRIDE_S3_BASEURL=http://minio:9000
+   ```
 
-# Run everything, the DB, the API and the UI together
-docker-compose --env-file docker.env -f docker-compose.yml -f docker-compose.ui.yml up --build -d
+3. Unless you can use `NETWORK_MODE=host` with docker, then you will need to add minio to the hosts file
 
-```
+   ``` hosts
+   127.0.0.1  minio
+   ```
 
-## Viewing logs in the Docker containers
+   The minio Web UI will be available at `http://minio:9090`. You can use this to add buckets and users etc.
 
-Each container is named so you can easily view the logs using docker on the CMD line
+   The minio S3 API is at `http://minio:9000`, this is used as the override base URL for S3.
 
-``` bash
-# UI logs
-docker logs superagent-ui -f
+4. Start up Minio along with Superagent, e.g the following starts API, DB, UI and Minio with docker.
 
-# API logs
-docker logs superagent-api -f
+   ``` bash
+   docker-compose --env-file docker.env -f docker-compose.yml -f docker-compose.ui.yml -f docker-compose.minio.yml up --build -d
+   ```
 
-# DB logs
-docker logs superagent-pgdb -f
+5. Setup a bucket in the Minio Console (UI)
+   - Open the minio ui at `http:minio:9090`
+   - login with the user from `MINIO_ROOT_USER` and password from `MINIO_ROOT_PASSWORD`
+   - On the buckets tab, create a bucket called `superagent`, save it
+   - Open the bucket, Set the policy to `Public`, this won't make it publicly available on the internet of course (unlike AWS S3), it just means you can read the files with their direct URL without using the API to authenticate.
 
-## Adminer logs
-docker logs superagent-adminer -f
-```
+You should now be able to upload files via Superagent, you can see the files in the object browser, where you can manage / delete them if needed

.docker/docker-compose.minio.yml

@@ -0,0 +1,23 @@
+version: "3"
+
+services:
+  # Use minio to mock s3
+  minio:
+    container_name: minio
+    network_mode: ${NETWORK_MODE}
+    image: "quay.io/minio/minio:latest"
+    restart: unless-stopped
+    ports:
+      # S3 compatible API
+      - "9000:9000"
+      # Minio console (Web UI)
+      - "9090:9090"
+    environment:
+      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
+      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
+    volumes:
+      - "minio:/data/minio"
+    command: minio server /data/minio --console-address ":9090"
+
+volumes:
+  minio:

.docker/docker-compose.yml

@@ -17,8 +17,6 @@ services:
     depends_on:
       pgdb:
         condition: service_healthy
-      pgadmin:
-        condition: service_started
     restart: unless-stopped
 
   pgdb:
@@ -31,7 +29,7 @@ services:
         source: pgdb-data
         target: "/var/lib/postgresql/data"
     environment:
-      POSTGRES_DB: postgres
+      POSTGRES_DB: admin
       POSTGRES_USER: ${DBUSER}
       POSTGRES_PASSWORD: ${DBPASS}
     ports:
@@ -51,6 +49,8 @@ services:
       PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_DEFAULT_PASSWORD}
       PGADMIN_LISTEN_ADDRESS: 0.0.0.0
       PGADMIN_LISTEN_PORT: 5050
+    ports:
+      - 5050:5050
     volumes:
       - type: volume
         source: pgadmin-data

.docker/docker.env.example

@@ -11,3 +11,8 @@ DBNAME=superagent
 # Login details for pgadmin, http://localhost:5050 / http://pgadmin:5050
 [email protected]
 PGADMIN_DEFAULT_PASSWORD=password
+
+# Minio (Locally hostable S3 compatible API and UI)
+# Use root user and pass to login to Minio UI and auth with the Minio S3 API
+MINIO_ROOT_USER:root
+MINIO_ROOT_PASSWORD:password

.env.example

@@ -8,7 +8,7 @@ JWT_SECRET=
 VECTORSTORE=pinecone
 PINECONE_API_KEY=
 PINECONE_ENVIRONMENT=
-PINECONE_INDEX=
+PINECONE_INDEX=superagent
 
 SUPERAGENT_TRACING=False
 
@@ -35,3 +35,8 @@ AZURE_API_VERSION=
 GOOGLE_CLIENT_ID=
 AZURE_AD_CLIENT_ID=
 AZURE_AD_TENANT_ID=
+
+# Option Langsmith tracing
+LANGCHAIN_TRACING_V2=True
+LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
+LANGCHAIN_API_KEY=

.github/workflows/ci.yml

@@ -35,3 +35,8 @@ jobs:
           PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
           NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
         run: fern generate --group publish --log-level debug --version ${{ github.ref_name }}
+
+      - name: Update Docs
+        env:
+          FERN_TOKEN: ${{ secrets.FERN_TOKEN }}
+        run: fern generate --docs

Dockerfile

@@ -18,6 +18,10 @@ RUN poetry install --no-root && rm -rf $POETRY_CACHE_DIR
 
 FROM python:3.11 AS runtime
 
+RUN apt-get update && apt-get install -y curl && \
+    curl -sL https://deb.nodesource.com/setup_18.x | bash - && \
+    apt-get install -y nodejs
+
 ENV PATH="/app/.venv/bin:$PATH"
 ENV PORT="8080"
 

README.md

@@ -37,11 +37,11 @@ Checkout the [full documentation here](https://docs.Superagent.sh/).
 
 You can follow the [roadmap here](https://github.com/users/homanp/projects/4)
 
-## 🛠️ Getting Started
+## 🎥 Tutorials
 
-<div align="center">
-<img alt="Getting started" src="https://cdn.loom.com/sessions/thumbnails/7869ed5dc7614205b62249bedfbc49e0-1688459140526-with-play.gif" />
-</div>
+We post tutorials regularly on our [Youtube channel](https://www.youtube.com/channel/UCBeXnF8gh2EwAmOIwpmfjmA). Make sure to check them out! 
+
+## 🛠️ Getting Started
 
 To get started with Superagent, follow these steps:
 
@@ -63,7 +63,7 @@ To get started with Superagent, follow these steps:
 
 In the `.docker` folder there are multiple docker-compose files.
 
-The main `docker-conpose.yml` file will start up the API and a Postgres DB in docker. You can optionally also run the UI in docker too.
+The main `docker-compose.yml` file will start up the API and a Postgres DB in docker. You can optionally also run the UI in docker too.
 
 The other docker compose files can be used individually, or in combination to start up just the bits you need.
 
@@ -103,7 +103,7 @@ To run this project, you will need to add the following environment variables to
 
 `WOLFRAM_ALPHA_APPID` - Wolfram Alpha App ID (found in your Wolfram Alpha dashboard)
 
-`REPLICATE_API_TOKEN` - Replicate API token (found in uour [replicate](https://replication.com) dashboard)
+`REPLICATE_API_TOKEN` - Replicate API token (found in uour [replicate](https://replicate.com/) dashboard)
 
 **If you plan on using other Language Models**