Merge pull request #5722 from EnterpriseDB/2024-06-03a

Release: 2024-06-03a

advocacy_docs/edb-postgres-ai/ai-ml/install-tech-preview.mdx

@@ -10,17 +10,19 @@ The preview release of pgai is distributed as a self-contained Docker container
 ## Configuring and running the container image
 
 If you haven’t already, sign up for an EDB account and log in to the EDB container registry.
-Login to docker with your the username tech-preview and your EDB Repo 2.0 Subscription Token as your password:
 
-```
+
+Log in to docker with your the username tech-preview and your EDB Repo 2.0 Subscription Token as your password:
+
+```shell
 docker login docker.enterprisedb.com -u tech-preview -p <your_EDB_repo_token>
 __OUTPUT__
 Login Succeeded
 ```
 
 Download the pgai container image:
 
-```
+```shell
 docker pull docker.enterprisedb.com/tech-preview/pgai
 __OUTPUT__
 ...
@@ -30,26 +32,26 @@ docker.enterprisedb.com/tech-preview/pgai:latest
 
 Specify a password to use for Postgres in the environment variable PGPASSWORD. The tech preview container will set up Postgres with this password and use it to connect to it. In bash or zsh set it as follows:
 
-```
+```shell
 export PGPASSWORD=<your_password>
 ```
 
 You can use the pgai extension with encoder LLMs in Open AI or with open encoder LLMs from HuggingFace. If you want to use Open AI you also must provide your API key for that in the OPENAI_API_KEY environment variable:
 
-```
+```shell
 export OPENAI_API_KEY=<your_openai_key>
 ```
 
 You can use the pgai extension with AI data stored in Postgres tables or on S3 compatible object storage. To work with object storage you need to specify the ACCESS_KEY and SECRET_KEY environment variables:.
 
-```
+```shell
 export ACCESS_KEY=<your_access_key>
 export SECRET_KEY=<your_secret_key>
 ```
 
 Start the pgai tech preview container with the following command. It makes the tech preview PostgreSQL database available on local port 15432:
 
-```
+```shell
 docker run -d --name pgai \
        -e ACCESS_KEY=$ACCESS_KEY \
        -e SECRET_KEY=$SECRET_KEY \
@@ -65,14 +67,14 @@ docker run -d --name pgai \
 
 If you haven’t yet, install the Postgres command-line tools. If you’re on a Mac, using Homebrew, you can install it as follows:
 
-```
+```shell
 brew install libpq
 ```
 
 Connect to the tech preview PostgreSQL running in the container. Note that this relies on $PGPASSWORD being set - if you’re using a different terminal for this part, make sure you re-export the password:
 
-```
-$ psql -h localhost -p 15432 -U postgres postgres
+```shell
+psql -h localhost -p 15432 -U postgres postgres
 __OUTPUT__
 psql (16.1, server 16.3 (Debian 16.3-1.pgdg120+1))
 Type "help" for help.
@@ -84,15 +86,16 @@ postgres=#
 Install the pgai extension:
 
 ```sql
-postgres=# create extension pgai cascade;
+create extension pgai cascade;
 __OUTPUT__
 NOTICE:  installing required extension "plpython3u"
 NOTICE:  installing required extension "vector"
 CREATE EXTENSION
+postgres=# 
 ```
 
 ```sql
-postgres=# \dx
+\dx
 __OUTPUT__
                                List of installed extensions
     Name    | Version |   Schema   |                     Description

advocacy_docs/edb-postgres-ai/ai-ml/overview.mdx

@@ -9,19 +9,21 @@ At the heart of EDB Postgres® AI is the EDB Postgres AI database (pgai). This b
 
 The pgai extension is currently available as a tech preview. It will be continuously extended with new functions. Here comes a description of the major functionality available to date.
 
-![PGAI Overview](images/pgai-overview.png)
+![PGAI Overview](images/pgai-overview-withbackground.png)
 
-pgai introduced the concept of a “retriever” that you can create for a given type and location of AI data. Currently pgai supports unstructured plain text documents as well as a set of image formats. This data can either reside in regular columns of a Postgres table or it can reside in an S3 compatible object storage bucket.
+pgai introduces the concept of a “retriever” that you can create for a given type and location of AI data. Currently pgai supports unstructured plain text documents as well as a set of image formats. This data can either reside in regular columns of a Postgres table or it can reside in an S3 compatible object storage bucket.
 
-A retriever encapsulates all processing that is needed to make the AI data in the provided source location searchable and retrievable via similarity. The application just needs to create a retriever via the pgai.create_retriever() function. When auto_embedding=TRUE is specified the pgai extension will automatically generate embeddings for all the data in the source location. Otherwise it will be up to the application to request a bulk generation of embeddings via pgai.refresh_retriever(). 
+A retriever encapsulates all processing that is needed to make the AI data in the provided source location searchable and retrievable through similarity. The application just needs to create a retriever via the `pgai.create_retriever()` function. When `auto_embedding=TRUE` is specified the pgai extension will automatically generate embeddings for all the data in the source location. 
+
+Otherwise it will be up to the application to request a bulk generation of embeddings using `pgai.refresh_retriever()`. 
 
 Auto embedding is currently supported for AI data stored in Postgres tables and it automates the embedding updates using Postgres triggers. You can also combine the two options by using pgai.refresh_retriever() to embed all previously existing data and also setting `auto_embedding=TRUE` to generate embeddings for all new and changed data from now on.
 
 All embedding generation, storage, indexing and management is handled by the pgai extension internally. The application just has to specify the encoder LLM that the retriever should be using for this specific data and use case.
 
 Once a retriever is created and all embeddings are up to date, the application can just use pgai.retrieve() to run a similarity search and retrieval by providing a query input. When the retriever is created for text data, the query input is also a text term. For image retrievers the query input is an image. The pgai retriever makes sure to use the same encoder LLM for the query input, conducts a similarity search and finally returns the ranked list of similar data from the source location.
 
-pgai currently supports a broad list of open encoder LLMs from HuggingFace as well as a set of OpenAI encoders. Just consult the list of supported encoder LLMs in the pgai.encoders meta table. HuggingFace LLMs are running locally on the Postgres node, while OpenAI encoders involve a call out to the OpenAI cloud service.
+pgai currently supports a broad list of open encoder LLMs from HuggingFace as well as a set of OpenAI encoders. Consult the list of supported encoder LLMs in the pgai.encoders meta table. HuggingFace LLMs are running locally on the Postgres node, while OpenAI encoders involve a call out to the OpenAI cloud service.
 
 
 

advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/standalone-embedding-functions.mdx → advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/additional_functions.mdx

@@ -1,10 +1,12 @@
 ---
-title: Stand-alone Embedding Functions in pgai
-navTitle: Stand-alone Embedding Functions
-description: Use the pgai extension to generate embeddings for images and text.
+title: Additional functions and stand-alone embedding in pgai
+navTitle:  Additional functions
+description: Other pgai extension functions and how to generate embeddings for images and text.
 ---
 
-Use generate_single_image_embedding function to get embeddings for the given image. Currently, model_provider can only be openai or huggingface. You can check the list of valid embedding models and model providers from the Encoders Supported PGAI section. 
+## Standalone embedding
+
+Use the `generate_single_image_embedding` function to get embeddings for the given image. Currently, `model_provider` can only be `openai` or `huggingface`. You can check the list of valid embedding models and model providers from the Encoders Supported PGAI section. 
 
 ```sql
 SELECT pgai.generate_single_image_embedding(
@@ -37,6 +39,7 @@ __OUTPUT__
 (1 row)    
 ```
 
+## Supported encoders
 
 You can check the list of valid embedding models and model providers from pgai.encoders table
 
@@ -50,4 +53,30 @@ __OUTPUT__
 (2 rows)
 ```
 
+## Available functions 
+
+You can find the complete list of currently available functions of the pgai extension by selecting from `information_schema.routines` any `routine_name` belonging to the pgai routine schema:
+
+
+```
+SELECT routine_name from information_schema.routines WHERE routine_schema='pgai';
+__OUTPUT__
+          routine_name
+---------------------------------
+ init
+ create_pg_retriever
+ create_s3_retriever
+ _embed_table_update
+ refresh_retriever
+ retrieve
+ retrieve_via_s3
+ register_prompt_template
+ render_prompt
+ generate
+ ag
+ rag
+ generate_text_embedding
+ generate_single_image_embedding
+(14 rows)
+```
 

advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/working-with-ai-data-in-S3.mdx

@@ -14,12 +14,11 @@ First let’s create a retriever for images stored on s3-compatible object stora
 SELECT pgai.create_s3_retriever(
     'image_embeddings', -- Name of the similarity retrieval setup
     'public', -- Schema of the source table
-    'img_id', -- Primary key
     'clip-vit-base-patch32', -- Embeddings encoder model for similarity data
     'img', -- data type, could be either img or text
     'torsten', -- S3 bucket name
-    'https://s3.us-south.cloud-object-storage.appdomain.cloud', -- s3 endpoint address
-    '' -- prefix
+    '', -- prefix
+    'https://s3.us-south.cloud-object-storage.appdomain.cloud' -- s3 endpoint address
 );
 __OUTPUT__
  create_s3_retriever 
@@ -57,33 +56,3 @@ __OUTPUT__
  (1 row)
 ```
 
-If you set the `ACCESS_KEY` and `SECRET_KEY` you can use the following queries to run without an `s3_endpoint` setting using a command like this:
-
-```sql
-SELECT pgai.create_s3_retriever(
-   'img_file_embeddings', -- Name of the similarity retrieval setup
-   'demo', -- Schema of the source table
-   'img_id', -- Primary key
-   'clip-vit-base-patch32', -- Embeddings encoder model for similarity data
-   'img', -- data type
-   'bilge-ince-test' -- S3 bucket name
-);
-__OUTPUT__
- create_s3_retriever
----------------------
-
-(1 row)
-```
-
-```sql
-SELECT pgai.refresh_retriever('img_file_embeddings');
-```
-
-```sql
-SELECT data from pgai.retrieve_via_s3('img_file_embeddings',
-     1, 
-     'bilge-ince-test', 
-     'kirpis_small.jpg', 
-     'http://s3.eu-central-1.amazonaws.com'
-     );
-```

advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/working-with-ai-data-in-postgres.mdx

@@ -22,14 +22,14 @@ CREATE TABLE
 ```
 
 
-Now let’s create a retriever with the just created products table as the source. We specify product_id as the unique key column to and we define the product_name and description columns to use for the similarity search by the retriever. We use the text-embeddings-3-small open encoder model from HuggingFace. We set auto_embedding to True so that any future insert, update or delete to the source table will automatically generate, update or delete also the corresponding embedding. We provide a name for the retriever so that we can identify and reference it subsequent operations:
+Now let’s create a retriever with the just created products table as the source. We specify product_id as the unique key column to and we define the product_name and description columns to use for the similarity search by the retriever. We use the `all-MiniLM-L6-v2` open encoder model from HuggingFace. We set `auto_embedding` to True so that any future insert, update or delete to the source table will automatically generate, update or delete also the corresponding embedding. We provide a name for the retriever so that we can identify and reference it subsequent operations:
 
 ```sql
 SELECT pgai.create_pg_retriever(
     'product_embeddings_auto', -- Retriever name
     'public', -- Schema
     'product_id', -- Primary key
-    'text-embedding-3-small', -- embedding model
+    'all-MiniLM-L6-v2', -- embedding model
     'text', -- data type
     'products', -- Source table
     ARRAY['product_name', 'description'], -- Columns to vectorize
@@ -87,7 +87,7 @@ SELECT pgai.create_pg_retriever(
     'product_embeddings_bulk', -- Retriever name
     'public', -- Schema
     'product_id', -- Primary key
-    'text-embedding-3-small', -- embedding model
+    'all-MiniLM-L6-v2', -- embedding model
     'text', -- data type
     'products', -- Source table
     ARRAY['product_name', 'description'], -- Columns to vectorize
@@ -103,7 +103,7 @@ __OUTPUT__
 
 We created this second retriever on the products table after we have inserted the AI records there. If we run a retrieve operation now we would not get back any results:
 
-```
+```sql
 SELECT data FROM pgai.retrieve(
 'I like it', -- The query text to retrieve the top similar data
  5, -- top K
@@ -117,7 +117,7 @@ __OUTPUT__
 
 That’s why we first need to run a bulk generation of embeddings. This is achieved via the `refresh_retriever()` function:
 
-```
+```sql
 SELECT pgai.refresh_retriever(
 'product_embeddings_bulk' -- name of the retriever
 );
@@ -229,4 +229,4 @@ __OUTPUT__
 
 We used the two different retrievers for the same source data just to demonstrate the workings of auto embedding compared to explicit `refresh_retriever()`. In practice you may want to combine auto embedding and refresh_retriever() in a single retriever to conduct an initial embedding of data that existed before you created the retriever and then rely on auto embedding for any future data that is ingested, updated or deleted.
 
-You should consider relying on refresh_retriever() only, without auto embedding, if you typically ingest a lot of AI data at once in batch manner.
+You should consider relying on `refresh_retriever()` only, without auto embedding, if you typically ingest a lot of AI data at once in a batched manner.

advocacy_docs/supported-open-source/patroni/installing_patroni.mdx

@@ -7,16 +7,16 @@ tags:
   - Patroni
 ---
 
-EDB provides Patroni to customers via the `edb-patroni` package. This package provides Patroni itself and all its dependencies, so there is no need to install additional packages on the Postgres nodes.
+EDB provides Patroni to customers by way of the `edb-patroni` package. This package provides Patroni and all its dependencies, so you don't need to install additional packages on the Postgres nodes.
 
-As the dependencies no longer have to be installed separately, there is no need to install the python3-cdiff, python3-psutil, python3-psycopg2, python3-ydiff, python3-click, python3-click, python3-six, python3-dateutil, python3-prettytable, python3-pyyaml, python3-urllib3, python3-etcd, python3-dns or python3-certifi packages.
+Since the dependencies don't have to be installed separately, you don't need to install the python3-cdiff, python3-psutil, python3-psycopg2, python3-ydiff, python3-click, python3-click, python3-six, python3-dateutil, python3-prettytable, python3-pyyaml, python3-urllib3, python3-etcd, python3-dns, or python3-certifi packages.
 
 Packages are available for all subscribed customers with a valid EDB account under any entitlement (Community360, Standard, and Enterprise).
 
 !!! Note
-    The `edb-patroni` does not provide packages for the etcd server needed for the DCS cluster
+    The `edb-patroni` package doesn't provide packages for the etcd server needed for the DCS cluster.
 
-Once you have the EDB repository configured on all the nodes of the cluster, run the following commands depending on the Linux distribution you are using.
+Once you have the EDB repository configured on all the nodes of the cluster, run the following commands based on the Linux distribution you're using.
 
 ### Debian/Ubuntu
 
@@ -25,7 +25,7 @@ sudo apt-get install -y edb-patroni
 ```
 
 !!! Note
-    On Debian and Ubuntu installations, if you've previously installed the Patroni package named `patroni` using the EDB repositories, `apt upgrade` will not replace this package with the `edb-patroni` package. Executing `apt install edb-patroni` will install `edb-patroni` as a replacement of `patroni`.
+    On Debian and Ubuntu installations, if you previously installed the Patroni package named `patroni` using the EDB repositories, `apt upgrade` will not replace this package with the `edb-patroni` package. Executing `apt install edb-patroni` will install `edb-patroni` as a replacement of `patroni`.
 
 See [Quick start on Debian 11](debian11_quick_start/#4-patroni) for a more detailed configuration example.
 
@@ -39,5 +39,4 @@ See [Quick start on RHEL8](rhel8_quick_start/#4-patroni) for a more detailed con
 
 ### Installing community packages
 
-We also support community packages provided through PGDG repositories. Follow the [PGDG deb download instructions](https://www.postgresql.org/download/linux/debian/) to set up the `apt` repository, or the [PGDG rpm download instructions](https://yum.postgresql.org/) for the `yum` repository. Keep in mind that for PGDG rpm repositories you will need to configure Extra Packages for Enterprise Linux ([EPEL](https://fedoraproject.org/wiki/EPEL)).
-
+We also support community packages provided through PGDG repositories. Follow the [PGDG deb download instructions](https://www.postgresql.org/download/linux/debian/) to set up the `apt` repository or the [PGDG rpm download instructions](https://yum.postgresql.org/) for the `yum` repository. Keep in mind that you need to configure Extra Packages for Enterprise Linux ([EPEL](https://fedoraproject.org/wiki/EPEL)) for PGDG rpm repositories.

advocacy_docs/supported-open-source/patroni/installing_with_TPA.mdx

@@ -10,4 +10,4 @@ tags:
 
 ### Deploying a Patroni cluster with TPA
 
-The recommended way for deploying Patroni clusters is by doing so with TPA. We recommend going over the [TPA documentation](https://www.enterprisedb.com/docs/tpa/latest/) for further information on deploying M1 architectures with Patroni as the failover manager.
+We recommend deploying Patroni clusters using TPA. See the [TPA documentation](https://www.enterprisedb.com/docs/tpa/latest/) for more information on deploying M1 architectures with Patroni as the failover manager.

product_docs/docs/migration_toolkit/55/09_mtk_errors.mdx

@@ -259,19 +259,18 @@ To correct the problem, specify `-fetchSize 1` as a command-line argument when y
 
 ### Incorrect timestamps and Daylight Saving Time
 
-When migrating from SQL Server to PostgreSQL using the MSSQL JDBC driver there may be no errors observed. If the source database contains rows with timestamp values which are within a range of time when Daylight Savings Time was in effect, they will be migrated to the target database with the wrong timestamps.
+When migrating from SQL Server to PostgreSQL using the MSSQL JDBC driver, an error in the migration of timestamps may occur that is not reported. Specifically, if the source database contains rows with timestamp values that are within a range of time when Daylight Savings Time was in effect, those rows will be migrated to the target database with the wrong timestamps.
 
-To resolve this issue you can update the `runMTK.sh` file and provide the option `-Duser.timezone=GMT`. This will be in operation then when running the toolkit,
+To resolve this issue, update the `runMTK.sh` file, and provide the option `-Duser.timezone=GMT`. This option will then be in effect when you run the toolkit.
 
-For example, given the original line:
+For example, suppose this is the original line:
 
 ```text
 runJREApplication $JAVA_HEAP_SIZE -Dprop=$base/etc/toolkit.properties -cp $base/bin/edb-migrationtoolkit.jar:$base/lib/* com.edb.MigrationToolkit "$@"
 ```
 
-This should be updated with the `-Duser.timezone=GMT` inserted before the `-cp` option:
+Update this line by inserting `-Duser.timezone=GMT` before the `-cp` option:
 
 ```text
 runJREApplication $JAVA_HEAP_SIZE -Dprop=$base/etc/toolkit.properties -Duser.timezone=GMT -cp $base/bin/edb-migrationtoolkit.jar:$base/lib/* com.edb.MigrationToolkit "$@"
 ```
-