Redis vector store index and retrieval issues

[johnhu33]

Why redis vector store can't retrieve content even though it is exactly matched by the query? Secondly why it retrieves the same content even though it is retrieving from a different custom index? Isn't redis database separated by the index name?

[dosubot[bot]]

I found two similar closed issues that might help:

1. [Bug]: cant retrieve image which are indexed using redis..retrieved_image is an empty list...the same code works without issue for qdrant database which is in llaamaindex documentation ^[1].
2. [Bug]: VectorStoreIndex not working when created with RedisIndexStore and RedisVectorStore ^[2].

Possible Reasons and Solutions:

1. Schema Issues: Ensure the schema is correctly defined with required fields (id, doc_id, text, and vector). Redis expects an explicit schema definition.

2. Prefix Issues: Redis uses key prefixes to segment the keyspace. Inconsistent prefixes can lead to retrieval issues. Ensure the prefix is consistent and correctly set in the index schema.

3. Data vs Index: Redis treats records in the dataset and the index as different entities. Ensure the index is correctly built and updated.

4. Empty Queries with Metadata: Metadata must be indexed correctly at the time of index creation. Adding metadata after index creation can lead to empty queries.

5. Query Embedding: Ensure the query embedding is provided when querying. Missing query embedding will cause the query to fail.

6. Redis Errors: Handle potential Redis errors such as timeouts or other Redis-specific issues.

Example Code to Ensure Correct Schema and Prefix

from redisvl.schema import IndexSchema

custom_schema = IndexSchema.from_dict(
    {
        "index": {
            "name": "paul_graham",
            "prefix": "essay",
            "key_separator": ":",
        },
        "fields": [
            {"type": "tag", "name": "id"},
            {"type": "tag", "name": "doc_id"},
            {"type": "text", "name": "text"},
            {"type": "numeric", "name": "updated_at"},
            {"type": "tag", "name": "file_name"},
            {
                "type": "vector",
                "name": "vector",
                "attrs": {
                    "dims": 1024,
                    "algorithm": "hnsw",
                    "distance_metric": "cosine",
                },
            },
        ],
    }
)

vector_store = RedisVectorStore(
    schema=custom_schema,
    redis_client=redis_client,
    overwrite=True,
)

storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

Example Code to Query with Metadata Filters

from llama_index.core.vector_stores import (
    MetadataFilters,
    MetadataFilter,
    ExactMatchFilter,
)

retriever = index.as_retriever(
    similarity_top_k=3,
    filters=MetadataFilters(
        filters=[
            ExactMatchFilter(key="file_name", value="paul_graham_essay.txt"),
            MetadataFilter(
                key="updated_at",
                value=date_to_timestamp("2023-01-01"),
                operator=">=",
            ),
            MetadataFilter(
                key="text",
                value="learn",
                operator="text_match",
            ),
        ],
        condition="and",
    ),
)

result_nodes = retriever.retrieve("What did the author learn?")
for node in result_nodes:
    print(node)

Handling Redis Query Errors

redis_query = get_redis_query(
    return_fields=return_fields,
    top_k=query.similarity_top_k,
    vector_field=self._vector_field,
    filters=filters,
)

if not query.query_embedding:
    raise ValueError("Query embedding is required for querying.")

query_params = {
    "vector": array_to_buffer(query.query_embedding),
}
_logger.info(f"Querying index {self._index_name}")

try:
    results = self._redis_client.ft(self._index_name).search(
        redis_query, query_params=query_params  # type: ignore
    )
except RedisTimeoutError as e:
    _logger.error(f"Query timed out on {self._index_name}: {e}")
    raise
except RedisError as e:
    _logger.error(f"Error querying {self._index_name}: {e}")
    raise

if len(results.docs) == 0:
    raise ValueError(
        f"No docs found on index '{self._index_name}' with "
        f"prefix '{self._prefix}' and filters '{filters}'. "
        "* Did you originally create the index with a different prefix? "
        "* Did you index your metadata fields when you created the index?"
    )

Ensure the schema is correctly defined, the prefix is consistent, metadata is indexed correctly, and the query embedding is provided to avoid retrieval issues. Handle potential Redis errors appropriately ^[3][4].

To continue talking to Dosu, mention @dosu.

[johnhu33]

@dosu when query using vector_store.query(), it retrieved nodes from other index, even though customised index had been set

[dosubot[bot]]

vector_store.query() retrieves nodes from another index even though a customized index has been set because the VectorStoreIndex class defaults to using the SimpleVectorStore index if you don't provide your own Vector Store. This behavior is explicitly mentioned in the provided script:

This also flows upstream to the VectorStoreIndex class - which will default use SimpleVectorStore index if you don't provide your own Vector Store.

Additionally, in the VectorIndexRetriever class, the _vector_store attribute is set to the vector_store of the provided index:

self._vector_store = self._index.vector_store

Therefore, if you have not explicitly set your customized index in the VectorStoreIndex class, it will fall back to using the SimpleVectorStore index. Make sure to set your customized vector store in the VectorStoreIndex to avoid this issue.

For example, if you are using a RedisVectorStore, you need to ensure that it is properly initialized and set in the VectorStoreIndex:

from llama_index.core import StorageContext
from redis import Redis
from llama_index.vector_stores.redis import RedisVectorStore

# create a Redis client connection
redis_client = Redis.from_url("redis://localhost:6379")

# create the vector store wrapper
vector_store = RedisVectorStore(redis_client=redis_client, overwrite=True)

# load storage context
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# build and load index from documents and storage context
index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)

By ensuring that your customized vector store is properly set, you can avoid the issue of vector_store.query() retrieving nodes from another index ^[1][2][3].

To continue talking to Dosu, mention @dosu.