How to check if a document exists before inserting it into the index?

[johnhu33]

How to check if a document exists before inserting it into the index?

[dosubot[bot]]

To check if a document exists before inserting it into the index, you can use the _handle_duplicates or _handle_upserts methods from the IngestionPipeline class. These methods check for existing documents based on their hashes or IDs.

Here is an example of how you can use _handle_duplicates to check for document existence:

def check_document_exists(pipeline: IngestionPipeline, nodes: List[BaseNode]) -> List[BaseNode]:
    """Check if documents exist in the docstore before inserting.

    Args:
        pipeline: The ingestion pipeline instance.
        nodes: The nodes (documents) to check.

    Returns:
        List of nodes that do not exist in the docstore.
    """
    if pipeline.docstore is None:
        raise ValueError("Docstore is not configured in the pipeline.")

    existing_hashes = pipeline.docstore.get_all_document_hashes()
    nodes_to_insert = []

    for node in nodes:
        if node.hash not in existing_hashes:
            nodes_to_insert.append(node)

    return nodes_to_insert

# Example usage
pipeline = IngestionPipeline(...)  # Initialize your pipeline
nodes = [...]  # List of nodes (documents) to check
nodes_to_insert = check_document_exists(pipeline, nodes)

if nodes_to_insert:
    # Proceed with inserting nodes_to_insert into the index
    pass
else:
    print("All documents already exist in the index.")

This function check_document_exists will return a list of nodes that do not exist in the docstore, allowing you to insert only the new documents ^[1][2].

Additionally, the IngestionPipeline class provides a docstore_strategy attribute that can be set to handle different de-duplication strategies, such as UPSERTS, DUPLICATES_ONLY, and UPSERTS_AND_DELETE. This can be useful for more advanced document management scenarios ^[1].

To continue talking to Dosu, mention @dosu.