Interleaved Document Addition and Document Deletion makes the auto-batching inefficient.

[gmourier]

Asked on the Meilisearch slack community by Brent Dillingham.

With auto batching, I’ve noticed that document updates and deletes are batched separately, in the order in which the tasks entered the queue, which for our workload makes auto batching fairly ineffective since we have deletes interleaved with our document updates.

In other words, each batch is much smaller than they could have otherwise been. We have ~2M documents and are updating the index several times per minute, so we’re fairly reliant on auto batching to use Meilisearch effectively.

Is there any workaround that would allow the auto batcher to group updates and deletes together?

For example, is there a _delete: true special attribute I could use when performing a bulk update?

Use case:

• daily inventory changes from distributors of books regarding stock levels, pricing, etc
• we use sales numbers to influence our search result ranking - each time there is a checkout, we’re updating the product document with the number of sales
• we only delete documents when they should actually be deleted from the index, i.e. it’s a book that’s no longer carried by our distributor. There are ~10 million books in our distributor’s catalog and we only index about 2 million of those, so the deletes are more frequent than you might expect.

[gmourier]

Suggested by Brent Dillingham

Perhaps if we could send deletions as part of the partial update to the PUT /indexes/my_index/documents endpoint, it would remove some ambiguity on ordering since the final state might be easier to understand.

[
  {
    "id": 123,
    "title": "Shazam ⚡️"
  },
  {
    "id": 123,
    "_delete": true
  },
  {
    "id": 123,
    "color": "green"
  }
]

Final state of 123 should be:

{
  "id": 123,
  "color": "green"
}

[gmourier]

Would using a boolean field on the document to remove it from the search results with a filter rather than deleting it could be a decent workaround?

[brentd]

We considered this, but I was concerned about having all ~10 million documents in our meilisearch index, since index update time is currently proportional to the number of documents in the index.

[gmourier]

Hello @brentd 👋

Following up on that subject, before thinking of providing an API endpoint to that, we will explore a way to batch addition and deletion together. See meilisearch/meilisearch#3440

Thank you!

[gmourier]

Hello, everyone following this discussion 👋

We have just released the first RC (release candidate) of Meilisearch containing the batching of interleaved document addition and deletion!

You can test it by using

• the release assets
• the Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.1.0-rc.0

You are welcome to communicate any feedback about this new implementation in this discussion.

If you encounter any bugs, please report them here.
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 Official and stable release containing this change will be available on 3rd April 2023

⚠️ RC (release candidates) are not recommended for production

[brentd]

Thanks @gmourier, I'm looking forward to trying this out soon!

[gmourier]

Thanks, @brentd! Please let us know if it speeds up your workflow!

[gmourier]

Hey folks 👋

It has been released with v1.1 ✨