problem with indexing - sometimes index is not created after search is executed

[zjazd]

When multiple indexing jobs are running I noticed critical indexing problem.

I run search query ?q=*:*&limit=0 to reindex search index on newly replicated database.
Server starts indexing, response is returned but index is not created.

those are steps that shows problem:

• run search query ?q=*:*&limit=0
• wait for completion ~1 minute
• check _design/DESIGNDOC_NAME/_search_info/INDEX_NAME endpoint - pending_seq is not the same as DB update_seq
• again run search query ?q=*:*&limit=0 and i have to wait for index to recalculate
• wait for completion ~1 minute
• check _design/DESIGNDOC_NAME/_search_info/INDEX_NAME endpoint - pending_seq is not the same as DB update_seq
• ...

I stuck on this problem when replicating multiple DB's to new environment and tries to reindex all indexes in those DB's. It was about 50 db's with 1k - 1000k documents inside and from 10 - 150 indexes.

I wrote a program to reindex databases after replication, this program just called search queries on indexes 10 indexed at once. This is when those strange behavior happens.

In couchdb logs i have multiple info messages index xxx closed with reason lru, maybe this is relevant.

Clouseau is running on windows 2012R2 server.

[zjazd]

I think I found reason, in my refresh indexes tool I had a loop to refresh index info, this loop was quering _design/DESIGNDOC_NAME/_search_info/INDEX_NAME endpoints for all indexes.

It looks like querying _design/DESIGNDOC_NAME/_search_info/INDEX_NAME endpoint refreshes index in LRU algorithm and LRU removes index that was currently indexed -> progress was not commited...