[BUG] Fix HNSW resize conditions to exclude deleted items

[eculver]

Description of changes

We were seeing errors in compaction that looked like:

WriteSegments(ApplyMaterializatedLogsError(HnswIndex(FFIException(\"The number of elements exceeds the specified limit\"))))

This is because the resize logic uses len which removes deleted elements which then causes the conditional for resizing to be skipped. This adds a new method len_with_deleted to return the length of the index including the elements that have been deleted so that the resize is not skipped. See:

chroma/rust/worker/src/segment/distributed_hnsw_segment.rs

Lines 212 to 222 in 27564af

	let mut index = self.index.inner.upgradable_read();
	let index_len = index.len();
	let index_capacity = index.capacity();
	if index_len + 1 > index_capacity {
	index.with_upgraded(|index| {
	// Bump allocation by 2x
	index
	.resize(index_capacity * 2)
	.map(|_| ApplyMaterializedLogError::Allocation)
	})?;
	}

Test plan

A test was added. It will fail when using the existing len method, but passes when using the newly added len_with_deleted method.

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

N/A

Thanks to @sanketkedia for helping root cause this.

Fixes https://github.com/chroma-core/hosted-chroma/issues/855

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[eculver]

Going to merge since Sanket has already looked over it all.