kvBuckets are unexpectedly slow

[EgusCpp]

Greetings!
I am using NATS kvBuckets with FileStorage. Mostly to store key-value pair and delete it over few seconds. It was all good with performance, until it reached over 100k pairs. Write and erase operations were still fine, while listing keys ("kv ls bucket_name") or clearing memory ("kv compact") started taking enormous time, provided that nats-server process consuming 98-99% of cpu core it attached to. Literally, before calling "kv compact" I had 106,538 records using 16MB of disk space, and after waiting 30+ minutes, I terminated nats-cli and now I have 85,904 records using 13MB of disk space. I.e. so much time to free 3MB? I don't understand what is wrong here, maybe it is not supposed to use buckets with file storage having big number of key-value pairs?

Nats server v.2.10.7

upd: 48+ minutes to "compact" the rest

[ripienaar]

Our design is heavily optimised for Get, Put, Update, Create, Delete etc being fast as possible. Listing keys is particularly slow in how it is currently designed as that is a client side operation.

We have some work planned to make that faster but for now listing keys is not fast and probably something that in typical use cases should not be done that often.

Likewise compact has to essentially scan through every key, do some checks etc, all on the client. This is also a slow process inherently but again probably not something we'd imagine people do super commonly.

[EgusCpp]

Maybe there is a way to reuse purged pairs for new unique keys, and I am just missing something?

[ripienaar]

Delete in KV is still slow as it has to write each key and mark it as deleted for audit trail purposes.

Well using a hierarchy will mean you can very quickly purge entire sub trees out so I would suggest considering that. Our operations tend to be on nats wildcards from subjects so think about those.

Other option is to use a stream directly where you have more control of implementation specifics such as optimising deletes that don't write audit trails and purging out entire subsets of data in a way that means they wont need to call compact.

[EgusCpp]

And there is no way for kvBuckets to disable "audit trails" , history of changes, etc?

[ripienaar]

No, this is essentially what they are and it also serve another purpose that of making watch be aware of delete, we cant really turn that off.

Using a stream seems better for you

[EgusCpp]

Thank you for assistance! However, streams are also a bit different from behaviour expected from plain persistent key-value storage.

[ripienaar]

In what way?

[EgusCpp]

I would expect the following operations from persistent key-value storage: put key with value, get value for key, list available keys and delete key with value. There is no "consumer" entity supposed. Also NATS streams are providing messages deletion only by sequence number, as far as I can see, what is very incovenient when you need to address using only your own keys.

[ripienaar]

We have:

• Put key with value
• Get value for key
• List available keys (albeit its a bit slow)
• Delete key by key name not value (purge per subject which includes wildcards hence fast wildcard deletes of many keys)

I've not seen before a delete key with value in other KVs or do you mean only delete by key if the value is a certain value?

A KV bucket is a stream, so you can assume the stuff the KV API can do is also supported by streams.