Consume memory & disk overflow queues in order

[david-eckardt-sociomantic]

Some applications need the records to be in order. The current disk overflow push strategy can change the order dramatically. The current strategy is to

• try pushing the record to the memory queue,
• if the record does not fit, push it to the disk overflow instead.

The pop strategy is the same - try popping from the memory queue, if it is empty, pop from the disk overflow.
With this push strategy records can be pushed in the memory queue when a disk overflow is present, and these records will be popped earlier than the records in the disk overflow that have been pushed earlier. The order of records can be preserved by changing the the push strategy as follows:

• If there is disk overflow, push the record to it.
• If not, apply the above strategy.

That way, if there is a disk overflow, it will contain the most recently pushed records while the memory queue will contain the least recently pushed records -- the records stay in order.

So the way how pushing is carried out determines the order in which the records are popped. Looking at pushing and consuming application a problem arises here: It is the consuming application that cares about the order, the pushing application doesn't and shouldn't need to.
The proposed solution is to extend the Consume request with a flag determining the push strategy. The first consumer that subscribes to a to a DMQ channel requests the push strategy for that queue. Subsequent consumers are expected to request the same push strategy or the Consume request fails with an error for them. This is feasible because multiple consumers using the same subscriber name for the same channel are expected to be instances of the same application and therefore require the same push strategy.

Another problem is what happens if a queue (i.e. channel with or without subscribers) is created but the push strategy is not know at that time.

• If a Push request creates a new channel, the push strategy doesn't matter because records are anyway pushed to the memory queue first, then to the disk overflow, so they stay in order. The push strategy does matter only after the first consumer has started consuming, popping records from the memory queue.
• If the DMQ node starts and creates a channel from a memory dump without disk overflow data present, the situation is the same as in the previous point: The push strategy matters only after the first consumer has started consuming. That means that the push strategy does not need to be stored in the memory dump.
• If the DMQ node starts and creates a channel where disk overflow data is present, the push strategy needs to be known: If the memory queue is not full and a record is pushed before the first consumer has started consuming, it must be know whether this record should go in the memory queue or the disk overflow.

So the push strategy needs to be stored in the disk overflow index but not in the memory dump.

Finally there is still the Pop request. For now there is no need for any application that uses Pop to pop records in order so we can ignore that.

[nemanja-boric-sociomantic]

Would the permanent change of the push strategy be bad?

[david-eckardt-sociomantic]

Would the permanent change of the push strategy be bad?

Yes, it would be for some of the consumer processes in our system which need priority for live traffic.