Implement dynamic tag info functionality for accurate tag calculation with multiple queues configured per server

[sseshasa]

To better understand the fix, the problem with multiple queues configured per server
needs to be understood first. This problem is currently faced by Ceph (or Ceph OSDs)
which are clients of dmClock server. Ceph creates multiple op queues per OSD (a.k.a
OSD op queue shards) and each op queue runs independently with the same server
id (OSD ID). With this configuration, tags are calculated on each queue independently.
Therefore, a client could distribute requests across the multiple mClock queues but
the server cannot meet the clients' QoS settings due to the following problem:

Problem Description

Consider two queues configured on the same server with
items added in the following sequence:

   Enqueue--->|T5|T2|T0|--->Dequeue
                   Queue0

   Enqueue--->|T4|T3|T1|--->Dequeue
                   Queue1

Consider the request arrival times (arr_time) in each queue according
to the number associated with each tag, for example:

 - Req0 arrives on queue0 at time t0
 - Req1 arrives on queue1 at time t1
 - Req2 arrives on queue0 at time t2

and so on with the final Req5 arriving on queue0 at time t5

Consider x to be any of the client info parameters i.e. reservation, weight or limit

Tag calcs on Queue0	Tag calcs on Queue1
T0 = max((NO_TAG + (1/x)), arr_time)	T1 = max((NO_TAG + (1/x)), arr_time)
T2 = max((T0 + (1/x)), arr_time)	T3 = max((T1 + (1/x)), arr_time)
T5 = max((T2 + (1/x)), arr_time)	T4 = max((T3 + (1/x)), arr_time)

For simplicity, assume reservation and limit is set to 3 IOPS, AtLimit is
set to 'Wait' and all requests arrived within a few milliseconds on
both queues.

It's pretty clear that all the 3 requests from both the queues will be
scheduled resulting in 6 IOPS which is not correct.

Proposed Solution

The fix is better understood by applying the solution to the above problem.
The fix is applicable to both DelayedTagCalc and ImmediateTagCalc.

Pre-conditions:

• Dynamic tag info functionality (bool U2 in the pull constructor) is enabled (is_dynamic_tag_info_f).
• The dmClock client maintains a mapping of clientId to ReqTagInfo for each client.
• The dmClock client registers custom implementations for reqtag_updt_f and reqtag_info_f.

Solution:

The fix involves enabling the dynamic tag info for two operations by the following entities:

1. the server - to update (using reqtag_updt_f) the dmClock client with latest tag for a clientId on a given queue once it's calculated. This is called after tags are calculated in initial_tag() and update_next_tag().

2. the dmClock client - to calculate and update the tick interval before adding a request to the mClock queue. For a given queue, the tick interval is the number of request(s) added to other mClock queues before the current one arriving on itself. The latest tick interval is read by the server using reqtag_info_f before calculating the initial_tag(). See calc_interval_tag() on how the new tag is calculated and see the unit tests on how the client calculates the tick interval before adding the request to the queue. It's important to note that for DelayedTagCalc, the tag will be calculated as part of initial_tag() ONLY for non-zero tick intervals.

The tag calculation for each tag is shown below with the fix in place:
(Apologies for the table formatting. Please consider the tag calculation row as the start of a row)

Tag calcs on Queue0	Tag calcs on Queue1
T0 = max((NO_TAG + (1/x)), arr_time)	T1 = max((T0 + (i/x)), arr_time)
tick interval i = 0;	tick interval i = 1 because req1
	arrived after req0 on Queue0.
	[Note: If tick_inteval > 0, the tag
	is calculated as part of
	initial_tag() for DelayedTagCalc]
T2 = max((T1 + (i/x)), arr_time)	T3 = max((T2 + (i/x)), arr_time)
tick interval i = 1	tick interval i = 1
T5 = max((T3 + (i/x)), arr_time)	T4 = max((T3 + (1/x)), arr_time)
tick interval i = 2 because req5	tick interval i = 0 because req4
arrived two requests after T2.	follows immediately after req3 on
	the same queue.
	[Note: If tick_interval == 0, the tag
	is calculated only when it gets to the front
	as part of update_next_tag() as usual
	if DelayedTagCalc is enabled.]

For simplicity, assume reservation and limit is set to 3 IOPS, AtLimit is set to 'Wait' and all requests arrived within a second on both queues.

With the fix in place only T0, T1 and T2 will be scheduled as expected during the first phase since each request is spaced 1/res apart i.e., 1/3rd of a second. The same pattern will apply for the rest of the requests in the queues thus ensuring accurate scheduling of requests across all the queues with the same server.

Tests:
A lot of unit tests have been added to exercise the solution with both delayed and immediate tag calculation. Of particular importance are the tests that completely randomize queue additions and pulls from multiple(5 Nos.) queues. The tests also provide an insight into how the clients calculate the latest tick interval and how the latest tag is updated by the client via the dynamic functions. See:
- pull_reservation_randomize_delydtag
- pull_reservation_randomize_immtag
- pull_weight_randomize_delydtag
- pull_weight_randomize_immtag

Signed-off-by: Sridhar Seshasayee [email protected]

[sseshasa]

@athanatos @ivancich @rzarzynski @neha-ojha Please take time to review this PR. I know most of you will be out for Cephalocon. In the meantime, I am hoping to get the changes on the Ceph side ready and get some real tests going. Thanks!

[athanatos]

In your first example above, why can't we avoid the problem by simply setting per-queue reservation/limit to 1/3 (1/<num_queues>) of the value we want for the whole OSD?

[athanatos]

reqtag_updt_f would be updating memory shared between queues, right?

[sseshasa]

In your first example above, why can't we avoid the problem by simply setting per-queue reservation/limit to 1/3 (1/<num_queues>) of the value we want for the whole OSD?

This may not work in all scenarios. For e.g., if a subset of queues are active for a workload, the realized IOPS may be lower than what is set by the client.

I recall having tried this on a cluster a while ago but don't remember what the result looked like. I will try it again and get back.

But I tried this with the randomized unit test without dynamic tag info and the number of requests dequeued from the client was lower than expected but not by much. I suspect on an actual cluster we may observe lower than expected IOPS.

Let me run a few tests on a cluster and get back with more details.

[sseshasa]

reqtag_updt_f would be updating memory shared between queues, right?

Yes, that's correct. The following block diagram should help clarify things with respect to which entity maintains and creates the associated data structures and dynamic functions. More details will be added to this diagram as things progress:

[sseshasa]

Let me run a few tests on a cluster and get back with more details.

With a single client, I ran a test using rados bench with your suggestion i.e., divide res and lim by number of shards. The results show that client is unable to achieve the set limit. Res was set to 125 IOPS and Limit to 625 IOPS and wgt set to 1. On each shard, res and lim was divided by the number of shards. For comparison, I ran the test with a single shard. Here's the outcome:

client QoS: [res:125 wgt:1 lim:625]

With 5 shards & res and limit reduced by num_shards

The average IOPS is off by around 70 IOPS as shown below:

Total time run:         120.086
Total writes made:      66737
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     2.17087
Stddev Bandwidth:       0.0778568
Max bandwidth (MB/sec): 2.32812
Min bandwidth (MB/sec): 1.92969
Average IOPS:           555
Stddev IOPS:            19.9313
Max IOPS:               596
Min IOPS:               494
Average Latency(s):     0.0287745
Stddev Latency(s):      0.0245614
Max latency(s):         0.137659
Min latency(s):         0.000275006

With 1 OSD shard

Total time run:         120.026
Total writes made:      74925
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     2.43845
Stddev Bandwidth:       0.0278175
Max bandwidth (MB/sec): 2.46875
Min bandwidth (MB/sec): 2.15625
Average IOPS:           624
Stddev IOPS:            7.12127
Max IOPS:               632
Min IOPS:               552
Average Latency(s):     0.0256237
Stddev Latency(s):      0.0040078
Max latency(s):         0.143611
Min latency(s):         0.000396659