Use DescribeQueries rather than GetQueryResults to poll for status

[vcschapp]

User Story

As an application developer, I want my log queries to run as fast as possible.

Background

Currently (as of v1.2.0 and soon to be v1.3.0), Incite uses the CloudWatch Logs GetQueryResults API operation to poll the status of running queries.

This is limiting in several ways:

• The service quota limit for GetQueryResults is 5 TPS and this value cannot be increased.
• Only one running CWL query (Incite chunk") at a time may be queried in this method, which means:
  • at most 5 different CWL queries (Incite chunks) may be polled per second;
  • if Incite is consuming 100% of CWL capacity, it takes a full two-second cycle to poll all running chunks at full parallelism.

CloudWatch Logs recently introduced a new DescribeQueries API operation. The RPS limit doesn't seem to be onboarded to service quotas yet, but because the operation is capable of returning the status of many CWL queries (Incite chunks) at once, there is an opportunity to reduce the latency between when a query changes status and when Incite learns about it, even at 5 TPS.

Notes

• If we do this work, note that preview-mode streams would still have to poll with GetQueryResults.
• The work is a fairly big rock since it would entail:
  • creating another worker sub-type and differentiating between polling for status changes and polling for results; and
  • rewriting zillions of mocks.
• The DescribeQueries API isn't super-ergonomic for our use case, because while it has some filters, you can't combine the filters in a way that's efficient. So implementing this in a way that scales to many queries in a QM would require making the polling code pretty clever, and that is yet more work.