CR-1175649 Always wait for command completion through hwqueue in SHIM…

… layer (#7702)

Today, XRT coreutil can directly check the exec buf state to know if a
command is completed without calling into SHIM layer. In the command
submission path, however, XRT coreutil will always call into SHIM
layer to issue the cmd to HW. There is a little bit asymmetric in this
architecture, which makes it harder for SHIM to abstract device
specific behavior and hide it from coreutil. It gets leaked to
coreutil through the completion code path.

One particular example is that it is difficult for SHIM to insert
static probes in both submission and completion code path to do
performance profiling since SHIM may not be involved in completion
path. And this type of probe is specific to TurinAI due to the fact
that the driver is not involved during command handling at all. In IPU
case, where driver is fully involved in both completion and submission
path, we have inserted static probe in driver.

When this CR is fixed, the workaround implemented in CR-1167065 can be
reverted since CERT will not directly write to the exec buf state
field anymore. SHIM will hide it from coreutil. This revert will be
handled separately from this PR.

Signed-off-by: Soren Soe <[email protected]>

src/runtime_src/core/common/api/hw_queue.cpp

@@ -412,12 +412,17 @@ class qds_device : public hw_queue_impl
   std::cv_status
   wait(const xrt_core::command* cmd, size_t timeout_ms) override
   {
+    // Dispatch wait to shim hwqueue_handle rather than accessing
+    // pkt state directly.  This is done to allow shim direct control
+    // over command completion and command state.
+    if (m_qhdl->wait_command(cmd->get_exec_bo(), static_cast<int>(timeout_ms)) == 0)
+      return std::cv_status::timeout;
+
+    // Validate command state
     auto pkt = cmd->get_ert_packet();
-    while (pkt->state < ERT_CMD_STATE_COMPLETED) {
-      // return immediately on timeout
-      if (m_qhdl->wait_command(cmd->get_exec_bo(), static_cast<int>(timeout_ms)) == 0)
-        return std::cv_status::timeout;
-    }
+    if (pkt->state < ERT_CMD_STATE_COMPLETED)
+      // unexpected state
+      throw std::runtime_error("qds_device::wait() unexpected command state");
 
     // notify_host is not strictly necessary for unmanaged
     // command execution but provides a central place to update