Xclbin to ELF flow migration #8581

rbramand-xilinx · 2024-10-29T13:18:30Z

Problem solved by the commit

This PR enables new flow where ELF file is input instead of xclbin.
spec - https://confluence.amd.com/display/AIE/AIE+Compiler+Artifacts#AIECompilerArtifacts-Flow2

spec of new config elf - https://confluence.amd.com/pages/viewpage.action?pageId=1485002156#Profile/Config/CtrlcodeElfSpec-Config.elf

This new Elf has partition size and kernel signature.
It has multiple .ctrltext and .ctrldata sections and multiple .pdi sections.
Each ctrltext section represents one control code and pdi addresses along with kernel args needs to be patched into the control code buffer.
Control code that needs to be run is decided while creating xrt::kernel object using kernel name - <kernel_name>:<ctrl_code_id> eg: "DPU:0" runs with control code from section .ctrltext0 and "DPU:1" runs with control code from section .ctrlcode1

Sample test case :

#include "xrt/xrt_hw_context.h"
#include "xrt/xrt_kernel.h"
#include "xrt/xrt_elf.h"

int main(int argc, char** argv)
{
    std::string elf_path {argv[1]};   // elf path
    auto elf = xrt::elf(elf_path);

    auto device = xrt::device(0);
    auto ctx = xrt::hw_context(device, elf);

    std::string kernel_name = "DPU:1";
    auto kernel = xrt::ext::kernel(ctx, kernel_name);
    
    //  create args
    auto run = kernel(arg1, arg2 ...);
    run.wait();

    return 0;
}

XRT first class object changes :
xrt::hw_context - Added new APIs to create context using Elf instead of xclbin
xrt::kernel - Added new ext API to create kernel object using kernel name, added changes to construct kernel args from kernel signature.
xrt::elf - can now take new Elf with os abi 70 as input
xrt::module - added changes to parse this new Elf, also refactored code

TODO : Refactor code to parse ELF file using xrt::elf class and make it similar to xrt::xclbin. Created CR - https://jira.xilinx.com/browse/CR-1219757 for tracking this.

Bug / issue (if any) fixed, which PR introduced the bug, how it was discovered

This is a new feature

How problem was solved, alternative solutions (if any) and why they were rejected

Hw context can be created either with xclbin (traditional flow) or with xrt::elf. A new constructor is provided for same.
A new xrt::ext::kernel constructor is provided that takes ctx and kernel name as input.
Added code to parse new Elf file and patch it according to spec.
Rest of the flow remains same

Risks (if any) associated the changes in the commit

Low to medium
Tested the existing flows but needs more testing with all the available test cases.

What has been tested and how, request additional testing if necessary

Tested with new application flow on aie2p simnow (needs changes with respect to flow in amdxdna shim and firmware, changes are yet to be merged, so tested with local drops)

Tested existing test cases on phoenix hw (linux) and tests passes so existing flow didn't break.

TODO : check whether existing aie2ps test cases work

Documentation impact (if any)

Added doxygen comments in code for new APIs added, may be we need to document about new flow after it is stabilized.

stsoe

Overall looks good. I am not completely done reviewing, but maybe address my first points and I will review again.

src/runtime_src/core/common/api/hw_context_int.h

src/runtime_src/core/common/api/module_int.h

src/runtime_src/core/common/api/xrt_hw_context.cpp

src/runtime_src/core/common/api/xrt_kernel.cpp

stsoe · 2024-10-29T15:58:55Z

src/runtime_src/core/common/xclbin_parser.cpp

+kernel_properties::mailbox_type
+get_mailbox_from_ini(const std::string& kname)
+{
+  static auto mailbox_kernels = xrt_core::config::get_mailbox_kernels();
+  return (mailbox_kernels.find("/" + kname + "/") != std::string::npos)
+    ? xrt_core::xclbin::kernel_properties::mailbox_type::inout
+    : xrt_core::xclbin::kernel_properties::mailbox_type::none;
+}
+
+// Kernel auto restart counter offset
+// Needed until meta-data support (Vitis-1147)
+kernel_properties::restart_type
+get_restart_from_ini(const std::string& kname)
+{
+  static auto restart_kernels = xrt_core::config::get_auto_restart_kernels();
+  return (restart_kernels.find("/" + kname + "/") != std::string::npos)
+    ? 1
+    : 0;
+}


I would rather not. Instead of rebuilding the properties, please cache a pointer to the existing properties that have all the information

Do we need to export these functions?

stsoe · 2024-10-29T15:59:16Z

src/runtime_src/core/common/xclbin_parser.cpp

+bool
+get_sw_reset_from_ini(const std::string& kname)
+{
+  static auto reset_kernels = xrt_core::config::get_sw_reset_kernels();
+  return (reset_kernels.find("/" + kname + "/") != std::string::npos);
+}


Same export question

stsoe · 2024-10-29T15:59:36Z

src/runtime_src/core/common/xclbin_parser.h

+kernel_properties::mailbox_type
+get_mailbox_from_ini(const std::string& kname);
+
+kernel_properties::restart_type
+get_restart_from_ini(const std::string& kname);
+
+bool
+get_sw_reset_from_ini(const std::string& kname);
+


Lets avoid.

Still open?

I need these functions while creating properties in ELF flow as well @stsoe , should I use static functions there as well? I moved these functions in header so that there wont be code duplicity

AShivangi · 2024-10-29T17:21:29Z

@hlaccabu recently added some code to enable elf flow in xrt-smi validate for npu3. Please work with him to make sure that this change doesn't breaks the existing code.

hlaccabu

good to go from driver perspective. As long as driver's xrt submodule (and corresponding xrt_plugin.deb) contains these changes as well as the XRT .deb.

src/runtime_src/core/common/api/xrt_module.cpp

src/runtime_src/core/include/ert.h

rbramand-xilinx · 2024-10-30T10:08:04Z

good to go from driver perspective. As long as driver's xrt submodule (and corresponding xrt_plugin.deb) contains these changes as well as the XRT .deb.

@hlaccabu can you please check if your feature doesn't break with these changes

hlaccabu · 2024-10-30T16:54:34Z

good to go from driver perspective. As long as driver's xrt submodule (and corresponding xrt_plugin.deb) contains these changes as well as the XRT .deb.

@hlaccabu can you please check if your feature doesn't break with these changes

Yep you're all set, the xrt-smi tests that I enabled for npu3 still run

rbramand-xilinx · 2024-11-04T12:24:44Z

Hi @stsoe , created CR - https://jira.xilinx.com/browse/CR-1219757 for refactoring existing code to move ELF parsing to xrt::elf object and make it similar to xrt::xclbin.
Please review this PR, I will do the refactoring using next set of PRs

larry9523 · 2024-11-05T23:41:33Z

@rbramand-xilinx , Did we run any exsiting preemption test to make sure we won't break any?

rbramand-xilinx · 2024-11-06T14:54:30Z

@rbramand-xilinx , Did we run any exsiting preemption test to make sure we won't break any?

Hi @larry9523 , I have run preemption test cases with my changes and found 1 bug and fixed it. Thanks

stsoe

Some more comments. I did not review xrt_module.cpp.

src/runtime_src/core/common/api/module_int.h

src/runtime_src/core/common/api/xrt_hw_context.cpp

stsoe · 2024-11-08T16:43:51Z

src/runtime_src/core/common/api/xrt_kernel.cpp

+    // initialize kernel name and ctrl code index
+    auto i = nm.find(":");
+    if (i == std::string::npos) {
+      // default case - ctrl code 0 will be used
+      name = nm.substr(0, nm.size());
+      m_ctrl_code_index = 0;
+    }
+    else {
+      name = nm.substr(0, i);
+      m_ctrl_code_index = std::stoul(nm.substr(i+1, nm.size()-i-1));
+    }


Uhh, this is a little hard to parse.
This is the second place name is parsed at :. Can this parsing be re-factored into say

std::pair<std::string, uint32_t> static get_kernel_name_and_ctrl_code_index(const std::string& nm) { ... return {name, index}; }

Then use as

std::tie(name, m_ctrl_code_index) = get_name_and_ctrl_code_index(nm);

Maybe choose a better function name for the refactored code.

I will refactor the code Soren, my attempt was to use initializer list for most of the class members but yeah it does not look good.

stsoe · 2024-11-08T16:53:37Z

src/runtime_src/core/common/xclbin_parser.cpp

+kernel_properties::mailbox_type
+get_mailbox_from_ini(const std::string& kname)
+{
+  static auto mailbox_kernels = xrt_core::config::get_mailbox_kernels();
+  return (mailbox_kernels.find("/" + kname + "/") != std::string::npos)
+    ? xrt_core::xclbin::kernel_properties::mailbox_type::inout
+    : xrt_core::xclbin::kernel_properties::mailbox_type::none;
+}
+
+// Kernel auto restart counter offset
+// Needed until meta-data support (Vitis-1147)
+kernel_properties::restart_type
+get_restart_from_ini(const std::string& kname)
+{
+  static auto restart_kernels = xrt_core::config::get_auto_restart_kernels();
+  return (restart_kernels.find("/" + kname + "/") != std::string::npos)
+    ? 1
+    : 0;
+}


Do we need to export these functions?

stsoe · 2024-11-08T16:53:46Z

src/runtime_src/core/common/xclbin_parser.cpp

+bool
+get_sw_reset_from_ini(const std::string& kname)
+{
+  static auto reset_kernels = xrt_core::config::get_sw_reset_kernels();
+  return (reset_kernels.find("/" + kname + "/") != std::string::npos);
+}


Same export question

stsoe · 2024-11-08T16:54:02Z

src/runtime_src/core/common/xclbin_parser.h

+kernel_properties::mailbox_type
+get_mailbox_from_ini(const std::string& kname);
+
+kernel_properties::restart_type
+get_restart_from_ini(const std::string& kname);
+
+bool
+get_sw_reset_from_ini(const std::string& kname);
+


Still open?

src/runtime_src/core/include/experimental/xrt_module.h

stsoe

Looks good now. Let me know when you have resolved if we really need to export internal xclbin_parser functions or not.

src/runtime_src/core/common/api/xrt_hw_context.cpp

src/runtime_src/core/common/api/xrt_kernel.cpp

rbramand-xilinx · 2024-11-13T08:57:09Z

Looks good now. Let me know when you have resolved if we really need to export internal xclbin_parser functions or not.

Hi @stsoe, we don't need those functions to be exported. I have reverted the change and verified the testcase it passes.
Thanks for the insights.

Signed-off-by: rbramand <[email protected]>

rbramand-xilinx requested review from stsoe, rozumx, maxzhen, mamin506, uday610, chvamshi-xilinx and AShivangi as code owners October 29, 2024 13:18

rbramand-xilinx requested review from larry9523 and HimanshuChoudhary-Xilinx and removed request for mamin506, maxzhen, rozumx and AShivangi October 29, 2024 13:18

rbramand-xilinx changed the title ~~Enable new XRT test case flow without xclbin~~ Xclbin to ELF flow migration Oct 29, 2024

rbramand-xilinx force-pushed the xclbin_to_elf1 branch from ef27174 to 8539462 Compare October 29, 2024 15:46

stsoe requested changes Oct 29, 2024

View reviewed changes

AShivangi requested a review from hlaccabu October 29, 2024 17:19

hlaccabu approved these changes Oct 29, 2024

View reviewed changes

larry9523 reviewed Oct 29, 2024

View reviewed changes

src/runtime_src/core/common/api/xrt_module.cpp Outdated Show resolved Hide resolved

larry9523 reviewed Oct 29, 2024

View reviewed changes

src/runtime_src/core/include/ert.h Outdated Show resolved Hide resolved

rbramand-xilinx force-pushed the xclbin_to_elf1 branch from 8539462 to d199a7f Compare November 1, 2024 14:04

rbramand-xilinx requested a review from larry9523 November 1, 2024 14:05

rbramand-xilinx force-pushed the xclbin_to_elf1 branch 2 times, most recently from ca47bb5 to a899344 Compare November 4, 2024 12:12

rbramand-xilinx requested a review from stsoe November 4, 2024 12:26

rbramand-xilinx force-pushed the xclbin_to_elf1 branch from a899344 to 6e61f5f Compare November 6, 2024 14:45

stsoe requested changes Nov 8, 2024

View reviewed changes

rbramand-xilinx force-pushed the xclbin_to_elf1 branch 3 times, most recently from f3576d4 to fccc82e Compare November 12, 2024 10:07

rbramand-xilinx requested a review from stsoe November 12, 2024 10:09

stsoe reviewed Nov 13, 2024

View reviewed changes

src/runtime_src/core/common/api/xrt_hw_context.cpp Show resolved Hide resolved

src/runtime_src/core/common/api/xrt_kernel.cpp Show resolved Hide resolved

rbramand-xilinx force-pushed the xclbin_to_elf1 branch from 9cc746f to c2e4500 Compare November 13, 2024 05:25

stsoe approved these changes Nov 13, 2024

View reviewed changes

rbramand added 6 commits November 14, 2024 15:02

Enable flow without xclbin using new elf

b8cc343

Signed-off-by: rbramand <[email protected]>

Address comments on PR

2a41afb

Signed-off-by: rbramand <[email protected]>

Update copyright files

be132bb

Signed-off-by: rbramand <[email protected]>

Fix failure

bf564a4

Signed-off-by: rbramand <[email protected]>

Add missing comment fixes

94c6411

Signed-off-by: rbramand <[email protected]>

Remove unused exported functions

2c25976

Signed-off-by: rbramand <[email protected]>

rbramand-xilinx force-pushed the xclbin_to_elf1 branch from 30de0c6 to 46f9f30 Compare November 14, 2024 09:46

Fix merge conflicts

e86d2c8

Signed-off-by: rbramand <[email protected]>

rbramand-xilinx force-pushed the xclbin_to_elf1 branch from 46f9f30 to e86d2c8 Compare November 14, 2024 10:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Xclbin to ELF flow migration #8581

Xclbin to ELF flow migration #8581

rbramand-xilinx commented Oct 29, 2024 •

edited

Loading

stsoe left a comment

stsoe Oct 29, 2024

stsoe Nov 8, 2024

stsoe Oct 29, 2024

stsoe Nov 8, 2024

stsoe Oct 29, 2024

stsoe Nov 8, 2024

rbramand-xilinx Nov 10, 2024

AShivangi commented Oct 29, 2024

hlaccabu left a comment

rbramand-xilinx commented Oct 30, 2024

hlaccabu commented Oct 30, 2024

rbramand-xilinx commented Nov 4, 2024

larry9523 commented Nov 5, 2024

rbramand-xilinx commented Nov 6, 2024

stsoe left a comment

stsoe Nov 8, 2024

rbramand-xilinx Nov 11, 2024

stsoe Nov 8, 2024

stsoe Nov 8, 2024

stsoe Nov 8, 2024

stsoe left a comment

rbramand-xilinx commented Nov 13, 2024

Xclbin to ELF flow migration #8581

Are you sure you want to change the base?

Xclbin to ELF flow migration #8581

Conversation

rbramand-xilinx commented Oct 29, 2024 • edited Loading

Problem solved by the commit

Bug / issue (if any) fixed, which PR introduced the bug, how it was discovered

How problem was solved, alternative solutions (if any) and why they were rejected

Risks (if any) associated the changes in the commit

What has been tested and how, request additional testing if necessary

Documentation impact (if any)

stsoe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AShivangi commented Oct 29, 2024

hlaccabu left a comment

Choose a reason for hiding this comment

rbramand-xilinx commented Oct 30, 2024

hlaccabu commented Oct 30, 2024

rbramand-xilinx commented Nov 4, 2024

larry9523 commented Nov 5, 2024

rbramand-xilinx commented Nov 6, 2024

stsoe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsoe left a comment

Choose a reason for hiding this comment

rbramand-xilinx commented Nov 13, 2024

rbramand-xilinx commented Oct 29, 2024 •

edited

Loading