Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

omnitrace-avail fails on ROCM 5.3 and RX 6800XT #281

Closed
vchuravy opened this issue May 21, 2023 · 2 comments
Closed

omnitrace-avail fails on ROCM 5.3 and RX 6800XT #281

vchuravy opened this issue May 21, 2023 · 2 comments
Labels
bug Something isn't working omnitrace-avail Involves the omnitrace-avail executable (info tool) rocprofiler GPU kernel HW counters wontfix This will not be worked on

Comments

@vchuravy
Copy link

Using the following docker container:

FROM rocm/rocm-terminal:5.3

RUN sudo apt-get update && \
    sudo apt-get dist-upgrade -y && \
    sudo apt-get install -y wget vim

RUN wget https://github.com/AMDResearch/omnitrace/releases/download/v1.10.0/omnitrace_1.10.0-ubuntu20.04.ROCm50300.PAPI.OMPT.Python3_amd64.deb && \
    sudo dpkg -i omnitrace_1.10.0-ubuntu20.04.ROCm50300.PAPI.OMPT.Python3_amd64.deb && \
    rm *.deb

Running with: docker run -it --device=/dev/kfd --device=/dev/dri --group-add video

*******                  
Agent 2                  
*******                  
  Name:                    gfx1030                            
  Uuid:                    GPU-849e7f8e4045a584               
  Marketing Name:          AMD Radeon RX 6800 XT              
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE   

rocm-user@0010d1870e71:~$ /opt/omnitrace/bin/omnitrace-avail                    
[omnitrace][184][0][fatal] 
[omnitrace][184][0][fatal] ERROR :: rocprofiler_iterate_info(), ImportMetrics(), Bad metric 'L2CacheHit', var 'TCC_HIT[0]' is not found


[omnitrace][184][0] Signal 6 caught : Aborted (Signal sent by tkill() 184 1000)

### ERROR ### [omnitrace][PID=184][TID=0] signal=6 (SIGABRT) abort program (formerly SIGIOT). code: -6
Backtrace:
[PID=184][TID=0][0/9] __restore_rt
[PID=184][TID=0][1/9] gsignal +0xcb
[PID=184][TID=0][2/9] abort +0x12b
[PID=184][TID=0][3/9] _ZN9omnitrace11rocprofiler5fatalERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE.cold +0x5
[PID=184][TID=0][4/9] _ZN9omnitrace11rocprofiler17rocm_check_statusE12hsa_status_tRKSt3setIS1_St4lessIS1_ESaIS1_EE +0xa0
[PID=184][TID=0][5/9] _ZN9omnitrace11rocprofiler12rocm_metricsEv +0x156
[PID=184][TID=0][6/9] main +0x3d47
[PID=184][TID=0][7/9] __libc_start_main +0xf3
[PID=184][TID=0][8/9] _start +0x2e

Backtrace (demangled):
[PID=184][TID=0][0/9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f02b03f1420]
[PID=184][TID=0][1/9] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f02ae30200b]
[PID=184][TID=0][2/9] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f02ae2e1859]
[PID=184][TID=0][3/9] /opt/omnitrace/bin/omnitrace-avail(+0x129aab) [0x55b1fe3d5aab]
[PID=184][TID=0][4/9] /opt/omnitrace/bin/omnitrace-avail(+0x6f99b0) [0x55b1fe9a59b0]
[PID=184][TID=0][5/9] /opt/omnitrace/bin/omnitrace-avail(+0x6fac16) [0x55b1fe9a6c16]
[PID=184][TID=0][6/9] /opt/omnitrace/bin/omnitrace-avail(+0x4b87b7) [0x55b1fe7647b7]
[PID=184][TID=0][7/9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f02ae2e3083]
[PID=184][TID=0][8/9] /opt/omnitrace/bin/omnitrace-avail(+0x4f3b8e) [0x55b1fe79fb8e]

/proc/184/maps:
    55b1fe2ac000-55b1fe3cb000 r--p 00000000 00:18 40421                      /opt/omnitrace/bin/omnitrace-avail
    55b1fe3cb000-55b20030f000 r-xp 0011f000 00:18 40421                      /opt/omnitrace/bin/omnitrace-avail
    55b20030f000-55b20086a000 r--p 02063000 00:18 40421                      /opt/omnitrace/bin/omnitrace-avail
    55b20086b000-55b2008b2000 r--p 025be000 00:18 40421                      /opt/omnitrace/bin/omnitrace-avail
    55b2008b2000-55b20092b000 rw-p 02605000 00:18 40421                      /opt/omnitrace/bin/omnitrace-avail
    55b20092b000-55b200f5d000 rw-p 00000000 00:00 0 
    55b201875000-55b20250a000 rw-p 00000000 00:00 0                          [heap]
    7f0197a00000-7f0198241000 r--p 00000000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f0198241000-7f019d738000 r-xp 00841000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f019d738000-7f019f908000 r--p 05d38000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f019f908000-7f019f909000 ---p 07f08000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f019f909000-7f019fe4c000 r--p 07f08000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f019fe4c000-7f019fe9c000 rw-p 0844b000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f019fe9c000-7f019fefc000 rw-p 00000000 00:00 0 
    7f01a0000000-7f01a0021000 rw-p 00000000 00:00 0 
    7f01a0021000-7f01a4000000 ---p 00000000 00:00 0 
    7f01a5800000-7f02a5800000 ---p 00000000 00:00 0 
    7f02a6cae000-7f02a8000000 rw-p 00000000 00:00 0 
    7f02a8000000-7f02a8021000 rw-p 00000000 00:00 0 
    7f02a8021000-7f02ac000000 ---p 00000000 00:00 0 
    7f02ac59c000-7f02acb76000 rw-p 00000000 00:00 0 
    7f02acb76000-7f02acc07000 rw-p 00000000 00:00 0 
    7f02acc67000-7f02acca7000 rw-p 00000000 00:00 0 
    7f02acca7000-7f02accb1000 r--p 00000000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02accb1000-7f02acce4000 r-xp 0000a000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02acce4000-7f02accef000 r--p 0003d000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02accef000-7f02accf0000 ---p 00048000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02accf0000-7f02accf3000 r--p 00048000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02accf3000-7f02acd00000 rw-p 0004b000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02acd00000-7f02ace01000 rw-p 00000000 00:00 0 
    7f02ace09000-7f02ace3a000 rw-p 00000000 00:00 0 
    7f02ace6b000-7f02acf00000 rw-p 00000000 00:00 0 
    7f02acf00000-7f02ad001000 rw-p 00000000 00:00 0 
    7f02ad015000-7f02ad05b000 rw-p 00000000 00:00 0 
    7f02ad05b000-7f02ad069000 r--p 00000000 00:18 1307                       /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
    7f02ad069000-7f02ad078000 r-xp 0000e000 00:18 1307                       /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
    7f02ad078000-7f02ad086000 r--p 0001d000 00:18 1307                       /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
    7f02ad086000-7f02ad08a000 r--p 0002a000 00:18 1307                       /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
    7f02ad08a000-7f02ad08b000 rw-p 0002e000 00:18 1307                       /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
    7f02ad08b000-7f02ad08c000 ---p 00000000 00:00 0 
    7f02ad08c000-7f02ad88c000 rwxp 00000000 00:00 0 
    7f02ad88c000-7f02ad88d000 ---p 00000000 00:00 0 
    7f02ad88d000-7f02ae08d000 rwxp 00000000 00:00 0 
    7f02ae08d000-7f02ae0dd000 rw-p 00000000 00:00 0 
    7f02ae0dd000-7f02ae173000 r--p 00000000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae173000-7f02ae264000 r-xp 00096000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae264000-7f02ae2ad000 r--p 00187000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae2ad000-7f02ae2ae000 ---p 001d0000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae2ae000-7f02ae2b9000 r--p 001d0000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae2b9000-7f02ae2bc000 rw-p 001db000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae2bc000-7f02ae2bf000 rw-p 00000000 00:00 0 
    7f02ae2bf000-7f02ae2e1000 r--p 00000000 00:18 1187                       /usr/lib/x86_64-linux-gnu/libc-2.31.so
    7f02ae2e1000-7f02ae459000 r-xp 00022000 00:18 1187                       /usr/lib/x86_64-linux-gnu/libc-2.31.so
    7f02ae459000-7f02ae4a7000 r--p 0019a000 00:18 1187                       /usr/lib/x86_64-linux-gnu/libc-2.31.so
    7f02ae4a7000-7f02ae4ab000 r--p 001e7000 00:18 1187                       /usr/lib/x86_64-linux-gnu/libc-2.31.so
    7f02ae4ab000-7f02ae4ad000 rw-p 001eb000 00:18 1187                       /usr/lib/x86_64-linux-gnu/libc-2.31.so
    7f02ae4ad000-7f02ae4b1000 rw-p 00000000 00:00 0 
    7f02ae4b1000-7f02ae4be000 r--p 00000000 00:18 1229                       /usr/lib/x86_64-linux-gnu/libm-2.31.so
    7f02ae4be000-7f02ae565000 r-xp 0000d000 00:18 1229                       /usr/lib/x86_64-linux-gnu/libm-2.31.so
    7f02ae565000-7f02ae5fe000 r--p 000b4000 00:18 1229                       /usr/lib/x86_64-linux-gnu/libm-2.31.so
    7f02ae5fe000-7f02ae5ff000 r--p 0014c000 00:18 1229                       /usr/lib/x86_64-linux-gnu/libm-2.31.so
    7f02ae5ff000-7f02ae600000 rw-p 0014d000 00:18 1229                       /usr/lib/x86_64-linux-gnu/libm-2.31.so
    7f02ae600000-7f02ae61e000 r--p 00000000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02ae61e000-7f02ae99e000 r-xp 0001e000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02ae99e000-7f02aea8e000 r--p 0039e000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02aea8e000-7f02aea8f000 ---p 0048e000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02aea8f000-7f02aea95000 r--p 0048e000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02aea95000-7f02affc9000 rw-p 00494000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02affc9000-7f02affda000 rw-p 00000000 00:00 0 
    7f02affe6000-7f02affe7000 rw-p 00000000 00:00 0 
    7f02affe8000-7f02afff0000 rw-s 107ae6000 00:b8 15                        /dev/dri/renderD128
    7f02afff2000-7f02afff4000 rw-p 00000000 00:00 0 
    7f02afff5000-7f02afff6000 rw-p 00000000 00:00 0 
    7f02afff7000-7f02b0000000 rw-p 00000000 00:00 0 
    7f02b0000000-7f02b00ef000 r--p 00000000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b00ef000-7f02b0100000 r-xp 000ef000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b0100000-7f02b0193000 r--p 00100000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b0193000-7f02b0194000 ---p 00193000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b0194000-7f02b023c000 r--p 00193000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b023c000-7f02b029d000 rw-p 0023b000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b029d000-7f02b029f000 rw-p 00000000 00:00 0 
    7f02b029f000-7f02b02a4000 rw-p 00000000 00:00 0 
    7f02b02a4000-7f02b02a7000 r--p 00000000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02a7000-7f02b02ac000 r-xp 00003000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02ac000-7f02b02ae000 r--p 00008000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02ae000-7f02b02af000 ---p 0000a000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02af000-7f02b02b0000 r--p 0000a000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02b0000-7f02b02b1000 rw-p 0000b000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02b1000-7f02b02b4000 r--p 00000000 00:18 3907                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
    7f02b02b4000-7f02b02bb000 r-xp 00003000 00:18 3907                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
    7f02b02bb000-7f02b02bd000 r--p 0000a000 00:18 3907                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
    7f02b02bd000-7f02b02be000 r--p 0000b000 00:18 3907                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
    7f02b02be000-7f02b02bf000 rw-p 0000c000 00:18 3907                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
    7f02b02bf000-7f02b02c4000 r--p 00000000 00:18 3905                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
    7f02b02c4000-7f02b02d1000 r-xp 00005000 00:18 3905                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
    7f02b02d1000-7f02b02d5000 r--p 00012000 00:18 3905                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
    7f02b02d5000-7f02b02d6000 r--p 00015000 00:18 3905                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
    7f02b02d6000-7f02b02d7000 rw-p 00016000 00:18 3905                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
    7f02b02d7000-7f02b02da000 r--p 00000000 00:18 18872                      /usr/lib/x86_64-linux-gnu/libelf-0.176.so
    7f02b02da000-7f02b02ec000 r-xp 00003000 00:18 18872                      /usr/lib/x86_64-linux-gnu/libelf-0.176.so
    7f02b02ec000-7f02b02f1000 r--p 00015000 00:18 18872                      /usr/lib/x86_64-linux-gnu/libelf-0.176.so
    7f02b02f1000-7f02b02f2000 r--p 00019000 00:18 18872                      /usr/lib/x86_64-linux-gnu/libelf-0.176.so
    7f02b02f2000-7f02b02f3000 rw-p 0001a000 00:18 18872                      /usr/lib/x86_64-linux-gnu/libelf-0.176.so
    7f02b02f3000-7f02b02f5000 rw-p 00000000 00:00 0 
    7f02b02f5000-7f02b02f8000 r--p 00000000 00:18 1212                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
    7f02b02f8000-7f02b030a000 r-xp 00003000 00:18 1212                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
    7f02b030a000-7f02b030e000 r--p 00015000 00:18 1212                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
    7f02b030e000-7f02b030f000 r--p 00018000 00:18 1212                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
    7f02b030f000-7f02b0310000 rw-p 00019000 00:18 1212                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
    7f02b0310000-7f02b032d000 r--p 00000000 00:18 4560                       /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
    7f02b032d000-7f02b038d000 r-xp 0001d000 00:18 4560                       /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
    7f02b038d000-7f02b03a2000 r--p 0007d000 00:18 4560                       /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
    7f02b03a2000-7f02b03a4000 r--p 00091000 00:18 4560                       /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
    7f02b03a4000-7f02b03a5000 rw-p 00093000 00:18 4560                       /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
    7f02b03a5000-7f02b03a7000 r--p 00000000 00:18 40701                      /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
    7f02b03a7000-7f02b03b2000 r-xp 00002000 00:18 40701                      /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
    7f02b03b2000-7f02b03b5000 r--p 0000d000 00:18 40701                      /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
    7f02b03b5000-7f02b03b6000 r--p 0000f000 00:18 40701                      /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
    7f02b03b6000-7f02b03b7000 rw-p 00010000 00:18 40701                      /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
    7f02b03b7000-7f02b03c1000 rw-p 00000000 00:00 0 
    7f02b03c1000-7f02b03c3000 r--p 00000000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03c3000-7f02b03d4000 r-xp 00002000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03d4000-7f02b03da000 r--p 00013000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03da000-7f02b03db000 ---p 00019000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03db000-7f02b03dc000 r--p 00019000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03dc000-7f02b03dd000 rw-p 0001a000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03dd000-7f02b03e3000 r--p 00000000 00:18 1281                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    7f02b03e3000-7f02b03f4000 r-xp 00006000 00:18 1281                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    7f02b03f4000-7f02b03fa000 r--p 00017000 00:18 1281                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    7f02b03fa000-7f02b03fb000 r--p 0001c000 00:18 1281                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    7f02b03fb000-7f02b03fc000 rw-p 0001d000 00:18 1281                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    7f02b03fc000-7f02b0400000 rw-p 00000000 00:00 0 
    7f02b0400000-7f02b0418000 r--p 00000000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b0418000-7f02b04ef000 r-xp 00018000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b04ef000-7f02b0558000 r--p 000ef000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b0558000-7f02b0559000 ---p 00158000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b0559000-7f02b0561000 r--p 00158000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b0561000-7f02b069c000 rw-p 00160000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b069c000-7f02b06a1000 rw-p 00000000 00:00 0 
    7f02b06a1000-7f02b06a2000 r--p 00000000 00:18 40657                      /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
    7f02b06a2000-7f02b06a8000 r-xp 00001000 00:18 40657                      /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
    7f02b06a8000-7f02b06ab000 r--p 00007000 00:18 40657                      /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
    7f02b06ab000-7f02b06ac000 r--p 00009000 00:18 40657                      /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
    7f02b06ac000-7f02b06ad000 rw-p 0000a000 00:18 40657                      /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
    7f02b06ad000-7f02b06af000 rw-p 00000000 00:00 0 
    7f02b06af000-7f02b06b1000 r--p 00000000 00:18 1285                       /usr/lib/x86_64-linux-gnu/librt-2.31.so
    7f02b06b1000-7f02b06b5000 r-xp 00002000 00:18 1285                       /usr/lib/x86_64-linux-gnu/librt-2.31.so
    7f02b06b5000-7f02b06b7000 r--p 00006000 00:18 1285                       /usr/lib/x86_64-linux-gnu/librt-2.31.so
    7f02b06b7000-7f02b06b8000 r--p 00007000 00:18 1285                       /usr/lib/x86_64-linux-gnu/librt-2.31.so
    7f02b06b8000-7f02b06b9000 rw-p 00008000 00:18 1285                       /usr/lib/x86_64-linux-gnu/librt-2.31.so
    7f02b06b9000-7f02b06ba000 r--p 00000000 00:18 1198                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so
    7f02b06ba000-7f02b06bc000 r-xp 00001000 00:18 1198                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so
    7f02b06bc000-7f02b06bd000 r--p 00003000 00:18 1198                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so
    7f02b06bd000-7f02b06be000 r--p 00003000 00:18 1198                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so
    7f02b06be000-7f02b06bf000 rw-p 00004000 00:18 1198                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so
    7f02b06bf000-7f02b06c6000 r--p 00000000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b06c6000-7f02b0700000 r-xp 00007000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b0700000-7f02b070d000 r--p 00041000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b070d000-7f02b070e000 ---p 0004e000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b070e000-7f02b070f000 r--p 0004e000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b070f000-7f02b0710000 rw-p 0004f000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b0710000-7f02b0715000 r--p 00000000 00:18 4566                       /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
    7f02b0715000-7f02b0742000 r-xp 00005000 00:18 4566                       /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
    7f02b0742000-7f02b0750000 r--p 00032000 00:18 4566                       /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
    7f02b0750000-7f02b0751000 r--p 0003f000 00:18 4566                       /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
    7f02b0751000-7f02b0752000 rw-p 00040000 00:18 4566                       /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
    7f02b0752000-7f02b0756000 rw-p 00000000 00:00 0 
    7f02b0756000-7f02b0757000 rw-s 00000000 00:bc 4                          /dev/shm/rocm_smi_card0
    7f02b0757000-7f02b0758000 rw-p 00000000 00:00 0 
    7f02b0758000-7f02b0759000 rw-p 00000000 00:00 0 
    7f02b0759000-7f02b075a000 rw-s 3e64800000000000 00:b8 11                 /dev/kfd
    7f02b075a000-7f02b075b000 rw-p 00000000 00:00 0 
    7f02b075b000-7f02b075c000 rw-s 00000000 00:bc 3                          /dev/shm/hsakmt_shared_mem
    7f02b075c000-7f02b075d000 r--p 00000000 00:18 1165                       /usr/lib/x86_64-linux-gnu/ld-2.31.so
    7f02b075d000-7f02b0780000 r-xp 00001000 00:18 1165                       /usr/lib/x86_64-linux-gnu/ld-2.31.so
    7f02b0780000-7f02b0788000 r--p 00024000 00:18 1165                       /usr/lib/x86_64-linux-gnu/ld-2.31.so
    7f02b0788000-7f02b0789000 rw-s 00000000 00:bc 2                          /dev/shm/sem.hsakmt_semaphore
    7f02b0789000-7f02b078a000 r--p 0002c000 00:18 1165                       /usr/lib/x86_64-linux-gnu/ld-2.31.so
    7f02b078a000-7f02b078b000 rw-p 0002d000 00:18 1165                       /usr/lib/x86_64-linux-gnu/ld-2.31.so
    7f02b078b000-7f02b078c000 rw-p 00000000 00:00 0 
    7ffe321ac000-7ffe321ee000 rwxp 00000000 00:00 0                          [stack]
    7ffe321ee000-7ffe321f0000 rw-p 00000000 00:00 0 
    7ffe321f3000-7ffe321f7000 r--p 00000000 00:00 0                          [vvar]
    7ffe321f7000-7ffe321f9000 r-xp 00000000 00:00 0                          [vdso]
    ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

Backtrace (demangled):
[PID=184][TID=0][0/9] __restore_rt
[PID=184][TID=0][1/9] gsignal +0xcb
[PID=184][TID=0][2/9] abort +0x12b
[PID=184][TID=0][3/9] omnitrace::rocprofiler::fatal(std::string const&) [clone .cold] +0x5
[PID=184][TID=0][4/9] omnitrace::rocprofiler::rocm_check_status(hsa_status_t, std::set<hsa_status_t, std::less<hsa_status_t>, std::allocator<hsa_status_t>> const&) +0xa0
[PID=184][TID=0][5/9] omnitrace::rocprofiler::rocm_metrics() +0x156
[PID=184][TID=0][6/9] main +0x3d47
[PID=184][TID=0][7/9] __libc_start_main +0xf3
[PID=184][TID=0][8/9] _start +0x2e

Backtrace (lineinfo):
[PID=184][TID=0][0/7]
    [/lib/x86_64-linux-gnu/libpthread.so.0:?] __restore_rt
[PID=184][TID=0][1/7]
    [/usr/lib/x86_64-linux-gnu/libc-2.31.so:?] abort
[PID=184][TID=0][2/7]
    [/home/omnitrace/source/lib/omnitrace/library/rocprofiler.cpp:75] omnitrace::rocprofiler::fatal(std::string const&)
[PID=184][TID=0][3/7]
    [/home/omnitrace/source/lib/omnitrace/library/rocprofiler.cpp:100] omnitrace::rocprofiler::rocm_check_status(hsa_status_t, std::set<hsa_status_t, std::less<hsa_status_t>, std::allocator<hsa_status_t>> const&)
[PID=184][TID=0][4/7]
    [/home/omnitrace/source/lib/omnitrace/library/rocprofiler.cpp:370] omnitrace::rocprofiler::rocm_metrics()
    [/usr/include/c++/9/bits/stl_set.h:281] std::set<hsa_status_t, std::less<hsa_status_t>, std::allocator<hsa_status_t>>::~set()
    [/usr/include/c++/9/bits/stl_tree.h:1000] std::_Rb_tree<hsa_status_t, hsa_status_t, std::_Identity<hsa_status_t>, std::less<hsa_status_t>, std::allocator<hsa_status_t>>::~_Rb_tree()
[PID=184][TID=0][5/7]
    [/home/omnitrace/source/bin/omnitrace-avail/avail.cpp:513] main
    [/usr/include/c++/9/bits/stl_vector.h:916] std::vector<tim::hardware_counters::info, std::allocator<tim::hardware_counters::info>>::size() const
[PID=184][TID=0][6/7]
    [/usr/lib/x86_64-linux-gnu/libc-2.31.so:?] __libc_start_main

[omnitrace][184] Finalizing afer signal 6 ::  Signal:    SIGABRT (signal number:   6)          abort program (formerly SIGIOT)

[omnitrace][184] Killing process 184 with signal 6...
@vchuravy
Copy link
Author

This works when switching to ROCM 5.4

FROM rocm/rocm-terminal:5.4

RUN sudo apt-get update && \
    sudo apt-get dist-upgrade -y && \
    sudo apt-get install -y wget vim

RUN wget https://github.com/AMDResearch/omnitrace/releases/download/v1.10.0/omnitrace_1.10.0-ubuntu20.04.ROCm50400.PAPI.OMPT.Python3_amd64.deb && \
    sudo dpkg -i omnitrace_1.10.0-ubuntu20.04.ROCm50400.PAPI.OMPT.Python3_amd64.deb && \
    rm *.deb

@jrmadsen
Copy link
Collaborator

FYI, this is probably a rocprofiler bug, not an omnitrace bug. There have been a lot of issues with its stability between minor releases recently.

@jrmadsen jrmadsen added bug Something isn't working wontfix This will not be worked on omnitrace-avail Involves the omnitrace-avail executable (info tool) rocprofiler GPU kernel HW counters labels Jun 22, 2023
@ppanchad-amd ppanchad-amd closed this as not planned Won't fix, can't repro, duplicate, stale Oct 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working omnitrace-avail Involves the omnitrace-avail executable (info tool) rocprofiler GPU kernel HW counters wontfix This will not be worked on
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vchuravy @jrmadsen @ppanchad-amd