invalid output with gfx803 and BUILD_WITH_TENSILE_HOST ON

[xuhuisheng]

Finally, I can reproduce gfx803 with BUILD_WITH_TENSILE_HOST errors using a small sample.

OS	Ubuntu-20.04.1
linux	Linux 5.4.0-64-generic
ROCm	4.0.1
GPU	RX 580 8G
pytorch	1.7.1

test codes : https://github.com/xuhuisheng/rocm-build/blob/feature/check/check/test-pytorch-rocblas.py
In this code, there is one full connection layer Y = w0 * x0 + w1 * x1 + b

var	value
input num	2
output num	1
weight0	1
weight1	1
bias	10
features	800
batch	32

For simple case, I init weight with [1,1], bias with 10, so it should not change anything, loss always should be 0.
But on the 12th steps, loss changed to 50. It is reproducable on my computer.

I print the X, Y, output, loss, and find out that the first 16 of output is 12 which is correct. the last 12 of output is 2 which is wrong. And seems the last 16 of output miss the bias, I guess.

     X tensor([[1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.]], device='cuda:0')
     Y tensor([1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01,
        1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01,
        1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01,
        1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01,
        1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01, 1.2000e+01,
        1.2000e+01, 1.2000e+01], device='cuda:0')
weight tensor([[1., 1.]], device='cuda:0')
  bias tensor([10.], device='cuda:0')
output tensor([[1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [1.2000e+01],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00],
        [2.0000e+00]], device='cuda:0', grad_fn=<AddmmBackward>)
     l tensor(50., device='cuda:0', grad_fn=<MseLossBackward>)

If I used BUILD_WITH_TENSILE_HOST=OFF and re-compile rocBLAS, the error will gone.

And It is weired if I comment out l.backward(), the error will change to happen on 21th step, and output change to all 3.
Why it is unstable? Maybe gfx803 need PCI3 Atomics operation? or gfx803 just miss some waitcnt() instructions?

     X tensor([[1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.],
        [1., 1.]], device='cuda:0')
     Y tensor([12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000,
        12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000,
        12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000,
        12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000, 12.0000],
       device='cuda:0')
weight tensor([[1., 1.]], device='cuda:0')
  bias tensor([10.], device='cuda:0')
output tensor([[3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.],
        [3.]], device='cuda:0', grad_fn=<AddmmBackward>)
     l tensor(81., device='cuda:0', grad_fn=<MseLossBackward>)

So I believe it caused by the new Tensile client which used assembly codes to optimized. I am not familiar with GCN assemly, So I create an issue to tensile wish someone could help.
ROCm/Tensile#1287

[xuhuisheng]

Thank @https://github.com/cgmb to show how to use rocblas-test to verify the issue of rocblas on gfx803.
ROCm/rocBLAS#1218.
Keep investigating.