Disabled building tests default (like LLVM), enabled for our CIs

[AlexandreEichenberger]

As mentioned in #1059, our default builds require onnx to be installed.

This PR implements the changes requested in that issue:

1. all extensive tests are predicated by ONNX_MLIR_BUILD_TESTS being ON
2. default ONNX_MLIR_BUILD_TESTS value switched to off.
3. Our CIs add the ONNX_MLIR_BUILD_TESTS=ON define.

That way, folks just building a default onnx-mlir will not build the tests, and thus we are not dependent on onnx third party to be installed.

For folks using their own scripts for testing, you must now add

        -DONNX_MLIR_BUILD_TESTS=ON \

to your cmake command.

To respond to issue #1059, I also pip uninstal onnx and tested that we can build a working compiler without onnx on my local Mac machine, and it worked.

[AlexandreEichenberger]

From the log from linux/amd

[...]
[100%] Built target numerical
[100%] Running the ONNX-MLIR numerical regression tests
Test project /workdir/onnx-mlir/build/test/numerical
      Start  1: TestConv
      Start  2: TestMatMul2D
      Start  3: TestMatMulBroadcast
      Start  4: TestGemm
      Start  5: TestLSTM
      Start  6: TestRNN
      Start  7: TestGRU
      Start  8: TestLoop
      Start  9: TestScan
      Start 10: TestElementwise
      Start 11: TestSoftplus


[...]

[100%] Built target check-onnx-backend
bringing up nodes...
bringing up nodes...

530 passed, 2348 skipped in 103.80s (0:01:43)

So it appears to build & test the tests in the CI for Linux/AMD

I will double check that it worked too for s390 once the CI completes.

[tungld]

@AlexandreEichenberger I am curious to know if without building onnx, can you compile a .onnx model or not? Numerical tests do not use .onnx files as input, so it does not tell whether we can compile a .onnx model. Just to double check.

[gongsu832]

I don't think this does anything. EXCLUDE_FROM_ALL is a target property. Setting a variable called EXCLUDE_FROM_ALL won't do anything. The proper use of EXCLUDE_FROM_ALL should be like the code in MLIR.cmake:

  if (EXCLUDE_FROM_ALL)
    add_executable(${name} EXCLUDE_FROM_ALL ${ARG_UNPARSED_ARGUMENTS})
  else()
    add_executable(${name} ${ARG_UNPARSED_ARGUMENTS})
  endif()

The variable EXCLUDE_FROM_ALL is used to decide whether to add the property EXCLUDE_FROM_ALL to a target.

[AlexandreEichenberger]

@gongsu832 I reused this pattern from the current onnx-mlir. I know it does not make it right, but on my Mac it just seems to work. When I set the variable as ON, it builds all the tests as before. When I set it to off, the resulting build does not recognize the make check-onyx-backend as a valid target.

Let me run the experiment on LoZ too.

[gongsu832]

I assume you meant check-onnx-backend since there is no check-onyx-backend.

Regardless of how the variable is set, check-onnx-backend should always be a valid target. The difference is whether it's included in the default all target.

[gongsu832]

Also note that add_executable, add_library, etc. behave differently from add_custom_target. By default, add_executable, add_library, etc. will add the target to all unless you specify EXCLUDE_FROM_ALL on the target. add_custom_target, on the other hand, by default will not add the target to all unless you specify ALL on the target.

[AlexandreEichenberger]

@gongsu832 Are there specific steps you would like me to take? Or if it takes you less time, do you want to take over this PR to do it the right way (if the current approach is not the preferred approach).

[gongsu832]

I can do it but probably won't be able to get to it until some time next week.

[AlexandreEichenberger]

Thanks, there is no rush, glad it will be done using the proper approach

[AlexandreEichenberger]

I am curious to know if without building onnx, can you compile a .onnx model or not? Numerical tests do not use .onnx files as input, so it does not tell whether we can compile a .onnx model. Just to double check.

@tungld Here is the experiment I ran. I executed a pip3 uninstall onnx until it said there were none; then I ran with the -DONNX_MLIR_BUILD_TESTS=OFF, build an onnx-mlir and tried the compiler on an mlir and onnx input, both of which worked. When I built it with ON, the make process would fail due to the lack of installed onnx.

That was on a Mac, let me retry all this on a LoZ

[tungld]

I ran with the -DONNX_MLIR_BUILD_TESTS=OFF, build an onnx-mlir and tried the compiler on an mlir and onnx input, both of which worked

Thank you for the confirmation! That sounds good. Let @gongsu832 give the final decision since the most changes are about CMakeFiles.