6 tests fail

[yurivict]

These tests fail:

          1 - serial_tests (Failed)
          2 - test-backtrace (Failed)
         10 - test-command-args (Failed)
         12 - test-coordinates-initialization (BAD_COMMAND)
         36 - test-squash (Failed)
         37 - test-stopCheck (BAD_COMMAND)

Here is the LastTest.log file.

Version: 5.1.1
clang-18
Python-3.11
FreeBSD 14.1

[ZedThree]

Please could you post the errors?

[yurivict]

All errors are in the linked log file, see above.

[ZedThree]

I have an idea about some of these.

serial_tests

Several failed tests here. First one is a death test:

/usr/ports/science/bout++/work/BOUT-dev-5.1.1/tests/unit/src/test_bout++.cxx:77: Failure
Death test: std::raise(11)
    Result: threw an exception.
 Error msg:
[  DEATH   ] 
[  DEATH   ] Sighandler called on process 0 with sig 11
[  DEATH   ] 
[  DEATH   ] /usr/ports/science/bout++/work/BOUT-dev-5.1.1/tests/unit/src/test_bout++.cxx:77:: Caught std::exception-derived exception escaping the death test statement. Exception message: 
[  DEATH   ] ****** SEGMENTATION FAULT CAUGHT ******
[  DEATH   ] 
[  DEATH   ] 
[  DEATH   ] 

[  FAILED  ] SignalHandlerTestDeathTest.SegFault (64 ms)

These are not super robust on clang, so might be sensible to turn off BOUT_ENABLE_SIGNAL. Would be nice if we could detect this and turn it off automatically.

Timer test:

Timer::resetTime
Timer name         | Total time (s) | % of top | Hits | Mean time/hit (s)
-------------------|----------------|----------|------|------------------
one                | 0.003241364    | 1.00     | 1    | 0.003241364      
two                | 0.002160308    | 0.67     | 2    | 0.001080154      
18 characters long | 9.6e-08        | 0.00     | 1    | 9.6e-08          
/usr/ports/science/bout++/work/BOUT-dev-5.1.1/externalpackages/googletest/googletest/src/gtest-port.cc:745: Failure
Value of: is_valid_
  Actual: false
Expected: true
Regular expression "one *| 0\.\d+ | 1    | 0\.\d+" is not a valid POSIX Extended regular expression.

/usr/ports/science/bout++/work/BOUT-dev-5.1.1/tests/unit/sys/test_timer.cxx:274: Failure
Value of: cout_capture.str()
Expected: contains regular expression "one *| 0\\.\\d+ | 1    | 0\\.\\d+"
  Actual: "Timer name         | Total time (s) | % of top | Hits | Mean time/hit (s)\n-------------------|----------------|----------|------|------------------\none                | 0.003241364    | 1.00     | 1    | 0.003241364      \ntwo                | 0.002160308    | 0.67     | 2    | 0.001080154      \n18 characters long | 9.6e-08        | 0.00     | 1    | 9.6e-08          \n"

/usr/ports/science/bout++/work/BOUT-dev-5.1.1/externalpackages/googletest/googletest/src/gtest-port.cc:745: Failure
Value of: is_valid_
  Actual: false
Expected: true
Regular expression "two *| 0 * | 2    | 0\.\d+" is not a valid POSIX Extended regular expression.

/usr/ports/science/bout++/work/BOUT-dev-5.1.1/tests/unit/sys/test_timer.cxx:275: Failure
Value of: cout_capture.str()
Expected: contains regular expression "two *| 0 * | 2    | 0\\.\\d+"
  Actual: "Timer name         | Total time (s) | % of top | Hits | Mean time/hit (s)\n-------------------|----------------|----------|------|------------------\none                | 0.003241364    | 1.00     | 1    | 0.003241364      \ntwo                | 0.002160308    | 0.67     | 2    | 0.001080154      \n18 characters long | 9.6e-08        | 0.00     | 1    | 9.6e-08          \n"

Looks like it doesn't like the regex we're using. I guess it's \d that it doesn't like, and this needs replacing with [0-9] instead for FreeBSD.

test-backtrace:

2/50 Test: test-backtrace
Command: "/usr/ports/science/bout++/work/.build/tests/integrated/test-backtrace/runtest"
Directory: /usr/ports/science/bout++/work/.build/tests/integrated/test-backtrace
"test-backtrace" start time: Sep 19 11:55 PDT
Output:
----------------------------------------------------------
Making backtrace environment variable test
Fail: did not detect offending function name in output when expected
=> Some failed tests
<end of output>
Test time =   0.57 sec
----------------------------------------------------------
Test Failed.
"test-backtrace" end time: Sep 19 11:55 PDT
"test-backtrace" time elapsed: 00:00:00
----------------------------------------------------------

Might be due to name mangling or the backtrace library giving us different output than on Linux? It would be helpful to print the full output in failure here.

test-command-args

======================================================================
FAIL: testDirectoryArgumentNonDirectory (__main__.TestCommandLineArgs.testDirectoryArgumentNonDirectory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/science/bout++/work/.build/tests/integrated/test-command-args/runtest", line 113, in testDirectoryArgumentNonDirectory
    self.assertIn(
AssertionError: '"runtest" is not a directory' not found in '' : FAIL: Error message not printed when missing input directory

======================================================================
FAIL: testDirectoryArgumentNonExistentDirectory (__main__.TestCommandLineArgs.testDirectoryArgumentNonExistentDirectory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/science/bout++/work/.build/tests/integrated/test-command-args/runtest", line 102, in testDirectoryArgumentNonExistentDirectory
    self.assertIn(
AssertionError: '"non_existent" does not exist' not found in '' : FAIL: Error message not printed when missing input directory

======================================================================
FAIL: testNoArgumentsNoDirectory (__main__.TestCommandLineArgs.testNoArgumentsNoDirectory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/science/bout++/work/.build/tests/integrated/test-command-args/runtest", line 34, in testNoArgumentsNoDirectory
    self.assertIn(
AssertionError: '"data" does not exist' not found in '' : FAIL: Error message not printed when missing input directory

----------------------------------------------------------------------
Ran 11 tests in 2.573s

Looks like FreeBSD is giving us empty error messages for some filesystem operations that we're expecting to fail. We're using POSIX struct stat when we could switch to std::filesystem and be more portable

test-squash

36/50 Test: test-squash
Command: "/usr/ports/science/bout++/work/.build/tests/integrated/test-squash/runtest" "/usr/ports/science/bout++/work/BOUT-dev-5.1.1/tests/integrated/test-squash/../../../bin"
Directory: /usr/ports/science/bout++/work/.build/tests/integrated/test-squash
"test-squash" start time: Sep 19 11:55 PDT
Output:
----------------------------------------------------------
/bin/sh: bout-squashoutput: not found
Making Squash test
Run once to get normal data
  0.62480021s ./squash -q -q -q nout=2
  0.01217532s mv data/BOUT.dmp.0.nc f1.nc
Parallel test
  0.01162004s rm -f f2.nc
getmpirun: using the default mpirun -np
  0.66829133s ./squash -q -q -q nout=2
  0.01016903s bout-squashoutput -qdcl 9 data --outputname ../f2.nc
Traceback (most recent call last):
  File "/usr/ports/science/bout++/work/.build/tests/integrated/test-squash/runtest", line 103, in <module>
    timed_shell_safe("{} -qdcl 9 data --outputname ../f2.nc".format(bout_squashoutput))
  File "/usr/ports/science/bout++/work/.build/tests/integrated/test-squash/runtest", line 39, in timed_shell_safe
    shell_safe(cmd, *args, **kwargs)
  File "/usr/ports/science/bout++/work/BOUT-dev-5.1.1/tools/pylib/boututils/run_wrapper.py", line 268, in shell_safe
    raise RuntimeError(
RuntimeError: Run failed with 127.
Command was:
bout-squashoutput -qdcl 9 data --outputname ../f2.nc

bout-squashoutput: not found is the issue, could be due to PATH not being set as we expect? Or the symlink not working/submodule not checked out?

This is maybe a test that should be in boutdata now? Sort of straddles the two repos

test-stopCheck and test-coordinates-initialization

These both fail with BAD COMMAND and don't have any output in the log file. Probably need set -xe in the shell script. I guess the issue we're using some shell command that's not in FreeBSD