Failing tests (and isolating network tests to be skipped)

[mcepl]

Sometimes it's useful to have the tests that use the network marked so they can be skipped easily when we know the network is not available.

This is useful for example on SUSE and openSUSE's build servers. When building our packages the network is disabled so we can assure reproducible builds (among other benefits). With this mark, it's easier to skip tests that can not succeed.

The %check section of our SPEC file is:

%check
# set up working directory
export BASETEMP=$(mktemp -d -t cachelib_test.XXXXXX)
trap "rm -rf ${BASETEMP}" EXIT
# Allow finding memcached
export PATH="%{_sbindir}/:$PATH"
export PYTEST_ADDOPTS="--capture=tee-sys --tb=short --basetemp=${BASETEMP}"
%pytest -rs

(%pytest means basically pytest -v with some variables set to take into consideration building environment).

When running only plain pytest, I get result “11 failed, 117 passed, 1 skipped, 3 errors”.

Complete build log in this situation

Obviously the hot candidate are tests using DynamoDB, which is completely inaccessible, so I have created this patch to mark these tests as network-requiring so they can be easily skipped:

---
 setup.cfg                          |    3 +++
 tests/test_dynamodb_cache.py       |    1 +
 tests/test_interface_uniformity.py |    1 +
 tests/test_redis_cache.py          |    2 +-
 4 files changed, 6 insertions(+), 1 deletion(-)

--- a/setup.cfg
+++ b/setup.cfg
@@ -34,11 +34,14 @@ python_requires = >= 3.7
 where = src
 
 [tool:pytest]
+addopts = --strict-markers
 testpaths = tests
 filterwarnings = 
        error
        default::DeprecationWarning:cachelib.uwsgi
        default::DeprecationWarning:cachelib.redis
+markers =
+       network: mark a test which requires net access
 
 [coverage:run]
 branch = True
--- a/tests/test_dynamodb_cache.py
+++ b/tests/test_dynamodb_cache.py
@@ -29,5 +29,6 @@ def cache_factory(request):
         request.cls.cache_factory = _factory
 
 
+@pytest.mark.network
 class TestDynamoDbCache(CommonTests, ClearTests, HasTests):
     pass
--- a/tests/test_interface_uniformity.py
+++ b/tests/test_interface_uniformity.py
@@ -19,6 +19,7 @@ def create_cache_list(request, tmpdir):
     request.cls.cache_list = [FileSystemCache(tmpdir), mc, rc, SimpleCache()]
 
 
+@pytest.mark.network
 @pytest.mark.usefixtures("redis_server", "memcached_server")
 class TestInterfaceUniformity:
     def test_types_have_all_base_methods(self):

Just by applying this patch (and adding -k "not network" to my call of pytest) I get much better results: “117 passed, 1 skipped, 12 deselected, 2 errors”. Again, Complete build log in this situation.

Unfortunately, I don’t know how to skip those remaining two erroring tests. Both of them use so complicated constructs that I don’t know where to put @pytest.mark.skip or @pytest.mark.network and any of my attempts failed to make any difference. The only method which actually works (but I really don’t like it) is --ignore=tests/test_redis_cache.py --ignore=tests/test_memcached_cache.py, which truly make test suite to pass.

Any ideas, how to make the test suite working even without network access? Do I do something wrong in arranging my test environment?

Environment:

• Python version: various versions, this particular errors are from “Python 3.8.16”
• CacheLib version: 0.10.2 from the tarball from PyPI.