common: git retry feature

[david-baylibre]

We may face some "too many requests" failures (429 error) with some servers. It's better to retry the failed git command on a single repo rather than restart the whole process later which is likely to fail again

[makohoek]

Hi,

While I like the overall idea, I observe new test errors when running make test:

----------------------------------------------------------------------
Ran 34 tests in 65.325s

FAILED (failures=1, errors=2)
fast: 1.356859823999912 slow: 1.3492181369999798
make: *** [Makefile:18: test] Error 1

The failure is:

======================================================================
FAIL: test_jobs_limit (repo_resource.test_check.TestCheck.test_jobs_limit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/repo_resource/test_check.py", line 412, in test_jobs_limit
    self.assertTrue(fast_duration < slow_duration)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: False is not true

That's "ok" because the test is flaky.

However, the 2 new tests that generate errors should be investigated:

======================================================================
ERROR: test_unreachable_projects_in_manifest (repo_resource.test_check.TestCheck.test_unreachable_projects_in_manifest)
----------------------------------------------------------------------
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ~~~~^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/root/repo_resource/common.py", line 91, in multi_run_wrapper
    return getRevision(*args)
  File "/usr/local/lib/python3.13/site-packages/retrying.py", line 56, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/retrying.py", line 266, in call
    raise attempt.get()
          ~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/retrying.py", line 301, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/six.py", line 719, in reraise
    raise value
  File "/usr/local/lib/python3.13/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
                      ~~^^^^^^^^^^^^^^^^^
  File "/root/repo_resource/common.py", line 127, in getRevision
    g.ls_remote(remoteUrl+'/'+project, headRef).split()
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/git/cmd.py", line 741, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
                                   ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/git/cmd.py", line 1315, in _call_process
    return self.execute(call, **exec_kwargs)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/git/cmd.py", line 1109, in execute
    raise GitCommandError(redacted_command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git ls-remote https://gitlab.com/baylibreci/concourse-ci/utils/docker-projects.git master
  stderr: 'fatal: could not read Username for 'https://gitlab.com': terminal prompts disabled'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/repo_resource/test_check.py", line 194, in test_unreachable_projects_in_manifest
    check.check(instream)
    ~~~~~~~~~~~^^^^^^^^^^
  File "/root/repo_resource/check.py", line 57, in check
    raise e
  File "/root/repo_resource/check.py", line 54, in check
    .update_manifest(jobs=check_jobs) \
     ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/root/repo_resource/common.py", line 517, in update_manifest
    raise e
  File "/root/repo_resource/common.py", line 481, in update_manifest
    revisionList = pool.map(multi_run_wrapper, projects)
  File "/usr/local/lib/python3.13/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/multiprocessing/pool.py", line 774, in get
    raise self._value
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git ls-remote https://gitlab.com/baylibreci/concourse-ci/utils/docker-projects.git master
  stderr: 'fatal: could not read Username for 'https://gitlab.com': terminal prompts disabled'

@david-baylibre did you run the tests ?

[david-baylibre]

My bad @makohoek I thought these were tests requiring the SSH key failing