migrate.sh fails on new installation

[mccalluc]

Following the steps on https://github.com/opendp/dpcreator/blob/develop/README-DEVELOPMENT.md,
after step 2, I have a running webapp, but I hit the same error reported in #860. Continuing to read the instructions I see:

The first time you run (or anytime schema changes have been made), open a separate Terminal, cd into the dpcreator/ directory and manually run this migration:

docker-compose run server ./migrate.sh

This fails for me with OSError: /usr/local/lib/python3.9/site-packages/opendp/lib/libopendp.so: cannot open shared object file: No such file or directory

DJANGO_SETTINGS_MODULE opendp_project.settings.development
Traceback (most recent call last):
  File "/code/server/manage.py", line 29, in <module>
    main()
  File "/code/server/manage.py", line 25, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 453, in execute
    self.check()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 485, in check
    all_issues = checks.run_checks(
  File "/usr/local/lib/python3.9/site-packages/django/core/checks/registry.py", line 88, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
  File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 42, in check_url_namespaces_unique
    all_namespaces = _load_all_namespaces(resolver)
  File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 61, in _load_all_namespaces
    url_patterns = getattr(resolver, "url_patterns", [])
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 715, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 708, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/code/server/opendp_project/urls.py", line 23, in <module>
    from opendp_apps.analysis.views.analysis_plan_view import AnalysisPlanViewSet, AnalysisPlanListViewSet
  File "/code/server/opendp_apps/analysis/views/analysis_plan_view.py", line 12, in <module>
    from opendp_apps.analysis.serializers import \
  File "/code/server/opendp_apps/analysis/serializers.py", line 9, in <module>
    from opendp_apps.analysis.validate_release_util import ValidateReleaseUtil
  File "/code/server/opendp_apps/analysis/validate_release_util.py", line 27, in <module>
    from opendp_apps.analysis.tools.dp_count_spec import DPCountSpec
  File "/code/server/opendp_apps/analysis/tools/dp_count_spec.py", line 7, in <module>
    from opendp.accuracy import laplacian_scale_to_accuracy
  File "/usr/local/lib/python3.9/site-packages/opendp/__init__.py", line 1, in <module>
    from opendp.mod import Transformation, Measurement, OpenDPException, UnknownTypeException
  File "/usr/local/lib/python3.9/site-packages/opendp/mod.py", line 4, in <module>
    from opendp._lib import AnyMeasurement, AnyTransformation
  File "/usr/local/lib/python3.9/site-packages/opendp/_lib.py", line 31, in <module>
    lib = ctypes.cdll.LoadLibrary(os.path.join(lib_dir, lib_name))
  File "/usr/local/lib/python3.9/ctypes/__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "/usr/local/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/local/lib/python3.9/site-packages/opendp/lib/libopendp.so: cannot open shared object file: No such file or directory

It seems possible that the errors are related. It doesn't look like the tests (https://github.com/opendp/dpcreator/blob/develop/.github/workflows/cypress.yml) exercise this. Since I'm new to this codebase, one debugging strategy is to go back through the history and try to find an earlier version that works, and then see what changed after that.

[mccalluc]

I get the same OSError back to #838 from August last year, so it's not something new in the code.

Since Docker should be isolating me from any details of my environment, it must be something outside that has changed. A docker image should run consistently, but a docker build has no such guarantees -- It looks like we pin dependencies, but could an unpinned transitive dependency cause problems?

Next:

• Look at CI logs: Does it build there?
• Try to understand this error: Should the .so be there? Or is the error expecting it to be there?

[mccalluc]

Trying for a minimal reproducer: docker image build server works! This is good. Looking at the original stacktrace:

dpcreator-celery-queue-1  | Traceback (most recent call last):
dpcreator-celery-queue-1  |   File "/usr/local/bin/celery", line 8, in <module>
dpcreator-celery-queue-1  |     sys.exit(main())
dpcreator-celery-queue-1  |   File "/usr/local/lib/python3.9/site-packages/celery/__main__.py", line 15, in main
dpcreator-celery-queue-1  |     sys.exit(_main())
dpcreator-celery-queue-1  |   File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 213, in main
...
dpcreator-celery-queue-1  |   File "/usr/local/lib/python3.9/site-packages/opendp/_lib.py", line 31, in <module>
dpcreator-celery-queue-1  |     lib = ctypes.cdll.LoadLibrary(os.path.join(lib_dir, lib_name))
dpcreator-celery-queue-1  |   File "/usr/local/lib/python3.9/ctypes/__init__.py", line 452, in LoadLibrary
dpcreator-celery-queue-1  |     return self._dlltype(name)
dpcreator-celery-queue-1  |   File "/usr/local/lib/python3.9/ctypes/__init__.py", line 374, in __init__
dpcreator-celery-queue-1  |     self._handle = _dlopen(self._name, mode)
dpcreator-celery-queue-1  | OSError: /usr/local/lib/python3.9/site-packages/opendp/lib/libopendp.so: cannot open shared object file

It's not the build, it's the celery startup.

Looking at CI logs: There is a separate docker-compose-ci-step2.yml, and it also starts celery, and seems roughly the same: Biggest change is just adding envvars: SKIP_PDF_CREATION_FOR_TESTS=True and SKIP_EMAIL_RELEASE_FOR_TESTS=True. In that context, celery starts successfully.

One thing I've been assuming is that the error I get in the UI has its root cause here, but it's possible that's not the case.

[mccalluc]

Same OSError error when using the CI docker compose files, so it's not the difference between dev and CI.