Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add assertions and workflow updates to debug ogr2ogr failure. #2849

Merged
merged 3 commits into from
Sep 11, 2023

Conversation

zaneselvans
Copy link
Member

PR Overview

Something is suddenly breaking our use of ogr2ogr in a subprocess to extract the Census DP1 GeoDataBase, but only in CI not locally. This PR is an attempt to debug why.

  • Update tox-pytest workflow to use setup-micromamba and checkout@v4
  • Add some assertions about the inputs & output paths involved in running ogr2ogr

PR Checklist

  • Merge the most recent version of the branch you are merging into (probably dev).
  • All CI checks are passing. Run tests locally to debug failures
  • Make sure you've included good docstrings.
  • For major data coverage & analysis changes, run data validation tests
  • Include unit tests for new functions and classes.
  • Defensive data quality/sanity checks in analyses & data processing functions.
  • Update the release notes and reference reference the PR and related issues.
  • Do your own explanatory review of the PR to help the reviewer understand what's going on and identify issues preemptively.

@bendnorman bendnorman mentioned this pull request Sep 11, 2023
8 tasks
@zaneselvans
Copy link
Member Author

Welp, all the input and output files and directories that we expect to be there are there, but for mysterious reasons the expected libabsl_cord.so shared library isn't being found. But it seems to work fine locally.

input = None, capture_output = True, timeout = None, check = True
popenargs = ([PosixPath('/home/runner/micromamba/envs/pudl-test/bin/ogr2ogr'), '/tmp/pytest-of-runner/pytest-0/pudl0/output/censusdp1tract.sqlite', '/tmp/tmpxktrwrjv/Profile-County_Tract.gdb'],)
kwargs = {'stderr': -1, 'stdout': -1}
process = <Popen: returncode: 127 args: [PosixPath('/home/runner/micromamba/envs/pudl-...>
stdout = b''
stderr = b'/home/runner/micromamba/envs/pudl-test/bin/ogr2ogr: error while loading shared libraries: libabsl_cord.so.2301.0.0: cannot open shared object file: No such file or directory\n'
retcode = 127

@jdangerx
Copy link
Member

Hmm, this libabsl seems to be from libabseil, and a new version got pushed up to conda over the weekend. Let me pin that to an older version and see if things work. If that doesn't fix it I'll look into a more interactive debugging approach than "push and hope."

@codecov
Copy link

codecov bot commented Sep 11, 2023

Codecov Report

Patch coverage: 100.0% and no project coverage change.

Comparison is base (b3567e5) 88.5% compared to head (9a89e6d) 88.5%.
Report is 4 commits behind head on dev.

Additional details and impacted files
@@          Coverage Diff          @@
##             dev   #2849   +/-   ##
=====================================
  Coverage   88.5%   88.5%           
=====================================
  Files         90      90           
  Lines      10134   10139    +5     
=====================================
+ Hits        8978    8983    +5     
  Misses      1156    1156           
Files Changed Coverage Δ
src/pudl/convert/censusdp1tract_to_sqlite.py 88.5% <100.0%> (+1.9%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

zaneselvans and others added 3 commits September 11, 2023 16:29
The error we're getting is

l_cord.so.2301.0.0: cannot open shared object file: No such file or directory

and 2301.0.0 seems suspiciously like the 2301 in the libabseil version number.
@zaneselvans zaneselvans merged commit e8d176c into dev Sep 11, 2023
5 checks passed
@zaneselvans zaneselvans deleted the ogr2ogr-debug branch September 11, 2023 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants