Nightly Build Failure 2024-11-12 #3958

jdangerx · 2024-11-12T14:21:18Z

Overview

What do you think happened, if anything?

Something funky going on with the Zenodo API. Getting a 500 back when trying to get the latest record. Also they just updated master yesterday

This is important to fix ASAP since we're going to try to do the quarterly release this week.

Next steps

Try to make that specific request and see what the actual Zenodo error message is:

     response = self.retry_request(
         method="POST",
         url=f"{self.base_url}/records/{record_id}/versions",
         headers=self.auth_headers,
         timeout=5,
     )

check the API documentation - probably this is a request to the new API and they've changed the API
it's also possible they've changed the API without changing the documentation so we might have to experiment.

Verify that everything is fixed!

Once you've applied any necessary fixes, make sure that the nightly build outputs are all in their right places.

Tasks

Give feedback

S3 distribution bucket was updated at the expected time
GCP distribution bucket was updated at the expected time
GCP internal bucket was updated at the expected time
Datasette PUDL version points at the same hash as nightly
Zenodo sandbox record was updated to the record number in the logs (search for zenodo_data_release.py and Draft in the logs, to see what the new record number should be!)
Options

Relevant logs

Google Batch
link to build logs from internal distribution bucket

2024-11-12 10:47:38.121 UTC File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 244, in get_empty_draft
2024-11-12 10:47:38.121 UTC new_version = self.zenodo_client.new_record_version(latest_record.id_)
2024-11-12 10:47:38.121 UTC ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-12 10:47:38.121 UTC File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 151, in new_record_version
2024-11-12 10:47:38.121 UTC return _NewRecord(**response.json())
2024-11-12 10:47:38.121 UTC ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-12 10:47:38.121 UTC File "/home/mambauser/env/lib/python3.12/site-packages/pydantic/main.py", line 212, in __init__
2024-11-12 10:47:38.121 UTC validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
2024-11-12 10:47:38.121 UTC ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-12 10:47:38.121 UTC pydantic_core._pydantic_core.ValidationError: 2 validation errors for _NewRecord
2024-11-12 10:47:38.121 UTC id
2024-11-12 10:47:38.121 UTC Field required [type=missing, input_value={'status': 500, 'message'...'Misconfigured search.'}, input_type=dict]
2024-11-12 10:47:38.121 UTC For further information visit https://errors.pydantic.dev/2.9/v/missing
2024-11-12 10:47:38.121 UTC files
2024-11-12 10:47:38.121 UTC Field required [type=missing, input_value={'status': 500, 'message'...'Misconfigured search.'}, input_type=dict]

The text was updated successfully, but these errors were encountered:

e-belfer · 2024-11-12T15:17:32Z

I ran the following:

from zenodo.zenodo_data_release import ZenodoClient, InitialDataset

zenodo_client = ZenodoClient(env="sandbox")
rec_id = 5563
source_dir = "gs://builds.catalyst.coop/2024-11-12-0603-2255a06f0-main/"

completed_draft = (
    InitialDataset(zenodo_client=zenodo_client, record_id=rec_id)
    .get_empty_draft()
    .sync_directory(source_dir)
    .update_metadata()
)

I started and stopped this before it completed, but it seemed to be working as expected and set up this draft and created the following logs

2024-11-12 10:10:02,042: INFO - Getting new version for 5563 (zenodo_data_release.py:244)
2024-11-12 10:10:03,660: INFO - Draft 129578 has 0 existing files, deleting... (zenodo_data_release.py:249)
2024-11-12 10:10:03,661: INFO - Syncing files from gs://builds.catalyst.coop/2024-11-12-0603-2255a06f0-main/ to draft 129578... (zenodo_data_release.py:283)
2024-11-12 10:10:04,151: INFO - <Response [200]> (zenodo_data_release.py:120)
2024-11-12 10:10:04,151: INFO - {"created": "2024-11-12T10:47:37.785637+00:00", "modified": "2024-11-12T14:50:29.890062+00:00", "id": 129578, "conceptrecid": "5563", "conceptdoi": "10.5072/zenodo.5563", "metadata": {"title": "PUDL Data Release", "access_right": "open", "creators": [{"name": "Catalyst Cooperative", "affiliation": null}], "license": "cc-by-4.0", "imprint_publisher": "Zenodo", "communities": [{"identifier": "catalyst-cooperative"}], "upload_type": "dataset", "prereserve_doi": {"doi": "10.5281/zenodo.129578", "recid": 129578}}, "title": "PUDL Data Release", "links": {"self": "https://sandbox.zenodo.org/api/deposit/depositions/129578", "html": "https://sandbox.zenodo.org/deposit/129578", "badge": "https://sandbox.zenodo.org/badge/doi/.svg", "files": "https://sandbox.zenodo.org/api/deposit/depositions/129578/files", "bucket": "https://sandbox.zenodo.org/api/files/f1532252-0e19-458b-9cec-8e89f350f96a", "latest_draft": "https://sandbox.zenodo.org/api/deposit/depositions/129578", "latest_draft_html": "https://sandbox.zenodo.org/deposit/129578", "publish": "https://sandbox.zenodo.org/api/deposit/depositions/129578/actions/publish", "edit": "https://sandbox.zenodo.org/api/deposit/depositions/129578/actions/edit", "discard": "https://sandbox.zenodo.org/api/deposit/depositions/129578/actions/discard", "newversion": "https://sandbox.zenodo.org/api/deposit/depositions/129578/actions/newversion", "registerconceptdoi": "https://sandbox.zenodo.org/api/deposit/depositions/129578/actions/registerconceptdoi"}, "record_id": 129578, "owner": 133, "files": [], "state": "unsubmitted", "submitted": false} (zenodo_data_release.py:121)
2024-11-12 10:10:05,886: INFO - Reading gs://builds.catalyst.coop/2024-11-12-0603-2255a06f0-main/2024-11-12-0603-2255a06f0-main-pudl-etl.log (zenodo_data_release.py:273)
2024-11-12 10:10:08,280: INFO - Uploading file to https://sandbox.zenodo.org/api/files/f1532252-0e19-458b-9cec-8e89f350f96a/2024-11-12-0603-2255a06f0-main-pudl-etl.log (zenodo_data_release.py:192)
2024-11-12 10:10:19,472: INFO - Uploading to https://sandbox.zenodo.org/api/files/f1532252-0e19-458b-9cec-8e89f350f96a/2024-11-12-0603-2255a06f0-main-pudl-etl.log got {"created": "2024-11-12T15:10:08.676643+00:00", "updated": "2024-11-12T15:10:19.387334+00:00", "version_id": "55f70197-30b6-4704-b065-c2a16d39d34d", "key": "2024-11-12-0603-2255a06f0-main-pudl-etl.log", "size": 6775287, "mimetype": "application/octet-stream", "checksum": "md5:a9a90beaaf1d42519918907f7f6396cc", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/f1532252-0e19-458b-9cec-8e89f350f96a/2024-11-12-0603-2255a06f0-main-pudl-etl.log", "version": "https://sandbox.zenodo.org/api/files/f1532252-0e19-458b-9cec-8e89f350f96a/2024-11-12-0603-2255a06f0-main-pudl-etl.log?versionId=55f70197-30b6-4704-b065-c2a16d39d34d", "uploads": "https://sandbox.zenodo.org/api/files/f1532252-0e19-458b-9cec-8e89f350f96a/2024-11-12-0603-2255a06f0-main-pudl-etl.log?uploads"}} (zenodo_data_release.py:308)
2024-11-12 10:10:19,474: INFO - Reading gs://builds.catalyst.coop/2024-11-12-0603-2255a06f0-main/censusdp1tract.sqlite (zenodo_data_release.py:273)
2024-11-12 10:15:15,250: INFO - Uploading file to https://sandbox.zenodo.org/api/files/f1532252-0e19-458b-9cec-8e89f350f96a/censusdp1tract.sqlite (zenodo_data_release.py:192)

So my read on this is that it was an intermittent Zenodo bork, not a major API change or structural problem. If this is still a problem tomorrow, I'll re-investigate.

jdangerx · 2024-11-12T16:42:55Z

Thanks for investigating further! Getting a 500 back does point to "problem is on Zenodo's end" vs. an API change that would in theory kick out a 400. 🤞

zaneselvans · 2024-11-15T16:48:11Z

The Zenodo step of last night's nightly build (2024-11-15) and the v2024.11.0 release both failed, but in different ways.

Nightly build

The draft deposition in the Zenodo sandbox seems to have all of the files.
The creation of the deposition failed due to a Pydantic validation error on the _LegacyDeposition but the logs don't contain the full validation error. It looks like it got missing values for the required id, conceptrecid, links, and metadata fields. See this stack trace below.
Those are actually all the fields that make up _LegacyDeposition so it seems like it's just getting an empty response, or a response with totally different contents. We should check interactively.

Full stack trace

Traceback (most recent call last):
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 410, in <module>
    pudl_zenodo_data_release()
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 399, in pudl_zenodo_data_release
    .update_metadata()
     ^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 348, in update_metadata
    self.zenodo_client.update_deposition_metadata(self.record_id, metadata=metadata)
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 167, in update_deposition_metadata
    return _LegacyDeposition(**response.json())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/pydantic/main.py", line 212, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 4 validation errors for _LegacyDeposition
id
  Field required [type=missing, input_value={'error_id': '11d6cc4ea15...cation.', 'status': 500}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
conceptrecid
  Field required [type=missing, input_value={'error_id': '11d6cc4ea15...cation.', 'status': 500}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
links
  Field required [type=missing, input_value={'error_id': '11d6cc4ea15...cation.', 'status': 500}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
metadata
  Field required [type=missing, input_value={'error_id': '11d6cc4ea15...cation.', 'status': 500}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing

v2024.11.0 release

A draft deposition was created, but it only contains a couple of files, and they aren't all complete.
The creation of the deposition failed due to some network timeouts. The one that ultimately killed it was uploading the 8GB pudl_parquet.zip file, but an earlier file upload of the also large pudl.sqlite.zip also had a couple of retries and did not complete before the script bailed.

Zenodo deposition logs

Creating a new PUDL data release on Zenodo.
2024-11-15 11:58:35,561: INFO - Using Zenodo token: e8a6...MVnQ (zenodo_data_release.py:88)
2024-11-15 11:58:35,561: INFO - Getting new version for 3653158 (zenodo_data_release.py:242)
2024-11-15 11:58:37,616: INFO - Draft 14169544 has 0 existing files, deleting... (zenodo_data_release.py:247)
2024-11-15 11:58:37,616: INFO - Syncing files from /home/mambauser/pudl_work/output to draft 14169544... (zenodo_data_release.py:281)
2024-11-15 11:58:38,231: INFO - Reading file:///home/mambauser/pudl_work/output/pudl.sqlite.zip (zenodo_data_release.py:271)
2024-11-15 11:58:38,231: INFO - Uploading file to https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/pudl.sqlite.zip (zenodo_data_release.py:190)
2024-11-15 12:11:11,414: WARNING - Attempt #1 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 2 s (zenodo_data_release.py:103)
2024-11-15 12:11:18,689: WARNING - Attempt #2 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 4 s (zenodo_data_release.py:103)
2024-11-15 12:12:04,000: INFO - Uploading to https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/pudl.sqlite.zip got {"created": "2024-11-15T12:11:23.181387+00:00", "updated": "2024-11-15T12:12:03.883771+00:00", "version_id": "d3b125cc-3f34-4a60-b5c6-8ec32209607e", "key": "pudl.sqlite.zip", "size": 628645704, "mimetype": "application/zip", "checksum": "md5:c62e1582d4be70e3af7cc8eed5ba5d58", "is_head": true, "delete_marker": false, "links": {"self": "https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/pudl.sqlite.zip", "version": "https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/pudl.sqlite.zip?versionId=d3b125cc-3f34-4a60-b5c6-8ec32209607e", "uploads": "https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/pudl.sqlite.zip?uploads"}} (zenodo_data_release.py:306)
2024-11-15 12:12:04,001: INFO - Reading file:///home/mambauser/pudl_work/output/ferc2_dbf.sqlite.zip (zenodo_data_release.py:271)
2024-11-15 12:12:04,001: INFO - Uploading file to https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/ferc2_dbf.sqlite.zip (zenodo_data_release.py:190)
2024-11-15 12:12:10,125: INFO - Uploading to https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/ferc2_dbf.sqlite.zip got {"created": "2024-11-15T12:12:04.464217+00:00", "updated": "2024-11-15T12:12:10.009024+00:00", "version_id": "63f6d62d-cabb-4464-b36a-e406aeaccd6b", "key": "ferc2_dbf.sqlite.zip", "size": 74502005, "mimetype": "application/zip", "checksum": "md5:8fa62784666976ca3c25337cb7045752", "is_head": true, "delete_marker": false, "links": {"self": "https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/ferc2_dbf.sqlite.zip", "version": "https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/ferc2_dbf.sqlite.zip?versionId=63f6d62d-cabb-4464-b36a-e406aeaccd6b", "uploads": "https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/ferc2_dbf.sqlite.zip?uploads"}} (zenodo_data_release.py:306)
2024-11-15 12:12:10,126: INFO - Reading file:///home/mambauser/pudl_work/output/ferc1_xbrl.sqlite.zip (zenodo_data_release.py:271)
2024-11-15 12:12:10,126: INFO - Uploading file to https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/ferc1_xbrl.sqlite.zip (zenodo_data_release.py:190)
2024-11-15 12:12:20,536: INFO - Uploading to https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/ferc1_xbrl.sqlite.zip got {"created": "2024-11-15T12:12:10.602750+00:00", "updated": "2024-11-15T12:12:20.394215+00:00", "version_id": "40360330-aa73-42ef-9f7c-d6e0965b0e3c", "key": "ferc1_xbrl.sqlite.zip", "size": 142739341, "mimetype": "application/zip", "checksum": "md5:75b97d29fead28082b24a65070dcf98e", "is_head": true, "delete_marker": false, "links": {"self": "https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/ferc1_xbrl.sqlite.zip", "version": "https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/ferc1_xbrl.sqlite.zip?versionId=40360330-aa73-42ef-9f7c-d6e0965b0e3c", "uploads": "https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/ferc1_xbrl.sqlite.zip?uploads"}} (zenodo_data_release.py:306)
2024-11-15 12:12:20,536: INFO - Reading file:///home/mambauser/pudl_work/output/pudl_parquet.zip (zenodo_data_release.py:271)
2024-11-15 12:12:20,537: INFO - Uploading file to https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/pudl_parquet.zip (zenodo_data_release.py:190)
2024-11-15 12:12:30,020: WARNING - Attempt #1 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 2 s (zenodo_data_release.py:103)
2024-11-15 12:12:36,626: WARNING - Attempt #2 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 4 s (zenodo_data_release.py:103)
2024-11-15 12:12:48,066: WARNING - Attempt #3 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 8 s (zenodo_data_release.py:103)
2024-11-15 12:13:06,876: WARNING - Attempt #4 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 16 s (zenodo_data_release.py:103)

Full stack trace

2024-11-15 12:12:20,537: INFO - Uploading file to https://zenodo.org/api/files/ff78adba-5476-4c1a-b8f6-ef4e83d831ce/pudl_parquet.zip (zenodo_data_release.py:190)
2024-11-15 12:12:30,020: WARNING - Attempt #1 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 2 s (zenodo_data_release.py:103)
2024-11-15 12:12:36,626: WARNING - Attempt #2 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 4 s (zenodo_data_release.py:103)
2024-11-15 12:12:48,066: WARNING - Attempt #3 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 8 s (zenodo_data_release.py:103)
2024-11-15 12:13:06,876: WARNING - Attempt #4 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 16 s (zenodo_data_release.py:103)
Traceback (most recent call last):
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1336, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1382, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1130, in _send_output
    self.send(chunk)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1055, in send
    self.sock.sendall(data)
  File "/home/mambauser/env/lib/python3.12/ssl.py", line 1210, in sendall
    v = self.send(byte_view[count:])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/ssl.py", line 1179, in send
    return self._sslobj.write(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mambauser/env/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 801, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/util/retry.py", line 552, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1336, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1382, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1130, in _send_output
    self.send(chunk)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1055, in send
    self.sock.sendall(data)
  File "/home/mambauser/env/lib/python3.12/ssl.py", line 1210, in sendall
    v = self.send(byte_view[count:])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/ssl.py", line 1179, in send
    return self._sslobj.write(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError('The write operation timed out'))

e-belfer · 2024-11-15T17:07:12Z

Those are actually all the fields that make up _LegacyDeposition so it seems like it's just getting an empty response, or a response with totally different contents. We should check interactively.

Yes, if you see the response input it shows it made a query to the existing deposition and got a 500 error, which it is then trying to shunt into _LegacyDeposition and getting an error.

zaneselvans · 2024-11-15T17:39:07Z

The error 500 that we're getting on the Sandbox server doesn't happen until after all of the files have been uploaded, and we don't get that far with the deposition on the Production server because of the timeouts, so it seems possible that even without the timeouts, we'll get the same problem on the production server. But also Zenodo has been bad about synchronizing the behavior of the two servers so who knows.

zaneselvans · 2024-11-15T20:36:53Z

Attempting to re-run the Zenodo data release

Created VM on GCP called pudl-build-env that uses their "Container optimized OS" and pulls the latest version of the docker.io/catalystcoop/pudl-etl container.
Started that VM, SSHed into it, ran the pudl-build-env container with command tail -f /dev/null so it stays running.
Used docker exec -it /bin/bash to get into the container
Activated our conda environment so I can run the zenodo release script.
Copied all the files to be deployed to Zenodo to a temporary path under gs://builds.catalyst.coop/tmp/
These files are *.zip, *.json, *.log -- none of the Parquet files get archived on Zenodo directly. Just the big zip archive of all of them.
Needed to run gcloud auth to get permissions to access the builds output. Tried using the service account we have set up for the nightly builds, but it didn't like that, so I used my personal account (not ideal)
Couldn't use outputs in gs://pudl.catalyst.coop/ because the zenodo release script can't handle a requester pays bucket.
This mostly worked. It experienced a single retry after timeout, but that retry, but failed to fully upload the pudl_parquet.zip file (size was 7.65GB rather than 8.4GB, MD5 sum did not match).
It also resulted in the same error status 500 after attempting to upload everything.
Trying again to upload but not publish a draft on the production server and.... everything uploads. No timeouts, big files seem to be the right sizes, and there's no 500 errors that crop up after the uploads.
What was different between the nightly build vs. release deployments?
- Sandbox vs. Production server
- Sandbox publishes automatically, Production just creates a draft.
What is different about what am I doing now?
- Only trying to publish a draft on both of them
- Using my personal gcloud authentication rather than the service account.
- Deploying form a GCS bucket path rather than the local filesystem.

zaneselvans · 2024-11-18T03:48:31Z

Last night (2024-11-17) two different builds kicked off within half an hour of each other. One was a nightly-build-cruft branch deployment, and the other was the normal nightly build on main. Both failed to create new Zenodo sandbox Data Releases due to timeouts (rather than Status 500). One timed out when uploading a relatively small file, and the other while attempting to delete files from an existing draft. The two builds don't seem to have interacted with each other at all.

Logs from Zenodo data release on main

Creating a new PUDL data release on Zenodo.
2024-11-17 11:42:40,034: INFO - Using Zenodo token: yUlx...XDrR (zenodo_data_release.py:88)
2024-11-17 11:42:40,034: INFO - Getting new version for 5563 (zenodo_data_release.py:242)
2024-11-17 11:42:41,691: INFO - Draft 133399 has 3 existing files, deleting... (zenodo_data_release.py:247)
2024-11-17 11:42:47,005: WARNING - Attempt #1 Got HTTPSConnectionPool(host='sandbox.zenodo.org', port=443): Read timed out. (read timeout=5), retrying in 2 s (zenodo_data_release.py:103)
2024-11-17 11:42:51,319: WARNING - Attempt #2 Got HTTPSConnectionPool(host='sandbox.zenodo.org', port=443): Read timed out. (read timeout=2), retrying in 4 s (zenodo_data_release.py:103)
2024-11-17 11:42:59,635: WARNING - Attempt #3 Got HTTPSConnectionPool(host='sandbox.zenodo.org', port=443): Read timed out. (read timeout=4), retrying in 8 s (zenodo_data_release.py:103)
2024-11-17 11:43:15,952: WARNING - Attempt #4 Got HTTPSConnectionPool(host='sandbox.zenodo.org', port=443): Read timed out. (read timeout=8), retrying in 16 s (zenodo_data_release.py:103)
Traceback (most recent call last):
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 462, in _make_request
    httplib_response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1428, in getresponse
    response.begin()
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/socket.py", line 720, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/ssl.py", line 1251, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/ssl.py", line 1103, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mambauser/env/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 801, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/util/retry.py", line 552, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 469, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='sandbox.zenodo.org', port=443): Read timed out. (read timeout=16)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 410, in <module>
    pudl_zenodo_data_release()
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 397, in pudl_zenodo_data_release
    .get_empty_draft()
     ^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 251, in get_empty_draft
    self.zenodo_client.delete_deposition_file(new_rec_id, f.id_)
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 174, in delete_deposition_file
    return self.retry_request(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 107, in retry_request
    return requests.request(method=method, url=url, timeout=timeout, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/requests/adapters.py", line 713, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='sandbox.zenodo.org', port=443): Read timed out. (read timeout=16)

This one had several uploads with retries before one of them finally failed. Maybe we should be more generous with the number of retries?

Logs from Zenodo data release on nightly-build-cruft

Creating a new PUDL data release on Zenodo.
+ [[ sandbox == \p\r\o\d\u\c\t\i\o\n ]]
+ [[ sandbox == \s\a\n\d\b\o\x ]]
+ /home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py --publish --env sandbox --source-dir /home/mambauser/pudl_work/output
2024-11-17 11:16:49,609: INFO - Using Zenodo token: yUlx...XDrR (zenodo_data_release.py:88)
2024-11-17 11:16:49,609: INFO - Getting new version for 5563 (zenodo_data_release.py:242)
2024-11-17 11:16:51,390: INFO - Draft 133399 has 0 existing files, deleting... (zenodo_data_release.py:247)
2024-11-17 11:16:51,390: INFO - Syncing files from /home/mambauser/pudl_work/output to draft 133399... (zenodo_data_release.py:281)
2024-11-17 11:16:51,998: INFO - Reading file:///home/mambauser/pudl_work/output/pudl.sqlite.zip (zenodo_data_release.py:271)
2024-11-17 11:16:51,998: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/pudl.sqlite.zip (zenodo_data_release.py:190)
2024-11-17 11:19:41,680: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/pudl.sqlite.zip got {"created": "2024-11-17T11:16:52.444367+00:00", "updated": "2024-11-17T11:19:41.580922+00:00", "version_id": "0ca030f6-0ded-4270-b43a-4b1fa5e735fb", "key": "pudl.sqlite.zip", "size": 2775440358, "mimetype": "application/zip", "checksum": "md5:c640ea80e56cc38e81536f44c8d2f870", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/pudl.sqlite.zip", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/pudl.sqlite.zip?versionId=0ca030f6-0ded-4270-b43a-4b1fa5e735fb", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/pudl.sqlite.zip?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:19:41,680: INFO - Reading file:///home/mambauser/pudl_work/output/ferc2_dbf.sqlite.zip (zenodo_data_release.py:271)
2024-11-17 11:19:41,680: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_dbf.sqlite.zip (zenodo_data_release.py:190)
2024-11-17 11:19:47,935: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_dbf.sqlite.zip got {"created": "2024-11-17T11:19:42.322103+00:00", "updated": "2024-11-17T11:19:47.835438+00:00", "version_id": "9ba1c216-adbc-4429-bf2c-2f927123efcf", "key": "ferc2_dbf.sqlite.zip", "size": 74502005, "mimetype": "application/zip", "checksum": "md5:a2eef58dc83e149a239fb37a1f440033", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_dbf.sqlite.zip", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_dbf.sqlite.zip?versionId=9ba1c216-adbc-4429-bf2c-2f927123efcf", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_dbf.sqlite.zip?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:19:47,936: INFO - Reading file:///home/mambauser/pudl_work/output/ferc1_xbrl.sqlite.zip (zenodo_data_release.py:271)
2024-11-17 11:19:47,936: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc1_xbrl.sqlite.zip (zenodo_data_release.py:190)
2024-11-17 11:19:58,172: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc1_xbrl.sqlite.zip got {"created": "2024-11-17T11:19:48.377809+00:00", "updated": "2024-11-17T11:19:58.072793+00:00", "version_id": "d14fd4c1-5ac2-47fd-af90-28c1bcbd7a90", "key": "ferc1_xbrl.sqlite.zip", "size": 143241492, "mimetype": "application/zip", "checksum": "md5:64ca39a6b6818ed498238bbaa5aa3105", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc1_xbrl.sqlite.zip", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc1_xbrl.sqlite.zip?versionId=d14fd4c1-5ac2-47fd-af90-28c1bcbd7a90", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc1_xbrl.sqlite.zip?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:19:58,172: INFO - Reading file:///home/mambauser/pudl_work/output/pudl_parquet.zip (zenodo_data_release.py:271)
2024-11-17 11:19:58,172: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/pudl_parquet.zip (zenodo_data_release.py:190)
2024-11-17 11:49:03,167: WARNING - Attempt #1 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 2 s (zenodo_data_release.py:103)
2024-11-17 11:49:10,243: WARNING - Attempt #2 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 4 s (zenodo_data_release.py:103)
2024-11-17 11:49:21,205: WARNING - Attempt #3 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 8 s (zenodo_data_release.py:103)
2024-11-17 11:49:40,094: WARNING - Attempt #4 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 16 s (zenodo_data_release.py:103)
2024-11-17 11:51:50,021: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/pudl_parquet.zip got {"created": "2024-11-17T11:49:56.710047+00:00", "updated": "2024-11-17T11:51:49.917211+00:00", "version_id": "daaaa27a-444d-4126-b677-05c2760f9dfb", "key": "pudl_parquet.zip", "size": 1611776261, "mimetype": "application/zip", "checksum": "md5:ce5a4a2b7af040d9baaeb450b5ff286e", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/pudl_parquet.zip", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/pudl_parquet.zip?versionId=daaaa27a-444d-4126-b677-05c2760f9dfb", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/pudl_parquet.zip?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:51:50,022: INFO - Reading file:///home/mambauser/pudl_work/output/2024-11-17-0523-6d1f0d582-nightly-build-cruft.log (zenodo_data_release.py:271)
2024-11-17 11:51:50,023: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/2024-11-17-0523-6d1f0d582-nightly-build-cruft.log (zenodo_data_release.py:190)
2024-11-17 11:51:52,313: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/2024-11-17-0523-6d1f0d582-nightly-build-cruft.log got {"created": "2024-11-17T11:51:50.817565+00:00", "updated": "2024-11-17T11:51:52.216137+00:00", "version_id": "48285244-fbbf-4a0f-bad3-48aaee2d91ef", "key": "2024-11-17-0523-6d1f0d582-nightly-build-cruft.log", "size": 6688160, "mimetype": "application/octet-stream", "checksum": "md5:018ad4b4c22addf1812b0ecb26e7dbab", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/2024-11-17-0523-6d1f0d582-nightly-build-cruft.log", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/2024-11-17-0523-6d1f0d582-nightly-build-cruft.log?versionId=48285244-fbbf-4a0f-bad3-48aaee2d91ef", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/2024-11-17-0523-6d1f0d582-nightly-build-cruft.log?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:51:52,313: INFO - Reading file:///home/mambauser/pudl_work/output/ferc60_xbrl_datapackage.json (zenodo_data_release.py:271)
2024-11-17 11:51:52,314: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_xbrl_datapackage.json (zenodo_data_release.py:190)
2024-11-17 11:51:53,762: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_xbrl_datapackage.json got {"created": "2024-11-17T11:51:52.765291+00:00", "updated": "2024-11-17T11:51:53.662586+00:00", "version_id": "0df31d59-0e52-47a0-b26f-d36975ef022f", "key": "ferc60_xbrl_datapackage.json", "size": 964801, "mimetype": "application/json", "checksum": "md5:d2ed6ba576f7fbe3fb85986203cd9ecc", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_xbrl_datapackage.json", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_xbrl_datapackage.json?versionId=0df31d59-0e52-47a0-b26f-d36975ef022f", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_xbrl_datapackage.json?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:51:53,762: INFO - Reading file:///home/mambauser/pudl_work/output/ferc60_xbrl_taxonomy_metadata.json (zenodo_data_release.py:271)
2024-11-17 11:51:53,762: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_xbrl_taxonomy_metadata.json (zenodo_data_release.py:190)
2024-11-17 11:51:55,369: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_xbrl_taxonomy_metadata.json got {"created": "2024-11-17T11:51:54.198687+00:00", "updated": "2024-11-17T11:51:55.266753+00:00", "version_id": "be1e0bf5-a252-433c-8dab-a6b9568b2dd4", "key": "ferc60_xbrl_taxonomy_metadata.json", "size": 1856818, "mimetype": "application/json", "checksum": "md5:3761d6fc6a9e4901476793076d338ab4", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_xbrl_taxonomy_metadata.json", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_xbrl_taxonomy_metadata.json?versionId=be1e0bf5-a252-433c-8dab-a6b9568b2dd4", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_xbrl_taxonomy_metadata.json?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:51:55,369: INFO - Reading file:///home/mambauser/pudl_work/output/ferc2_xbrl_datapackage.json (zenodo_data_release.py:271)
2024-11-17 11:51:55,369: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl_datapackage.json (zenodo_data_release.py:190)
2024-11-17 11:51:59,473: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl_datapackage.json got {"created": "2024-11-17T11:51:55.805268+00:00", "updated": "2024-11-17T11:51:59.374009+00:00", "version_id": "faeb24be-74fc-4332-81a6-6c4d3e98358d", "key": "ferc2_xbrl_datapackage.json", "size": 2581417, "mimetype": "application/json", "checksum": "md5:94fc1f642648e8eb7b629c2e95fe0c43", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl_datapackage.json", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl_datapackage.json?versionId=faeb24be-74fc-4332-81a6-6c4d3e98358d", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl_datapackage.json?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:51:59,473: INFO - Reading file:///home/mambauser/pudl_work/output/ferc2_xbrl.sqlite.zip (zenodo_data_release.py:271)
2024-11-17 11:51:59,473: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl.sqlite.zip (zenodo_data_release.py:190)
2024-11-17 11:52:02,314: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl.sqlite.zip got {"created": "2024-11-17T11:51:59.909628+00:00", "updated": "2024-11-17T11:52:02.214836+00:00", "version_id": "3c47a895-3f5a-4b6d-96d3-5f5b69a9a17e", "key": "ferc2_xbrl.sqlite.zip", "size": 19987676, "mimetype": "application/zip", "checksum": "md5:de97a28a59b5f1dd9e9692f1471279fe", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl.sqlite.zip", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl.sqlite.zip?versionId=3c47a895-3f5a-4b6d-96d3-5f5b69a9a17e", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl.sqlite.zip?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:52:02,314: INFO - Reading file:///home/mambauser/pudl_work/output/ferc714_xbrl_datapackage.json (zenodo_data_release.py:271)
2024-11-17 11:52:02,314: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc714_xbrl_datapackage.json (zenodo_data_release.py:190)
2024-11-17 11:52:03,281: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc714_xbrl_datapackage.json got {"created": "2024-11-17T11:52:02.749273+00:00", "updated": "2024-11-17T11:52:03.184718+00:00", "version_id": "9323df26-ac62-46c7-b889-3099952f8523", "key": "ferc714_xbrl_datapackage.json", "size": 85393, "mimetype": "application/json", "checksum": "md5:1f13dd784a809aa54c9ff11cdb4b759a", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc714_xbrl_datapackage.json", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc714_xbrl_datapackage.json?versionId=9323df26-ac62-46c7-b889-3099952f8523", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc714_xbrl_datapackage.json?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:52:03,281: INFO - Reading file:///home/mambauser/pudl_work/output/ferc2_xbrl_taxonomy_metadata.json (zenodo_data_release.py:271)
2024-11-17 11:52:03,281: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl_taxonomy_metadata.json (zenodo_data_release.py:190)
2024-11-17 11:52:10,124: WARNING - Attempt #1 Got HTTPSConnectionPool(host='sandbox.zenodo.org', port=443): Read timed out. (read timeout=5), retrying in 2 s (zenodo_data_release.py:103)
2024-11-17 11:52:14,437: WARNING - Attempt #2 Got HTTPSConnectionPool(host='sandbox.zenodo.org', port=443): Read timed out. (read timeout=2), retrying in 4 s (zenodo_data_release.py:103)
2024-11-17 11:52:22,752: WARNING - Attempt #3 Got HTTPSConnectionPool(host='sandbox.zenodo.org', port=443): Read timed out. (read timeout=4), retrying in 8 s (zenodo_data_release.py:103)
2024-11-17 11:52:32,093: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl_taxonomy_metadata.json got {"created": "2024-11-17T11:52:31.417639+00:00", "updated": "2024-11-17T11:52:31.992775+00:00", "version_id": "4a454035-9b49-46ab-a205-026828610f4e", "key": "ferc2_xbrl_taxonomy_metadata.json", "size": 0, "mimetype": "application/json", "checksum": "md5:d41d8cd98f00b204e9800998ecf8427e", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl_taxonomy_metadata.json", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl_taxonomy_metadata.json?versionId=4a454035-9b49-46ab-a205-026828610f4e", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc2_xbrl_taxonomy_metadata.json?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:52:32,093: INFO - Reading file:///home/mambauser/pudl_work/output/ferc60_dbf.sqlite.zip (zenodo_data_release.py:271)
2024-11-17 11:52:32,093: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_dbf.sqlite.zip (zenodo_data_release.py:190)
2024-11-17 11:52:33,995: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_dbf.sqlite.zip got {"created": "2024-11-17T11:52:32.528455+00:00", "updated": "2024-11-17T11:52:33.899414+00:00", "version_id": "8f0c2ea1-d845-47d5-a101-479245bfdf36", "key": "ferc60_dbf.sqlite.zip", "size": 2882665, "mimetype": "application/zip", "checksum": "md5:d711db66e3e467de9a515646a2a79235", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_dbf.sqlite.zip", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_dbf.sqlite.zip?versionId=8f0c2ea1-d845-47d5-a101-479245bfdf36", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc60_dbf.sqlite.zip?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:52:33,995: INFO - Reading file:///home/mambauser/pudl_work/output/ferc714_xbrl.sqlite.zip (zenodo_data_release.py:271)
2024-11-17 11:52:33,995: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc714_xbrl.sqlite.zip (zenodo_data_release.py:190)
2024-11-17 11:52:46,052: INFO - Uploading to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc714_xbrl.sqlite.zip got {"created": "2024-11-17T11:52:34.437756+00:00", "updated": "2024-11-17T11:52:45.953003+00:00", "version_id": "edfce243-2bfd-4b32-b49b-d735503290c3", "key": "ferc714_xbrl.sqlite.zip", "size": 157650589, "mimetype": "application/zip", "checksum": "md5:21a854cf9827b28b39c786c17b710187", "is_head": true, "delete_marker": false, "links": {"self": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc714_xbrl.sqlite.zip", "version": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc714_xbrl.sqlite.zip?versionId=edfce243-2bfd-4b32-b49b-d735503290c3", "uploads": "https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc714_xbrl.sqlite.zip?uploads"}} (zenodo_data_release.py:306)
2024-11-17 11:52:46,052: INFO - Reading file:///home/mambauser/pudl_work/output/ferc1_dbf.sqlite.zip (zenodo_data_release.py:271)
2024-11-17 11:52:46,052: INFO - Uploading file to https://sandbox.zenodo.org/api/files/a7670db4-4420-47bb-a5ef-03971172515a/ferc1_dbf.sqlite.zip (zenodo_data_release.py:190)
2024-11-17 11:53:00,723: WARNING - Attempt #1 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 2 s (zenodo_data_release.py:103)
2024-11-17 11:53:07,934: WARNING - Attempt #2 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 4 s (zenodo_data_release.py:103)
2024-11-17 11:53:20,065: WARNING - Attempt #3 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 8 s (zenodo_data_release.py:103)
2024-11-17 11:53:39,100: WARNING - Attempt #4 Got ('Connection aborted.', TimeoutError('The write operation timed out')), retrying in 16 s (zenodo_data_release.py:103)
Traceback (most recent call last):
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1336, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1382, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1130, in _send_output
    self.send(chunk)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1055, in send
    self.sock.sendall(data)
  File "/home/mambauser/env/lib/python3.12/ssl.py", line 1210, in sendall
    v = self.send(byte_view[count:])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/ssl.py", line 1179, in send
    return self._sslobj.write(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mambauser/env/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 801, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/util/retry.py", line 552, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/mambauser/env/lib/python3.12/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1336, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1382, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1130, in _send_output
    self.send(chunk)
  File "/home/mambauser/env/lib/python3.12/http/client.py", line 1055, in send
    self.sock.sendall(data)
  File "/home/mambauser/env/lib/python3.12/ssl.py", line 1210, in sendall
    v = self.send(byte_view[count:])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/ssl.py", line 1179, in send
    return self._sslobj.write(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError('The write operation timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 413, in <module>
    pudl_zenodo_data_release()
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 401, in pudl_zenodo_data_release
    .sync_directory(source_dir)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 303, in sync_directory
    response = self.zenodo_client.create_bucket_file(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 194, in create_bucket_file
    response = self.retry_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/devtools/zenodo/zenodo_data_release.py", line 107, in retry_request
    return requests.request(method=method, url=url, timeout=timeout, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError('The write operation timed out'))

zaneselvans · 2024-11-18T19:32:50Z

I emailed Zenodo support about the consistent timeouts today (2024-11-18)

zaneselvans · 2024-11-20T06:15:52Z

We've had one successful nightly sandbox deployment and several successful fast ETL sandbox deployments so maybe this is back to working magically? If it passes tomorrow morning 2024-11-20 I'll close.

zaneselvans · 2024-11-20T16:02:49Z

We got another successful sandbox release last night, and Zenodo Support responded saying that they made a faulty deployment to the Sandbox on Nov 12th that could have created issues, so I'm going to call this fixed for the moment. 🤞🏼

jdangerx added bug Things that are just plain broken. nightly-builds Anything having to do with nightly builds or continuous deployment. labels Nov 12, 2024

github-project-automation bot added this to Catalyst Megaproject Nov 12, 2024

github-project-automation bot moved this to New in Catalyst Megaproject Nov 12, 2024

jdangerx moved this from New to Backlog in Catalyst Megaproject Nov 12, 2024

e-belfer self-assigned this Nov 12, 2024

e-belfer moved this from Backlog to In progress in Catalyst Megaproject Nov 12, 2024

e-belfer moved this from In progress to In review in Catalyst Megaproject Nov 12, 2024

zaneselvans mentioned this issue Nov 15, 2024

PUDL Data Release v2024.11.0 #3960

Closed

zaneselvans mentioned this issue Nov 16, 2024

Clean up some nightly build infrastructure cruft #3962

Merged

zaneselvans moved this from In review to In progress in Catalyst Megaproject Nov 17, 2024

zaneselvans closed this as completed Nov 20, 2024

github-project-automation bot moved this from In progress to Done in Catalyst Megaproject Nov 20, 2024

zaneselvans reopened this Nov 20, 2024

zaneselvans closed this as completed Nov 20, 2024

github-project-automation bot moved this from In review to Done in Catalyst Megaproject Nov 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nightly Build Failure 2024-11-12 #3958

Nightly Build Failure 2024-11-12 #3958

jdangerx commented Nov 12, 2024 •

edited by e-belfer

Loading

Tasks

e-belfer commented Nov 12, 2024

jdangerx commented Nov 12, 2024

zaneselvans commented Nov 15, 2024 •

edited

Loading

e-belfer commented Nov 15, 2024

zaneselvans commented Nov 15, 2024

zaneselvans commented Nov 15, 2024 •

edited

Loading

zaneselvans commented Nov 18, 2024

zaneselvans commented Nov 18, 2024

zaneselvans commented Nov 20, 2024

zaneselvans commented Nov 20, 2024

Nightly Build Failure 2024-11-12 #3958

Nightly Build Failure 2024-11-12 #3958

Comments

jdangerx commented Nov 12, 2024 • edited by e-belfer Loading

Overview

Next steps

Verify that everything is fixed!

Tasks

Relevant logs

e-belfer commented Nov 12, 2024

jdangerx commented Nov 12, 2024

zaneselvans commented Nov 15, 2024 • edited Loading

Nightly build

v2024.11.0 release

e-belfer commented Nov 15, 2024

zaneselvans commented Nov 15, 2024

zaneselvans commented Nov 15, 2024 • edited Loading

zaneselvans commented Nov 18, 2024

zaneselvans commented Nov 18, 2024

zaneselvans commented Nov 20, 2024

zaneselvans commented Nov 20, 2024

jdangerx commented Nov 12, 2024 •

edited by e-belfer

Loading

zaneselvans commented Nov 15, 2024 •

edited

Loading

zaneselvans commented Nov 15, 2024 •

edited

Loading