Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Uncaught Exceptions During Re-Indexing #2908

Open
Samk13 opened this issue Nov 12, 2024 · 0 comments
Open

Uncaught Exceptions During Re-Indexing #2908

Samk13 opened this issue Nov 12, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@Samk13
Copy link
Member

Samk13 commented Nov 12, 2024

Package version (if known): V12

Describe the bug

During restore tests, running invenio rdm rebuild-all-indices command results in a PreconditionFailed (406) error after a period of operation, most likely due to a lack of resources (using default Helm chart resource settings).

Steps to Reproduce

  1. Restore an instance that already has names, records, vocabularies, etc ...
  2. Run invenio rdm rebuild-all-indices.
  3. Monitor logs for any PreconditionFailed or InvalidCommand errors over time.

Expected behavior

Tasks should complete without unhandled exceptions, even under resource limitations.

Screenshots (if applicable)

[2024-11-11 14:54:27,790: INFO/ForkPoolWorker-2] Task invenio_records_resources.tasks.manage_indexer_queues[90e690b2-e4de-42df-b161-b3fdaea7958d] succeeded in 0.01700770197203383s: None
[2024-11-11 14:54:37,773: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[8d10f48b-1329-4d4f-b8c1-1b08223d376e] received
[2024-11-11 14:54:37,796: INFO/ForkPoolWorker-2] Task invenio_records_resources.tasks.manage_indexer_queues[8d10f48b-1329-4d4f-b8c1-1b08223d376e] succeeded in 0.02198187098838389s: None
[2024-11-11 14:54:47,772: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[14913fda-f976-4e14-8aa5-4d2e1f4d4c98] received
[2024-11-11 14:54:47,790: INFO/ForkPoolWorker-2] Task invenio_records_resources.tasks.manage_indexer_queues[14913fda-f976-4e14-8aa5-4d2e1f4d4c98] succeeded in 0.017010738025419414s: None
[2024-11-11 14:54:57,772: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[05360362-87a1-4624-89d4-1b8d87cc140f] received
[2024-11-11 14:54:57,790: INFO/ForkPoolWorker-2] Task invenio_records_resources.tasks.manage_indexer_queues[05360362-87a1-4624-89d4-1b8d87cc140f] succeeded in 0.016327430959790945s: None
[2024-11-11 14:55:00,002: INFO/MainProcess] Task invenio_stats.tasks.process_events[cf738094-d174-4090-a6cb-7567263accb5] received
[2024-11-11 14:55:00,025: ERROR/ForkPoolWorker-2] Task invenio_stats.tasks.process_events[cf738094-d174-4090-a6cb-7567263accb5] raised unexpected: PreconditionFailed(406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 2 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more', (0, 0), '')
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 515, in channel
    return self.channels[channel_id]
           ~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask_celeryext/app.py", line 71, in __call__
    return Task.__call__(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/invenio_stats/tasks.py", line 43, in process_events
    results.append((event_name, processor.run()))
                                ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/invenio_stats/processors.py", line 223, in run
    return search.helpers.bulk(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/opensearchpy/helpers/actions.py", line 423, in bulk
    for ok, item in streaming_bulk(client, actions, ignore_status=ignore_status, *args, **kwargs):  # type: ignore
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/opensearchpy/helpers/actions.py", line 326, in streaming_bulk
    for bulk_data, bulk_actions in _chunk_actions(
                                   ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/opensearchpy/helpers/actions.py", line 167, in _chunk_actions
    for action, data in actions:
                        ^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/invenio_stats/processors.py", line 189, in actionsiter
    for msg in self.queue.consume():
               ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/invenio_queues/queue.py", line 109, in consume
    with self.create_consumer() as consumer:
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/invenio_queues/queue.py", line 97, in create_consumer
    with self.consumer(conn) as consumer:
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/invenio_queues/queue.py", line 75, in consumer
    return Consumer(
           ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kombu/compat.py", line 102, in __init__
    self.backend = connection.channel()
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kombu/connection.py", line 303, in channel
    chan = self.transport.create_channel(self.connection)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kombu/transport/pyamqp.py", line 168, in create_channel
    return connection.channel()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 518, in channel
    channel.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 293, in _on_close
    raise error_for_code(
amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 2 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
[2024-11-11 14:55:07,772: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[1df09154-c81a-49f2-9839-c9d801e3467d] received
[2024-11-11 14:55:07,789: INFO/ForkPoolWorker-2] Task invenio_records_resources.tasks.manage_indexer_queues[1df09154-c81a-49f2-9839-c9d801e3467d] succeeded in 0.01606845302740112s: None
[2024-11-11 14:55:17,773: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[d00be27d-0536-4df2-aa61-9973a23d1e64] received
[2024-11-11 14:55:17,789: INFO/ForkPoolWorker-2] Task invenio_records_resources.tasks.manage_indexer_queues[d00be27d-0536-4df2-aa61-9973a23d1e64] succeeded in 0.015239176980685443s: None
[2024-11-11 14:55:27,773: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[1f2cbea9-21c9-4a86-be3f-c38588051a43] received
[2024-11-11 14:55:27,789: INFO/ForkPoolWorker-2] Task invenio_records_resources.tasks.manage_indexer_queues[1f2cbea9-21c9-4a86-be3f-c38588051a43] succeeded in 0.015507553005591035s: None
[2024-11-11 14:55:37,773: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[c8edf51e-6ac4-48ca-937f-0dc897991a38] received
[2024-11-11 14:55:37,792: INFO/MainProcess] Task invenio_indexer.tasks.process_bulk_queue[fadd883b-77c5-4a59-a3db-87f915face6c] received
[2024-11-11 14:55:37,803: INFO/ForkPoolWorker-2] Task invenio_records_resources.tasks.manage_indexer_queues[c8edf51e-6ac4-48ca-937f-0dc897991a38] succeeded in 0.029242602991871536s: None
[2024-11-11 14:55:37,809: ERROR/ForkPoolWorker-1] Task invenio_indexer.tasks.process_bulk_queue[fadd883b-77c5-4a59-a3db-87f915face6c] raised unexpected: PreconditionFailed(406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 2 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more', (0, 0), '')
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 515, in channel
    return self.channels[channel_id]
           ~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask_celeryext/app.py", line 71, in __call__
    return Task.__call__(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/invenio_indexer/tasks.py", line 33, in process_bulk_queue
    indexer.process_bulk_queue(search_bulk_kwargs=search_bulk_kwargs)
  File "/usr/local/lib/python3.12/site-packages/invenio_indexer/api.py", line 281, in process_bulk_queue
    consumer = Consumer(
               ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kombu/compat.py", line 102, in __init__
    self.backend = connection.channel()
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kombu/connection.py", line 303, in channel
    chan = self.transport.create_channel(self.connection)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kombu/transport/pyamqp.py", line 168, in create_channel
    return connection.channel()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 518, in channel
    channel.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 292, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 177, in _do_revive
    self.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 293, in _on_close
    raise error_for_code(
amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 2 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
[2024-11-11 14:55:47,773: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[367cd240-c575-446c-97c4-80bb56d7e67f] received
[2024-11-11 14:55:47,787: INFO/MainProcess] Task invenio_indexer.tasks.process_bulk_queue[76d8c737-20d5-49a8-8166-98e8b92d7f20] received
[2024-11-11 14:55:47,793: ERROR/ForkPoolWorker-1] Task invenio_indexer.tasks.process_bulk_queue[76d8c737-20d5-49a8-8166-98e8b92d7f20] raised unexpected: InvalidCommand(503, 'COMMAND_INVALID - unimplemented method', (20, 41), 'Channel.close_ok')
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 515, in channel
    return self.channels[channel_id]
           ~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask_celeryext/app.py", line 71, in __call__
    return Task.__call__(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/invenio_indexer/tasks.py", line 33, in process_bulk_queue
    indexer.process_bulk_queue(search_bulk_kwargs=search_bulk_kwargs)
  File "/usr/local/lib/python3.12/site-packages/invenio_indexer/api.py", line 281, in process_bulk_queue
    consumer = Consumer(
               ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kombu/compat.py", line 102, in __init__
    self.backend = connection.channel()
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kombu/connection.py", line 303, in channel
    chan = self.transport.create_channel(self.connection)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kombu/transport/pyamqp.py", line 168, in create_channel
    return connection.channel()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 518, in channel
    channel.open()
  File "/usr/local/lib/python3.12/site-packages/amqp/channel.py", line 448, in open
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.12/site-packages/amqp/connection.py", line 668, in _on_close
    raise error_for_code(reply_code, reply_text,
amqp.exceptions.InvalidCommand: Channel.close_ok: (503) COMMAND_INVALID - unimplemented method
[2024-11-11 14:55:47,801: INFO/ForkPoolWorker-2] Task invenio_records_resources.tasks.manage_indexer_queues[367cd240-c575-446c-97c4-80bb56d7e67f] succeeded in 0.027210702013690025s: None
[2024-11-11 14:55:57,773: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[85807ee8-7eb0-452f-9de1-95d4c081c463] received
[2024-11-11 14:55:57,783: INFO/MainProcess] Task invenio_indexer.tasks.process_bulk_queue[c00fe135-1c06-4620-abb9-4de905c47190] received
[2024-11-11 14:55:57,793: INFO/ForkPoolWorker-2] Task invenio_records_resources.tasks.manage_indexer_queues[85807ee8-7eb0-452f-9de1-95d4c081c463] succeeded in 0.01827888801926747s: None
[2024-11-11 14:55:57,869: INFO/ForkPoolWorker-1] POST http://invenio-opensearch:9200/_bulk [status:200 request:0.065s]
[2024-11-11 14:55:57,873: INFO/ForkPoolWorker-1] Task invenio_indexer.tasks.process_bulk_queue[c00fe135-1c06-4620-abb9-4de905c47190] succeeded in 0.08873315097298473s: None
[2024-11-11 14:56:07,773: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[6c3970d4-646d-4b7e-b5d1-5a65780092cf] received
@Samk13 Samk13 added the bug Something isn't working label Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Samk13