Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash when transaction can't complete #811

Closed
deric opened this issue Sep 12, 2024 · 3 comments
Closed

Crash when transaction can't complete #811

deric opened this issue Sep 12, 2024 · 3 comments
Labels
crash
Milestone
1.3.0

Comments

@deric
Copy link

deric commented Sep 12, 2024

Transaction failure results in service crash

pq: could not serialize access due to read/write dependencies among transactions
                                                         can't perform "INSERT INTO \"icingadb_instance\" (\"icinga2_performance_data_enabled\", \"responsible\", \"icinga2_version\", \"icinga2_event_handlers_enabled\", \"endpoint_id\", \"heartbeat\", \"id\", \"icinga2_start_time\", \"icinga2_flap_detection_enabled\", \"icinga2_active_host_checks_enabled\", \"environment_id\", \"icinga2_active_service_checks_enabled\", \"icinga2_notifications_enabled\") VALUES (:icinga2_performance_data_enabled,:responsible,:icinga2_version,:icinga2_event_handlers_enabled,:endpoint_id,:heartbeat,:id,:icinga2_start_time,:icinga2_flap_detection_enabled,:icinga2_active_host_checks_enabled,:environment_id,:icinga2_active_service_checks_enabled,:icinga2_notifications_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \"icinga2_performance_data_enabled\" = EXCLUDED.\"icinga2_performance_data_enabled\",\"responsible\" = EXCLUDED.\"responsible\",\"icinga2_version\" = EXCLUDED.\"icinga2_version\",\"icinga2_event_handlers_enabled\" = EXCLUDED.\"icinga2_event_handlers_enabled\",\"endpoint_id\" = EXCLUDED.\"endpoint_id\",\"heartbeat\" = EXCLUDED.\"heartbeat\",\"id\" = EXCLUDED.\"id\",\"icinga2_start_time\" = EXCLUDED.\"icinga2_start_time\",\"icinga2_flap_detection_enabled\" = EXCLUDED.\"icinga2_flap_detection_enabled\",\"icinga2_active_host_checks_enabled\" = EXCLUDED.\"icinga2_active_host_checks_enabled\",\"environment_id\" = EXCLUDED.\"environment_id\",\"icinga2_active_service_checks_enabled\" = EXCLUDED.\"icinga2_active_service_checks_enabled\",\"icinga2_notifications_enabled\" = EXCLUDED.\"icinga2_notifications_enabled\""
                                                         github.com/icinga/icingadb/internal.CantPerformQuery
                                                                 github.com/icinga/icingadb/internal/internal.go:30
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).realize.func1
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:368
                                                         github.com/icinga/icingadb/pkg/retry.WithBackoff
                                                                 github.com/icinga/icingadb/pkg/retry/retry.go:60
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).realize
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:284
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:221
                                                         runtime.goexit
                                                                 runtime/asm_amd64.s:1695
                                                         can't perform "INSERT INTO \"icingadb_instance\" (\"icinga2_flap_detection_enabled\", \"icinga2_active_host_checks_enabled\", \"environment_id\", \"icinga2_active_service_checks_enabled\", \"icinga2_notifications_enabled\", \"icinga2_event_handlers_enabled\", \"icinga2_performance_data_enabled\", \"responsible\", \"icinga2_version\", \"endpoint_id\", \"heartbeat\", \"id\", \"icinga2_start_time\") VALUES (:icinga2_flap_detection_enabled,:icinga2_active_host_checks_enabled,:environment_id,:icinga2_active_service_checks_enabled,:icinga2_notifications_enabled,:icinga2_event_handlers_enabled,:icinga2_performance_data_enabled,:responsible,:icinga2_version,:endpoint_id,:heartbeat,:id,:icinga2_start_time) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \"icinga2_flap_detection_enabled\" = EXCLUDED.\"icinga2_flap_detection_enabled\",\"icinga2_active_host_checks_enabled\" = EXCLUDED.\"icinga2_active_host_checks_enabled\",\"environment_id\" = EXCLUDED.\"environment_id\",\"icinga2_active_service_checks_enabled\" = EXCLUDED.\"icinga2_active_service_checks_enabled\",\"icinga2_notifications_enabled\" = EXCLUDED.\"icinga2_notifications_enabled\",\"icinga2_event_handlers_enabled\" = EXCLUDED.\"icinga2_event_handlers_enabled\",\"icinga2_performance_data_enabled\" = EXCLUDED.\"icinga2_performance_data_enabled\",\"responsible\" = EXCLUDED.\"responsible\",\"icinga2_version\" = EXCLUDED.\"icinga2_version\",\"endpoint_id\" = EXCLUDED.\"endpoint_id\",\"heartbeat\" = EXCLUDED.\"heartbeat\",\"id\" = EXCLUDED.\"id\",\"icinga2_start_time\" = EXCLUDED.\"icinga2_start_time\"": pq: could not serialize access due to read/write dependencies among transactions
                                                         github.com/icinga/icingadb/pkg/retry.WithBackoff
                                                                 github.com/icinga/icingadb/pkg/retry/retry.go:75
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).realize
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:284
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:221
                                                         runtime.goexit
                                                                 runtime/asm_amd64.s:1695
                                                         HA aborted
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).abort.func1
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:131
                                                         sync.(*Once).doSlow
                                                                 sync/once.go:74
                                                         sync.(*Once).Do
                                                                 sync/once.go:65
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).abort
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:129
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:233
                                                         runtime.goexit
                                                                 runtime/asm_amd64.s:1695
                                                         HA exited with an error
                                                         main.run
                                                                 github.com/icinga/icingadb/cmd/icingadb/main.go:335
                                                         main.main
                                                                 github.com/icinga/icingadb/cmd/icingadb/main.go:37
                                                         runtime.main
                                                                 runtime/proc.go:271
                                                         runtime.goexit
                                                                 runtime/asm_amd64.s:1695
icingadb.service: Main process exited, code=exited, status=1/FAILURE
icingadb.service: Failed with result 'exit-code'.
  • Icinga DB version: v1.2.0
  • Icinga 2 version: 2.14.2
  • Operating System and version: Debian 12
@oxzi oxzi added the crash label Sep 13, 2024
@oxzi
Copy link
Member

oxzi commented Sep 13, 2024

Thanks for your bug report. Such an issue should be retried, as being addressed in Icinga/icinga-go-library#59. However, based on your stack trace, I am unsure if it wasn't even retried. Could you please supply more logs, best at least the last 30 minutes prior to the crash?

@deric
Copy link
Author

deric commented Sep 13, 2024

Sure, the server was overloaded I have upgrade it. However icingadb shouldn't be crashing...

Thu 2024-09-12 17:03:24 UTC icingadb[1121651]: runtime-updates: Upserted 10 ServiceState items
Thu 2024-09-12 17:03:39 UTC icingadb[1121651]: overdue-sync: Synced 44 service overdue indicators
Thu 2024-09-12 17:03:59 UTC icingadb[1121651]: overdue-sync: Synced 19 service overdue indicators
Thu 2024-09-12 17:04:08 UTC icingadb[1121651]: heartbeat: Lost Icinga heartbeat
Thu 2024-09-12 17:04:08 UTC icingadb[1121651]: high-availability: Lost heartbeat
Thu 2024-09-12 17:04:08 UTC icingadb[1121651]: Handing over
Thu 2024-09-12 17:05:08 UTC icingadb[1121651]: heartbeat: Waiting for Icinga heartbeat
Thu 2024-09-12 17:05:38 UTC icingadb[1121651]: heartbeat: Received Icinga heartbeat
Thu 2024-09-12 17:05:38 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:38 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:39 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:41 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:41 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:44 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:44 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:46 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:48 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:51 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:53 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:06:53 UTC icingadb[1121651]: high-availability: Instance updated/inserted successfully after error
Thu 2024-09-12 17:06:57 UTC icingadb[1121651]: history-sync: Synced 1 state history items
Thu 2024-09-12 17:08:05 UTC icingadb[1121651]: pq: could not serialize access due to read/write dependencies among transactions
                                               can't perform "INSERT INTO \"icingadb_instance\" (\"icinga2_performance_data_enabled\", \"responsible\", \"icinga2_version\", \"icinga2_event_hand>
                                               github.com/icinga/icingadb/internal.CantPerformQuery
                                                       github.com/icinga/icingadb/internal/internal.go:30
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).realize.func1
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:368
                                               github.com/icinga/icingadb/pkg/retry.WithBackoff
                                                       github.com/icinga/icingadb/pkg/retry/retry.go:60
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).realize
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:284
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:221
                                               runtime.goexit
                                                       runtime/asm_amd64.s:1695
                                               can't perform "INSERT INTO \"icingadb_instance\" (\"icinga2_flap_detection_enabled\", \"icinga2_active_host_checks_enabled\", \"environment_id\", >
                                               github.com/icinga/icingadb/pkg/retry.WithBackoff
                                                       github.com/icinga/icingadb/pkg/retry/retry.go:75
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).realize
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:284
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:221
                                               runtime.goexit
                                                       runtime/asm_amd64.s:1695
                                               HA aborted
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).abort.func1
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:131
                                               sync.(*Once).doSlow
                                                       sync/once.go:74
                                               sync.(*Once).Do
                                                       sync/once.go:65
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).abort
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:129
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:233
                                               runtime.goexit
                                                       runtime/asm_amd64.s:1695
                                               HA exited with an error
                                               main.run
                                                       github.com/icinga/icingadb/cmd/icingadb/main.go:335
                                               main.main
                                                       github.com/icinga/icingadb/cmd/icingadb/main.go:37
                                               runtime.main
                                                       runtime/proc.go:271
                                               runtime.goexit
                                                       runtime/asm_amd64.s:1695
Thu 2024-09-12 17:08:05 UTC systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Thu 2024-09-12 17:08:05 UTC systemd[1]: icingadb.service: Failed with result 'exit-code'.
Thu 2024-09-12 17:08:05 UTC systemd[1]: icingadb.service: Consumed 3min 35.643s CPU time.
Thu 2024-09-12 17:24:38 UTC systemd[1]: Starting icingadb.service - Icinga DB...
Thu 2024-09-12 17:24:38 UTC icingadb[1184381]: Starting Icinga DB daemon (v1.2.0)
Thu 2024-09-12 17:24:38 UTC systemd[1]: Started icingadb.service - Icinga DB.
Thu 2024-09-12 17:24:38 UTC icingadb[1184381]: Connecting to database at 'icm.de.recombee.net:0'
Thu 2024-09-12 17:24:38 UTC icingadb[1184381]: Connecting to Redis at 'localhost:6380'
Thu 2024-09-12 17:24:38 UTC icingadb[1184381]: Starting history sync
Thu 2024-09-12 17:24:39 UTC icingadb[1184381]: heartbeat: Received Icinga heartbeat
Thu 2024-09-12 17:24:39 UTC icingadb[1184381]: high-availability: Another instance is active
Thu 2024-09-12 17:24:58 UTC icingadb[1184381]: history-sync: Synced 31 notification history items
Thu 2024-09-12 17:24:58 UTC icingadb[1184381]: history-sync: Synced 2745 state history items
Thu 2024-09-12 17:25:18 UTC icingadb[1184381]: history-sync: Synced 24 state history items
Thu 2024-09-12 17:25:38 UTC icingadb[1184381]: history-sync: Synced 2 notification history items
Thu 2024-09-12 17:25:38 UTC icingadb[1184381]: history-sync: Synced 79 state history items
Thu 2024-09-12 17:25:58 UTC icingadb[1184381]: history-sync: Synced 18 state history items
Thu 2024-09-12 17:26:18 UTC icingadb[1184381]: history-sync: Synced 80 state history items
Thu 2024-09-12 17:26:18 UTC icingadb[1184381]: history-sync: Synced 1 notification history items
Thu 2024-09-12 17:26:38 UTC icingadb[1184381]: history-sync: Synced 61 state history items
Thu 2024-09-12 17:26:38 UTC icingadb[1184381]: history-sync: Synced 2 notification history items
Thu 2024-09-12 17:26:58 UTC icingadb[1184381]: history-sync: Synced 47 state history items
Thu 2024-09-12 17:27:18 UTC icingadb[1184381]: history-sync: Synced 1 notification history items
Thu 2024-09-12 17:27:18 UTC icingadb[1184381]: history-sync: Synced 25 state history items
Thu 2024-09-12 17:27:38 UTC icingadb[1184381]: history-sync: Synced 36 state history items
Thu 2024-09-12 17:27:58 UTC icingadb[1184381]: history-sync: Synced 4 notification history items
Thu 2024-09-12 17:27:58 UTC icingadb[1184381]: history-sync: Synced 37 state history items
Thu 2024-09-12 17:28:18 UTC icingadb[1184381]: history-sync: Synced 34 state history items
Thu 2024-09-12 17:28:38 UTC icingadb[1184381]: history-sync: Synced 1 notification history items
Thu 2024-09-12 17:28:38 UTC icingadb[1184381]: history-sync: Synced 51 state history items
Thu 2024-09-12 17:28:58 UTC icingadb[1184381]: history-sync: Synced 53 state history items
Thu 2024-09-12 17:29:18 UTC icingadb[1184381]: history-sync: Synced 62 state history items
Thu 2024-09-12 17:29:38 UTC icingadb[1184381]: high-availability: Another instance is active
Thu 2024-09-12 17:29:38 UTC icingadb[1184381]: history-sync: Synced 78 state history items
Thu 2024-09-12 17:29:58 UTC icingadb[1184381]: history-sync: Synced 56 state history items
Thu 2024-09-12 17:30:10 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:11 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:14 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:15 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:15 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:17 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:18 UTC icingadb[1184381]: history-sync: Synced 2 notification history items
Thu 2024-09-12 17:30:18 UTC icingadb[1184381]: history-sync: Synced 27 state history items
Thu 2024-09-12 17:30:18 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:19 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:05:44 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:44 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:46 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:48 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:51 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:53 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:06:53 UTC icingadb[1121651]: high-availability: Instance updated/inserted successfully after error
Thu 2024-09-12 17:06:57 UTC icingadb[1121651]: history-sync: Synced 1 state history items

@yhabteab
Copy link
Member

Should be fixed by Icinga/icinga-go-library#59

@yhabteab yhabteab added this to the 1.3.0 milestone Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
crash
Projects
None yet
Milestone
1.3.0
Development

No branches or pull requests
3 participants
@deric @oxzi @yhabteab