fix: Persist inbound stripe webhooks

[ancorcruz]

Context

Currently, when our application receives webhooks from third-party services (e.g., Stripe, GoCardless, Adyen), their payloads are immediately queued in Sidekiq for background processing. While this approach works in most cases, it introduces a critical reliability issue, especially in self-hosted environments where Sidekiq may lose jobs if workers are abruptly terminated or overloaded.

This unreliability has led to issues like pending transactions that are never resolved despite receiving valid webhooks. For example, a reported issue involved a wallet credit purchase where the payment settled on Stripe, the webhook was received, but the transaction remained pending indefinitely until manually resolved.

To address this, we aim to ensure that no webhook payload is lost and processing remains consistent, even under adverse conditions.

Description

This PR introduces a robust mechanism for handling inbound webhooks by persisting them in the database before processing. Key changes include:

Database Persistence:

• Webhooks are now stored in a new inbound_webhooks table immediately upon receipt. This ensures all webhook payloads are safely persisted in a reliable system before processing.

Background Processing:

• A new InboundWebhooks::CreateService is introduced to persist the webhook and enqueue it for processing.
• A new InboundWebhooks::ProcessJob Sidekiq worker processes the persisted webhooks, invoking the InboundWebhooks::ProcessService
• A new InboundWebhooks::ProcessService processes the persisted webhooks invoking the appropriate payment provider service.

Retry Mechanisms:

• Failed webhooks are marked as such and can be retried manually or automatically via a clock job.
• A cleanup job is introduced to remove processed webhook records older than 1 month.

Controller Update:

• The WebhooksController actions (stripe, adyen, and gocardless) now use the InboundWebhooks::CreateService and respond with 200 OK once the webhook is persisted successfully.

Backward Compatibility:

• No webhooks will be missed during the transition. The change is designed to integrate seamlessly into both cloud and self-hosted environments.

Key Changes

• Added InboundWebhook ActiveRecord model to persist webhook payloads with fields such as source, event_type, payload, status.
• Implemented InboundWebhooks::CreateService to handle webhook persistence and Sidekiq job enqueueing.
• Added InboundWebhooks::ProcessJob to process persisted webhooks from Sidekiq queue.
• Added InboundWebhooks::ProcessService to call the respective provider's handle incoming webhook service.
• Updated PaymentProviders::Stripe::HandleIncomingWebhookService to process webhooks from the database instead of relying solely on direct Sidekiq queues.
• Updated WebhooksController#stripe action to integrate with the new service and workflow.
• Introduced a clock job for:

1. Retrying pending webhooks older than 2 hours.
2. Cleaning up processed webhooks older than 90 days.

Benefits

Reliability: Ensures no webhook is lost, even if Sidekiq workers fail.
Resilience: Provides a retry mechanism for failed webhook processing.
Transparency: Makes webhook processing traceable with statuses (pending, processing, processed, failed).

[vincent-pochet]

🙌

[vincent-pochet]

It looks like we are not relying on transaction right? So I guess this after_commit block is not mandatory

[ancorcruz]

this ensures the job is not added to the queue and tried to be processed until the DB transaction is committed. This will avoid race condition between sidekiq and postgres where sidekiq picks the job but it is unable to find the resource in DB.

[vincent-pochet]

I agree, but here InboundWebhooks::CreateService is called directly from the controller and no transaction is created in it so the job will always be called after the InboundWebhook creation transaction.
That's why I was asking. No a major issue anyway ;)

[nudded]

without indexes on these columns, this will be a full table scan always 🤔

[ancorcruz]

add two combined indices [status, processing_at] and [status, created_at] should do it.