Email Timeouts on Request Form Submit

[kevinreiss]

Expected behavior

When the request form is submitted the system sends one or more email messages out to the person submitting the request and staff members who process the request (this varies on the type of request).

Actual behavior

Requests that utilize APIs are still being placed correctly but emails associated with these requests can fail.

Steps to replicate

Seems to randomly happen upon form submit. Difficult to replicate.

Impact of this bug

E.g. "I can't work until this is fixed" or "I have a workaround"

Honeybadger link and code snippet, if applicable

Example: https://app.honeybadger.io/projects/54399/faults/112537713

Net::ReadTimeout: Net::ReadTimeout with #<TCPSocket:(closed)>

Implementation Notes

Some months ago we tried adjusting the timeout on mail actions: https://github.com/pulibrary/orangelight/blame/d77257f966d024e8b53a01bc8329b911fb9d31ad/config/environments/production.rb#L86. We could up this, but to make this process more robust we need to move to a mail strategy that involves retrying messages that fail. This commit in pulfalight could be example to model to do that: pulibrary/pulfalight#1366.

[acozine]

@maxkadel and I looked at the logs on lib-ponyexpr-prod today. The /var/log/mail.err file is empty, but we saw this error in /var/log/mail.log:

postfix/smtpd[52650]: lost connection after CONNECT from catalog2.princeton.edu

The timing seems to correspond to the Honeybadger errors. I'm not sure this gets us much closer to a solution, but it may help us narrow in on the problem. The error happens with all 3 catalog prod VMs; no pattern there.

[rladdusaw]

Sending emails with send_later requires anything passed into the email to be a serializable object. I'm working on creating a custom serializer for the Submission object to make this work.

[maxkadel]

@kevinreiss will follow up with @rladdusaw on the status of this ticket.

[rladdusaw]

What I got stuck on was implementing the deserializer, it requires a Patron object but there doesn't seem to be a good way to instantiate one without a User, which requires a token. I tried adding a method to get the user by netid but the test user wasn't in the database. I think adding a similar method and mocking its return value should work.

The serialize method is missing some values like the bib, but that should be straightforward.

The last thing I can think of is I modified one of the email methods to take a hash instead of the Submissions::Clancy object to avoid have to serialize it as well, passing it an array of error hashes. It might be better to do a serializer there as well. Any code calling that email method will need to be updated to supply the required hash.

[kevinreiss]

I created a draft PR to see the failing specs related to the branch.