Unisender Go: new ESP #352

Arondit · 2024-01-10T15:39:38Z

#349

…changes

…nder Go

medmunds · 2024-01-10T19:48:27Z

Thanks for this!

I'll take a closer look over the weekend

Arondit · 2024-01-12T09:25:15Z

Sorry, this time it must pass.

medmunds · 2024-01-12T20:24:38Z

Sorry, I don't know why it requires me to re-approve the tests workflow every time.

You can run the tests on your own machine with python runtests.py in the project root. More options: https://anymail.dev/en/stable/contributing/#testing

Incidentally, I'm trying to get an anymail.dev subdomain set up in a Unisender Go account to run integration tests, but they seem to have some problems validating subdomains. (And in general, I'm seeing a lot of 50x
and 429 errors just trying to navigate their dashboard—though I suppose that might be some sort of geopolitical filtering issue, since I'm connecting from the U.S.)

Arondit · 2024-01-16T11:23:25Z

I connected the service through U.S. VPN, but haven't got any errors. I am not sure, if the problems is somehow related with geopolitical issues. Tell me pls, how can i help you to test the service? If it matters, I already use all this code in production with >1000 emails per day. It just lays in project directory, which is not very scalable.

Also, if the problem remains, i can ask service tech support to help somehow.

medmunds

Thanks again for this PR!

I've added several comments/requests. Most of them are about making things more consistent with other Anymail backends.

CHANGELOG.rst

anymail/backends/unisender_go.py

anymail/webhooks/unisender_go.py

tests/test_unisender_go_payload.py

docs/esps/index.rst

anymail/webhooks/unisender_go.py

medmunds · 2024-01-17T01:18:48Z

I did manage to get an Anymail test domain set up in Unisender Go. (Their dashboard seems to be working fine now; maybe they were just having a bad day last week.)

I'll try to put together some live integration tests.

Arondit · 2024-01-19T15:47:19Z

Hello again. Thank you for comments, it will take a bit of time to fix them all. They also require adding new tests for extra parameters, which i did not include before. I will fix them all and update PR on Monday.

…tions

Arondit · 2024-01-24T17:22:10Z

I don't really understand now, what went wrong again. Locally everything works fine. Also the error is weird, AttributeError: module 'importlib' has no attribute 'util'. But python's had importlib.util since 3.3. I have no idea, how i could break it.
Sorry for so much mess-up.

medmunds · 2024-01-24T21:43:38Z

@Arondit the 'importlib' has no attribute 'util' error isn't a problem in your code, it's a problem in the hatch packaging tool that Anymail uses. One of hatch's dependencies changed out from under it a few ~~days~~ hours ago, in a way that surfaced this bug in hatch. (We run weekly builds to catch problems like this; tomorrow's test run would have noticed it, but your PR got there first.)

There's a PR just opened in hatch to fix the problem, but if that doesn't make it into a release soon, I'll try to update Anymail to work around it.

Until then, if the tests run fine on your local machine, that should be good.

medmunds · 2024-01-25T17:15:39Z

Tests are working again. (Bug in hatch was fixed and a patch released last night.)

Arondit · 2024-01-29T12:25:31Z

Hello again. So, tests are working, all checks are passed and comments are closed. Is there anything else I should do? Also, did it work with integration tests for this ESP?

medmunds · 2024-01-29T21:37:33Z

There are still a handful of unresolved change requests from earlier. (They might be hiding behind a not-so-helpful GitHub "load more" link like this

in the discussion above.)

The two most important ones are handling for cc and bcc recipients (which I think will currently overwrite the to list), and parsing the send status out of Unisender Go's API response.

Also, while the unit tests are helpful, it might be a good idea to also implement at least one or two basic functional tests using the RequestsBackendMockAPITestCase, to verify that the API key, url, and basic payload structure are as expected—and more importantly, that future changes don't break that. (Here's an example from the Postmark tests. And another that would have caught the cc issue.)

medmunds · 2024-02-12T19:44:17Z

@Arondit I'd be happy to work on resolving the remaining handful of issues from the original review, unless you are already working on them. Just let me know.

Also, do you know if it's true that Unisender Go does not support "cc" or "bcc"? (Or did I miss something in their docs?)

Arondit · 2024-02-19T08:18:15Z

@medmunds i'm working on it, just was busy at work those days, sorry for 2 weeks of silence.

About "bcc" and "cc". Yes, you're right. They don't support it through their WEB API. There is a possibility to use SMTP API instead, which supports "cc" and "bcc" in a strict mode. Is it a problem?

medmunds · 2024-02-19T18:40:19Z

@Arondit no worries, and no hurry, just didn't want to conflict with any work you might be doing.

On cc/bcc, I was asking related to this review comment. We can just handle cc/bcc as unsupported features. (Rather than trying to use their SMTP API. Anymail's current design doesn't really work with SMTP.)

Arondit · 2024-02-20T10:20:47Z

Fix all missed comments (yeah, you're right, GitHub've hidden them from me).

medmunds · 2024-02-22T00:28:51Z

Thanks, it's looking great. I have a few things I wanted to double check, and then can get this merged.

I added some live integration tests. If you want to try them locally:

export ANYMAIL_TEST_UNISENDER_GO_API_KEY="your-api-key"
export ANYMAIL_TEST_UNISENDER_GO_API_URL="https://go1.unisender.ru/ru/transactional/api/v1"  # or "go2"
export ANYMAIL_TEST_UNISENDER_GO_DOMAIN="your.validated.sending.domain"
# optional:
export ANYMAIL_TEST_UNISENDER_GO_TEMPLATE_ID="id of email template in your account"

python runtests.py tests.test_unisender_go_integration

(The integration tests send actual email to an anymail.dev null mailbox, so each run will use up about five sends from your quota.)

…datetime format)

Arondit · 2024-02-22T20:59:01Z

After small fixes tests worked. Thank you. Also, '/' in the end of ANYMAIL_TEST_UNISENDER_GO_API_URL
appeared to be significant.

- Make "/" at end of api_url optional (so URLs in Unisender Go API docs work) - Map Unisender Go "failed_emails" status to normalized AnymailRecipientStatus status codes - Correctly handle Unisender Go's response for duplicate emails - Remove unused code - Don't send unnecessary API payload fields - Fix amp-html payload construction - Fix attachment "name" field for inline images (must use cid) - Support per-recipient merge_metadata - Format send_at per Unisender Go docs - Handle boolean values for track_clicks, track_opens - Don't duplicate Unisender Go's restrictions on attachment filenames (their API error message is adequate)

- Most errors detected by the receiving email server should use RejectReason.BOUNCED. (INVALID is for address _format_ errors. BLOCKED is for Unisender Go policies.) - No need to modify ESP payload data by adding esp_name. - Most missing AnymailTrackingEvent params should be set to None. Missing metadata must be set to an empty dict.

In recipient metadata, change name of generated tracking id from "message_id" to "anymail_id" to reduce chance of conflicts with caller metadata. (And for consistency with same feature in SendGrid backend.) In webhooks, remove "anymail_id" from metadata before passing to signal handler.

When not generating unique per-recipient message ids, use Unisender Go's "job_id" as Anymail's `message_id` for all recipients. The same job_id is provided in the send API response and tracking webhooks.

medmunds · 2024-02-29T23:12:41Z

Fixed a handful of issues and consistency with other ESP backends.

I just want to update the docs a little and then will merge.

medmunds · 2024-03-01T00:46:26Z

Hmm, it looks like cc and bcc are possible through the web API: https://godocs.unisender.ru/cc-and-bcc. (I think they just updated the API docs—I don't remember seeing that a few weeks ago.)

Basically, we'd need to generate an entry in "recipients" for each of the to, cc, and bcc emails, and then construct "To" and "CC" headers that list all of their respective addresses. Unisender Go (effectively) just treats those headers as text to include verbatim in the generated message, without trying to parse them.

There might be some tricky cases to consider when using Anymail's batch sending mode. (I think for batch send we just skip the common "To" header, but keep the common "CC" header. But need to think about this a little more.)

Also we'd want to test whether non-ASCII display names require special handling.

Match method order used by superclass, other ESP backends.

- Parsing bug when to/cc header contains quoted display name with comma ('"ExampleCo, Ltd." <[email protected]>') - `reply_to_name` not properly quoted (like `to_name` and `from_name` are)

Arondit · 2024-03-04T15:58:07Z

Add one unit test and fix integration test (bcc address needs to be sent on approved domain). Also thank you very much for work around added cc/bcc feature. I started to do the same today earlier and noticed, you'd already done it, but better. I guess, now it is ready to merge, isn't it?

medmunds · 2024-03-05T02:33:17Z

Good catch on the integration test, thanks.

I emailed Unisender Go tech support earlier today about the display-name quoting bugs and a couple other issues. Hoping to hear back from them once they get into the office, but if not I'll go ahead and merge this with the workarounds.

- Fix capitalization of "Unisender Go" - Alphabetize ESPs in description

Unisender Go tech support says not only commas, but also < > @ are prohibited in To and Cc header display names, even if properly quoted.

medmunds · 2024-03-05T19:52:27Z

@Arondit OK, this is merged and will be in the next release!

Thanks again for all your work on this.

medmunds · 2024-03-12T21:20:06Z

(Released in Anymail 10.3)

Arondit · 2024-03-15T15:21:59Z

nice, thank you

Arondit added 5 commits January 10, 2024 18:22

Unisender Go: Implement backend for sending emails

c4ca211

Unisender Go: Implement webhooks for getting info about email status …

009a47a

…changes

Unisender Go: Test backend and webhook

624ba44

Unisender Go: Add info and infrastructure details about new ESP Unise…

6c30ede

…nder Go

Unisender Go: Write docs about new ESP

c10da29

medmunds added new ESP esp:Unisender Go labels Jan 10, 2024

Arondit added 3 commits January 11, 2024 14:01

Unisender Go: Fix style problems

77d414b

Unisender Go: Fix failing test

ef75750

Unisender Go: Fix failing test and fix docs typo

441c884

medmunds requested changes Jan 17, 2024

View reviewed changes

Arondit added 4 commits January 22, 2024 18:28

Unisender Go: Fix docs inconsistency and add more info about extra op…

9d8974d

…tions

Unisender Go: Add fields to backend, fix typos, test changes

80f382f

Unisender Go: Fix esp_extra parameters getting (use update_deep)

e707ec3

Merge branch 'main' into feature/unisender-esp

3a6f13a

Merge branch 'main' into feature/unisender-esp

819efb8

Add live integration tests

03c064b

medmunds approved these changes Feb 22, 2024

View reviewed changes

Unisender Go: Fix integration tests (not support merge_metadata, fix …

a755ece

…datetime format)

medmunds added 5 commits February 25, 2024 16:47

Revert a755ece

8dcc27b

Use job_id as message_id if generation disabled

7d6e98b

When not generating unique per-recipient message ids, use Unisender Go's "job_id" as Anymail's `message_id` for all recipients. The same job_id is provided in the send API response and tracking webhooks.

medmunds and others added 7 commits March 2, 2024 11:05

Rearrange backend methods for consistency

ebec01f

Match method order used by superclass, other ESP backends.

Support cc/bcc

80d870f

Merge branch 'main' into fork/feature/unisender-esp

2c02202

Update docs

ed25682

Provide variables to integration test workflow

036ef24

Workaround Unisender Go display-name bugs

58aab53

- Parsing bug when to/cc header contains quoted display name with comma ('"ExampleCo, Ltd." <[email protected]>') - `reply_to_name` not properly quoted (like `to_name` and `from_name` are)

Unisender Go: Fix integrations bcc test and add unit test for cc/bcc

5c5f926

medmunds added 3 commits March 4, 2024 18:39

pyproject.toml cosmetic tweaks

2435af9

- Fix capitalization of "Unisender Go" - Alphabetize ESPs in description

Add unisender_go to integration workflow

dd47e56

Update display-name workaround

8d40a36

Unisender Go tech support says not only commas, but also < > @ are prohibited in To and Cc header display names, even if properly quoted.

medmunds merged commit a71a0d9 into anymail:main Mar 5, 2024
58 checks passed

medmunds mentioned this pull request Mar 5, 2024

Add support to Unisender Go #349

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unisender Go: new ESP #352

Unisender Go: new ESP #352

Arondit commented Jan 10, 2024 •

edited

Loading

medmunds commented Jan 10, 2024

Arondit commented Jan 12, 2024

medmunds commented Jan 12, 2024

Arondit commented Jan 16, 2024 •

edited

Loading

medmunds left a comment

medmunds commented Jan 17, 2024

Arondit commented Jan 19, 2024

Arondit commented Jan 24, 2024

medmunds commented Jan 24, 2024 •

edited

Loading

medmunds commented Jan 25, 2024

Arondit commented Jan 29, 2024

medmunds commented Jan 29, 2024

medmunds commented Feb 12, 2024

Arondit commented Feb 19, 2024

medmunds commented Feb 19, 2024

Arondit commented Feb 20, 2024

medmunds commented Feb 22, 2024

Arondit commented Feb 22, 2024

medmunds commented Feb 29, 2024

medmunds commented Mar 1, 2024

Arondit commented Mar 4, 2024

medmunds commented Mar 5, 2024

medmunds commented Mar 5, 2024

medmunds commented Mar 12, 2024

Arondit commented Mar 15, 2024

Unisender Go: new ESP #352

Unisender Go: new ESP #352

Conversation

Arondit commented Jan 10, 2024 • edited Loading

medmunds commented Jan 10, 2024

Arondit commented Jan 12, 2024

medmunds commented Jan 12, 2024

Arondit commented Jan 16, 2024 • edited Loading

medmunds left a comment

Choose a reason for hiding this comment

medmunds commented Jan 17, 2024

Arondit commented Jan 19, 2024

Arondit commented Jan 24, 2024

medmunds commented Jan 24, 2024 • edited Loading

medmunds commented Jan 25, 2024

Arondit commented Jan 29, 2024

medmunds commented Jan 29, 2024

medmunds commented Feb 12, 2024

Arondit commented Feb 19, 2024

medmunds commented Feb 19, 2024

Arondit commented Feb 20, 2024

medmunds commented Feb 22, 2024

Arondit commented Feb 22, 2024

medmunds commented Feb 29, 2024

medmunds commented Mar 1, 2024

Arondit commented Mar 4, 2024

medmunds commented Mar 5, 2024

medmunds commented Mar 5, 2024

medmunds commented Mar 12, 2024

Arondit commented Mar 15, 2024

Arondit commented Jan 10, 2024 •

edited

Loading

Arondit commented Jan 16, 2024 •

edited

Loading

medmunds commented Jan 24, 2024 •

edited

Loading