IMAP process for attachment-downloader

[rpedrica]

Hi James,

First thanks for your fantastic tool - I've had a continual problem getting parsedmarc (as part of dmarc-visualiser) to do this (it's supposed to be an internal function) but it simply refuses, so I've been looking for something else to get the attachments off of my dmarc reports.

Everything generally runs correctly but I have some emails where the attachments are not being processed. I can't see any difference between these attachments and those from successful emails ... issue #33 appears to be similar/same. Out of 72 emails in that folder originally, attachments for just over 40 were processed and the emails then moved, while attachments on the other emails are not processed and the emails remain behind.

My current config:

attachment-downloader --host xxx --username xxx --password xxx --imap-folder DMARCReports --output /opt/attachment-downloader/reports --delete --delete-copy-folder DMARCDone

Log of a successful email (where the attachment is downloaded and the email is then moved to DMARCDone):

2024-10-18 10:24:56,678 - INFO - Processing message 'b'5'' subject 'Report Domain: capetownvascular.co.za Submitter: protection.outlook.com Report-ID: ac9650175b8c4b2fbf3a6fd30f16d694' send to xxx'
2024-10-18 10:24:56,678 - INFO - Downloading attachment 'protection.outlook.com!capetownvascular.co.za!1728518400!1728604800.xml.gz' to path /opt/attachment-downloader/reports/protection.outlook.com!capetownvascular.co.za!1728518400!1728604800.xml.gz
2024-10-18 10:24:56,678 - INFO - Copy UID 5 to "DMARCDone" folder
2024-10-18 10:24:56,843 - INFO - Mark UID 5 with \Deleted FLAG and expunge.
2024-10-18 10:24:56,923 - INFO - Downloaded and parsed mail 'Report Domain: capetownvascular.co.za; Submitter: Mail.Ru; Report-ID: 54502103726632498361728604800' with 1 attachments

And a log of an email where the attachment is not downloaded:

2024-10-18 10:24:57,393 - INFO - Processing message 'b'12'' subject 'Report domain: aatech.co.za Submitter: google.com Report-ID: 6481668288816146378' send to 'xxx'
2024-10-18 10:24:57,435 - INFO - Downloaded and parsed mail 'Report domain: capetownvascular.co.za Submitter: google.com
Report-ID: 17483065721708887824' with 0 attachments

I've checked the emails with issues in the mailbox, and the attachments are definitely there ... I've even downloaded a couple of the attachments to confirm their contents (the xml reports are there).

Any ideas?

Regards, Robby

[rpedrica]

Hi James

I've found a repeatable pattern. Emails with attachment names starting with google* or mimecast* seem to be problematic:

mimecast.org!emailstor.co.za!1729036800!1729123199!228c7ec2b2a854f849b352c34277377c60d012cfb6f08d7d5a8decc981c7039f.xml.gz
google.com!aatech.co.za!1729036800!1729123199.zip

ad entry for 1st above =

2024-10-18 11:02:46,464 - INFO - Downloaded and parsed mail 'Report domain: emailstor.co.za Submitter: mimecast.org Report-ID:
228c7ec2b2a854f849b352c34277377c60d012cfb6f08d7d5a8decc981c7039f' with 0 attachments
2024-10-18 11:02:46,464 - INFO - Processing message 'b'79'' subject 'Report domain: emailstor.co.za Submitter: mimecast.org Report-ID:
228c7ec2b2a854f849b352c34277377c60d012cfb6f08d7d5a8decc981c7039f' send to 'xxx'

Everything else appears to be fine. Working examples:

enterprise.protection.outlook.com!emailstor.co.za!1728950400!1729036800.xml.gz
protection.outlook.com!capetownvascular.co.za!1728604800!1728691200.xml.gz
mail.ru!capetownvascular.co.za!1728691200!1728777600.xml.gz
yahoo.com!capetownvascular.co.za!1728864000!1728950399.xml.gz

ad entry for last one above:

2024-10-18 10:25:00,176 - INFO - Downloaded and parsed mail 'Report Domain: capetownvascular.co.za Submitter: yahoo.com Report-ID: <1728954518.536287>' with 1 attachments
2024-10-18 10:25:00,176 - INFO - Processing message 'b'41'' subject 'Report Domain: capetownvascular.co.za Submitter: yahoo.com Report-ID: <1728954518.536287>' send to 'xxx'
2024-10-18 10:25:00,176 - INFO - Downloading attachment 'yahoo.com!capetownvascular.co.za!1728864000!1728950399.xml.gz' to path /opt/attachment-downloader/reports/yahoo.com!capetownvascular.co.za!1728864000!1728950399.xml.gz

I can't see name length being an issue as there are long and short names for both working and non-working samples. Same with chars - there's nothing special about the non-working samples.

Regards, Robby