Question about cancelling a Pop3 message fetch

[davidei1955]

I'm using MimeKit, MailKit and Pop3 to download and parse email from several gmail servers. It works very well. A simplified version of my ReadMail method looks like this:

    async Task<MailReaderResults> ReadMailAsync(LoginInfo login) {
          CancellationToken token = ReaderTokenSource!.Token;
          using (var pop3 = new Pop3Client()) {
              try {
                  int messageCount = await AuthenticateMailAsync(login, readerResults, pop3, token);
                  for (int messageIndex = 0; messageIndex < messageCount; messageIndex++) {
                      MimeMessage message = await pop3.GetMessageAsync(messageIndex, token);
                      message.WriteTo(tempMessageFileSpec);
                  }
              } catch (OperationCanceledException oce) {
                  return readerResults.Update(ReaderStatus.Cancelled, oce);
              } catch (Exception e) {
                  return readerResults.Update(ReaderStatus.Exception, e);
              } finally {
                  pop3.Disconnect(true);  //Disconnect cleanly, so messages that have already been read are deleted from the server
              }
          }
          return readerResults.Update(ReaderStatus.Success, stopwatch);
      }

Cancelling an active pop3.GetMessageAsync(messageIndex, token); then re-running the method at a later time is a normal part of my workflow. I would like the messages that have been successfully downloaded before ReaderTokenSource.Cancel(); is invoked (on another thread) to be deleted from the server. The problem is that it appears that none of the messages are deleted in this scenario and I end up downloading some of the messages twice.

Is there a way to force the deletion of messages that have been downloaded successfully, or failing that, detecting duplicate email messages?

[jstedfast]

Some POP3 servers support UIDs (but many do not). If your server supports UIDs, then you could keep a local record of which message UIDs you've already processed. You can check for this using the following snippet:

if (pop3.Capabilities.HasFlag (Pop3Capabilities.UIDL))
    ...

If that is available, then it opens up APIs like pop3.GetMessageUids() which allows you to map a message index to a message UID:

var uids = pop3.GetMessageUids ();

for (int i = 0; i < pop3.Count; i++) {
    var message = pop3.GetMessage (i);
    string uid = uids[i];
}

But even if the server supports UIDs, you probably still want to at least try to prune messages that have already been processed...

async Task<MailReaderResults> ReadMailAsync(LoginInfo login) {
          CancellationToken token = ReaderTokenSource!.Token;
          using (var pop3 = new Pop3Client()) {
              int lastIndex = -1;

              try {
                  int messageCount = await AuthenticateMailAsync(login, readerResults, pop3, token);
                  for (int messageIndex = 0; messageIndex < messageCount; messageIndex++) {
                      MimeMessage message = await pop3.GetMessageAsync(messageIndex, token);
                      message.WriteTo(tempMessageFileSpec);
                      lastIndex = messageIndex;
                  }
              } catch (OperationCanceledException oce) {
                  return readerResults.Update(ReaderStatus.Cancelled, oce);
              } catch (Exception e) {
                  return readerResults.Update(ReaderStatus.Exception, e);
              } finally {
                  if (lastIndex >= 0 && !pop3.IsConnected)
                      await AuthenticateMailAsync(login, readerResults, pop3, token);

                  if (lastIndex ==  0)
                      await pop3.DeleteMessageAsync (0);
                  else if (lastIndex > 0)
                      await pop3.DeleteMessagesAsync (0, lastIndex);

                  await pop3.DisconnectAsync(true);  //Disconnect cleanly, so messages that have already been read are deleted from the server
              }
          }
          return readerResults.Update(ReaderStatus.Success, stopwatch);
      }

If the pop3 server supports UIDs, you could improve the logic a bit more by adding a HashSet<string> KnownUids property on LoginInfo, for example:

static void CleanUpKnownUids (ref HashSet<string> knownUids, IList<string> uids)
{
    var newKnownUids = new HashSet<string> ();

    foreach (var uid in uids) {
        if (knownUids.Contains (uid))
            newKnownUids.Add (uid);
    }

    knownUids = newKnownUids;
}

async Task<MailReaderResults> ReadMailAsync(LoginInfo login) {
          CancellationToken token = ReaderTokenSource!.Token;
          using (var pop3 = new Pop3Client()) {
              int lastIndex = -1;

              try {
                  int messageCount = await AuthenticateMailAsync(login, readerResults, pop3, token);
                  IList<string> uids = null;

                  if (pop3.Capabilities.HasFlag (Pop3Capabilities.UIDL)) {
                      uids = pop3.GetMessageUidsAsync ();

                      // remove UIDs that no longer exist from our KnownUids so the list doesn't keep growing forever
                      CleanUpKnownUids (ref login.KnownUids, uids);
                  }

                  for (int messageIndex = 0; messageIndex < messageCount; messageIndex++) {
                      if (uids != null && login.KnownUids.Contains (uids[messageIndex]))
                          continue;

                      MimeMessage message = await pop3.GetMessageAsync(messageIndex, token);
                      message.WriteTo(tempMessageFileSpec);
                      lastIndex = messageIndex;

                      if (uids != null)
                          login.KnownUids.Add(uids[messageIndex]);
                  }
              } catch (OperationCanceledException oce) {
                  return readerResults.Update(ReaderStatus.Cancelled, oce);
              } catch (Exception e) {
                  return readerResults.Update(ReaderStatus.Exception, e);
              } finally {
                  if (lastIndex >= 0 && !pop3.IsConnected)
                      await AuthenticateMailAsync(login, readerResults, pop3, token);

                  if (lastIndex ==  0)
                      await pop3.DeleteMessageAsync (0);
                  else if (lastIndex > 0)
                      await pop3.DeleteMessagesAsync (0, lastIndex);

                  await pop3.DisconnectAsync(true);  //Disconnect cleanly, so messages that have already been read are deleted from the server
              }
          }
          return readerResults.Update(ReaderStatus.Success, stopwatch);
      }

[davidei1955]

Using MessageUids to detect and remove duplicates seems to be working well. Thanks!