Problems with fetching latest messages

[sphh]

I have the following two problems, which might be related:

I get the uids of the messages, which arrive after after a certain time, with a search:

client.select_folder('INBOX', readonly=True)
while True:
    res = client.folder_status('INBOX', what='UIDNEXT')
    uid_next = res[b'UIDNEXT']
    # Wait for new messages to arrive, either by `client.idle()` or `time.sleep()`
    new_uids = client.search(['UID', f'{uid_next-1}:*', 'UNDELETED', 'UNSEEN'])

This is my first problem: The list of new_uids only includes the uids of the messages from uid_next-1 up to but the last message – the uid of the very last message, which arrived latest, is missing.

How do I get this last uid included in my list of new_uids?

I tried to use just ['UID', f'{uid_next-1}:*'] as a search criterion, but that did not help.

I then get the flags and message envelope of all new messages:

    res = client.fetch(new_uids, ['ENVELOPE', 'FLAGS'])

but the result also contains all recently deleted messages, e.g. with new_uids = [56398]:

defaultdict(<class 'dict'>, {
    967: {
        b'SEQ': 967,
        b'FLAGS': (
            b'\\Deleted',
            b'\\Seen',
            b'NonJunk')},
    56398: {
        b'SEQ': 968,
        b'FLAGS': (
            b'NonJunk',),
        b'ENVELOPE': Envelope(...)}
   })

Why is the deleted message included in the response? (Please note, that the message uid key is actually the sequence id.)

I could check for a missing b'ENVELOPE key or if the b'SEQ' value is equal to the key, but is there another way to not have the deleted message(s) included in the response? Mind you, their uids were not included in the new_uid list.

Is this a problem of the IMAP server (dovecot) or of this great library?

[mjs]

This is very curious as according to the docs any message sequence range with :* should include the last UID in the folder even if the start of the range is higher than any UID in use.

Have you tried:

client.search(['UID', f'{uid_next-1}:*', 'UNDELETED'])

as the Seen flag is fairly unreliable?

If that doesn't help, can you post a more complete example of the code you're running in case there's something happening outside of the lines you've shown?

Another things that would be helpful is to turn on debug logging in case IMAPClient is sending a poorly formed query. Instructions are here: https://imapclient.readthedocs.io/en/2.3.0/concepts.html#logging

[p-baum]

I just ran into this. Starting a new connection or calling NOOP beforehand is a functioning workaround. I dont know why. Seems there is some unwanted caching somewhere?

[mjs]

There's no caching in IMAPClient but it's possible that the server is providing a static view of the folder which isn't always updated.

Calling NOOP, re-selecting the folder (select_folder) or a new connection (a bit more heavyweight) are all viable workarounds if this is what's happening