Search is inefficient

[loxK]

Issue

The search in Nextcloud Mail is awful, I doubt it that it does trigger a real IMAP search. I searched for an email I sent a while ago, using the recipient address: not a single result.

It is a dealbreaker. It may be good to keep as a backup webmail, but certainly not for daily use, nor for working.

Mail app

Mail app version: 1.10.5
Mailserver or service: IMAP

Server configuration

Operating system: Ubuntu

Web server: Apache

Database: MariaDB

PHP version: 8.0

Nextcloud Version: 22.2.0

Client configuration

Browser: Firefox 92.0

Operating system: Ubuntu

[miaulalala]

Hi @loxK , we don't offer a full text search for IMAP yet. We're aware that this needs some more love.

You can use filters like from: to: cc: subject: etc to filter.

If you would like to search just the mail app you can use the filter in:mail at the end of the search expression.

Thanks for letting us know!

[JackDrydenUK]

You can use filters like from: to: cc: subject: etc to filter.

If you would like to search just the mail app you can use the filter in:mail at the end of the search expression.

Thanks @miaulalala, great to know these filters can be used when searching!

I think the biggest issue which makes the search inefficient is that it does not search from or to names, only for email addresses. (As noted on issue #3562) e.g. if I want to find an email from Joe Bloggs [email protected], searching 'from:Joe Bloggs' will not bring up anthing. It only works if I search 'from:[email protected]'.

This is quite a big problem because you are far more likely to remember the name of someone who emailed you than the exact format of their email address.

[loxK]

@miaulalala this is not user-friendly. In rainloop or roundcube, I go in the folder I want to search in, and just input a name, or an email and have the proper results. If needed I can also refine my search criteria with to:, from:, body: and so on. I also have a much better preview of the results than the small search bubble of NextCloud.

Does it search in current folder or only Inbox ? Does it trigger an IMAP search ?

[miaulalala]

@JackDrydenUK thanks for your comment, that is a fair point re: the names.

@loxK I understand your frustration, but again, this is something we're aware is an issue and are planning on improving at some point in the future. As I've said, we do not do an IMAP search at the moment.

[loxK]

@miaulalala I am not frustrated, how could I ? Nextcloud is awesome and open source. I only wanted to be sure that the "deal-breaking for serious work" aspect of the issue was well understood. It seems like it is, so, all good. Thanks for the hard work.

[ArnY]

On the subject, the search keywords from: and to: will only work if you enter a whole address and will not return any result with you use a firstname, lastname or a domain which is a bit disappointing.

[kartoffelheinz]

Has there been any progress on this?

We are in 2021, even the worst mail application you can think of implements default "wildcards" in their search and if you just search for standard text without any modifiers, it searches the message including Subject, Sender, Body etc.

I was hoping I could finally replace my googlemail with this. The absence of the most basic search function is a real showstopper for this app. And I sure am not the only one pushed away by this.

[miaulalala]

@kartoffelheinz I understand your frustration. We're currently working on the next major release so we're a bit pressed for time. Feel free to take a shot at this yourself. Otherwise, I can only ask you to be patient. You will see if there's been progress by way of a PR.

[cm-schl]

I don't have any Nextcloud development experience at the moment and I can imagine that something like this would be to much for someone who would like to begin - but could you tell generally what are the steps needed to improve the search function? I looked at the developer manual and found this chapter about the unified search https://docs.nextcloud.com/server/latest/developer_manual/digging_deeper/search.html.

Starting from lib/Search/Provider.php I came to

mail/lib/Db/MessageMapper.php

Line 734 in 3fcf569

public function findIdsGloballyByQuery(IUser $user, SearchQuery $query, ?int $limit, array $uids = null): array {

which seems to build the query that is used for searching the mails. I imagine that adding "just" some fields to this query wouldn't solve the whole issue...? (this isn't sarcasm - I really don't know it and would only like to learn 😄 )

[loxK]

@cm-schl we need IMAP search. Period. Anything else would be inefficient and over complex. IMAP has a very good search feature, that can be optimized mail server wises, let it do what it was designed for.

[ChristophWurst]

I agree that this can be tuned by the mail hoster but many people don't have control over that and we've seen IMAP implementations that do not handle search efficiently. Often it is also not just about search but it's about being able to sort search results.
Imagine users with 200,000 messages in their inbox. If you search for a broad term then you get back a lot of search hits. This causes an overhead in transfer and memory consumption. In the end the majority of data would not be used because only a fraction is shown to the user at a time. To remedy this you can split the result into pages, but this pagination needs consistency. To match the front end we would paginate by the message date. In order to sort by date you need the SORT capabilities. These are not always available. Horde falls back to a local sort if it's not there and then things get really slow because it downloads all UIDs+timestamps, sorts locally and fetches only the relevant messages.
I'm open for any pointers or suggestions if I'm missing something.

[piiskop]

I can't find emails from a specific human. I use to:fred in:mail and Nextcloud just doesn't find anything. I have many mails from and to frederer. It only finds if I put in the whole mail address. What if I don't know the whole mail address and only remember the name?

[Spartachetto]

I understand the technical points raised by @ChristophWurst , yet as I receive too many emails, I would not manage to use an email client with a search function less than complete. What @loxK describes is what I use several times every day in my work.

Said that, let me repeat verbatim a point: I only wanted to be sure that the "deal-breaking for serious work" aspect of the issue was well understood.

[JotaPin]

I unfortunately see that my users testing it in a SME considers this lack of good seach implementation as a no-go, despite they like the rest of the Nexcloud apps. Am I right that there is no real workaround in Nextcloud for seach in emails? (as in other email integrations other problems appear? rainloop: cardav sync to only one account for all users? Afterlogic: no way to sync contacts with Nextcloud?). Just to be sure because I really want them to adopt Nextcloud. :-(

[piiskop]

How does Gmail do this?

[Spartachetto]

@piiskop they are sure that the search will not be made by a Raspberry Pi and that instead one of the world's biggest server farms will be used.

I am not an expert but I guess that it helps the search speed!

[cm-schl]

I unfortunately see that my users testing it in a SME considers this lack of good seach implementation as a no-go, despite they like the rest of the Nexcloud apps. Am I right that there is no real workaround in Nextcloud for seach in emails? (as in other email integrations other problems appear? rainloop: cardav sync to only one account for all users? Afterlogic: no way to sync contacts with Nextcloud?). Just to be sure because I really want them to adopt Nextcloud. :-(

Be aware that Rainloop (the Rainloop core, not the Rainloop integration in Nextcloud) has some open issues - actually it's not clear if the project is maintained. This is surly important for business use.
See:

• Jan 2022: Is rainloop still maintained? RainLoop/rainloop-webmail#2162
• Security Contact for high severity Security Issue RainLoop/rainloop-webmail#2142

So for business use the Nextcloud Mail App cloud become more and more important. For professional use (as mentioned before by others) the fact that the search isn't ideal at the moment is kind of a show stopper. Thanks @miaulalala for adding the nextcloud-gmbh tag 👍.

[xkill]

Basically seems that the mail app does not support the IMAP command search text "" when the IMAP supports it:

2 login user password
2 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE COMPRESS=DEFLATE] Logged in
3 select Inbox
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft NotJunk $Forwarded NonJunk Junk $label1 $Junk $NotJunk unknown-8)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NotJunk $Forwarded NonJunk Junk $label1 $Junk $NotJunk unknown-8 \*)] Flags permitted.
* 2375 EXISTS
* 0 RECENT
* OK [UNSEEN 6] First unseen.
* OK [UIDVALIDITY 1188148348] UIDs valid
* OK [UIDNEXT 206749] Predicted next UID
* OK [HIGHESTMODSEQ 3828] Highest
3 OK [READ-WRITE] Select completed (0.001 + 0.000 secs).
4 search text "FACT278249"
* SEARCH 2182
4 OK Search completed (0.562 + 0.000 + 0.001 secs).

This mailbox is very big and the search was very fast, but when search for FACT278249 on Nextcloud Mail App, nothing appears (also tested using Roundcube webmail and works fine.

[NablaMetal]

Is there some kind of roadmap for improving the usability of the mail app ? Whether IMAP support or tweaking the existing technology ?
As a lot of mentioned already, the app is great, the integration to nexcloud is great, but the lack of a efficient user-friendly search is a deal breaker. If I had enough time and skill I would contribute.
thanks anyway for this mail app.

[loxK]

In Nextcloud mail I barely find what I search for (impossible for older unsynced emails). With a client using IMAP search (untweaked basic email server) I can very easily, and I have lots of emails. Not a single user I have hasn't complained about it to the point that I had to ditch Nextcloud mail and replace it with Roundcube.
It is that simple.

About every IMAP server supports search, they are designed for it and do it pretty well.

[sazanof]

#6546
This PR implements basic search

[ChristophWurst]

About every IMAP server supports search, they are designed for it and do it pretty well.

That is not the case unfortunately. This app used IMAP search in the past and we found many hosters where sorting by sent date wasn't supported by the server. The fallback is to download all UIDs and send dates, do the sorting client-side and only fetch the relevant page. But this doesn't scale with larger mailboxes as too much data needs to be transferred.

[cm-schl]

Actually I'm in the same situation like @loxK and looking for a alternative for Nextcloud Mail (Roundcube is also on the list). But the integration of files, contacts etc. would be nicer in Nextcloud.

@ChristophWurst I see the need of a backward compatibility for servers that does not support an IMAP search. Would the PR #6546 of @sazanof help in our situation with big / giant mailboxes? Other idea: what about the possibility to let the admin activate / deactivate IMAP search autonomously when the server supports it?

[NablaMetal]

I think this would be great, but would double the work for devs to implement both (imap & not imap).
imao nextcloud mail should ditch compatibility with hosters not supporting imap as it is likely that most users would support it.

[loxK]

I think this would be great, but would double the work for devs to implement both (imap & not imap). imao nextcloud mail should ditch compatibility with hosters not supporting imap as it is likely that most users would support it.

Nextcloud mail is an IMAP only client. What do you call "not IMAP" ?

[NablaMetal]

I was referring to mail servers not supporting imap, not the nextcloud client

[loxK]

This is therefore completely irrelevant.

[ChristophWurst]

Would the PR #6546 of @sazanof help in our situation with big / giant mailboxes?

It still searches on our database cache, not IMAP.

[loxK]

But the integration of files, contacts etc. would be nicer in Nextcloud

You still can get some good integration for files and contacts

[the-djmaze]

if I want to find an email from Joe Bloggs <[email protected]>, searching 'from:Joe Bloggs' will not bring up anthing.

That must be a bug as IMAP can search fine for that.

[kartoffelheinz]

Of course IMAP can search fine for that. And if this app used IMAP search, this would indeed be a bug.
But the whole point of this Issue is that the Mail App does not use IMAP search at all...

[alpianon]

cc @kappapiana

I'm testing Nextcloud to see if we can use it in production and I immediately ran into this issue.

IIUC Nextcloud mail client fetches all message metadata from the mailbox and stores them in its database, and has just a very basic search functionality (no wildcards, no logic expressions) only for message metadata in its database.

This does not make any sense. I have a dovecot IMAP server with full text search based on elasticsearch. It is very fast, Roundcube just uses it, and it works like a charm even on a 50GB mailbox.

I do not even dare to open that mailbox with Nextcloud: I expect that it would get stuck forever... (see #7644)

IMHO there is a general design issue: why do not Nextcloud simply use and integrate already existing open source tools, instead of trying to do everything internally? Why reinventing the wheel every time?

[ChristophWurst]

This does not make any sense.

#7770

[dodedodo]

@alpianon

IMHO there is a general design issue: why do not Nextcloud simply use and integrate already existing open source tools, instead of trying to do everything internally? Why reinventing the wheel every time?

I don't think that's completely true. Nextcloud mail is based on the horde libraries. And horde already implements imapsearch classes if I'm not mistaken. The choice to avoid imapsearch in the nextcloud imap client is just that, a choice.

You might want to take a look at the snappymail nextcloud plugin. It's forked from rainloop, which had been abandonned in the past years. I'm not sure if it's ready for prime time yet, but development seems to be very rapid, and I've had a very good and stable experience with standalone rainloop in the past. It's also a distant cousin of afterlogic

[ChristophWurst]

With #8645 and #8897 merged I believe this ticket can be closed. v3.5 will search for recipients case insensitive and using wildcards. Body search on IMAP is supported too.