Advanced search feature

[indyteo]

This feature was originally requested on our internal Fider instance and will be developed by our team.

The current search feature allow us to find posts matching words, and we can use a few filters on the home page to display only some posts. This is great, but can be improved. Here come the...

Advanced search

This feature will be split in two parts, and a third may be realized later:

1. Search operators (exact, author, votes, etc...)
2. Advanced search zone to generate query from a visual panel
3. Saved views based on a query that can be quickly applied and creating search templates

Search operators

Presentation

The traditional simple search will still exist with words without any operator (i.e. Webhook, mention users or sticky posts...).
An operator will be a simple word with an associated value, with the following syntax: operator:"value".
Operators could be inverted by prefixing them with an exclamation mark (i.e. !operator:"value").
The values could be combined using a comma, to symbolize a OR combination (i.e. operator:"val1,val2,val3").
When applicable (elements can be ordered like numbers or dates), values can themselves be ranges, opened or closed (i.e. operator:"valmin..", operator:"..valmax", operator:"valmin..valmax").
They will be combined with AND when several are used at the same time (i.e. operator1:"value1" operator2:"value2").

First list of operators

On terms

• exact:"term" = must contains exactly the given sequence of characters
• title:"term" = same as exact but only in the title
• desc:"term" = same as exact but only in the description
• comment:"term" = same as exact but only in the comments

On dates

• open:"date" = match the post creation date
• active:"date" = match the post last activity date
• response:"date" = match the post response date

On meta-data

• author:"user" = only from the given author
• status:"status" = only with a given status
• votes:"number" = match the votes count
• with:"user" = with the participation of a user (in the comments)

Special values

Some special values would allow us to create "generic" search query.

• @me = replaced with our username
• @now = replaced with the current date
• @week = replaced with the date one week ago
• @month = replaced with the date a month ago

Example queries

• Posts containing "link", opened since the 17th of December 2020, neither by indyteo nor Guilherme Oenning:

  link open:"2020-12-17.." !author:"indyteo,Guilherme Oenning"
  

• Posts with a recent activity in the past month but not this week (it would be approximately the same as !active:"..@month,@week.."), with at least 10 votes and which I didn't participated (commented):

  active:"@month..@week" votes:"10.." !with:"@me"
  

• Posts opened during the past week, still opened, and containing exactly "Fider" in their title:

  title:"Fider" status:"opened" active:"@week.."
  

• Posts opened today by someone else:

  open:"@now" !author:"@me"
  

Advanced search zone

This will come after the operators and this is too early to state anything about it. Maybe it'll be a popup with fields and a "Search" button that generate the corresponding filter and paste it into the actual search bar.

First thoughts on the technical implementation

It is planed to create a query parser, to interpret the search query and transform it into a Go model, and then use this model to perform the actual search into the database. A little rework of the current requests may happen, as it seems to be very complex for the moment, even with a simple idea (textual search). Perhaps a better model or the usage of database stored procedures could help.

Then, to create the advanced search zone, a query builder will be created, to generate queries based on input fields of a form.

No extra modules or packages should be included.

---

The definition of this feature may evolve as it's not currently developed. But it will look like this 😄

[goenning]

Oh wow, that's looking great. It reminds me of GitHub.

I'd just suggest you start small, maybe break it down into multiple deliverables to simplify reviewing and also start getting value early out of this feature.

Questions:

1. status:"opened" how would this work with the existing status dropdown?
2. Translating (i18n) the queries can be complex. What's your thoughts on it?
3. Beware of SQL Injection :)

[indyteo]

It reminds me of GitHub.

Yes, I took the inspiration from GitHub, Twitter, and a bit from Google too.

I'd just suggest you start small

I think the first iteration will contain a basic parsing (i.e. being able to parse something like that word1 word2 operator1:"value1" word3 operator2:"value2" into that {words: ["word1", "word2", "word3"], operators: {operator1: "value1", operator2: "value2"}}). I'll start by doing the existing search features such as simple word, status, and tags. Then it's evolve later.

However, concerning the intermediate reviews, it'll be complex.
Indeed, we'll start working on this on our internal fork, because the request is to add it to our Fider's instance. And this fork is outdated, but cannot be updated, since we still have mandatory features we developed and that aren't here.
The most important thing being the whole native LDAP authentication system, and there's absolutely no way our company drop it.
So until every major features we created are merged, we're stuck and must develop new features both on our fork and here in order to "publish" them. I'll try to create a draft pull request as soon as the first iteration will be over, and update it with new features.

My thoughts on your questions:

1. The dropdowns will stay there, and corresponding operators will exists: status:"opened", tags:"tag1,tag2,tag3", filter:"my-votes", sort:"most-wanted". Under the hoods, dropdowns will simply build an advanced search query that will replace the current GET parameters request style (with a single GET param maybe query or q).
2. Yep, it would be tricky. This is why I think it won't be translated 😅 I think it's fair enough not to translate it as soon as there will be a UI section, translated, with buttons and text inputs to interact and build a query without looking at it.
   But if we estimate that translation is necessary, thanks to server-side locale, we would be able to get the translated set of operators and map them to the functions. And maybe allowing the use of English operators even if the site has another locale would be a good thing as users will be able to perform advanced queries without knowing the word corresponding to the operator in the other language. Because checking the docs (if we maintain a list of all translated operators) may be annoying.
   TL;DR: I think we should keep them in English 😄
3. This is exactly why I was thinking of a little rework of the way the SQL queries are built. Because now it's really hard to follow the path of the request building, and even harder to understand the final query. I strongly think stored procedures would REALLY make this part of the code cleaner and easier to evolve.

And by the way, as a little preview, I made the parsing regexp, for a basic input: Checkout POC in playground