Datafilters

A small library that allow to convert a string to a generic IFilter object. Highly inspired by the elastic query syntax, it offers a powerful way to build and query data with a syntax that's not bound to a peculiar datasource.

Disclaimer

This project adheres to Semantic Versioning.

Major version zero (0.y.z) is for initial development. Anything MAY change at any time.

The public API SHOULD NOT be considered stable.

Parsing
Filters syntax
Equals
Starts with
Ends with
Contains
Is null
Any of
Is not null
Interval expressions
Regular expression
Logical operators
- And
- Or
- Not
Special character handling
Sorting
How to install
How to use
- On the client
- On the backend
  - Building expression trees to filtering data from any datasource
  - Extending IFIlters

The idea came to me when working on a set of REST APIs and trying to build /search endpoints. I wanted to have a uniform way to query a collection of resources whilst abstracting away underlying datasources.

Let's say your API handles vigilante resources :

public class Vigilante
{
    public string Firstname { get; set; }
    public string Lastname { get; set; }
    public string Nickname {get; set; }
    public int Age { get; set; }
    public string Description {get; set;}
    public IEnumerable<string> Powers {get; set;}
    public IEnumerable<Vigilante> Acolytes {get; set;} 
}

JSON Schema

{
  "id": "vigilante_root",
  "title": "Vigilante",
  "type": "object",
  "properties": {
    "firstname": {
      "required": true,
      "type": "string"
    },
    "lastname": {
      "required": true,
      "type": "string"
    },
    "nickname": {
      "required": true,
      "type": "string"
    },
    "age": {
      "required": true,
      "type": "integer"
    },
    "description": {
      "required": true,
      "type": "string"
    },
    "powers": {
      "required": true,
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "acolytes": {
      "required": true,
      "type": "array",
      "items": {
        "$ref": "vigilante_root"
      }
    }
  }
}

and the base URL of your API is https://my-beautiful/api.

vigilante resources could then be located at https://my-beautiful/api/vigilantes/

Wouldn't it be nice to be able to search any resource like so https://my-beautiful/api/vigilantes/search?nickname=Bat*|Super* ?

This is exactly what this project is about : giving you an uniform syntax to query resources without having to think about the underlying datasource.

Parsing

This is the first step on filtering data. Thanks to SuperPower, the library supports a custom syntax that can be used to specified one or more criteria resources must fullfill. The currently supported syntax mimic the query string syntax : a key-value pair separated by ampersand (& character) where :

field is the name of a property of the resource to filter
value is an expression which syntax is highly inspired by the Lucene syntax

To parse an expression, simply call ToFilter<T> extension method (see unit tests for more details on the syntax)

Filters syntax

Several expressions are supported and here's how you can start using them in your search queries.

	`string`	numeric types (`int`, `short`, ...)	Date and time types (`DateTime`, `DateTimeOffset`, ...)
EqualTo	✅	✅	✅
StartsWith	✅	N/A	N/A
Ends with	✅	N/A	N/A
Contains	✅	N/A	N/A
IsNull	✅	N/A	N/A
IsNotNull	✅	N/A	N/A
LessThanOrEqualTo	N/A	✅	✅
GreaterThanOrEqualTo	N/A	✅	✅
bracket expression	N/A	✅	✅

Equals

Search for any vigilante resources where the value of the property nickname is manbat

Query string	JSON	C#
`nickname=manbat`	`{ "field":"nickname", "op":"eq", "value":"manbat" }`	`new Filter(field: "nickname", @operator : FilterOperator.EqualsTo, value : "manbat")`

Starts with

Search for any vigilante resources where the value of the property nickname starts with "bat"

Query string	JSON	C#
`nickname=bat*`	`{ "field":"nickname", "op":"startswith", "value":"bat" }`	`new Filter(field: "nickname", @operator : FilterOperator.StartsWith, value : "bat")`

Ends with

Search for vigilante resources where the value of the property nickname ends with man.

Query string	JSON	C#
`nickname=*man`	`{ "field":"nickname", "op":"endswith", "value":"man" }`	`new Filter(field: "nickname", @operator : FilterOperator.EndsWith, value : "man")`

Contains

Search for any vigilante resources where the value of the property nickname contains "bat".

Query string	JSON	C#
`nickname=bat`	`{ "field":"nickname", "op":"contains", "value":"bat" }`	`new Filter(field: "nickname", @operator : FilterOperator.Contains, value : "bat")`

💡 contains also work on arrays. powers=*strength* will search for vigilantes who have strength related powers.

Search for vigilante resources that have no powers.

Query string	JSON	C#
`powers=!*`	`{ "field":"powers", "op":"isempty" }`	`new Filter(field: "powers", @operator : FilterOperator.IsEmpty)`

Is null

Search for vigilante resources that have no powers.

Query string	JSON	C#
`N/A`	`{ "field":"powers", "op":"isnull" }`	`new Filter(field: "powers", @operator : FilterOperator.IsNull)` or `new Filter(field: "powers", @operator : FilterOperator.EqualsTo, value: null)`

Any of

Search for vigilante resources that have at least one of the specified powers.

Query string	JSON
`powers={strength\|speed\|size}`	N/A

will result in a IFilter instance equivalent to

IFilter filter = new MultiFilter
{
     Logic = Or,
     Filters = new IFilter[]
     {
         new Filter("powers", EqualTo, "strength"),
         new Filter("powers", EqualTo, "speed"),
         new Filter("powers", EqualTo, "size")
     }
};

Is not null

Search for vigilante resources that have no powers.

Query string	JSON	C#
`N/A`	`{ "field":"powers", "op":"isnotnull" }`	`(new Filter(field: "powers", @operator : FilterOperator.IsNull)).Negate()` or `new Filter(field: "powers", @operator : FilterOperator.NotEqualTo, value: null)`

Interval expressions

Interval expressions are delimited by upper and a lower bound. The generic syntax is

<field>=<min> TO <max>

where

field is the name of the property current interval expression will be apply to
min is the lowest bound of the interval
max is the highest bound of the interval

Greater than or equal

Search for vigilante resources where the value of age property is greater than or equal to 18

Query string	JSON	C#
`age=[18 TO *[`	`{"field":"age", "op":"gte", "value":18}`	`new Filter(field: "age", @operator : FilterOperator.GreaterThanOrEqualTo, value : 18)`

Less than or equal

Search for vigilante resource where the value of age property is lower than 30

Query string	JSON	C#
`age=]* TO 30]`	`{"field":"age", "op":"lte", "value":30}`	`new Filter(field: "age", @operator : FilterOperator.LessThanOrEqualTo, value : 30)`

Between

Search for vigilante resources where age property is between 20 and 35

Query string	JSON	C#
`age=[20 TO 35]`	`{"logic": "and", filters[{"field":"age", "op":"gte", "value":20}, {"field":"age", "op":"lte", "value":35}]}`	`new MultiFilter { Logic = And, Filters = new IFilter[] { new Filter ("age", GreaterThanOrEqualTo, 20), new Filter("age", LessThanOrEqualTo, 35) } }`

💡 You can exclude the lower (resp. upper) bound by using ] (resp. [).

age=]20 TO 35[ means age strictly greater than 20 and strictly less than35
age=[20 TO 35[ means age greater than or equal to 20 and strictly less than35
age=]20 TO 35] means age greater than 20 and less than or equal to 35

💡 Dates, times and durations must be specified in ISO 8601 format

Examples :

]1998-10-26 TO 2000-12-10[
my/beautiful/api/search?date=]1998-10-26 10:00 TO 1998-10-26 10:00[
]1998-10-12T12:20:00 TO 13:30[ is equivalent to ]1998-10-12T12:20:00 TO 1998-10-12T13:30:00[

💡 You can apply filters to any sub-property of a given collection

Example : acolytes["name"]='robin' will filter any vigilante resource where at least one item in acolytes array with name equals to robin.

The generic syntax for filtering on in a hierarchical tree property["subproperty"]...["subproperty-n"]=<expression>

you can also use the dot character (.). property["subproperty"]["subproperty-n"]=<expression> and property.subproperty["subproperty-n"]=<expression> are equivalent

Regular expression

The library offers a limited support of regular expressions. To be more specific, only bracket expressions are currently supported. A bracket expression. Matches a single character that is contained within the brackets.

For example:

[abc] matches a, b, or c
[a-z] specifies a range which matches any lowercase letter from a to z.

BracketExpressions can be, as any other expressions, combined with any other expressions to build more complex expressions.

Logical operators

Logicial operators can be used combine several instances of IFilter together.

And

Use the coma character , to combine multiple expressions using logical AND operator

Query string	JSON
`nickname=Bat,man`	`{"logic": "and", filters[{"field":"nickname", "op":"startswith", "value":"Bat"}, {"field":"nckname", "op":"endswith", "value":"man"}]}`

will result in a IFilter instance equivalent to

IFilter filter = new MultiFilter
{
    Logic = And,
    Filters = new IFilter[]
    {
        new Filter("nickname", StartsWith, "Bat"),
        new Filter("nickname", EndsWith, "man")
    }
}

Or

Use the pipe character | to combine several expressions using logical OR operator Search for vigilante resources where the value of the nickname property either starts with "Bat" or ends with "man"

Query string	JSON
`nickname=Bat\|man`	`{"logic": "or", filters[{"field":"nickname", "op":"startswith", "value":"Bat"}, {"field":"nckname", "op":"endswith", "value":"man"}]}`

will result in

IFilter filter = new MultiFilter
{
    Logic = Or,
    Filters = new IFilter[]
    {
        new Filter("nickname", StartsWith, "Bat"),
        new Filter("nickname", EndsWith, "man")
    }
}

Not

To negate a filter, simply put a ! before the expression to negate

Search for vigilante resources where the value of nickname property does not starts with "B"

Query string	JSON
`nickname=!B*`	`{"field":"nickname", "op":"nstartswith", "value":"B"}`

will be parsed into a IFilter instance equivalent to

IFilter filter = new Filter("nickname", DoesNotStartWith, "B");

Expressions can be arbitrarily complex.

"nickname=(Bat*|Sup*)|(*man|*er)"

Explanation :

The criteria under construction will be applied to the value of nickname property and can be read as follow :

Searchs for vigilante resources that starts with Bat or Sup and ends with man or er.

will be parsed into a

IFilter filter = new MultiFilter
{
    Logic = Or,
    Filters = new IFilter[]
    {
        new MultiFilter
        {
            Logic = Or,
            Filters = new IFilter[]
            {
                new Filter("Firstname", StartsWith, "Bat"),
                new Filter("Firstname", StartsWith, "Sup"),
            }
        },
        new MultiFilter
        {
            Logic = Or,
            Filters = new IFilter[]
            {
                new Filter("Firstname", EndsWith, "man"),
                new Filter("Firstname", EndsWith, "er"),
            }
        },
    }
}

The ( and ) characters allows to group two expressions together so that this group can be used as a more complex expression unit.

Special character handling

Sometimes, you'll be looking for a filter that match exactly a text that contains a character which has a special meaning.

The backslash character (\) can be used to escape characters that will be otherwise interpreted as a special character.

Query string	JSON	C#
`comment=*\!`	`{"field":"comment", "op":"endswith", "value":"!"}`	`new Filter(field: "comments", @operator: FilterOperator.EndsWith, value: "!")`

💡 Escaping special characters can be a tedious task when working with longer texts. Just use a text expression instead by wrapping the text between double quotes (").

Query string	JSON	C#
`comment=*"!"`	`{"field":"comment", "op":"endswith", "value":"!"}`	`new Filter(field: "comments", @operator: FilterOperator.EndsWith, value: "!")`

Example :

I'm a long text with some \"special characters\"  in it and each one must be escaped properly`

can be rewritten

"I'm a long text with some \"special characters\"  in it and each one must be escaped properly !`

When using text expressions, only \ and " characters need to be escaped.

Sorting

This library also supports a custom syntax to sort elements.

sort=nickname or sort=+nickname sort items by their nickname properties in ascending order.

You can sort by several properties at once by separating them with a ,.

For example sort=+nickname,-age allows to sort by nickname ascending, then by age property descending.

How to install

run dotnet install DataFilters : you can already start building IFilter instances 😉 !
install one or more DataFilters.XXXX extension packages to convert IFilter instances to various target.

How to use

So you have your API and want provide a great search experience ?

On the client

The client will have the responsability of building search criteria. Go to filtering and sorting sections to see example on how to get started.

On the backend

One way to start could be by having a dedicated resource which properties match the resource's properties search will be performed onto.

Continuing with our vigilante API, we could have

// Wraps the search criteria for Vigilante resources.
public class SearchVigilanteQuery
{
    public string Firstname {get; set;}

    public string Lastname {get; set;}

    public string Nickname {get; set;}

    public int? Age {get; set;}
}

and the following endpoint

using DataFilters;

public class VigilantesController
{
    // code omitted for brievity

    [HttpGet("search")]
    [HttpHead("search")]
    public ActionResult Search([FromQuery] SearchVigilanteQuery query)
    {
        IList<IFilter> filters = new List<IFilter>();

        if(!string.IsNullOrWhitespace(query.Firstname))
        {
            filters.Add($"{nameof(Vigilante.Firstname)}={query.Firstname}".ToFilter<Vigilante>());
        }

        if(!string.IsNullOrWhitespace(query.Lastname))
        {
            filters.Add($"{nameof(Vigilante.Lastname)}={query.Lastname}".ToFilter<Vigilante>());
        }

        if(!string.IsNullOrWhitespace(query.Nickname))
        {
            filters.Add($"{nameof(Vigilante.Nickname)}={query.Nickname}".ToFilter<Vigilante>());
        }

        if(query.Age.HasValue)
        {
            filters.Add($"{nameof(Vigilante.Age)}={query.Age.Value}".ToFilter<Vigilante>());
        }


        IFilter  filter = filters.Count() == 1
            ? filters.Single()
            : new MultiFilter{ Logic = And, Filters = filters };

        // filter now contains our search criteria and is ready to be used 😊

    }
}

Some explanation on the controller's code above :

The endpoint is bound to incoming HTTP GET and HEAD requests on /vigilante/search
The framework will parse incoming querystring and feeds the query parameter accordingly.
From this point we test each criterion to see if it's acceptable to turn it into a IFilter instance. For that purpose, the handy .ToFilter<T>() string extension method is available. It turns a query-string key-value pair into a full IFilter.
we can then either :
- use the filter directly is there was only one filter
- or combine them using composite filter when there is more than one criterion.

💡 Remarks

You may have noticed that SearchVigilanteQuery.Age property is nullable whereas Vigilante.Age property is not. This is to distinguish if the Age criterion was provided or not when calling the vigilantes/search endpoint.

Building expression trees to filtering data from any datasource

Most of the time, once you have an IFilter, you want to use it against a datasource. Using Expression<Func<T, bool>> is the most common type used for this kind of purpose. DataFilters.Expressions library adds ToExpression<T>() extension method on top of IFilter instance to convert it to an equivalent System.Expression<Func<T, bool>> instance. Using the example of the VigilantesController, we can turn our filter into a Expression<Func<T, bool>>

IFilter filter = ...
Expression<Func<Vigilante, bool>> predicate = filter.ToExpression<Vigilante>();

The predicate expression can now be used against any datasource that accepts Expression<Func<Vigilante, bool>> (👋🏾 EntityFramework and the likes )

Extending `IFIlter`s

What to do when you cannot use expression trees when querying your datasource ? Well, you can write your own method to render it duh !!!

DataFilters.Queries adds ToWhere<T>() extension method on top of IFilter instance to convert it to an equivalent IWhereClause instance. IWhereClause is an interface from the Queries that can later be translated a secure SQL string. You can find more info on that directly in the Github repository.

Package	Downloads	Description
		provides core functionalities of parsing strings and converting to IFilter instances.
		adds `ToExpression<T>()` extension method on top of IFilter instance to convert it to an equivalent `System.Linq.Expressions.Expression<Func<T, bool>>` instance.
		adds `ToWhere<T>()` extension method on top of IFilter instance to convert it to an equivalent `IWhereClause` instance.

Name		Name	Last commit message	Last commit date
Latest commit History 860 Commits
.config		.config
.github		.github
.nuke		.nuke
build		build
src		src
test		test
.editorconfig		.editorconfig
.gitattributes		.gitattributes
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
DataFilters.lutconfig		DataFilters.lutconfig
DataFilters.sln		DataFilters.sln
GitVersion.yml		GitVersion.yml
LICENSE		LICENSE
NuGet.config		NuGet.config
README.md		README.md
build.cmd		build.cmd
build.ps1		build.ps1
build.sh		build.sh
build.yaml		build.yaml
core.props		core.props
global.json		global.json
renovate.json		renovate.json
tests.props		tests.props

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Datafilters

Disclaimer

Parsing

Filters syntax

Equals

Starts with

Ends with

Contains

Is null

Any of

Is not null

Interval expressions

Greater than or equal

Less than or equal

Between

Regular expression

Logical operators

And

Or

Not

Special character handling

Sorting

How to install

How to use

On the client

On the backend

Building expression trees to filtering data from any datasource

Extending `IFIlter`s

About

Releases 13

Packages

Contributors 4

Languages

License

candoumbe/DataFilters

Folders and files

Latest commit

History

Repository files navigation

Datafilters

Disclaimer

Parsing

Filters syntax

Equals

Starts with

Ends with

Contains

Is null

Any of

Is not null

Interval expressions

Greater than or equal

Less than or equal

Between

Regular expression

Logical operators

And

Or

Not

Special character handling

Sorting

How to install

How to use

On the client

On the backend

Building expression trees to filtering data from any datasource

Extending IFIlters

About

Resources

License

Stars

Watchers

Forks

Releases 13

Packages 0

Contributors 4

Languages

Extending `IFIlter`s

Packages