Proposal: Generic Filter Functionality

[ajnavarro]

Description

We are suffering from Feature Creep on the filtering side of GraphQL API implementation, and we can still not cover most of the use cases (we cannot filter blocks by the proposer, for example).

I did a quick research and it is possible to implement a plugin on gqlgen that can generate the filter objects we need to be able to filter by all the fields we want.

Proposal

Implement a gqlgen plugin

We can create a plugin implementing CodeGenerator interface:

type CodeGenerator interface {
	GenerateCode(cfg *codegen.Data) error
}

That will allow us to scan all existing objects in the schema, get their fields with their types, and generate the needed objects for applying the filtering.

Proposed Filter structure

A filter will contain And filters, Or filters, Not filter, and field filters. They will be composed as follows (we are using Block filter as an example):

type FilterBlock struct {
	And []*FilterBlock
	Or []*FilterBlock
	Not *FilterBlock

	hash *FilterString

	num_txs *FilterNumber
   
	txs *NestedFilterBlockTransaction
}

type FilterString struct {
	Exists bool
    Eq       string
    Neq    string

    Like    string
    Nlike  string

	// TODO: maybe there are more useful ones?
}

That will allow us to create filters like:

{
   "_and":[
      {
         "num_txs":{
            "gt":10,
            "lt":100
         }
      },
      {
         "proposer_address_raw":{
            "eq":"ADDR"
         },
         "_or":[
            {
               "tx":{
                  "hash":{
                     "eq":"TXHASH"
                  }
               }
            }
         ]
      }
   ],
   "_or":[
      {
         "consensus_hash":{
            "eq":"HASH"
         }
      }
   ]
}

That will be the same as:

WHERE (((num_txs > 10 AND num_txs < 100) AND proposer_address_raw = "ADDR") OR tx_hash="TXHASH") OR consensus_hash = "HASH"

Schema tagging for filter creation

We can scan the existing graphql schema to find specific annotations, and from that, generate the needed filters. Example:

# plugin:filter
type Block {

  # plugin:filter
  hash: String!

  # plugin:filter
  height: Int!

  version: String!

  # plugin:filter
  chain_id: String!

  # plugin:filter
  time: Time!
	
[...]

That will allow us to create filters only from the types and fields we want/need. In the future we can be more specific, like only creating eq filter for a field: (# plugin:filter(types:[eq])))

Special cases

We have filters that are used outside the filtering logic, for example, to query the storage (block height). We need to expose this values to retrieve them. One idea (still thinking if it is the best) will be add a special tag to these values to expose a method to retrieve the max and min values from the filters, something like:

  # plugin:filter(expose:[minmax])
  height: Int!

Generating the following method in the generated filter struct:

func (f *FilterBlock) GetMinMaxHeight()(min int, max int) {
	// TODO check height filters values (eq, gt, lt and so on) in a recursive way, and return the min value and max value
}

[zivkovicmilos]

I love this, I think it's great 💯

I was bummed to find out we need to directly modify the existing objects in order to index specific block fields, like the proposer, but if this is a long term solution then I'm all for it

@jinoosss what do you think?

[jinoosss]

@zivkovicmilos @ajnavarro ,
I think that's a really good suggestion, too.
We can provide consistent and intuitive filters.

[zivkovicmilos]

Closed by #90