Skip to content

Latest commit

 

History

History
438 lines (310 loc) · 14.9 KB

README.md

File metadata and controls

438 lines (310 loc) · 14.9 KB
Raw

EPCC Search AST Helper

Introduction

This project is designed to help consume the EP-Internal-Search-Ast-v* headers. In particular, it provides functions for processing these headers in a variety of use cases.

Retrieving an AST

The GetAst() function will convert the JSON header into a struct that can be then be processed by other functions:

package example

import epsearchast_v3 "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3"

func Example(headerValue string) (*epsearchast_v3.AstNode, error) {
	
	ast, err := epsearchast_v3.GetAst(headerValue)
	
	if err != nil { 
		return nil, err
    } else { 
		return ast, nil
    }
	
}

If the error that comes back is a ValidationErr you should treat it as a 400 to the caller.

Aliases

This package provides a way to support aliases for fields, this will allow a user to specify multiple different names for a field, and still have it validated and converted properly:

package example

import epsearchast_v3 "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3"

func Example(ast *epsearchast_v3.AstNode) error {
	
	//The ast from the user will be converted into a new one, and if the user specified a payment_status field, the new ast will have it recorded as status. 
	aliasedAst, err := ApplyAliases(ast, map[string]string{"payment_status": "status"})

	if err != nil { 
		return err
    }
	
	DoSomethingElse(aliasedAst)
	
	return err
}

Regular Expressions

Aliases can also match Regular Expressions. Regular expresses are specified starting with the ^ and ending with $, as the key to the alias. The regular expression can include capture groups and use the same syntax as Regexp.Expand() to refer to the groups in the replacement (e.g., $1).

Note: Regular expressions are an advanced use case, and care is needed as the validation involved is maybe more limited than expected. In general if more than one regular expression can a key, then it's not defined which one will be used. Some errors may only be caught at runtime.

Note: Another catch concerns the fact that . is a wild card in regex and often a path seperator in JSON, so if you aren't careful you can allow or create inconsistent rules. In general, you should escape . in separators to \. and use ([^.]+) to match a wild card part of the attribute name (or maybe even [a-zA-Z0-9_-]+)

Incorrect: ^attributes.locales..+.description$ - This would match attributesXlocalesXXXdescription, it would also match attributes.locales.en-US.foo.bar.description

Correct: ^attributes\.locales\.([a-zA-Z0-9_-]+)\.description$

Validation

This package provides a concise way to validate that the operators and fields specified in the header are permitted, as well as contrain the allowed values to specific types such as Boolean, Int64, and Float64:

package example

import epsearchast_v3 "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3"

func Example(ast *epsearchast_v3.AstNode) error {
	var err error
	// The following is an implementation of all the filter operators for orders https://elasticpath.dev/docs/orders/orders-api/orders-api-overview#filtering
	err = epsearchast_v3.ValidateAstFieldAndOperators(ast, map[string][]string {
		"status": {"eq"},
		"payment": {"eq"},
		"shipping": {"eq"},
		"name": {"eq", "like"},
		"email": {"eq", "like"},
		"customer_id": {"eq", "like"},
		"account_id": {"eq", "like"},
		"account_member_id": {"eq", "like"},
		"contact.name": {"eq", "like"},
		"contact.email": {"eq", "like"},
		"shipping_postcode": {"eq", "like"},
		"billing_postcode": {"eq", "like"},
		"with_tax": {"gt", "ge", "lt", "le"},
		"without_tax": {"gt", "ge", "lt", "le"},
		"currency": {"eq"},
		"product_id": {"eq"},
		"product_sku": {"eq"},
		"created_at": {"eq", "gt", "ge", "lt", "le"},
		"updated_at": {"eq", "gt", "ge", "lt", "le"}, 
    })
	
	if err != nil { 
		return err
    }
	
	// You can additionally create aliases which allows for one field to reference another:
	// In this case any headers that search for a field of `order_status` will be mapped to `status` and use those rules instead. 
	err = epsearchast_v3.ValidateAstFieldAndOperatorsWithAliases(ast, map[string][]string {"status": {"eq"}}, map[string]string {"order_status": "status"})
	if err != nil {
		return err
	}
	
	// You can also supply validators on fields, which may be necessary in some cases depending on your data model or to improve user experience.
	// Validation is provided by the go-playground/validator package https://github.com/go-playground/validator#usage-and-documentation
	err = epsearchast_v3.ValidateAstFieldAndOperatorsWithValueValidation(ast, map[string][]string {"status": {"eq"}}, map[string]string {"status": "oneof=incomplete complete processing cancelled"})
	
	if err != nil {
		return err
    }
	
	// Finally you can also restrict certain fields to types, which may be necessary in some cases depending on your data model or to improve user experience.
   err = epsearchast_v3.ValidateAstFieldAndOperatorsWithFieldTypes(ast, map[string][]string {"with_tax": {"eq"}}, map[string]epsearchast_v3.FieldType{"with_tax": epsearchast_v3.Int64})

   if err != nil {
      return err
   }
   
   // All of these options together can be done with  epsearchast_v3.ValidateAstFieldAndOperatorsWithAliasesAndValueValidationAndFieldTypes
	return err
}

Regular Expressions

Regular Expressions can also be set when using the Validation functions, the same rules apply as for aliases (see above). In general aliases are resolved prior to validation rules and operator checks.

Generating Queries

GORM/SQL

The following examples shows how to generate a Gorm query with this library.

package example

import epsearchast_v3 "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3"
import epsearchast_v3_gorm "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3/gorm"
import "gorm.io/gorm"

func Example(ast *epsearchast_v3.AstNode, query *gorm.DB, tenantBoundaryId string) error {
	var err error
	
	// Not Shown: Validation
	
	// Create query builder
	var qb epsearchast_v3.SemanticReducer[epsearchast_v3_gorm.SubQuery] = epsearchast_v3_gorm.DefaultGormQueryBuilder{}

	
	sq, err := epsearchast_v3.SemanticReduceAst(ast, qb)

	if err != nil {
		return err
	}

	// Don't forget to add additional filters 
	query.Where("tenant_boundary_id = ?", tenantBoundaryId)
	
	// Don't forget to expand the Args argument with ...
	query.Where(sq.Clause, sq.Args...)
}
Limitations
  1. The GORM builder does not support aliases (easy MR to fix).
  2. The GORM builder does not support joins (fixable in theory).
  3. There is no way currently to specify the type of a field for SQL, which means everything gets written as a string today (fixable with MR).
  4. The text operator implementation makes a number of assumptions, and you likely will want to override it's implementation:
    • English is hard coded as the language.
    • Postgres recommends using a distinct tsvector column and using a stored generated column. The current implementation does not support this and, you would need to override the method to support it. A simple MR could be made to allow for the Gorm query builder to know if there is a tsvector column and use that.
Advanced Customization

In some cases you may want to change the behaviour of the generated SQL, the following example shows how to do that in this case, we want all eq queries for emails to use the lower case, comparison, and for cart_items field to be numeric.

package example

import (
	epsearchast_v3 "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3"
	"strconv"
)
import epsearchast_v3_gorm "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3/gorm"
import "gorm.io/gorm"


func Example(ast *epsearchast_v3.AstNode, query *gorm.DB, tenantBoundaryId string) error {
	var err error

	// Not Shown: Validation

	// Create query builder
	var qb epsearchast_v3.SemanticReducer[epsearchast_v3_gorm.SubQuery] = &CustomQueryBuilder{}

	sq, err := epsearchast_v3.SemanticReduceAst(ast, qb)

	if err != nil {
		return err
	}

	// Don't forget to add additional filters 
	query.Where("tenant_boundary_id = ?", tenantBoundaryId)
	
	// Don't forget to expand the Args argument with ...
	query.Where(sq.Clause, sq.Args...)
}

type CustomQueryBuilder struct {
	epsearchast_v3_gorm.DefaultGormQueryBuilder
}

func (l *CustomQueryBuilder) VisitEq(first, second string) (*epsearchast_v3_gorm.SubQuery, error) {
	if first == "email" {
		return &epsearchast_v3_gorm.SubQuery{
			Clause: fmt.Sprintf("LOWER(%s::text) = LOWER(?)", first),
			Args:   []interface{}{second},
		}, nil
	} else if first == "cart_items" {
		n, err := strconv.Atoi(second)
		if err != nil {
			return nil, err
		}
		return &epsearchast_v3_gorm.SubQuery{
			Clause: fmt.Sprintf("%s = ?", first),
			Args:   []interface{}{n},
		}, nil
	} else {
		return DefaultGormQueryBuilder.VisitEq(l.DefaultGormQueryBuilder, first, second)
	}
}

Mongo

The following examples shows how to generate a Mongo Query with this library.

package example

import (
	"context"
	epsearchast_v3 "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3"
	epsearchast_v3_mongo "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3/mongo"
	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
)

func Example(ast *epsearchast_v3.AstNode, collection *mongo.Collection, tenantBoundaryQuery bson.M)  (*mongo.Cursor, error) {
	// Not Shown: Validation

	// Create query builder
	var qb epsearchast_v3.SemanticReducer[bson.D] = DefaultMongoQueryBuilder{}

	// Create Query Object
	queryObj, err := epsearchast_v3.SemanticReduceAst(ast, qb)

	if err != nil {
		return nil, err
	}

	mongoQuery := bson.D{
		{"$and",
			bson.A{
				tenantBoundaryQuery,
				queryObj,
			},
		}}
	
	
	return collection.Find(context.TODO(), mongoQuery)
}
Limitations
  1. The Mongo Query builder is designed to produce filter compatible with the filter argument in a Query, if a field in the API is a projection that requires computation via the aggregation pipeline, then we would likely need code changes to support that.
  2. The $text operator in Mongo has a number of limitations that make it unsuitable for arbitrary queries. In particular in mongo you can only search a collection, not fields for text data, and you must declare a text index. This means that any supplied field in the filter, is just dropped. It is recommended that when using text with Mongo, you only allow users to search text(*,search) , i.e., force them to use a wildcard as the field name. It is also recommended that you use a Wildcard index to avoid the need of having to remove and modify it over time.
Advanced Customization
Field Types

In some cases, depending on how data is stored in Mongo you might need to instruct the query builder what the type of the field is. The following example shows how to do that in this case we want to specify that with_tax is a number.

package example

import (
	"context"
	epsearchast_v3 "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3"
	epsearchast_v3_mongo "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3/mongo"
	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"strings"
)

func Example(ast *epsearchast_v3.AstNode, collection *mongo.Collection, tenantBoundaryQuery *bson.M)  (*mongo.Cursor, error) {
	// Not Shown: Validation

	// Create query builder
	var qb epsearchast_v3.SemanticReducer[bson.D] = &epsearchast_v3_mongo.DefaultMongoQueryBuilder{
		FieldTypes: map[string]epsearchast_v3_mongo.FieldType{"with_tax": epsearchast_v3_mongo.Int64},
    }

	// Create Query Object
	queryObj, err := epsearchast_v3.SemanticReduceAst(ast, qb)

	if err != nil {
		return nil, err
	}

	mongoQuery := bson.D{
		{"$and",
			bson.A{
				tenantBoundaryQuery,
				queryObj,
			},
		}}
	
	return collection.Find(context.TODO(), mongoQuery)
}
Custom Queries

In some cases you may want to change the behaviour of the generated Mongo, the following example shows how to do that in this case we want to change emails because we store them only in lower case in the db.

package example

import (
	"context"
	epsearchast_v3 "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3"
	epsearchast_v3_mongo "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3/mongo"
	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"strings"
)

func Example(ast *epsearchast_v3.AstNode, collection *mongo.Collection, tenantBoundaryQuery *bson.M)  (*mongo.Cursor, error) {
	// Not Shown: Validation

	// Create query builder
	var qb epsearchast_v3.SemanticReducer[bson.D] = &LowerCaseEmail{}

	// Create Query Object
	queryObj, err := epsearchast_v3.SemanticReduceAst(ast, qb)

	if err != nil {
		return nil, err
	}

	mongoQuery := bson.D{
		{"$and",
			bson.A{
				tenantBoundaryQuery,
				queryObj,
			},
		}}
	
	return collection.Find(context.TODO(), mongoQuery)
}

type LowerCaseEmail struct {
	epsearchast_v3_mongo.DefaultMongoQueryBuilder
}

func (l *LowerCaseEmail) VisitEq(first, second string) (*bson.D, error) {
	if first == "email" {
		return &bson.D{{first, bson.D{{"$eq", strings.ToLower(second)}}}}, nil
	} else {
		return DefaultMongoQueryBuilder.VisitEq(l.DefaultMongoQueryBuilder, first, second)
	}
}

You can of course use the FieldTypes and CustomQueryBuilder together.

Elastic Search (Open Search)

The following examples shows how to generate a Elastic Search Query with this library.

package example
import epsearchast_v3 "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3"
import epsearchast_v3_es "github.com/elasticpath/epcc-search-ast-helper/external/epsearchast/v3/els"

func Example(ast *epsearchast_v3.AstNode, tenantBoundaryId string)  (string, error) {
   // Not Shown: Validation

   // Create query builder
   var qb epsearchast_v3.SemanticReducer[epsearchast_v3_es.JsonObject] = epsearchast_v3_es.LowerCaseEmail{}

   // Create Query Object
   queryObj, err := epsearchast_v3.SemanticReduceAst(ast, qb)

   if err != nil {
      return nil, err
   }

   ...
}

type LowerCaseEmail struct {
   epsearchast_v3_es.DefaultEsQueryBuilder
}

func (l *LowerCaseEmail) VisitEq(first, second string) (*bson.D, error) {
   ....
}

FAQ

Design

Why does validation include alias resolution, why not process aliases first?

When validation errors occur, those errors go back to the user, so telling the user the error that occurred using the term they specified improves usability.