Build rich mongodb search interfaces with cross collection lookups while preserving the security and business logic in your FeathersJS hooks.
FeathersJS imposes great structure on a web application project with the hooks system. In the before hooks for a service one can transform data, check permissions and perform other tasks appropriate before a DB operation, and in the after hooks one can perform side effects and implement further transformations.
Let's take query restriction and data redaction as examples of each. A service can have a before hook that restricts the user to their own records by adding their _id to the query. And another service might have an after hook to redact passwords or other credentials.
Now what happens if you want to build a client side web page or app screen that uses the full power of the MongoDB aggregation framework as applied to your collections? This goes beyond simple CRUD operations as supported by feathers-mongodb and so you'll need to implement your own service. But in this service you'll need to re-implement all of that same restriction and redaction logic. All of this leads to additional highly coupled code in your project, leading to maintainability problems and security leaks and eventually higher costs.
This is what FMR-Searchkit is for. It's a drop-in solution for building a highly functional cross-collection search backed by a full MongoDB aggregation that re-uses the before and after hoooks from your feathers services while building the aggregations and handling the results. Does your user service redact passwords in an after find hook? FMR-Searchkit searches of your user collection will too, right out of the box - whether the user data appears as the main rows of your results or whether it appears as names on a facet filter restricting some other collection. Does your personal_info collection restrict people to their own records in a before find hook? So will FMR-Searchkit searches of that collection, right out of the box. And so on.
You can use the searchkit right now at my site The Logic Resource Center where it powers the Sequent, Derivation and Tableaux searches. It also features toward the end of this demo video for my multi-party food and beverage app Porter where it powers premium merchant interfaces for restautrant operations data.
Install fmr-searchkit
with your favorite package manager.
The module assumes the following
- In the folder for each service you will expose through the searchkit there is a standard Feathers
hooks
file - In the each of the same folders there is a BSON schema file called
schema
for the Mongo collection backing the service. - The db connection is made available at
app.mongodb.db
as demonstrated below
In app.ts
:
import mongodb, { ObjectId } from 'mongodb'
import { searchkit } from 'fmr-searchkit'
const client = await MongoClient.connect(/* your connection args */)
app.mongodb = { db: client.db() }
app.configure(
searchkit({
ObjectId,
services: [
'service1',
'service2',
'service3',
'service4'
],
servicesPath: `${__dirname}/services/`
})
)
You'll likely want to put some of your own hooks on the new search
and schema
services this creates, but other than that this is really it - you now have a highly functional search service that reuses the before
and after
all
and find
hooks on your services, even on other searches that run lookups and subqueries to the underlying collections.
The search
service takes a POST
or Feathers create
with a JSON stringified object specifying the search. The properties of this object:
Prop | Required | Default | Comment |
---|---|---|---|
collection |
true | none | base collection from which results will be drawn |
sortField |
false | none | |
sortDir |
false | none | asc or desc |
page |
false | 1 | |
pageSize |
false | 100 | |
include |
false | none | Returns all fields when unset |
includeSchema |
false | false | Return base schema with the response |
filters |
false | none | Array of Filter . See ./types |
charts |
false | none | Array of Chart . See ./types |
lookup |
false | none | Keyed object of Lookup . See ./types |
id |
false | none | Not used server-side |
So how do you use it? This is where FMR-Searchkit-React comes in. It allows your to build a wide variety of client- and server-rendered searches with React 18 and full Next 13 support. As you can see here the searchkit ships now with about 20 filters and charts. Instructions on how to use them are at FMR-Searchkit-React
The sad truth is that Mongo shines on single collections when the queries have been anticipated with good indexing and struggles with lookups to other collections when you get beyond maybe 20,000 records, often in ways that defy indexing. So what's the point?
You should think of the lookup
functionality here as a tool for prototying and offline layouts. There are plenty of applications in the course of a normal web development career where collections are small or users tolerate longer response times. Launching a new product or feature at all but the biggest companies would be an example, as would basic offline analytics functionality for employees.
When you need real performance you need to denormalize. Luckily the plan is simple: after every mutation of the records in your collection you run all necessary lookups and write the populated record to another collection. Just change your layout to search the new collection and you're done. The live data stays small and queries of the denormalized data perfom. So the lifecycle is like this:
- prototype and explore with free use of
lookup
. - at launch, index properly for the layouts you actually build - all filters will need to be supported by an index and in fact indexes will increase factorially in number with filters so choose wisely.
- when indexing still isn't getting you by drop in the
denormalizeAndUpsert
hook provided here and adjust the consuming searchkit layout to point at the new collection with appropriate adjustments.