Extending TypeBuilder and the compiler

[giggo1604]

Hi and thanks for this awesome lib.

We tested it in our code base as a replacement for my-zod and we really liked the developer experience. We also tested the compiler and can confirm the superior performance.

Unfortunately we hit a blocker when we tried to implement validation for javascript Date objects. As we are using it in combination with mongodb this is crucial.

Extending the TypeBuilder is straight forward but extending the compiler is currently not possible. We tried sanitizing the schemata before compiling them but this approach is flawed as this leads to all kind of issues e.g. with union types.

Are there any plans to get support for this in the future or are you open to add the Date type to the lib? Would you accept a PR for that? I guess a simple instanceof Date like it is done for the Uint8Array would do.

Or maybe if the TypeBuilder would define both the Type and the code generation for the validation it would be possible to do this is user space without the need to ensure that compiler and schema are compatible.

[sinclairzx81]

@giggo1604 Hi! Thanks :)

Custom Schemas (Date objects)

At this stage, the TypeCompiler only supports extending custom string Formats (so would only support serializable iso8601 strings or timestamps, but not non-serializable Date objects. There are some plans to enable user defined schema validators relatively soon-ish (some initial design discussions can be found here). However for now I would recommend using Ajv + addKeyword if you specifically need Date object validation. See the link here for configuring Ajv to use custom schemas tied to custom keywords.

Happy to discuss more in this thread if you have some thoughts on exposing an API for custom schemas through the TypeCompiler.

Mongo Schematics

If you're working against MongoDB, it's actually possible to use TypeBox to model Mongo's native bsonType schemas which could be more appropriate as Mongo advertises support for native Date and ObjectId bson types. Some information on Mongo schematics can be found here, and a reference example of how to use TypeBox map over Mongo schemas below.

TypeScript Link Here

Example

// ------------------------------------------------------------------------------
//
// TypeBox: MongoDB BsonType Reference
//
// ------------------------------------------------------------------------------

import { Type, Static, TUnsafe } from '@sinclair/typebox'
import { ObjectId, Binary } from 'mongodb'

export interface BsonObjectOptions {
  minProperties?: number
  maxProperties?: number
  additionalProperties?: boolean
}

export interface BsonArrayOptions {
  additionalItems?: boolean
  minItems?: number
  maxItems?: number
  uniqueItems?: number
}

export namespace BsonType {
  export function Boolean() {
    return Type.Unsafe<boolean>({ bsonType: 'bool' })
  }
  export function String() {
    return Type.Unsafe<string>({ bsonType: 'string' })
  }
  export function Double() {
    return Type.Unsafe<number>({ bsonType: 'double' })
  }
  export function Decimal() {
    return Type.Unsafe<number>({ bsonType: 'decimal' })
  }
  export function Uuid() {
    return Type.Unsafe<string>({ bsonType: 'uuid' })
  }
  export function BinData() {
    return Type.Unsafe<Binary>({ bsonType: 'binData' })
  }
  export function Long() {
    return Type.Unsafe<number>({ bsonType: 'long' })
  }
  export function Null() {
    return Type.Unsafe<number>({ bsonType: 'null' })
  }
  export function Date() {
    return Type.Unsafe<Date>({ bsonType: 'date' })
  }
  export function ObjectId() {
    return Type.Unsafe<ObjectId>({ bsonType: 'objectId' })
  }
  export function Array<T extends TUnsafe<any>>(schema: T, options: BsonArrayOptions = {}) {
    return Type.Unsafe<Static<T>[]>({
      ...options,
      bsonType: 'array',
      items: schema
    })
  }
  export function Object<T extends Record<string, TUnsafe<any>>>(properties: T, options: BsonObjectOptions = {}) {
    return Type.Unsafe<{[K in keyof T]: Static<T[K]>}>({
      ...options,
      bsonType: 'object',
      properties
    })
  }
}

// ------------------------------------------------------------------------------
//
// Example
//
// ------------------------------------------------------------------------------

type T = Static<typeof T>

const T = BsonType.Object({
  id: BsonType.ObjectId(),
  name: BsonType.String(),
  date: BsonType.Date(),
  items: BsonType.Array(Type.Object({
    x: BsonType.Double(),
    y: BsonType.Double(),
    z: BsonType.Double(),
  }))
})

Hope this helps!
S

[giggo1604]

Thanks for your in-depth reply. I already evaluated the option to use Ajv but this would be an additional dependency and from a performance perspective it can't compete with TypeBox. As we got like 99% of the way this would be a bummer.

Just to be on the same page on the mongodb problem. The cool thing about the mongodb node driver is that date objects will be stored as a date type automatically.

collection.insertOne({ createdAt: new Date() });

So in this example the driver will store createdAt as a proper date (which I guess means storing it as a BSON date) which is really convenient. Unfortunately right now we can not ensure this with Typebox.

[sinclairzx81]

@giggo1604 Perhaps as an alternative to extending the compiler (noting custom schemas may not make it in before the end of the year), it would be technically feasible to just implement a Type.Date() type in TypeBox. Historically I've been somewhat reluctant to introduce types that are either outside the JSON Schema specification or happen to be non-serializable....however....

TypeBox does include an Extended set of types which are documented as such and which are mostly used to express typed contractual interfaces in this downstream project. However the odd type out in the extended set is currently Uint8Array which is neither JSON Schema expressible or JSON serializable (and requires a binary message encoding for it to be useful at all). Given Uint8Array has been in for several months now and hasn't caused much confusion, it may be time to re-evaluate Date (which has actually been highly requested since this projects inception)

As it stands, the current extended type set is due for a refactor (mostly to clean up the specialized keyword, and to provide a consistent structure for Ajv to be able to implement these extensions via addKeyword). I may take a look at this either this week or next, and potentially include Type.Date() in the process (which would include implementation in the compiler, value and error reporting)

How's that sound?

[giggo1604]

@sinclairzx81 That sounds awesome. I could help out with a PR on that but I guess if you are already planning to refactor it, it is easier to not do so.

[sinclairzx81]

@giggo1604 I'll try take a look at this later in the week. I did have a couple of reservations around dates being neither json or binary serializable, but do note that @msgpack/msgpack does seem to support date serialization just fine. I think that ticks enough boxes for Type.Date() to finally be included in the extended type set.

import * as msgpack from '@msgpack/msgpack'

setInterval(() => {
    const encoded = msgpack.encode({ date: new Date() })
    const decoded = msgpack.decode(encoded)
    console.log(decoded) // seems correct
}, 1000)

I'll update this discussion when there's news to report.
Cheers
S

[sinclairzx81]

@giggo1604 TypeBox has been updated to finally include a Type.Date() type.

const C = TypeCompiler.Compile(Type.Date())

const OK   = C.Check(new Date())     // const OK   = true

const FAIL = C.Check(42)             // const FAIL = false

The updates for this are on 0.25.0 with information on the new type documented on #271. Have included extra documentation for Ajv users here with configurations for both Standard and Extended type sets (note, Ajv can only reasonably type check Type.UInt8Array(), Type.Date() and TypeBox's current interpretation of Type.Void()).

Have updated the TypeCompiler, Value, Conditional and Error Reporting API's to support the Date type by default.

Hope that helps!
S

[sinclairzx81]

@giggo1604 Hi,

Just an update, TypeBox now provides the ability to extend the compiler and value API's with user defined types. Documentation on this feature can be found here.

The following example shows creating a user defined Date type.

TypeScript Link Here

import { TypeCompiler } from '@sinclair/typebox/compiler'
import { Type, Kind } from '@sinclair/typebox'
import { Custom } from '@sinclair/typebox/custom'

Custom.Set('Date', (schema, value) => value instanceof Date) // custom type check

const T = Type.Unsafe<Date>({ [Kind]: 'Date' })              // custom type schema

const R = TypeCompiler.Compile(T).Check(new Date())          // true

As this functionality has landed a little earlier than expected, I'm going to let it settle till early next year, but will likely look to deprecate both Type.Date() and Type.Uint8Array() in 0.26.x (suggesting users construct their own user defined types moving forward). If you have an existing implementation currently using Type.Date(), would be interested to get some feedback on the design off Custom before flagging these types as deprecated.

Will mark this thread as resolved, but happy to discuss moving from Type.Date() to a custom type. (getting some insights here would be very helpful)
Cheers
S

[giggo1604]

@sinclairzx81 Hi,

Awesome that this feature already landed.

We currently have a PR up which is going to introduce TybeBox in our code base. We already introduced a wrapper package where we extend TypeBuilder. Currently we just added some convenience types like Nullable and a custom string type to it but I guess we could just do the same with a custom date type which would be a small change contained to our wrapper package.

I guess the real upshot is that it might be a good idea to document extending TypeBuilder. Otherwise people will end up with solutions which depend on strings matching by convention which is kind of error-prone. So I guess it would make sense to promote an Implementation like below if custom formats or types are needed.

// @acme/validate

import { TypeCompiler } from '@sinclair/typebox/compiler'
import { TypeBuilder } from '@sinclair/typebox';
import { Kind } from '@sinclair/typebox'
import { Custom } from '@sinclair/typebox/custom'

const KindDate = "CustomDate"

Custom.Set(KindDate, (schema, value) => value instanceof Date)

class AcmeTypeBuilder extends TypeBuilder {
    CustomDate() {
        return this.Unsafe<Date>({ [Kind]: KindDate })
    }
}

export const Type = new AcmeTypeBuilder();

// using it somewhere else 

const T = Type.CustomDate();
const R = TypeCompiler.Compile(T).Check(new Date());

[sinclairzx81]

@giggo1604 Hi, that's a good idea.

Historically extending the TypeBuilder was the actually recommended way to add additional schema types, but many users generally opt for Type.Unsafe() (which was actually introduced much much later). I think for framework implementers, extending the TypeBuilder is the most ideal way to go (as types can be added or overridden to match the constructs of the framework), where user space developers can leverage Type.Unsafe() for one off types.

Extending the TypeBuilder for framework integration has a precedent here if you're interested https://github.com/sinclairzx81/sidewinder/blob/master/libs/contract/type.ts#L92-L100

I think I will make an effort to document the two approaches when I next get some time to sit down with TypeBox.

Thanks for your inputs, let me know how you get on with your integration!
Cheers
S