koncierge 🛎

Evaluating AB testing variants, given an experiment definition and some context.

Rationale

We have a microservice that is in charge of knowing whether a user should be part of an experiment or not, and if they are, whether they should be in the control group, or the experiment group.

The reasons why a user might participate or not are very varied, and change drastically from experiment to experiment.

We want to have the experiment's microservice be somewhat agnostic of the rest of the microservice infrastructure; so whoever consumes the information must provide enough information about the user for us to choose what experiments they should be part of.

We found that knowing if a user participates in an experiment, and it figures out what variant they fall into are really similar concepts, so we came up with the idea of having "variants all the way". So an experiment is really just a variant on the world. The same syntax can apply to narrow down the focus of the experiment. A good example of this can be found in the Variant all the way example.

Syntax

We borrowed heavily for Mongo's query language.

Operators

These are the fields that start with a $. There are two families of operators, describe below.

Context changers

These operations change the context, either narrowing down the provided context, or by plucking some information from the World. Think of them as :: Context -> Context.

Name	Description	Example
any non-$ string	Will dive into the context to the key with the same name. (can deep dive by combining keys with .s)	{ "foo.bar": "yes" }
$rand	Chooses a value at random between 0 and 1. Uses the current context as seed*	{ "$rand": { "$gt": 0.5 } }
$chaos	Chooses a value at random between 0 and 1. This, compared to ☝️does not use the current context	{ "$chaos": { "$gt": 0.5 } }
$date	Changes the context with the current date. Useful when convincing with other comparison operator	{ "$date": { "$gt": "2020-01-01 15:00:00" } }
$size	Changes the context to the size of the current context. This can be an array length, a string length or an object number of fields, or 0 if null	{ "$size": { "$gt": "1" } }
$not	Negates the context.	{ "$not": { "$gt": "1" } }

Evaluators

These operations yield weather the context is or not par of the variant. Think of them as :: Context -> Bool.

Name	Description	Example
$lt, $gt	Will be enabled if the context if less than (lt) or greater than (gt) the value provided **	{ "$gt": 0.5 }
$or	Will be enabled if any of the sub-operators are enabled	{ "$or": { "beta": "yes", "$rand" : 0.5 } }
$always	Will always evaluate to its (boolean) value, regardless of the context. See the exists example	{ "$always": true }
$any, $all	Will be enabled if (any/all) the elements in the context (which should be an array) are enabled	{ "$any": { "$gt": 0.5 } }

Where a context changer can be nested, evaluators need to be terminal.

Even though there are both $and and $eq operators; they are rarely used, as they can be expressed more concisely. Refer to the And example for more information.

Examples

You can play around with different experiments and contexts in our koncierge playground.

Simple

Everyone is participating in the experimentEXP001, and only half of the users (chosen at random) will see the experiment.

{
    "EXP001": {
        "$children": {
            "participating": { "$rand": { "$gt": 0.5 } },
            "control": { "$always":  true }
        }
    }
}

Playground

The output for half the user base will be: EXP001.participating and EXP001.control for the other half.

Variant all the way

{
    "EXP001": {
        "$date": { "$gt": "2020-01-01 15:00:00" },
        "$children": {
            "participating": { "$rand": { "$gt": 0.5 } },
            "control": { "$always":  true }
        }
    }
}

Playground

The output will be the same as the simple example, but only if it is later than the January first 2020.

Missing children

{
    "EXP001": {
        "$children": {
          "never": { "$always":  false }
        }
    }
}

Playground The output will be EXP001 even if it has no active child.

And

{
    "EXP001": {
        "$date": { "$gt": "2020-01-01 15:00:00" },
        "beta": "yes"
    }
}

Playground The output will be EXP001 if the context has a key beta as yes and is queried after January first 2020.

Exists

{
    "EXP001": {
        "beta": {
            "$always": true,
            "$not": { "$eq": null }
        }
    }
}

Playground The output will be EXP001 if the context has a key beta, and it is not null.

Uncalled for

{
   "EXP001": {
        "$children": {
            "control": { "$always":  true },
            "participating": { "$rand": { "$gt": 0.5 } }
        }
    }
}

Playground

This will always output EXP001.control, as we parse children rules sequentially.

More examples

You can check out the /test folder for more examples and edge cases, as well as playing in the Playground

Usage

The library is divided into two major namespaces.

1. Parse (Left Kan).
2. Interpret (Right Kan).

A sub-section of the interpretation is also to validate that the given context can be matched to the rules.

Parser

A function that takes the full JSON definition of the experiments and transforms it into objects that can be then interpreted

Interpreter

A function that takes the parsed definition, the context provided, and a World and outputs the list of experiments (and variants) the context is part of

Context validation

We validate the context in an interpreter-agnostic way. This means that even though semantically, some rules we might never call (see the Uncalled for example), the context will still need to match every possible child rule.

Extras

Date

Both $gt and $lt only work with numbers, so dates will be compared by their epoch second. The parser can interpret some date formats (read more about the date formats here). Formats with no timezone, we choose the default system timezone. This is not recommended.

Format	Example	Notes
yyyy-MM-dd	2020-10-05	The time is at 00:00 from the system's timezone
EEE MMM dd yyyy HH:mm:ss 'GMT'Z (z)	Tue Oct 06 2020 00:30:00 GMT+0200 (Central European Summer Time)	JavaScript's default Date format
yyyy-MM-dd'T'HH:mm:ss.SSSZ	2020-10-05T22:20:00.000+0200	Recommended ISO 8601-2:2019 standard

Randomness

Even though $rand and $chaos might look similar; they differ on the seed for its randomness. With $rand, any random value generated with the same context and variant name, will be the same output.

For this reason, most of the times you'll want to narrow down the context before applying $rand. For example, given this context:

{
    "user_id": 3,
    "last_login": "2020-10-15T10:00:00.000Z"
}

the last_login will probably change throughout the time, but we need the experiment to be fixed by the user_id. In such a scenario, we can write the experiment as such:

{
    "EXP001": {
        "user_id": {
            "$rand": { "$gt": 0.5 }
        }
    }
}

Every time the user_id: 3 queries the experiment, the $rand value will be the same.

In contrast, $chaos will generate a new value each time, so there is no guarantee in what variant user_id: 3 will fall.

Rolling experiment updates

As we generate the same number for each context, you can safely update the threshold and rest assured that the minimum number of clients will change variants.

TODO

1. Escape the $ to be able to match to $ keys, and the . in keys to match not-nested keys with .