feat(generate): generate pkl testing

[toniphan21]

Introduce new command generate to generate some code for testing using pkl-lang.

Motivation

I want to have a testing system for an Authorization Model that:

• Independent from authorization model
• Easy to write with type safety and auto-completion
• Easy to read so non-technical people can read and verify the test scenario

Solution

Using pkl-lang to write tests instead of yaml. The process of testing is:

• Tests are written in pkl lang, which is type safe and have IDE integrations
• All tests will be translated to yaml file and use fga model --tests command to perform test

However, written tests in pkl lang is not the completed solution, we need something to decouple the business logic and authorization model. It means if the authorization model changed, the changes we make in pkl files should be minimum (it's not possible with yaml because all tuples are tired to the authorization model structure). So the final architecture looks like this:

• The Business Logic is written in pkl and never depends on any authorization model.
• The Adapters is an indirection layer which provide Type and Assertions for Business Logic. Adapters use generated code and adapt them to Business Logic layer
• Generate Pkl is a generated code based on the given authorization model (which is covered in this PR)

Usage example:

• In the system business logic has Employee, Company, and they have some rules need to be tested.
• Developers define the first authorization model, using fga generate pkl to generate code based on that model.
• Developers write Type and Assertions utilities functions on Adapters layer, then write Tests using these Type and Assertions.
• Tests are run and verify by fga model --tests, also they are easy to understand and readable for non-technical people.
• For some reasons, the authorization model need to be changed, say Company now will be renamed to Organization technically and introduce a tree-like structure. So all old business logics are still valid, but the model and tuples aren't. What we need to do is:
  • Write new model
  • Use fga generate pkl to regenerate code
  • Adapt Type and Assertions to the new model

By doing this way there are some advantages:

1. Firstly the tests are much easier to understand compare to yaml file
2. Developer can understand which kind of tuples need to be produced based on Type in the Adapter layer
3. Non-Technical people can participate in early-phase of a model development
4. All Type mistakes could be avoided when writing tests for the model

Changes description

Introduce a new command generate has a subcommand pkl which generate a small pkl code library with assertions and types from given openFGA model. The purpose of this command is provide functionality for Generate Pkl layer. The structure of generated directory looks like:

• testing/lib/test.pkl: small library file to match the syntax of pkl to yaml file for testing a model - this file always be overridden when run the command
• testing/lib/gen.pkl: generated types and assertions based on model - this file always be overridden when run the command
• testing/type.pkl: a types definitions where we can write some functions to make the tests setup looks nicer - only generated if the file is not found.
• testing/assertions.pkl: assertions library where we can write some functions to make the tests assertions looks nicer - only generated if the file is not found.
• testing/test_example.pkl: an example test - only generated if the file is not found.
• testing/run: a simple bash script helper to run a test or all tests - only generated if the file is not found.

More about pkl-lang: https://pkl-lang.org/

Example

Run this command with the model custom-roles in sample-store.

fga generate pkl  --file=path-to-model/model.fga 

The generated contents are:

testing/run

#!/bin/bash
param=$1
if [ "$param" == "all" ]; then
  tests=`ls ./test*.pkl`
  for test in $tests
  do
    echo "--- running test file: $test"
    pkl eval -f yaml $test | fga model test --tests /dev/stdin
    echo "--- finished"
  done
else
  pkl eval -f yaml $param | fga model test --tests /dev/stdin
fi

testing/type.pkl

import "lib/gen.pkl"
import "assertions.pkl"

class Asset extends gen.BaseAsset {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "asset"
  hidden fgaType: String = "asset"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

class AssetCategory extends gen.BaseAssetCategory {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "asset-category"
  hidden fgaType: String = "asset-category"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

class Org extends gen.BaseOrg {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "org"
  hidden fgaType: String = "org"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

class Role extends gen.BaseRole {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "role"
  hidden fgaType: String = "role"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

class Team extends gen.BaseTeam {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "team"
  hidden fgaType: String = "team"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

class User extends gen.BaseUser {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "user"
  hidden fgaType: String = "user"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

testing/assertions.pkl

import "lib/gen.pkl"

class AssetAssertions extends gen.BaseAssetAssertions {
  // you can put some custom assertions in this class

}

class AssetCategoryAssertions extends gen.BaseAssetCategoryAssertions {
  // you can put some custom assertions in this class

}

class OrgAssertions extends gen.BaseOrgAssertions {
  // you can put some custom assertions in this class

}

class RoleAssertions extends gen.BaseRoleAssertions {
  // you can put some custom assertions in this class

}

class TeamAssertions extends gen.BaseTeamAssertions {
  // you can put some custom assertions in this class

}

testing/lib/...

These files are too big, but it looks like this

....
abstract class BaseUser {
  id: String
  hidden type: String = "user"
  hidden fgaType: String = "user"

  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"

  function relation_assignee(i: BaseRole): Mapping = new Mapping {
    ["user"] = toFGAType()
    ["relation"] = "assignee"
    ["object"] = i.toFGAType()
  }

  function relation_member(i: BaseOrg|BaseTeam): Mapping = new Mapping {
    ["user"] = toFGAType()
    ["relation"] = "member"
    ["object"] = if (i.fgaType == "org") i.toFGAType()
      else if (i.fgaType == "team") i.toFGAType()
      else ""
  }

  function relation_owner(i: BaseOrg): Mapping = new Mapping {
    ["user"] = toFGAType()
    ["relation"] = "owner"
    ["object"] = i.toFGAType()
  }

  function assert_org(
    name: String,
    object: BaseOrg,
    customAssertions: BaseOrgAssertions,
    asserts: (BaseOrgAssertions) -> Listing
  ): Mapping = new Mapping {
    ["name"] = name
    ["check"] = new Listing {
      new Mapping {
        ["user"] = toFGAType()
        ["object"] = object.toFGAType()
        ["assertions"] = asserts.apply(customAssertions)
          .fold((new Mapping {}).toMap(), (r: Map, m: Mapping) -> r + m.toMap())
      }
    }
  }
....

Write Test after running the command

the type.pkl is where we put some setup helper functions, such as:

class Org extends gen.BaseOrg {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "org"
  hidden fgaType: String = "org"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  function has_user(i: User) = i.relation_member(this)

  // you could even set up more than 1 relation using this syntax
  function has_something(input: User) = new Listing {
    input.relation_member(this)
    input.relation_owner(this)
  }
}

In the code above, we use relation_member generated from the model to express that an org can has_user() with type safety. Then in the class User:

class User extends gen.BaseUser {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "user"
  hidden fgaType: String = "user"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
    assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

We express that we want to test a user in a specific org, and use generated assert helper handle the check.

Then we can start writing test for multiple-scenarios, each scenarios has it own setup and file, looks like test_example:

import "type.pkl"
import "lib/test.pkl"

// NOTE: this is where you define your type for testing, for example:
local Anna: type.User = new type.User { id = "Anna" }
local OpenFGA: type.Org = new type.Org { id = "OpenFGA" }

suite: test.OpenFGATestSuite = new {
  name = "Awesome test - RENAME ME"
  model = read("../model.fga")
  setup {
    // NOTE: setup your test here, for example:
    Anna.relation_owner(OpenFGA)
    OpenFGA.has_user(Anna)
  }
  tests {
    // NOTE: write your assertions here, for example:
    Anna.in_org(OpenFGA, (she) -> new Listing {
      she.should_be_member
      she.should_be_owner("comment or reason")
    })
  }
}

output { value = test.output_value(suite) }

as you see, in the setup we could use generated function or custom function (which is nicer), and in the tests we could test user Anna in a specific Organization with very nice syntax.

Now, to run the test use run utility, or by using raw command:

# raw command
pkl eval -f yaml test_example.pkl | fga model test --tests /dev/stdin

# run utility
./run test_example.pkl

# run all test file which start with `test*`
./run all

That's all.

Review Checklist

• I have clicked on "allow edits by maintainers".
• I have added documentation for new/changed functionality in this PR or in a PR to openfga.dev [Provide a link to any relevant PRs in the references section above]
• The correct base branch is being used, if not main
• I have added tests to validate that the change in functionality is working as expected

[linux-foundation-easycla]

• ✅login: toniphan21 / (9f28e15)

The committers listed above are authorized under a signed CLA.

[minder-by-stacklok]

Minder Vulnerability Report ✅

Minder analyzed this PR and found no vulnerable dependencies.

Vulnerability scan of ee0c4677:

• 🐞 vulnerable packages: 0
• 🛠 fixes available for: 0

[ewanharris]

Hey @toniphan21, thanks for this PR and the well thought out description to bring us up to speed! ❤️

Just to set some expectations, it might take us a little while to get around to fully reviewing this PR. We'd like to make sure we fully understand the pieces that are new to us (pkl, generation of files outside the model) and how the full model development cycle looks with this change (e.g. local and CI usage, potentially expanding to how this may fit into areas like the playground).

We're definitely excited to see the suggestions you have for improving the testing of a model. Out of curiosity is this something you're using to author tests currently?

[toniphan21]

Hey @ewanharris, thank you for reviewing this PR. I understand it's a big one, so I don't expect the review process to be fast. Also, you can see I haven't added any tests yet. The reason is that I want to gather your feedback before adding them.

To answer your question, yes, I do use these tests at work to verify the model with non-technical stakeholders. That's why readability is the main focus.

Let me walk you through the process of how I came up with the idea, and then I'll suggest some approaches for integrating pkl into openfga's ecosystem.

First, I started with the yaml file for testing a model:

name: test
model: ''
tuples:
- user: user:1
  relation: member
  object: org:1
tests:
- name: whatever
  check:
  - user: user:1
    object: object
    assertions:
      allow: true

Then I wrote a pkl file which could be converted to the exact yaml file above:

name: String = "test"
model: String = ""
tuples: Listing<Tuple> = new {
  new {
    user = "user:1"
    relation = "member"
    object = "org:1"
  }
}
tests: Listing<TestCase> = new {
  new {
    name = "whatever"
    check {
      new {
        user = "user:1"
        object = "object"
        assertions {
          ["allow"] = true
        }
      }
    }
  }
}

class Test {
  name: String
  model: String?
  tuples: Listing<Tuple>
  tests: Listing<TestCase>
}

class Tuple {
  user: String
  relation: String
  object: String
}

class TestCase {
  name: String?
  check: Listing<Check>
}

class Check {
  user: String
  object: String
  assertions: Mapping<String, Boolean>
}

As you can see, the raw pkl file is longer than the yaml file. However, there are some upsides to using pkl:

• I can separate a single pkl file using modules, which means I can import files.
• I can write functions to express business logic in a more readable way. For example:

new {
    user = "user:1"
    relation = "member"
    object = "org:1"
  }

becomes

// somewhere at the top of the pkl file
local Anna = new User { id = "1" }
local OpenFGA = new Org { id = "1" }

Anna.member(OpenFGA) // member is a function of class User which returns the structure above

The benefit is that I can name member whatever I want and still produce the same output. I can condense every setup (producing a tuple) into one line. Another advantage is that in the function, we can produce more than one tuple (for example, a user belongs to an org - one tuple, and the org has the member - another tuple).

In short, the pkl file is just a nicer way to express ideas or business logic; in the end, it will be converted to yaml and tested using the openfga cli.

However, I faced a problem where I have to write a pkl class every time my model changes. That’s why I want to generate pkl files automatically based on the model, saving time and avoiding mistakes.

I suggest that:

• Write documentation on how to convert a pkl file to yaml, merge this MR, and add the fga generate pkl command to the documentation to show users how to create YAML files faster with more readable syntax.
• Continue integrating listObject and listUser functionality into the pkl generation command.
• Integrate pkl files into the openfga CLI, which means running pkl directly instead of converting it to yaml (this task isn’t too big because the structure generated from the pkl file is very similar to yaml, even cleaner, and I’m happy to do it if OpenFGA thinks it makes sense).
• Once the fga cli can run pkl directly, we could integrate it into the playground.

I think this is a lot of work, and it’s not easy to integrate a new language into an ecosystem. However, I believe pkl offers a lot compared to yaml, and generating source code from a model is a good start. Please test this PR with any models in the sample store and give me feedback. I hope the idea is clearer after you've tried it.