Directives support, elaborator rework and query algebra simplification

[milessabin]

This excessively large PR is the result of the collision of three overlapping objectives. Whilst they could have been
split apart, that would have resulted in a non-trivial amount of rework as each incremental change was applied.

The three objectives are,

• Full support of custom query and schema directives.

  Directives are a major GraphQL feature (schema directives are an important part of typical GraphQL access control
  schemes) and it would be premature to declare a 1.0 release without support.

• Simplify the query algebra.

  The query algebra has accumulated a number of odd cases. For the most part these are unproblematic, but Rename in
  particular has proven to be awkward. It manifests in user code as the need to use the PossiblyRenamedSelect
  extractor in effect handlers, and this is confusing and error prone.

• Support threading of state through the query elaborator.

  This enables, amongst other things, a simpler implementation of CSS-style cascading field arguments (eg. to
  implement filter criteria which should be inherited by children).

Support for directives means adding directives attributes throughout the schema and at various places in the query
algebra, notably Select nodes. The most straightforward way of disposing of the Rename node was to incorporate the
alias it carries directly into the Select node. The combination of these two changes, along with the invariants
associated with them, made the current approach taken to the select elaborator likely to be too burdensome for end
users. As a result it seemed sensible to take another look at how select elaboration and rework it in the light of the
desire to also thread state through the process.

Hence the size of this PR.

Directive support

Directives can be applied to both the GraphQL schema and to queries. Typically directives will be processed by one or
more additional phases during query compilation. Included in the tests are two examples, one demonstrating query
directives (that is, directives which are added to the query by the client), and one demonstrating schema directives
(that is, directives which are added to the query by the application developer).

The query directive example adds an @upperCase directive which may be applied to query fields which have String
values,

type Query {
  user: User!
}
type User {
  name: String!
  handle: String!
  age: Int!
}
directive @upperCase on FIELD

If clients apply this directive to a String field, the result will be upper cased,

query {
  user {
    name
  }
}

yields,

{
  "data" : {
    "user" : {
      "name" : "Mary"
    }
  }
}

whereas,

query {
  user {
    name @upperCase
  }
}

yields,

{
  "data" : {
    "user" : {
      "name" : "MARY"
    }
  }
}

This is implemented as an additional phase which is added to the Mapping,

object QueryDirectivesMapping extends ValueMapping[IO] {
  // ... details elided ...

  // Transform which modifies the query to apply upper case logic to fields which
  // have the `@upperCase directive applied.
  object upperCaseElaborator extends Phase {
    override def transform(query: Query): Elab[Query] =
      query match {
        case UntypedSelect(nme, alias, _, directives, _) if directives.exists(_.name == "upperCase") =>
          for {
            c    <- Elab.context
            fc   <- Elab.liftR(c.forField(nme, alias))
            res  <- if (fc.tpe =:= ScalarType.StringType)
                      super.transform(query).map(TransformCursor(toUpperCase, _))
                    else
                      // We could make this fail the whole query by yielding Elab.failure here
                      Elab.warning(s"'@upperCase' may only be applied to fields of type String") *>
                        super.transform(query)
          } yield res
        case _ =>
          super.transform(query)
      }

    def toUpperCase(c: Cursor): Result[Cursor] =
      FieldTransformCursor[String](c, _.toUpperCase.success).success
  }

  // The phase is added to sequence of compiler phases.
  override def compilerPhases: List[QueryCompiler.Phase] =
    List(upperCaseElaborator, selectElaborator, componentElaborator, effectElaborator)
}

This phase identifies Select nodes with the @upperCase directive applied then, if the type of the corresponding
field is String, it adds a field transform which converts the value to upper case when the compiled query is
executed.

Note that this logic applies to all fields defined by the schema, without requiring any field-specific logic.

The schema directive example adds a simple access control mechanism along the lines of the one described here.

The schema defines @authenticated and @hasRole directives, along with an enum type which defines USER and
ADMIN roles. Fields and types with the @authenticated directive applied are only accessible to authenticated
users, and fields and types with the @hasRole(requries: <role>) directive applied require both authentication and
possession on the specified role.

type Query {
  products: [String!]!
  user: User! @authenticated
}

type Mutation @authenticated {
  needsBackup: Boolean!
  backup: Boolean! @hasRole(requires: ADMIN)
}

type User {
  name: String!
  email: String! @hasRole(requires: ADMIN)
  phone: String  @hasRole(requires: ADMIN)
}

directive @authenticated on FIELD_DEFINITION | OBJECT
directive @hasRole(requires: Role = ADMIN) on FIELD_DEFINITION | OBJECT

enum Role {
  ADMIN
  USER
}

Here user information is only accessible to authenticated users, and access to e-mail and phone information in
addition requires the ADMIN role. The type level @authenticate directive applied to Mutation means that mutatios
are only available to authenticated users.

object SchemaDirectivesMapping extends ValueMapping[IO] {
  // ... details elided ...

  case class AuthStatus(role: String)

  object permissionsElaborator extends Phase {
    override def transform(query: Query): Elab[Query] = {
      def checkPermissions(
        c: Context, name: String,
        status: Option[AuthStatus],
        query: Query,
        nullAndWarn: Boolean
      ): Elab[Query] = {
        val dirs = c.tpe.directives ++ c.tpe.fieldInfo(name).map(_.directives).getOrElse(Nil)
        val requiresAuth = dirs.exists(_.name == "authenticated")
        val roles =
          dirs.filter(_.name == "hasRole").flatMap(_.args.filter(_.name == "requires"))
            .map(_.value).collect {
              case EnumValue(role) => role
            }
        val requiresRole =
          if (roles.contains("ADMIN")) Some(AuthStatus("ADMIN"))
          else if (roles.contains("USER")) Some(AuthStatus("USER"))
          else None

        (status, requiresAuth, requiresRole) match {
          case (None, false, None) => Elab.pure(query)
          case (Some(_), _, None) => Elab.pure(query)
          case (Some(AuthStatus("ADMIN")), _, _) => Elab.pure(query)
          case (Some(AuthStatus("USER")), _, Some(AuthStatus("USER"))) =>
            Elab.pure(query)
          case _ =>
            if (!nullAndWarn) Elab.failure(s"Unauthorized")
            else
              for {
                _ <- Elab.warning(s"Unauthorized access to field '$name' of type '${c.tpe}'")
              } yield TransformCursor(NullFieldCursor(_).success, query)
        }
      }

      query match {
        case s: UntypedSelect =>
          for {
            c      <- Elab.context
            status <- Elab.env[AuthStatus]("authStatus")
            query0 <- super.transform(query)
            res    <- checkPermissions(c, s.name, status, query0, s.name == "phone")
          } yield res

        case _ =>
          super.transform(query)
      }
    }
  }

  override def compilerPhases: List[QueryCompiler.Phase] =
    List(permissionsElaborator, selectElaborator, componentElaborator, effectElaborator)
}

As with the query directive example, this mechanism is implemented as a compiler phase. We have left it to the service
that Grackle is embedded in to assemble authentication information and provided to the compiler as an AuthStatus
value in the environment under the "authStatus" key. The phase transforms the query as before, this time checking
permissions at each select. If the permission check is successful the select is left unmodified. If the permission
check is unsuccessful then either the entire query fails or, in the case of the phone field of User the permission
violation generates a warning and the field value is returned as null.

Additional changes

• Added directives to all applicable operation, query and type elements.
• Added full directive location, argument and repetition validation logic.
• Schema parser/renderer now has full support for directives.
• Renamed Directive to DirectiveDef. The type Directive is now an application of a directive.
• Added DirectiveDefs for skip, include and deprecated rather than special casing in introspection and directive
  processing.
• Properly handle directive locations in introspection.
• Query minimizer now has full support for directives.
• Added query directive test as a demo of custom query directives.
• Added a schema directive based access control/authorization test as a demo of custom schema directives.

Elaborator rework

To support the threading of state through the query elaboration process an elaboration monad Elab has been
introduced. This,

• Provides access to the schema, context, variables and fragments of a query.
• Supports the transformation of the children of Select to provide semantics for field arguments.
• Supports the addition of contextual data to the resulting query both to drive run time behaviour and to support
  propagation of context to the elaboration of children.
• Supports the addition of queries for additional attributes to the resulting query.
• Supports context queries against the query being elaborated.
• Supports reporting of errors or warnings during elaboration.

The following, taken from the Star Wars demo illustrates the change from the old select elaborator to the new scheme,

// Old select elaborator
override val selectElaborator = new SelectElaborator(Map(
  QueryType -> {
    case Select("hero", List(Binding("episode", TypedEnumValue(e))), child) =>
      Episode.values.find(_.toString == e.name).map { episode =>
        Select("hero", Nil,
          Unique(Filter(Eql(CharacterType / "id", Const(hero(episode).id)), child))
        ).success
      }.getOrElse(Result.failure(s"Unknown episode '${e.name}'"))

    case Select(f@("character" | "human" | "droid"), List(Binding("id", IDValue(id))), child) =>
      Select(f, Nil, Unique(Filter(Eql(CharacterType / "id", Const(id)), child))).success
  }
))

// New select elaborator
override val selectElaborator = SelectElaborator {
  case (QueryType, "hero", List(Binding("episode", EnumValue(e)))) =>
    for {
      episode <- Elab.liftR(Episode.values.find(_.toString == e).toResult(s"Unknown episode '$e'"))
      _       <- Elab.transformChild(child =>
                   Unique(Filter(Eql(CharacterType / "id", Const(hero(episode).id)), child))
                 )
    } yield ()

  case (QueryType, "character" | "human" | "droid", List(Binding("id", IDValue(id)))) =>
    Elab.transformChild(child => Unique(Filter(Eql(CharacterType / "id", Const(id)), child)))
}

The most obvious differences visible here are,

• Instead of the select elaborator being parametrized with a Map[TypeRef, PartialFunction[Select, Query]] we instead
  have a PartialFunction[(Type, String, List[Binding]), Elab[Unit]. This means that instead of user code having to
  unpack all the contents of a Select, including a possible alias and directives, it's handed only the typically
  relevant components. Moving the Type into the argument of the partial function also allows for sharing of logic
  between similar fields of different GraphQL types.
• Instead of constucting a complete new Select node (which is error prone) the Elab monad allows a function to be
  applied to the child only, leaving the fiddly work of reconstructing the resulting Select to the library.

As well as covering the existing use cases more safely, the new elaborator supports more complex scenarios smoothly.
The cascade demo/test provides an example of CSS-style argument cascades from parent fields to child fields,

type Query {
  foo(filter: FooFilter, limit: Int): Foo
}
type Foo {
  bar(filter: BarFilter, limit: Int): Bar
}
type Bar {
  foo(filter: FooFilter, limit: Int): Foo
}
input FooFilter {
  foo: String
  fooBar: Int
}
input BarFilter {
  bar: Boolean
  fooBar: Int
}

Here we want the value of fooBar in a FooFilter argument of a foo field to be propagated to the value of
fooBar in a BarFilter argument of a child bar field, and vice versa, recursively. Whilst this was possible
previously it required a fairly complex initial pass over the query to propagate the values. Under the new schema this
is a great deal more straightforward,

case class CascadedFilter(
  foo: Option[String],
  bar: Option[Boolean],
  fooBar: Option[Int],
  limit: Option[Int]
)

override val selectElaborator = SelectElaborator {
  case (QueryType | BarType, "foo",
         List(Binding("filter", FooFilter(filter)), Binding("limit", TriInt(limit)))) =>
    for {
      current0 <- Elab.env[CascadedFilter]("filter")
      current  =  current0.getOrElse(CascadedFilter.empty)
      _        <- Elab.env("filter", filter(current).withLimit(limit))
    } yield ()

  case (FooType, "bar",
         List(Binding("filter", BarFilter(filter)), Binding("limit", TriInt(limit)))) =>
    for {
      current0 <- Elab.env[CascadedFilter]("filter")
      current  =  current0.getOrElse(CascadedFilter.empty)
      _        <- Elab.env("filter", filter(current).withLimit(limit))
    } yield ()
}

Here we take advantage of the ability to read values from an environment inherited from parent nodes in the query and
to update values in the environment that is passed to child nodes and the elaboration proceeds.

Query algebra changes

• Removed Skipped, UntypedNarrows and Wrap query algebra cases.
• Added UntypedFragmentSpread and UntypedInlineFragment query algebra cases primarily to provide a position in the
  query algebra for directives with the FRAGMENT_SPREAD and INLINE_FRAGMENT location to live.
• Removed Rename query algebra case and added alias field to Select, The args and directives fields of
  Select have been moved to a new UntypedSelect query algebra case.
• Count nodes now correspond to values rather than fields and so no longer have a field name and must be nested
  within a Select.
• Added operation name and directives to untyped operation types.
• Renamed Query.mapFields to mapFieldsR and introduced a non-Result using mapFields.

Type/schema changes

• The Field.isDeprecated and deprecationReason fields have been removed and redefined in terms of directives.
• Schema.{queryType, mutationType, subscriptionType} now correctly handle the case where the declared type is
  nullable.
• Added fieldInfo method to Type.
• Removed redundant Type.withField and withUnderlyingField.
• Renamed EnumValue to EnumValueDefinition.
• Renamed EnumType.value to valueDefinition. EnumType.value new returns an EnumValue.
• Added Value.VariableRef.
• Collected all value elaboration and validation into object Value.

Query parser/compiler changes

• QueryParser.parseText incompatible signature change. Now returns full set of operations and fragments.
• QueryParser.parseDocument incompatible signature change. Now returns full set of operations and fragments.
• QueryParser.parseXxx removed redundant fragments argument.
• QueryParser directive processing generalized from skip/include to general directives.
• QueryParser.parseValue logic moved to SchemaParser.
• QueryCompiler.compileUntyped renamed to compileOperation.
• Fragments are now comiled in the context of each operation where there are more than one. This means that fragments
  are now correctly evaluated in the scope of any operation specific variable bindings.
• The parser now recognises arguments as optional in directive definitions.
• The parser now supports directives on operations and variable definitions.

Mapping/interpreter changes

• Removed redundant QueryExecutor trait.
• Renamed Mapping.compileAndRunOne and Mapping.compileAndRunAll to compileAndRun and compileAndRunSubscription
  respectively.
• Removed Mapping.run methods because they don't properly accumulate GraphQL compilation warnings.
• Added RootEffect.computeUnit for effectful queries/mutations which don't need to modify the query or any special
  handling for root cursor creation.
• RootEffect.computeCursor and RootStream.computeCursor no longer take the query as a argument.
• RootEffect.computeQuery and RootStream.computeQuery have been replaced by computeChild methods which transform
  only the child of the root query. Uses of computeQuery were the main places where PossiblyRenamedSelect had to
  be used by end users, because the name and alias of the root query had to be preserved. By only giving the end user
  the opportunity to compute the child query and having the library deal with the top level, this complication has
  been removed.
• RootStream.computeCursor no longer takes the query as a argument.
• QueryInterpreter.run no longer generates the full GraphQL response. This is now done in
  Mapping.compileAndRun/compileAndRunSubscription, allowing compilation warnings to be incorporated in the GraphQL
  response.
• Various errors now correctly identifed as internal rather than GraphQL errors.

Miscellaneous changes

• Added Result.pure and unit.
• Fixes to the order of accumulation of GraphQL problems in Result.
• Fixed ap of Parallel[Result]'s Applicative to accumulate problems correctly.
• Unnested Context and Env from object Cursor because they are now used in non-cursor scenarios.
• Added NullFieldCursor which transforms a cursor such that its children are null.
• Added NullCursor which transforms a cursor such that it is null.
• Moved QueryMinimizer to its own source file.
• Updated tutorial and demo.

[tpolecat]

Whew, you're not kidding, this is big. I'm going to build this branch and update our code to use it and see how it goes.

[tpolecat]

This is a big improvement and it all works fine with the latest changes. 👍🏻