DevGuide Overview

Overview

vizier/backend/src/info/vizierdb

• types: A repository of Enums and typecasts for the entire project
• Vizier: The entry point into Vizier -- Main function, initialization, etc...
• VizierAPI: Jetty routes (to handlers in .api)
• VizierURLs: URL constructors for the routes in VizierAPI (would be nice to automate this eventually).

vizier/backend/src/info/vizierdb/catalog

Objects for the Vizier ORM. Classes in this package must not create DB sessions; Instead mark methods with implicit session parameters where needed.

• Metadata: Preferences/general settings for the local install
• Project: A single project/notebook
  • has many Artifact: A dataset, file, exported function, etc... created in the course of workflow execution.
  • has many Branch: A sequence of workflows, iterated over time. The most recent workflow is referred to as the head
    • has many Workflow: A sequence of cells.
      • has many Cell: A module at a specific position in a workflow and any resulting execution state.
        • has one Module: A command identifier coupled with a set of arguments to that command. A single module may be shared by multiple cells.
        • may have one Result: The outcome of executing a cell. A single result may be shared by multiple cells. See here for more details.
          • has many [Input]ArtifactRef: An artifact referenced in the execution of this cell.
            • has one Artifact
          • has many [Output]ArtifactRef: An artifact created by executing this cell.
            • has one Artifact
          • has many Message: Logging or data visualization outputs generated as a result of executing a cell.

A Workflow is the heart of Vizier, and represents an ordered sequence of Modules in the notebook. Cell serves as a many-many relationship table, assigning specific Modules to (i) Their position in the workflow, and (ii) The Result of executing them at that point in the workflow. This may seem a little overcomplicated, but it is necessary.

• A single Module may be re-used across workflows. In principle, we could create a copy of the entire workflow every time we modify it, but apart from wasting space, this makes it much harder to track relationships between workflows.
• A single Result may also be re-used across workflows (e.g., when we discover that a module's inputs --- InputArtifactRef that is --- remain unchanged).
• Thus, Cell acts as a 3-way many/many relationship between Workflow, Module and Result. Note that the Result relationship is optional (e.g., if we have not yet figured out whether the cell needs to be recomputed). See here for more details.

The Artifact class is a little tricky for two reasons. First, an Artifact can be of multiple types (see types.ArtifactType), dictating how the artifact behaves. Second, two of these types are odd:

• An ArtifactType.DATASET does not live in Vizier, but rather in Mimir/Spark.
• An ArtifactType.FILE also does not live in Vizier, but rather in the local filesystem.

The Artifact records is mainly present as a placeholder in both cases.

The Schema class is a master specification of the database state that backs the ORM. Scalikejdbc only has limited support for structure creation, and no support for Schema migrations. The Schema class does both.

Gotcha: When changing one of the ORM classes listed above, make sure to apply relevant changes to the Schema class. Additionally as of right now the Schema is locked in: Any changes must be made by adding migrations.

The binders object contains instructions for Scalikejdbc to translate Enums in info.vizierdb.types.

vizier/backend/src/info/vizierdb/commands

Implementation of module command implementations, each of which extend the Command trait. A command has four parts:

• A name that is how the command is displayed in the UI
• A list of parameters that describe the fields that appear in the UI
• A format that generates a short summary text describing what the command is doing based on its arguments.
• A handler that implements the command logic.

To register a Command, use Commands.register (see Commands itself for examples). Commands are registered as part of a package (with a string identifier), and each command has a unique commandId (also a string).

There are three main classes involved in implementing a Command:

Implementations of the Parameter trait are used to describe arguments to the cell. Each Parameter implementation corresponds to a different widget in the UI. Specific details vary for each parameter type, but all Parameter constructors have the following fields:

• id: A unique identifier for this parameter (used in conjunction with Arguments)
• name: A human-readable description for the parameter. This is what is shown in the UI.
• required: If true, the UI will not allow users to omit/leave blank this parameter (default: true)
• default: A default value for the parameter (default: null/None)
• hidden: If true, the UI will pass around this parameter, but will not display it to the user (default: false)

At time of writing, available parameters include:

• BooleanParameter: A checkbox
• CodeParameter: A coding block. The language field must be set to a language supported by the UI (currently sql, scala, python, or r)
• ColIdParameter: A column selector. When this parameter is used, there must also be a DatasetParameter with id = "dataset"; The list of columns will be drawn from this dataset. The "value" of this parameter is the Int position of the selected column.
• DatasetParameter: A dataset selector. The list of datasets used will be appropriate for the point in the notebook. The returned value will be the String userFacingName of the dataset.
• DecimalParameter: A field where the user can enter any Double.
• FileParameter: A file upload area. The uploaded/linked file will be provided as a FileArgument
• IntParameter: A field where the user can enter any Int.
• ListParameter: A variable-height 2-d grid of elements, each specified as a parameter. The name field of component parameters will be used as the column header.
• RecordParameter: A grouping of multiple parameters into a single row in the UI. Behaves like a ListParameter, but with only a single row.
• RowIdParameter: A field where the user can enter any row identifier as a String. Mainly for use in the spreadsheet, and not for human use.
• EnumerableParameter: A drop-down menu. The values are EnumerableValue instances including a name (the user-facing name) and a value (a String) that is provided during command execution.
• StringParameter: A field where the user can enter any String.

---

The Arguments class is used to streamline access to parameter values provided by the UI (which are natively Json objects). Arguments are accessed via four methods:

• get[Type](id): Returns the parameter with id id, casting it to type Type. (It would be fantastic if we could make this more strongly typed, but that level of Scala-fu is not worth it at this point)
• getOpt[Type](id): Like get, but returns an Option[Type].
• getList(id): For use with ListParameter, returns a Seq[Arguments] with one entry per row in the list.
• getRecord(id): For use with RecordParameter, returns the nested Arguments object.

---

Finally, ExecutionContext provides key/value access to artifacts created by preceding cells, and is passed to Command.handle. This class is currently a little slapdash, and needs clean-up, but basically ALL interactions with artifacts should go through it (since it also automatically records accesses). All artifact access is case-insensitive.

Basic functions include

• artifactExists(name): Returns true if the specified artifact is in-scope
• artifact(name): Returns the Artifact object associated with the specified name, or None if the name is undefined
• output(name, artifact): Inserts/replaces the artifact associated with name with artifact
• error(message): Logs an error message.
• message([mimeType, ]content): Logs a status message. If the optional mimeType is provided, one of several special display handlers in the UI can be invoked. For example passing MIME.HTML (="text/html") will cause the message to be rendered as HTML rather than plaintext.

A few additional utility methods also exist to streamline common tasks (e.g., displaying datasets in the output).

Two methods in particular are notable: outputDataset and outputFile. Both datasets and files live outside of the Vizier state, and the corresponding Artifact objects are simply placeholders. Thus, calling outputDataset or outputFile, creates the placeholder, but it is still up to the caller to create the resulting object.

Finally, I'll mention the updateArguments and updateJsonArguments methods. Both methods do basically the same thing: Create a new module with some its arguments replaced and relink the current cell to the new module. If misused, this HAS THE POTENTIALY TO DO VERY VERY BAD THINGS TO REPRODUCIBILITY, SUMMON A HELLSWARM, AND CAUSE THE END OF DAYS. They exist for two very specific use-cases:

1. As part of its normal operation, the Command is able to infer some of its parameters from the input. For example, LoadDataset is able to infer the schema of the loaded dataset. When parameters are inferred, it often makes sense to codify the inferred parameters in the module specification to allow the user to override them. This is *only safe if you can guarantee that executing the module before and after the replacement will produce exactly the same output.

2. As part of its normal operation, the Command has non-deterministic behavior. For example, command.sample._ need to generate a random sample seed. It is safe to store seed values to make subsequent cell executions deterministic.

vizier/backend/src/info/vizierdb/viztrails

Most of the execution logic lives here, between four classes: Provenance, Scheduler, RunningWorkflow and RunningCell.

Provenance is a set of utility methods for dealing with sequences of Cells (i.e., subsets of Workflows). Its methods serve three main roles:

• Given a sequence of Cell objects, link their OutputArtifactRefs together to compute the state (scope) of a workflow after those cells were executed.
• Determine whether the Result for a Cell is safe for re-use from a previous execution (by comparing the InputArtifactRefs to the scope generated by the preceding cells).
• Compute the next Cell in an incomplete Workflow to need re-execution.

In general Provenance is also responsible for maintaining the Cell state machine documented here.

Scheduler is the entry point for workflow re-execution. Its main role is to maintain and manage a list of RunningWorkflow objects, and the corresponding WorkerPools. There are four main methods here:

• schedule(workflowId): Starts up a new RunningWorkflow for the specified workflow, if one does not already exist.
• abort(workflowId): If the specified Workflow has executing cells, cancel execution.
• isWorkflowPending(workflowId): Return true if the specified Workflow is currently using resources on execution.
• joinWorkflow(workflowId): Block the current thread until there is no further computation happening on the specified Workflow.

RunningWorkflow handles identification of cells that need (re-)execution. Broadly, it looks for Cells in the STALE state, executes them by creating a RunningCell, and then invokes Provenance to update the remaining Cell state machines.

RunningCell handles execution of a cell. Most of this code is setup and teardown code, needed to e.g., create an ExecutionContext for the cell, parse arguments, measure timing, and interact with the delta bus.

Gotcha: Scheduler operates asynchronously, so it's important that any Workflow construction is completed and committed before you call Scheduler.schedule. Essentially, make sure that you call Scheduler.schedule outside of the DB.autocommit block used to create the workflow. See, for example, info.vizierdb.api.AppendModule

Currently, the heavy asynchronous lifting is done by Java's ForkJoinPool and ForkJoinTask (which is subclassed by Scheduler.WorkflowExecution). These iteratively identify the next Cell for execution, and then call Scheduler.processSynchronously, which loads all relevant context, allocates a Result object, and invokes the appropriate Command handler.

---

The last thing in this package is the MutableProject class. This is mainly meant for testing purposes, but kind of acts like a "client" wrapper for Catalog. It's not really complete, but is meant to abstract away some of the immutability of the append-only catalog model.

vizier/backend/src/info/vizierdb/api

This package includes all of the handlers referenced in VizierAPI. These adopt the API handler model used by mimir-api. Handlers are expected to implement a handle method that produces a Result. Several flavors of Result exist, each producing different forms of output (e.g, errors, Json data, etc...)

VizierServelet, which is stored with VizierAPI is where most of the action happens. It's a little crufty, but the basic flow is that the handle method is a big ol' routing table stored as a giant match statement. Pre-defined regular expressions (e.g., PROJECT, BRANCH, etc...) govern how the query is routed. At each endpoint, we use the respond nested method to map HTTP verbs to the corresponding response. The respond method exists mainly to allow us to automatically synthesize a response to the OPTIONS verb, which needs to know which HTTP verbs are supported at this endpoint. Responses are generated by either the process or processJson methods. The former simply calls the handle method of the passed Request (along with a standard array of try/catch blocks, etc...), while the latter first tries to decode the input as Json corresponding to an object of the method's type parameter. processJson can optionally add fields to the object (e.g., parsed out of the path).

It's crufty, but it works for now, is compact, and standard enough to be easily readable. One thing that I would like to fix at some point is to make the response constructors lazy. That leads me to:

Gotcha: At present, Requests handed by process are constructed before we decide which HTTP verb handles the request. This can lead to some odd exceptions, for example if one of the calls to .toLong fail.

vizier/backend/src/info/vizierdb/delta

This package contains logic for asynchronous ('push') notifications of updates to workflows, mainly for use by the Websocket. State changes are a bit scattered around the Catalog and Scheduler, so DeltaBus creates a central clearinghouse for them. State is published to DeltaBus, and clients (e.g., the websocket) can subscribe.