Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: convert root dir to primary dagger module #196

Merged
merged 31 commits into from
Dec 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
5d51616
refactor: remove repo directory access workarounds
aweris Nov 21, 2023
a990237
refactor: make workflow walk func private
aweris Nov 22, 2023
fefac13
refactor: extract workflows to separate module
aweris Nov 22, 2023
9c27885
refactor: merge repo module back to gale
aweris Nov 22, 2023
3f9edf8
refactor: merge workflows module back to gale
aweris Nov 23, 2023
d465116
refactor: decouple context from task runner
aweris Nov 23, 2023
5d36aee
refactor: move task pkg to common
aweris Nov 23, 2023
6096592
refactor: remove unused runner cache and id
aweris Nov 23, 2023
523b232
refactor: extract workflow selection to separate function
aweris Nov 24, 2023
2bbc660
refactor: centralize internal components in module scope
aweris Nov 29, 2023
26417bf
refactor: create runner container part of the workflow run
aweris Nov 29, 2023
d96176b
refactor: introduce run context
aweris Nov 30, 2023
fe84cd9
refactor: implement opts types for internal processing
aweris Nov 30, 2023
5a82f03
fix: remove unnecessary unset env var
aweris Nov 30, 2023
32fea0f
refactor: move non workflow config to context
aweris Nov 30, 2023
1a17a0d
refactor: connect workflows to repo as sub entrypoint
aweris Nov 30, 2023
8ae70c3
refactor: move github workflow config to dagger module
aweris Nov 30, 2023
aea8683
fix: use path if workflow name is not provided
aweris Nov 30, 2023
a3a0ae3
refactor: use job_run.json as source of truth
aweris Dec 1, 2023
9f43c0f
refactor: make log group creation for task optional
aweris Dec 1, 2023
e0aaa24
refactor: simplify runner container
aweris Dec 1, 2023
81aa249
refactor: move workflow run config to context
aweris Dec 1, 2023
24990dc
refactor: move workflow orchestration to dagger module
aweris Dec 4, 2023
e613698
refactor: do not expose ghx.Source to outside
aweris Dec 4, 2023
85c1438
refactor: extract container to runner again
aweris Dec 5, 2023
2f7064d
refactor: remove job execution plan
aweris Dec 5, 2023
1352c56
refactor: extract job run report to module
aweris Dec 5, 2023
783832e
refactor: move workflow run report to dagger module
aweris Dec 5, 2023
5fc1e4d
refactor: convert root dir to primary dagger module
aweris Dec 9, 2023
2803aec
refactor: prefix step index to step id to keep execution order steps …
aweris Dec 10, 2023
5316ae4
docs: update inline docs
aweris Dec 11, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 0 additions & 42 deletions .github/workflows/lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,45 +22,3 @@ jobs:
uses: reviewdog/action-golangci-lint@v2
with:
workdir: ${{ matrix.workdir }}

daggerverse-lint:
runs-on: ubuntu-latest

steps:
- name: Check out code
uses: actions/checkout@v3

- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.21

- name: Install Dagger CLI
run: cd /usr/local && { curl -L https://dl.dagger.io/dagger/install.sh | sh; cd -; }

- name: Sync dagger modules
run: dagger export -m ci daggerverse sync --output daggerverse

- name: Check if daggerverse is up to date
run: git diff --exit-code || (echo 'Uncommitted changes detected' && exit 1)

gha-gen-lint:
runs-on: ubuntu-latest

steps:
- name: Check out code
uses: actions/checkout@v3

- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.21

- name: Install Dagger CLI
run: cd /usr/local && { curl -L https://dl.dagger.io/dagger/install.sh | sh; cd -; }

- name: Generate GitHub Actions modules
run: dagger download -m ci gha generate -o daggerverse/gha/

- name: Check if generated Github Actions modules is up to date
run: git diff --exit-code || (echo 'Uncommitted changes detected' && exit 1)
28 changes: 6 additions & 22 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,27 +1,11 @@
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib

# Test binary, built with `go test -c`
*.test

# Output of the go coverage tool, specifically when used with LiteIDE
*.out

# Dependency directories (remove the comment below to include it)
# vendor/

# project directories
build/
bin/
dist/
.gale/
# Dagger generated files
/dagger.gen.go
/internal/querybuilder/
/querybuilder/

# IDE files
.idea/

# Local configuration files
.envrc
.envrc
.gale/
4 changes: 3 additions & 1 deletion .golangci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,6 @@ linters-settings:
- github.com/google/uuid
- github.com/caarlos0/env/v9
- github.com/go-git/go-git/v5
- github.com/dave/jennifer/jen
- github.com/dave/jennifer/jen
- github.com/Khan/genqlient/graphql
- github.com/99designs/gqlgen/graphql
274 changes: 73 additions & 201 deletions LICENSE

Large diffs are not rendered by default.

99 changes: 36 additions & 63 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# GitHub Action Local Executor

![dagger-min-version](https://img.shields.io/badge/dagger%20version-v0.9.3-green)
![dagger-min-version](https://img.shields.io/badge/dagger%20version-v0.9.4-green)

Welcome to project `gale`!

Expand All @@ -26,21 +26,33 @@ To start using `gale`, make sure your computer has these tools:

1. **Docker**: Dagger, requires [Docker](https://www.docker.com/) running on your host system.

2. **Dagger CLI**: You need to install the Dagger CLI , version >= v0.9.1, to use `gale`. [Install the Dagger CLI](https://docs.dagger.io/quickstart/729236/cli)
2. **Dagger CLI**: You need to install the Dagger CLI , version >= v0.9.4, to use `gale`. [Install the Dagger CLI](https://docs.dagger.io/quickstart/729236/cli)

Once you have these tools, you are ready to install and use `gale`.

## How to Use

### Setup Dagger Module

To avoid adding `-m github.com/aweris/gale/daggerverse/gale` to every command, you can add run the following command to
To avoid adding `-m github.com/aweris/gale` to every command, you can add run the following command to
set `DAGGER_MODULE` environment variable:

```shell
export DAGGER_MODULE=github.com/aweris/gale/daggerverse/gale
export DAGGER_MODULE=github.com/aweris/gale
```

### Global Flags for `gale`

The following table lists the global flags available for use with the `gale` module:

| Flag | Condition | Description |
|-------------------|-------------|------------------------------------------------------------------------------------------|
| `--repo` | Conditional | Specify the repository in 'owner/name' format. Used if `--source` is not provided. |
| `--source` | Conditional | Path to the repository source directory. Takes precedence over `--repo`. |
| `--tag` | Conditional | Tag name to check out. Takes precedence over `--branch`. Required when `--repo` is used. |
| `--branch` | Conditional | Branch name to check out. Used if `--tag` is not specified and `--repo` is used. |
| `--workflows-dir` | Optional | Path to the workflows' directory. Defaults to `.github/workflows`. |

### Listing a Workflows

To get a list of workflows, you can use the dagger call workflows list command.
Expand All @@ -54,26 +66,21 @@ Usage:
dagger call list [flags]

Flags:
--branch string Branch name to check out. Only one of branch or tag can be used. Precedence is as follows: tag, branch.
-h, --help help for list
--repo string The name of the repository. Format: owner/name.
--source Directory The directory containing the repository source. If source is provided, rest of the options are ignored.
--tag string Tag name to check out. Only one of branch or tag can be used. Precedence is as follows: tag, branch.
--workflows-dir string Path to the workflows' directory. (default: .github/workflows)
```

#### Examples

List all workflows for current repository:

```shell
dagger call list --source "."
dagger -m github.com/aweris/gale call --source "." list
```

List workflows for a specific repository and directory:

```shell
dagger call list --repo aweris/gale --branch main --workflows-dir examples/workflows
dagger -m github.com/aweris/gale call --repo aweris/gale --branch main --workflows-dir examples/workflows list
```

### Run a Workflow
Expand All @@ -89,68 +96,34 @@ Usage:
dagger call run [command]

Available Commands:
config Configuration for the workflow run.
directory Directory returns the directory of the workflow run information.
sync Sync runs the workflow and returns the container that ran the workflow.

Flags:
--branch string Branch name to check out. Only one of branch or tag can be used. Precedence is as follows: tag, branch.
--container Container Container to use for the runner (default: ghcr.io/catthehacker/ubuntu:act-latest).
--event string Name of the event that triggered the workflow. e.g. push
--event-file File File with the complete webhook event payload.
-h, --help help for run
--job string Name of the job to run. If empty, all jobs will be run.
--repo string The name of the repository. Format: owner/name.
--runner-debug Enables debug mode.
--source Directory The directory containing the repository source. If source is provided, rest of the options are ignored.
--tag string Tag name to check out. Only one of branch or tag can be used. Precedence is as follows: tag, branch.
--token Secret GitHub token to use for authentication.
--workflow string Name of the workflow to run.
--workflow-file File External workflow file to run.
--workflows-dir string Path to the workflows directory. (default: .github/workflows)
```

#### Sub-Commands

- **sync**: Sync evaluates the workflow run and returns the container at executed the workflow.

```shell
dagger call workflow run ... --workflow build sync
```

- **directory**: Exports workflow run data.

The help output for directory option
```shell
Usage:
dagger download run directory [flags]

Aliases:
directory, Directory

Flags:
-h, --help help for directory
--include-artifacts Adds the uploaded artifacts to the exported directory.
--include-event Adds the event file to the exported directory.
--include-repo Adds the repository source to the exported directory.
--include-secrets Adds the mounted secrets to the exported directory.
```

Example command for exporting the workflow run data:
```shell
dagger download workflow run ... --workflow build directory --export-path .gale/exports --include-artifacts
data Returns the directory containing the workflow run data.
log Returns all job run logs as a single file.
sync Returns the container for the given job id. If there is on one job in the workflow run, then job id is not required.

Flags:
--container Container Container to use for the runner(default: ghcr.io/catthehacker/ubuntu:act-latest).
--docker-host string Sets DOCKER_HOST to use for the native docker support. (default "unix:///var/run/docker.sock")
--event string Name of the event that triggered the workflow. e.g. push (default "push")
--event-file File File with the complete webhook event payload.
-h, --help help for run
--job string Name of the job to run. If empty, all jobs will be run.
--runner-debug Enables debug mode.
--token Secret GitHub token to use for authentication.
--use-dind Enables docker-in-dagger support to be able to run docker commands isolated from the host. Enabling DinD may lead to longer execution times.
--use-native-docker Enables native Docker support, allowing direct execution of Docker commands in the workflow. (default true)
--workflow string Name of the workflow to run.
--workflow-file File External workflow file to run.
```

##### Examples

Running a workflow for remote repository and downloading exporting the workflow run data and artifacts:

```shell
dagger download --focus=false run --repo kubernetes/minikube --branch master --workflow build --job build_minikube --token $GITHUB_TOKEN directory --export-path .gale/exports --include-artifacts
dagger -m github.com/aweris/gale export --repo kubernetes/minikube --branch master run --workflow build --job build_minikube --token $GITHUB_TOKEN data --output .gale/exports
```

**Notes for Above Example:**
- `--focus=false` is used to disable focus mode. Required for displaying the execution logs.
- `--token` is optional however it is required for the workflow in this example.

## Feedback and Collaboration
Expand Down
131 changes: 131 additions & 0 deletions actions.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
package main

import (
"context"
"fmt"

"gopkg.in/yaml.v3"

"github.com/aweris/gale/common/model"
)

type Actions struct {
// The repository information
Repo *RepoInfo

// The workflows in the repository
Workflows *Workflows

// The steps in the action
Steps []Step
}

func (a *Actions) Action(
// ID of the step. Defaults to the step index in the job.
// +optional=true
stepID string,
// External workflow file to run.
// +optional=false
uses string,
// Environment variables for the action. Format: name=value.
// +optional=true
env []string,
// Input parameters for the action. Format: name=value.
// +optional=true
with []string,
) (*Actions, error) {
envVal, err := ParseKeyValuePairs(env)
if err != nil {
return nil, err
}

withVal, err := ParseKeyValuePairs(with)
if err != nil {
return nil, err
}

step := Step{
StepID: stepID,
Uses: uses,
Env: envVal,
With: withVal,
}

a.Steps = append(a.Steps, step)

return a, nil
}

func (a *Actions) Run(
// The action to run. it should be in the format of <action>@<version>
ctx context.Context,
// Name of the event that triggered the workflow. e.g. push
// +optional=true
// +default=push
event string,
// File with the complete webhook event payload.
// +optional=true
eventFile *File,
// Container to use for the runner(default: ghcr.io/catthehacker/ubuntu:act-latest).
// +optional=true
container *Container,
// Enables debug mode.
// +optional=true
// +default=false
runnerDebug bool,
// GitHub token to use for authentication.
// +optional=true
token *Secret,
) (*WorkflowRun, error) {
models := make([]model.Step, 0, len(a.Steps))

for _, step := range a.Steps {
sm := model.Step{
ID: step.StepID,
Uses: step.Uses,
Environment: ConvertKVSliceToMap(step.Env),
With: ConvertKVSliceToMap(step.With),
}

models = append(models, sm)
}

w := &model.Workflow{
Jobs: map[string]model.Job{
"dagger": {
Steps: models,
},
},
}

data, err := yaml.Marshal(w)
if err != nil {
return nil, err
}

fmt.Printf("workflow: %s\n", string(data))

if eventFile == nil {
eventFile = dag.Directory().WithNewFile("event.json", "{}").File("event.json")
}

if container == nil {
container = dag.Container().From("ghcr.io/catthehacker/ubuntu:act-latest")
}

planner := NewWorkflowExecutionPlanner(
a.Repo,
a.Workflows,
&WorkflowRunOpts{WorkflowFile: dag.Directory().WithNewFile("workflow.yml", string(data)).File("workflow.yml")},
&RunnerOpts{Ctr: container, Debug: runnerDebug},
&EventOpts{Name: event, File: eventFile},
&SecretOpts{Token: token},
)

executor, err := planner.Plan(ctx)
if err != nil {
return nil, err
}

return executor.Execute(ctx)
}
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"name": "actions-artifactcache-service",
"root": "../../..",
"root": "../..",
"sdk": "go"
}
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading
Loading