Skip to content

Commit

Permalink
Documenter, PkgTemplate setup
Browse files Browse the repository at this point in the history
  • Loading branch information
mahiki committed Aug 21, 2023
1 parent 1b1634c commit f7076cd
Show file tree
Hide file tree
Showing 34 changed files with 734 additions and 387 deletions.
73 changes: 73 additions & 0 deletions .github/workflows/CI.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
name: CI
on:
push:
branches:
- main
tags: ['*']
pull_request:
concurrency:
# Skip intermediate builds: always.
# Cancel intermediate builds: only if it is a pull request build.
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
jobs:
test:
name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
version:
- '1.6'
- '1.7'
- '1.8'
- '1.9'
os:
- ubuntu-latest
- macOS-latest
arch:
- x64
- x86
exclude:
- os: macOS-latest
arch: x86
steps:
- uses: actions/checkout@v3
- uses: julia-actions/setup-julia@v1
with:
version: ${{ matrix.version }}
arch: ${{ matrix.arch }}
- uses: julia-actions/cache@v1
- uses: julia-actions/julia-buildpkg@v1
- uses: julia-actions/julia-runtest@v1
- uses: julia-actions/julia-processcoverage@v1
- uses: codecov/codecov-action@v3
with:
files: lcov.info
docs:
name: Documentation
runs-on: ubuntu-latest
permissions:
contents: write
statuses: write
steps:
- uses: actions/checkout@v3
- uses: julia-actions/setup-julia@v1
with:
version: '1'
- name: Configure doc environment
run: |
julia --project=docs/ -e '
using Pkg
Pkg.develop(PackageSpec(path=pwd()))
Pkg.instantiate()'
- uses: julia-actions/julia-buildpkg@v1
- uses: julia-actions/julia-docdeploy@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: |
julia --project=docs -e '
using Documenter: DocMeta, doctest
using PrefectInterfaces
DocMeta.setdocmeta!(PrefectInterfaces, :DocTestSetup, :(using PrefectInterfaces); recursive=true)
doctest(PrefectInterfaces)'
16 changes: 16 additions & 0 deletions .github/workflows/CompatHelper.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: CompatHelper
on:
schedule:
- cron: 0 0 * * *
workflow_dispatch:
jobs:
CompatHelper:
runs-on: ubuntu-latest
steps:
- name: Pkg.add("CompatHelper")
run: julia -e 'using Pkg; Pkg.add("CompatHelper")'
- name: CompatHelper.main()
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
COMPATHELPER_PRIV: ${{ secrets.DOCUMENTER_KEY }}
run: julia -e 'using CompatHelper; CompatHelper.main()'
31 changes: 31 additions & 0 deletions .github/workflows/TagBot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: TagBot
on:
issue_comment:
types:
- created
workflow_dispatch:
inputs:
lookback:
default: 3
permissions:
actions: read
checks: read
contents: write
deployments: read
issues: read
discussions: read
packages: read
pages: read
pull-requests: read
repository-projects: read
security-events: read
statuses: read
jobs:
TagBot:
if: github.event_name == 'workflow_dispatch' || github.actor == 'JuliaTagBot'
runs-on: ubuntu-latest
steps:
- uses: JuliaRegistries/TagBot@v1
with:
token: ${{ secrets.GITHUB_TOKEN }}
ssh: ${{ secrets.DOCUMENTER_KEY }}
16 changes: 16 additions & 0 deletions .github/workflows/register.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: Register Package
on:
workflow_dispatch:
inputs:
version:
description: Version to register or component to bump
required: true
jobs:
register:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: julia-actions/RegisterAction@latest
with:
token: ${{ secrets.GITHUB_TOKEN }}
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2023 mahiki
Copyright (c) 2023 mahiki <[email protected]> and contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
7 changes: 6 additions & 1 deletion Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,9 @@ Match = "7eb4fadd-790c-5f42-8a69-bfa0b872bfbf"
Parameters = "d96e819e-fc66-5662-9728-84c9c7592b0a"

[compat]
julia = "^1.8"
CSV = "0.9, 0.10"
DataFrames = "1.5"
HTTP = "1.9"
JSON = "0.21"
Parameters = "0.12"
julia = "1.6"
102 changes: 11 additions & 91 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,27 @@
# PrefectInterfaces.jl
>May the Prefect Julia SDK arrive soon.
# PrefectInterfaces.jl
[![Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://mahiki.github.io/PrefectInterfaces.jl/stable/) [![Build Status](https://github.com/mahiki/PrefectInterfaces.jl/actions/workflows/CI.yml/badge.svg?branch=main)](https://github.com/mahiki/PrefectInterfaces.jl/actions/workflows/CI.yml?query=branch%3Amain) [![Coverage](https://codecov.io/gh/mahiki/PrefectInterfaces.jl/branch/main/graph/badge.svg)](https://codecov.io/gh/mahiki/PrefectInterfaces.jl)

You prefer to analyze and wrangle data in Julia, and you orchestrate your data workflows with Prefect (in python).
>May the official Prefect Julia SDK arrive soon.
PrefectInterfaces.jl helps you integrate Julia operations into a Prefect orchestration environment. Julia functions call your Prefect instance (Server or Cloud) and pull block information including secrets. Now you can use Prefect python flows to call your Julia process via `DockerContainer` or `ShellOperation` and deploy these in the normal way. The Julia process has access to resources and can read/write in the same environment as the rest of your orchestration code.


## INSTALLATION
* A Prefect server/cloud instance must be available via API URL to use these functions.
* See [Prefect local installation instructions here.](prefect/README.md)
This is a small package that helps you connect to a Prefect instance from a Julia process. This enables integrating your Julia code into your existing workflow orchestration managed by Prefect. Included in the package is a bootstrapped installation of a local Prefect instance, and an example `Dataset` type to demonstrate a concrete use-case.

## Installation
```julia
# Package is not yet registered
# Package registration pending
julia> Pkg.add("https://github.com/mahiki/PrefectInterfaces.jl")
```
## USAGE
These demo commands will work with the included prefect installation, [see installation reference](README.md#installation).

* List available Prefect blocks
* Load a secret from the Prefect DB
* Load a local file system block from Prefect DB
* Use the `read_path`, `write_path` methods from the FS Block.
* Notice the block implements a base path
* *NOTE:* these are defaulting to read/read via `CSV` module. That should get fixed in the future to handle any type of file.
## Usage
* A Prefect server/cloud instance must be available via API URL to use these functions, the examples below are hypothetical.
* See [Prefect local installation instructions.](docs/src/prefect/install-local-prefect-environment.md)

```julia
# provide a reference to the running Prefect REST API
julia> ENV["PREFECT_API_URL"] = "http://127.0.0.1:4300/api"

using PrefectInterfaces

# retrieve the names of blocks from your running Prefect instance
db = ls();
db.blocks
# 3-element Vector{String}:
Expand All @@ -40,78 +32,6 @@ db.blocks
secret_block = PrefectBlock("secret/necromancer")
# PrefectBlock("secret/necromancer", Main.PrefectInterfaces.SecretBlock("secret/necromancer", "secret", ####Secret####))

secret_block.block.value
# ####Secret####

secret_block.block.value.secret
# "abcd1234"

using DataFrames
df = DataFrame(
flag = [false, true, false, true, false, true]
, amt = [19.00, 11.00, 35.50, 32.50, 5.99, 5.99]
, qty = [1, 4, 1, 3, 21, 109]
, item = ["B001", "B001", "B020", "B020", "BX00", "BX00"]
, day = ["2021-01-01", "2021-01-01", "2112-12-12", "2020-10-20", "2021-05-04", "1984-07-04"]
);

fs_block = PrefectBlock("local-file-system/willowdata");
dump(fs_block)
# PrefectBlock
# blockname: String "local-file-system/willowdata"
# block: LocalFSBlock
# blockname: String "local-file-system/willowdata"
# blocktype: String "local-file-system"
# basepath: String "$HOME/willowdata/main"
# read_path: #4 (function of type PrefectInterfaces.var"#4#6"{String})
# basepath: String "$HOME/willowdata/main"
# write_path: #5 (function of type PrefectInterfaces.var"#5#7"{String})
# basepath: String "$HOME/willowdata/main"

datafile = fs_block.block.write_path("csv/dataset=test_block_write/data.csv", df)
# "$HOME/willowdata/main/csv/dataset=test_block_write/data.csv"

isfile(datafile)
# true

df2 = fs_block.block.read_path("csv/dataset=test_block_write/data.csv")
# 6×5 DataFrame
# Row │ flag amt qty item day
# │ Bool Float64 Int64 String7 Date
# ─────┼────────────────────────────────────────────
# 1 │ false 19.0 1 B001 2021-01-01
# 2 │ true 11.0 4 B001 2021-01-01
# 3 │ false 35.5 1 B020 2112-12-12
# 4 │ true 32.5 3 B020 2020-10-20
# 5 │ false 5.99 21 BX00 2021-05-04
# 6 │ true 5.99 109 BX00 1984-07-04
```

## DATASETS
On top of the Prefect API, this package includes a **Datasets** module that reads/writes dataframes to file locations based only on the name you give to the data artifact. [See the Julia-demo document for examples](julia-demo/Julia-demo.md#dataset-type).

## CALLING FROM PREFECT FLOW
The one thing the Julia process will need from the prefect flow is the PREFECT_API_URL. This is accessible from your Prefect application code via settings:

```py
from prefect import flow
from prefect_shell import ShellOperation
from prefect import settings
prefect_api = settings.PREFECT_API_URL.value()
# 'http://127.0.0.1:4300/api'

@flow(log_prints=True)
call_julia_script(prefect_api_url_arg=prefect_api):
result = ShellOperation(
stream_output=True
, command=["julia --project=. --load path/to/julia-script.jl"]
, working_dir="path/to/whatever"
, env={"PREFECT_API_URL": prefect_api_url_arg}
).run()
```

So you'll pass this to your `Docker` or `ShellOperation` either as an env variable or parameter to the julia command.


----------
More detailed design discussion, including why not `PythonCall/JuliaCall` in [Usage Explanation](info/Usage-and-Explanation.md).
```
3 changes: 3 additions & 0 deletions docs/Project.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[deps]
Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
PrefectInterfaces = "25d49962-0f22-42a0-bb44-b427e1ded1d4"
2 changes: 2 additions & 0 deletions docs/docs/Project.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[deps]
Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
38 changes: 38 additions & 0 deletions docs/make.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
using PrefectInterfaces
using Documenter

push!(LOAD_PATH,"../src/")

DocMeta.setdocmeta!(PrefectInterfaces, :DocTestSetup, :(using PrefectInterfaces); recursive=true)

makedocs(;
modules=[PrefectInterfaces],
authors="mahiki <[email protected]>",
repo="https://github.com/mahiki/PrefectInterfaces.jl/blob/{commit}{path}#{line}",
sitename="PrefectInterfaces.jl",
format=Documenter.HTML(;
prettyurls=get(ENV, "CI", "false") == "true",
canonical="https://mahiki.github.io/PrefectInterfaces.jl",
edit_link="main",
assets=String[],
),
pages=[
"Introduction" => "index.md"
, "Demonstration" => "julia-demo.md"
, "Detailed Explanation" => "usage-and-explanation.md"
, "Prefect" => [
"Prefect Install" => "prefect/install-local-prefect-environment.md"
, "Manual Install" => "prefect/setup-without-justfile.md"
]
, "API" => [
"Exported Names" => "lib/autodoc.md"
]
, "Developers" => "developers.md"
]

)

deploydocs(;
repo="github.com/mahiki/PrefectInterfaces.jl",
devbranch="main",
)
40 changes: 40 additions & 0 deletions docs/src/developers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Developers
Develop and test with

```bash
# launch the local prefect server if its not available
cd ./prefect
just launch

cd ./PrefectInterfaces
julia --project=. --startup-file=no --eval 'import Pkg; Pkg.test()'
# SERVER HEALTH CHECK #
# =================== #
Active Prefect Environment: main
┌ Info: Prefect Server must be running, i.e. `prefect server start`
│ Calling http://127.0.0.1:4300/api/health
└ Server reponse status: 200 OK

Test Summary: | Pass Total Time
All tests | 94 94 2.7s
Config | 9 9 0.4s
Block types, function tests | 58 58 1.8s
Dataset function | 27 27 0.5s

Testing PrefectInterfaces tests passed
```

Alternately, using the `justfile`
```julia
cd .../PrefectInterfaces

just test
```

Or, from the REPL:

`julia --project=.`
```julia
pkg> activate .
pkg> test
```
Loading

0 comments on commit f7076cd

Please sign in to comment.