Skip to content

A package for dynamically checkpointing program state

License

Notifications You must be signed in to change notification settings

invenia/Checkpoints.jl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Checkpoints

stable build status coverage

Checkpoints.jl allows packages to register checkpoints which can serialize objects to disk during the execution of an application program, if the application program configures them.

A minimal working example consists of the package:

module MyPackage

using Checkpoints

MODULE = "MyPackage"

__init__() = Checkpoints.register(MODULE, ["foo", ])

function foo(x)
    with_checkpoint_tags(:foo1 => 1, :foo2 => 2) do
        checkpoint(MODULE, "foo", :data => 2x)
    end
    return 2x
end

end

and the application program:

using Checkpoints

Checkpoints.config("MyPackage.foo", "./path/to/checkpoints")

for i in 1:2
    with_checkpoint_tags(:iteration => i) do
        MyPackage.foo(1.0)
    end
end

which results in recorded checkpoints at

./path/to/checkpoints/iteration=1/foo1=1/foo2=2/MyPackage/foo.jlso
./path/to/checkpoints/iteration=2/foo1=1/foo2=2/MyPackage/foo.jlso

You can use index_checkpoint_files to get an index of the files, which is a Tables.jl table and so can e.g. be passed to DataFrame (and then you can do things like groupby etc):

julia> using DataFrames

julia> DataFrame(index_checkpoint_files("./path/to/checkpoints/"))
2×6 DataFrame
 Row │ prefixes        checkpoint_name  iteration   foo1        foo2        checkpoint_path
     │ Tuple          SubString       SubString  SubString  SubString  PosixPath
─────┼────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ ("MyPackage",)  foo              1           1           2           ./path/to/checkpoints/iteration=
   2 │ ("MyPackage",)  foo              2           1           2           ./path/to/checkpoints/iteration=

or worked with directly:

julia> [checkpoint_path(out) for out in index_checkpoint_files("./path/to/checkpoints/") if out.iteration=="1"]
1-element Array{FilePathsBase.PosixPath,1}:
 p"./path/to/checkpoints/iteration=1/foo1=1/foo2=2/MyPackage/foo.jlso"