Skip to content

Commit

Permalink
Merge pull request #52 from JuliaCI/hard_limit
Browse files Browse the repository at this point in the history 
Restore hard limit functionality, replace `Pkg.gc` with manual version.
  • Loading branch information
@maleadt
maleadt authored Nov 8, 2024
2 parents 2864369 + 3ca82fe commit de61ece
Show file tree
Hide file tree
Showing 3 changed files with 138 additions and 74 deletions.
33 changes: 20 additions & 13 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,15 @@ steps:
## Options
* `version`: A version to download and use, examples are `1`, `1.6`, `1.5.3`, `1.7-nightly`.
* `isolated_depot`: a boolean which defaults to `true`, automatically configuring Julia to use a pipeline-specific depot. If `false`, the default depot (usually `$HOME/.julia`) is used.
* `persist_depot_dirs`: a string of comma-separated directories to persist from pipeline run to pipeline run within the isolated depot. Cannot be set if `isolated_depot` is `false`. Defaults to `"packages,artifacts,compiled,logs,datadeps,scratchspaces"`.
* `version`: A version to download and use, examples are `1`, `1.6`, `1.5.3`,
`1.7-nightly`.
* `isolated_depot`: a boolean which defaults to `true`, automatically
configuring Julia to use a pipeline-specific depot. If `false`, the default
depot (usually `$HOME/.julia`) is used.
* `persist_depot_dirs`: a string of comma-separated directories to persist from
pipeline run to pipeline run within the isolated depot. Cannot be set if
`isolated_depot` is `false`. Defaults to
`"packages,artifacts,compiled,logs,datadeps,scratchspaces"`.

### Advanced Options

Expand All @@ -27,17 +33,18 @@ steps:
installations, depots, etc. Defaults to
`${HOME}/.cache/julia-buildkite-plugin`. Persist this directory on your agents
to speed up subsequent builds.
* `cleanup_collect_delay`: a string specifying a period in seconds after which
package garbage collection, i.e.
[`Pkg.gc`](https://pkgdocs.julialang.org/v1/api/#Pkg.gc), will consider
orphaned items for cleanup. Defaults to `604800` seconds, i.e. 1 week.
* `pipeline_age_limit`: a string specifying a period in seconds after which the
pipeline-specific depot will be considered stale and removed. Defaults to
`2592000` seconds, i.e. 30 days.
* `compilecache_size_limit`: a string specifying the maximum size of the
compilecache in bytes. Defaults to `1073741824` bytes, i.e. 1 GiB.
* `compiled_size_limit`: a string specifying the maximum size of the compilation
cache, in bytes. Defaults to `1073741824` bytes, i.e. 1 GiB.
* `artifacts_size_limit`: a string specifying the maximum size of the artifacts
store, in bytes. Defaults to `10737418240` bytes, i.e. 10 GiB.
* `depot_size_limit`: a string specifying the maximum size of the entire depot,
in bytes. Defaults to 10 GiB over the previous two limits, i.e., 21 GiB.
* `pipeline_age_limit`: a string specifying a period in seconds after which a
depot will be considered stale and removed. Defaults to `2592000` seconds,
i.e. 30 days.
* `debug_plugin`: a boolean, which defaults to `false`, severely increasing the
verbosity of the plugin for debugging purposes.
* `python`: a string specifying the path to a Python 3 distribution. The plugin
will try to autodetect the location of a Python 3 installation by default.
* `update_registry`: a boolean, which defaults to `true`, indicating whether to update the package registry.
* `update_registry`: a boolean, which defaults to `true`, indicating whether to
update the package registry.
171 changes: 112 additions & 59 deletions hooks/pre-exit
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,97 +25,150 @@ if [[ "${BUILDKITE_PLUGIN_JULIA_ISOLATED_DEPOT:-true}" == "true" ]]; then
fi


### Step 2: Reduce the size of our depot by running `Pkg.gc`.

# since we remove most manifests at the start of each pipeline (cleaning the depot),
# there's no way for Pkg to actually track which artifacts are active, so we just invoke
# `gc()` with `collect_delay` set to zero to cause immediate collection on versions of
# julia that are new enough to have a generational Pkg GC.

# Allow the user to customize whether items are GC'ed immediately or not
# Defaults to deleting things after a week of inactivity
CLEANUP_COLLECT_DELAY="${BUILDKITE_PLUGIN_JULIA_CLEANUP_COLLECT_DELAY:-604800}" # 1 week

GC_CMD="""
using Pkg, Dates
collect_delay = Second(parse(Int64, ARGS[1]))
@info(\"Running Pkg.gc()\", collect_delay)
if VERSION >= v\"1.6-\"
# If we're on v1.6+, we can use verbose, which is nice
Pkg.gc(collect_delay=collect_delay, verbose=true)
elseif VERSION >= v\"1.3-\"
# If we're on v1.3+, we must set the collect_delay low
Pkg.gc(collect_delay=collect_delay)
else
# Otherwise, on truly old versions, the only thing we can do is call gc()
Pkg.gc()
end
"""
julia --color=yes -e "${GC_CMD}" "${CLEANUP_COLLECT_DELAY}" || true


### Step 3: Reduce the size of our depot by removing old precompilation files.
### Step 2: Reduce the size of our depot by removing old precompilation files.

# Julia does not track when a precompilation file was last used, so we simply
# remove files in the `compiled` directory until we are below the hard limit.
# remove files in the `compiled` directory until we are below the limit.

COMPILED_DIR="${JULIA_DEPOT_PATH}/compiled"

COMPILECACHE="${JULIA_DEPOT_PATH}/compiled"
# default limit: 1 GiB
COMPILED_LIMIT="${BUILDKITE_PLUGIN_JULIA_COMPILED_SIZE_LIMIT:-1073741824}"

if [[ "${BUILDKITE_PLUGIN_JULIA_ISOLATED_DEPOT:-true}" == "true" ]] && [[ -d "${COMPILECACHE}" ]]; then
CACHE_SIZE_HUMAN=$(du -h -s "${COMPILECACHE}" | cut -f 1)
if [[ "${BUILDKITE_PLUGIN_JULIA_ISOLATED_DEPOT:-true}" == "true" ]] && [[ -d "${COMPILED_DIR}" ]]; then
COMPILED_SIZE_HUMAN=$(du -h -s "${COMPILED_DIR}" | cut -f 1)
# `-k` gives consistently the number of kilobytes on both macOS and Linux,
# without it BSD `du` would give the number of multiples of 512 bytes.
CACHE_SIZE=$(($(du -k -s "${COMPILECACHE}" | cut -f 1) * 1024))
CACHE_LIMIT="${BUILDKITE_PLUGIN_JULIA_COMPILECACHE_SIZE_LIMIT:-1073741824}"
echo "The compilation cache size is: ${CACHE_SIZE_HUMAN}"
COMPILED_SIZE=$(($(du -k -s "${COMPILED_DIR}" | cut -f 1) * 1024))
echo "The compilation cache size is: ${COMPILED_SIZE_HUMAN}"

if [[ ${CACHE_SIZE} -gt ${CACHE_LIMIT} ]]; then
echo "This is greater than the hard limit (${CACHE_SIZE} > ${CACHE_LIMIT} bytes), so we will clear the compilation cache"
if [[ ${COMPILED_SIZE} -gt ${COMPILED_LIMIT} ]]; then
echo "This is greater than the limit (${COMPILED_SIZE} > ${COMPILED_LIMIT} bytes), so we will clear the compilation cache"

# Remove oldest files until we are below the hard limit
# Remove files until we are below the limit
# We do this in Julia to avoid platform portability issues.
julia --color=yes -e '
function main(compilecache, cache_size_str, cache_limit_str)
cache_size = parse(Int, cache_size_str)
cache_limit = parse(Int, cache_limit_str)
function main(compilecache, size_str, limit_str)
size = parse(Int, size_str)
limit = parse(Int, limit_str)
# Get all files with their modification times
files = []
for (root, _, files_in_dir) in walkdir(compilecache)
for file in files_in_dir
worklist = []
for (root, dirs, files) in walkdir(compilecache)
for file in files
path = joinpath(root, file)
push!(files, (; path, stat=stat(path)))
push!(worklist, (; path, stat=stat(path)))
end
end
# Sort by modification time (oldest first)
sort!(files; by=file->file.stat.mtime)
sort!(worklist; by=entry->entry.stat.mtime)
# Remove files until we are under the limit
for file in files
rm(file.path)
cache_size -= file.stat.size
cache_size <= cache_limit && break
for entry in worklist
rm(entry.path)
size -= entry.stat.size
size <= limit && break
end
end
main(ARGS...)
' "${COMPILECACHE}" "${CACHE_SIZE}" "${CACHE_LIMIT}" || true
' "${COMPILED_DIR}" "${COMPILED_SIZE}" "${COMPILED_LIMIT}" || true

# Cleanup empty directories
find "${COMPILECACHE}" -type d -empty -delete
find "${COMPILED_DIR}" -type d -empty -delete
fi
fi


### Step 3: Reduce the size of our depot by removing old artifacts.

# This is normally done by `Pkg.gc`, however, since manifests are ephemeral
# (and logs directories may not even persist) that doesn't work well.
# So instead we remove directories based on their creation time.

ARTIFACTS_DIR="${JULIA_DEPOT_PATH}/compiled"

# default limit: 10 GiB
ARTIFACTS_LIMIT="${BUILDKITE_PLUGIN_JULIA_ARTIFACTS_SIZE_LIMIT:-10737418240}"

if [[ "${BUILDKITE_PLUGIN_JULIA_ISOLATED_DEPOT:-true}" == "true" ]] && [[ -d "${ARTIFACTS_DIR}" ]]; then
ARTIFACTS_SIZE_HUMAN=$(du -h -s "${ARTIFACTS_DIR}" | cut -f 1)
# `-k` gives consistently the number of kilobytes on both macOS and Linux,
# without it BSD `du` would give the number of multiples of 512 bytes.
ARTIFACTS_SIZE=$(($(du -k -s "${ARTIFACTS_DIR}" | cut -f 1) * 1024))
echo "The artifact store size is: ${ARTIFACTS_SIZE_HUMAN}"

if [[ ${ARTIFACTS_SIZE} -gt ${ARTIFACTS_LIMIT} ]]; then
echo "This is greater than the limit (${ARTIFACTS_SIZE} > ${ARTIFACTS_LIMIT} bytes), so we will clear the artifact store"

# Remove directories until we are below the limit
# We do this in Julia to avoid platform portability issues.
julia --color=yes -e '
function main(artifacts, size_str, limit_str)
size = parse(Int, size_str)
limit = parse(Int, limit_str)
# Get all artifacts with their creation times
worklist = []
for dir in readdir(artifacts)
path = joinpath(artifacts, dir)
isdir(path) || continue
push!(worklist, (; path, stat=stat(path)))
end
# Sort by creation time (oldest first)
sort!(worklist; by=entry->entry.stat.ctime)
# Remove artifacts until we are under the limit
for entry in worklist
# artifacts can have funky permissions
run(`chmod -R u+w $(entry.path)`)
sz = parse(Int, split(read(`du -k -s $(entry.path)`, String))[1]) * 1024
rm(entry.path; recursive=true)
size -= sz
size <= limit && break
end
end
main(ARGS...)
' "${ARTIFACTS_DIR}" "${ARTIFACTS_SIZE}" "${ARTIFACTS_LIMIT}" || true

# Cleanup empty directories
find "${ARTIFACTS_DIR}" -type d -empty -delete
fi
fi


### Step 4: If the depot is still too big, remove it altogether.

# Data may be stored in unmanaged directories (e.g. `conda` or `scratchspaces`)

# default limit: COMPILED_LIMIT + ARTIFACTS_LIMIT + 10GiB (~20GiB normally)
DEPOT_LIMIT_DEFAULT=$(($COMPILED_LIMIT+$ARTIFACTS_LIMIT+10737418240))
DEPOT_LIMIT="${BUILDKITE_PLUGIN_JULIA_DEPOT_SIZE_LIMIT:-${DEPOT_LIMIT_DEFAULT}}"

if [[ "${BUILDKITE_PLUGIN_JULIA_ISOLATED_DEPOT:-true}" == "true" ]]; then
DEPOT_SIZE_HUMAN=$(du -h -s "${JULIA_DEPOT_PATH}" | cut -f 1)
# `-k` gives consistently the number of kilobytes on both macOS and Linux,
# without it BSD `du` would give the number of multiples of 512 bytes.
DEPOT_SIZE=$(($(du -k -s "${JULIA_DEPOT_PATH}" | cut -f 1) * 1024))
echo "The depot size is: ${DEPOT_SIZE_HUMAN}"

if [[ ${DEPOT_SIZE} -gt ${DEPOT_LIMIT} ]]; then
echo "This is greater than the limit (${DEPOT_SIZE} > ${DEPOT_LIMIT} bytes), so we will clear the entire depot"
rm -rf "${JULIA_DEPOT_PATH}"
mkdir -p "${JULIA_DEPOT_PATH}"
fi
fi


### Step 4: Remove old depots.
### Step 5: Remove old depots.

# We mark a depot as in-use by `touch`ing it at the start of the pipeline,
# so we can remove any depots that are older than a certain age.

PIPELINE_AGE_LIMIT="${BUILDKITE_PLUGIN_JULIA_PIPELINE_AGE_LIMIT:-2592000}"
DEPOT_AGE_LIMIT="${BUILDKITE_PLUGIN_JULIA_DEPOT_AGE_LIMIT:-2592000}"

julia --color=yes -e '
function main(depots_dir, age_limit_str)
Expand All @@ -136,4 +189,4 @@ julia --color=yes -e '
end
end
main(ARGS...)
' "${CACHE_DIR}"/depots "${PIPELINE_AGE_LIMIT}" || true
' "${CACHE_DIR}"/depots "${DEPOT_AGE_LIMIT}" || true
8 changes: 6 additions & 2 deletions plugin.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,13 @@ configuration:
type: boolean
persist_depot_dirs:
type: string
depot_hard_size_limit:
compiled_size_limit:
type: string
cleanup_collect_delay:
artifacts_size_limit:
type: string
depot_size_limit:
type: string
depot_age_limit:
type: string
python:
type: string
Expand Down

0 comments on commit de61ece

Please sign in to comment.