Hand-roll our version of Pkg.gc().

README.md

@@ -15,9 +15,15 @@ steps:
 
 ## Options
 
-* `version`: A version to download and use, examples are `1`, `1.6`, `1.5.3`, `1.7-nightly`.
-* `isolated_depot`: a boolean which defaults to `true`, automatically configuring Julia to use a pipeline-specific depot. If `false`, the default depot (usually `$HOME/.julia`) is used.
-* `persist_depot_dirs`: a string of comma-separated directories to persist from pipeline run to pipeline run within the isolated depot. Cannot be set if `isolated_depot` is `false`. Defaults to `"packages,artifacts,compiled,logs,datadeps,scratchspaces"`.
+* `version`: A version to download and use, examples are `1`, `1.6`, `1.5.3`,
+  `1.7-nightly`.
+* `isolated_depot`: a boolean which defaults to `true`, automatically
+  configuring Julia to use a pipeline-specific depot. If `false`, the default
+  depot (usually `$HOME/.julia`) is used.
+* `persist_depot_dirs`: a string of comma-separated directories to persist from
+  pipeline run to pipeline run within the isolated depot. Cannot be set if
+  `isolated_depot` is `false`. Defaults to
+  `"packages,artifacts,compiled,logs,datadeps,scratchspaces"`.
 
 ### Advanced Options
 
@@ -27,17 +33,18 @@ steps:
   installations, depots, etc. Defaults to
   `${HOME}/.cache/julia-buildkite-plugin`. Persist this directory on your agents
   to speed up subsequent builds.
-* `cleanup_collect_delay`: a string specifying a period in seconds after which
-  package garbage collection, i.e.
-  [`Pkg.gc`](https://pkgdocs.julialang.org/v1/api/#Pkg.gc), will consider
-  orphaned items for cleanup. Defaults to `604800` seconds, i.e. 1 week.
-* `pipeline_age_limit`: a string specifying a period in seconds after which the
-  pipeline-specific depot will be considered stale and removed. Defaults to
-  `2592000` seconds, i.e. 30 days.
-* `compilecache_size_limit`: a string specifying the maximum size of the
-  compilecache in bytes. Defaults to `1073741824` bytes, i.e. 1 GiB.
+* `compiled_size_limit`: a string specifying the maximum size of the compilation
+  cache, in bytes. Defaults to `1073741824` bytes, i.e. 1 GiB.
+* `artifacts_size_limit`: a string specifying the maximum size of the artifacts
+  store, in bytes. Defaults to `10737418240` bytes, i.e. 10 GiB.
+* `depot_size_limit`: a string specifying the maximum size of the entire depot,
+  in bytes. Defaults to 10 GiB over the previous two limits, i.e., 21 GiB.
+* `pipeline_age_limit`: a string specifying a period in seconds after which a
+  depot will be considered stale and removed. Defaults to `2592000` seconds,
+  i.e. 30 days.
 * `debug_plugin`: a boolean, which defaults to `false`, severely increasing the
   verbosity of the plugin for debugging purposes.
 * `python`: a string specifying the path to a Python 3 distribution. The plugin
   will try to autodetect the location of a Python 3 installation by default.
-* `update_registry`: a boolean, which defaults to `true`, indicating whether to update the package registry.
+* `update_registry`: a boolean, which defaults to `true`, indicating whether to
+  update the package registry.

hooks/pre-exit

@@ -25,100 +25,128 @@ if [[ "${BUILDKITE_PLUGIN_JULIA_ISOLATED_DEPOT:-true}" == "true" ]]; then
 fi
 
 
-### Step 2: Reduce the size of our depot by running `Pkg.gc`.
-
-# since we remove most manifests at the start of each pipeline (cleaning the depot),
-# there's no way for Pkg to actually track which artifacts are active, so we just invoke
-# `gc()` with `collect_delay` set to zero to cause immediate collection on versions of
-# julia that are new enough to have a generational Pkg GC.
-
-# Allow the user to customize whether items are GC'ed immediately or not
-# Defaults to deleting things after a week of inactivity
-CLEANUP_COLLECT_DELAY="${BUILDKITE_PLUGIN_JULIA_CLEANUP_COLLECT_DELAY:-604800}" # 1 week
-
-GC_CMD="""
-using Pkg, Dates
-
-collect_delay = Second(parse(Int64, ARGS[1]))
-
-@info(\"Running Pkg.gc()\", collect_delay)
-if VERSION >= v\"1.6-\"
-    # If we're on v1.6+, we can use verbose, which is nice
-    Pkg.gc(collect_delay=collect_delay, verbose=true)
-elseif VERSION >= v\"1.3-\"
-    # If we're on v1.3+, we must set the collect_delay low
-    Pkg.gc(collect_delay=collect_delay)
-else
-    # Otherwise, on truly old versions, the only thing we can do is call gc()
-    Pkg.gc()
-end
-"""
-julia --color=yes -e "${GC_CMD}" "${CLEANUP_COLLECT_DELAY}" || true
-
-
-### Step 3: Reduce the size of our depot by removing old precompilation files.
+### Step 2: Reduce the size of our depot by removing old precompilation files.
 
 # Julia does not track when a precompilation file was last used, so we simply
-# remove files in the `compiled` directory until we are below the hard limit.
+# remove files in the `compiled` directory until we are below the limit.
 
-COMPILECACHE="${JULIA_DEPOT_PATH}/compiled"
+COMPILED_DIR="${JULIA_DEPOT_PATH}/compiled"
 
 # default limit: 1 GiB
-CACHE_LIMIT="${BUILDKITE_PLUGIN_JULIA_COMPILECACHE_SIZE_LIMIT:-1073741824}"
+COMPILED_LIMIT="${BUILDKITE_PLUGIN_JULIA_COMPILED_SIZE_LIMIT:-1073741824}"
 
-if [[ "${BUILDKITE_PLUGIN_JULIA_ISOLATED_DEPOT:-true}" == "true" ]] && [[ -d "${COMPILECACHE}" ]]; then
-    CACHE_SIZE_HUMAN=$(du -h -s "${COMPILECACHE}" | cut -f 1)
+if [[ "${BUILDKITE_PLUGIN_JULIA_ISOLATED_DEPOT:-true}" == "true" ]] && [[ -d "${COMPILED_DIR}" ]]; then
+    COMPILED_SIZE_HUMAN=$(du -h -s "${COMPILED_DIR}" | cut -f 1)
     # `-k` gives consistently the number of kilobytes on both macOS and Linux,
     # without it BSD `du` would give the number of multiples of 512 bytes.
-    CACHE_SIZE=$(($(du -k -s "${COMPILECACHE}" | cut -f 1) * 1024))
-    echo "The compilation cache size is: ${CACHE_SIZE_HUMAN}"
+    COMPILED_SIZE=$(($(du -k -s "${COMPILED_DIR}" | cut -f 1) * 1024))
+    echo "The compilation cache size is: ${COMPILED_SIZE_HUMAN}"
 
-    if [[ ${CACHE_SIZE} -gt ${CACHE_LIMIT} ]]; then
-        echo "This is greater than the hard limit (${CACHE_SIZE} > ${CACHE_LIMIT} bytes), so we will clear the compilation cache"
+    if [[ ${COMPILED_SIZE} -gt ${COMPILED_LIMIT} ]]; then
+        echo "This is greater than the limit (${COMPILED_SIZE} > ${COMPILED_LIMIT} bytes), so we will clear the compilation cache"
 
-        # Remove oldest files until we are below the hard limit
+        # Remove files until we are below the limit
         # We do this in Julia to avoid platform portability issues.
         julia --color=yes -e '
-            function main(compilecache, cache_size_str, cache_limit_str)
-                cache_size = parse(Int, cache_size_str)
-                cache_limit = parse(Int, cache_limit_str)
+            function main(compilecache, size_str, limit_str)
+                size = parse(Int, size_str)
+                limit = parse(Int, limit_str)
 
                 # Get all files with their modification times
-                files = []
-                for (root, _, files_in_dir) in walkdir(compilecache)
-                    for file in files_in_dir
+                worklist = []
+                for (root, dirs, files) in walkdir(compilecache)
+                    for file in files
                         path = joinpath(root, file)
-                        push!(files, (; path, stat=stat(path)))
+                        push!(worklist, (; path, stat=stat(path)))
                     end
                 end
 
                 # Sort by modification time (oldest first)
-                sort!(files; by=file->file.stat.mtime)
+                sort!(worklist; by=entry->entry.stat.mtime)
 
                 # Remove files until we are under the limit
-                for file in files
-                    rm(file.path)
-                    cache_size -= file.stat.size
-                    cache_size <= cache_limit && break
+                for entry in worklist
+                    rm(entry.path)
+                    size -= entry.stat.size
+                    size <= limit && break
+                end
+            end
+
+            main(ARGS...)
+        ' "${COMPILED_DIR}" "${COMPILED_SIZE}" "${COMPILED_LIMIT}" || true
+
+        # Cleanup empty directories
+        find "${COMPILED_DIR}" -type d -empty -delete
+    fi
+fi
+
+
+### Step 3: Reduce the size of our depot by removing old artifacts.
+
+# This is normally done by `Pkg.gc`, however, since manifests are ephemeral
+# (and logs directories may not even persist) that doesn't work well.
+# So instead we remove directories based on their creation time.
+
+ARTIFACTS_DIR="${JULIA_DEPOT_PATH}/compiled"
+
+# default limit: 10 GiB
+ARTIFACTS_LIMIT="${BUILDKITE_PLUGIN_JULIA_ARTIFACTS_SIZE_LIMIT:-10737418240}"
+
+if [[ "${BUILDKITE_PLUGIN_JULIA_ISOLATED_DEPOT:-true}" == "true" ]] && [[ -d "${ARTIFACTS_DIR}" ]]; then
+    ARTIFACTS_SIZE_HUMAN=$(du -h -s "${ARTIFACTS_DIR}" | cut -f 1)
+    # `-k` gives consistently the number of kilobytes on both macOS and Linux,
+    # without it BSD `du` would give the number of multiples of 512 bytes.
+    ARTIFACTS_SIZE=$(($(du -k -s "${ARTIFACTS_DIR}" | cut -f 1) * 1024))
+    echo "The artifact store size is: ${ARTIFACTS_SIZE_HUMAN}"
+
+    if [[ ${ARTIFACTS_SIZE} -gt ${ARTIFACTS_LIMIT} ]]; then
+        echo "This is greater than the limit (${ARTIFACTS_SIZE} > ${ARTIFACTS_LIMIT} bytes), so we will clear the artifact store"
+
+        # Remove directories until we are below the limit
+        # We do this in Julia to avoid platform portability issues.
+        julia --color=yes -e '
+            function main(artifacts, size_str, limit_str)
+                size = parse(Int, size_str)
+                limit = parse(Int, limit_str)
+
+                # Get all artifacts with their creation times
+                worklist = []
+                for dir in readdir(artifacts)
+                    path = joinpath(artifacts, dir)
+                    isdir(path) || continue
+                    push!(worklist, (; path, stat=stat(path)))
+                end
+
+                # Sort by creation time (oldest first)
+                sort!(worklist; by=entry->entry.stat.ctime)
+
+                # Remove artifacts until we are under the limit
+                for entry in worklist
+                    # artifacts can have funky permissions
+                    run(`chmod -R u+w $(entry.path)`)
+                    sz = parse(Int, split(read(`du -k -s $(entry.path)`, String))[1]) * 1024
+                    rm(entry.path; recursive=true)
+                    size -= sz
+                    size <= limit && break
                 end
             end
 
             main(ARGS...)
-        ' "${COMPILECACHE}" "${CACHE_SIZE}" "${CACHE_LIMIT}" || true
+        ' "${ARTIFACTS_DIR}" "${ARTIFACTS_SIZE}" "${ARTIFACTS_LIMIT}" || true
 
         # Cleanup empty directories
-        find "${COMPILECACHE}" -type d -empty -delete
+        find "${ARTIFACTS_DIR}" -type d -empty -delete
     fi
 fi
 
 
 ### Step 4: If the depot is still too big, remove it altogether.
 
-# `Pkg.gc` doesn't always clean old artifacts, for reasons that are not entirely
-# clear yet. As a final resort, we can remove the entire depot if it is too big.
+# Data may be stored in unmanaged directories (e.g. `conda` or `scratchspaces`)
 
-# default limit: 20 GiB
-DEPOT_HARD_LIMIT="${BUILDKITE_PLUGIN_JULIA_DEPOT_HARD_SIZE_LIMIT:-21474836480}"
+# default limit: COMPILED_LIMIT + ARTIFACTS_LIMIT + 10GiB (~20GiB normally)
+DEPOT_LIMIT_DEFAULT=$(($COMPILED_LIMIT+$ARTIFACTS_LIMIT+10737418240))
+DEPOT_LIMIT="${BUILDKITE_PLUGIN_JULIA_DEPOT_SIZE_LIMIT:-${DEPOT_LIMIT_DEFAULT}}"
 
 if [[ "${BUILDKITE_PLUGIN_JULIA_ISOLATED_DEPOT:-true}" == "true" ]]; then
     DEPOT_SIZE_HUMAN=$(du -h -s "${JULIA_DEPOT_PATH}" | cut -f 1)
@@ -127,8 +155,8 @@ if [[ "${BUILDKITE_PLUGIN_JULIA_ISOLATED_DEPOT:-true}" == "true" ]]; then
     DEPOT_SIZE=$(($(du -k -s "${JULIA_DEPOT_PATH}" | cut -f 1) * 1024))
     echo "The depot size is: ${DEPOT_SIZE_HUMAN}"
 
-    if [[ ${DEPOT_SIZE} -gt ${DEPOT_HARD_LIMIT} ]]; then
-        echo "This is greater than the hard limit (${DEPOT_SIZE} > ${DEPOT_HARD_LIMIT} bytes), so we will clear the entire depot"
+    if [[ ${DEPOT_SIZE} -gt ${DEPOT_LIMIT} ]]; then
+        echo "This is greater than the limit (${DEPOT_SIZE} > ${DEPOT_LIMIT} bytes), so we will clear the entire depot"
         rm -rf "${JULIA_DEPOT_PATH}"
         mkdir -p "${JULIA_DEPOT_PATH}"
     fi
@@ -140,7 +168,7 @@ fi
 # We mark a depot as in-use by `touch`ing it at the start of the pipeline,
 # so we can remove any depots that are older than a certain age.
 
-PIPELINE_AGE_LIMIT="${BUILDKITE_PLUGIN_JULIA_PIPELINE_AGE_LIMIT:-2592000}"
+DEPOT_AGE_LIMIT="${BUILDKITE_PLUGIN_JULIA_DEPOT_AGE_LIMIT:-2592000}"
 
 julia --color=yes -e '
     function main(depots_dir, age_limit_str)
@@ -161,4 +189,4 @@ julia --color=yes -e '
         end
     end
     main(ARGS...)
-' "${CACHE_DIR}"/depots "${PIPELINE_AGE_LIMIT}" || true
+' "${CACHE_DIR}"/depots "${DEPOT_AGE_LIMIT}" || true

plugin.yml

@@ -12,9 +12,13 @@ configuration:
       type: boolean
     persist_depot_dirs:
       type: string
-    depot_hard_size_limit:
+    compiled_size_limit:
       type: string
-    cleanup_collect_delay:
+    artifacts_size_limit:
+      type: string
+    depot_size_limit:
+      type: string
+    depot_age_limit:
       type: string
     python:
       type: string