Merge remote-tracking branch 'origin/6.0/release' into 6.0/patch

README.md

@@ -97,39 +97,38 @@ we've decided to leverage the advantages offered by revision control. As such,
 we've adopted a well defined branching model for each third-party package.
 
 First of all, we have a Delphix repository on github for each third-party
-package that we build. Each repository has at least 2 branches: **master** and
-**upstreams/master**. The **master** branch of the package is the one we build,
-and contains Delphix changes. The **upstreams/master** branch is used to track
+package that we build. Each repository has at least 2 branches: **6.0/stage** and
+**upstreams/6.0/stage**. The **6.0/stage** branch of the package is the one we build,
+and contains Delphix changes. The **upstreams/6.0/stage** branch is used to track
 the upstream version of the package. For packages that are not provided by
-Ubuntu but are available on git, the **upstreams/master** branch usually just
-tracks the **master** branch of the project. For packages that are provided by
-Ubuntu, the **upstreams/master** branch instead tracks the source package that
+Ubuntu but are available on git, the **upstreams/6.0/stage** branch usually just
+tracks the **6.0/stage** branch of the project. For packages that are provided by
+Ubuntu, the **upstreams/6.0/stage** branch instead tracks the source package that
 is maintained by Ubuntu (i.e. the branch contains the files obtained from
 `apt-get source <source-package>`). This offers the advantage of using a version
 of the package tuned to work with our Ubuntu distribution.
 
-When updating a package, we first check if the **upstreams/master** branch is
+When updating a package, we first check if the **upstreams/6.0/stage** branch is
 up-to-date, by fetching the latest version of the upstream git repository or the
 Ubuntu source package. If changes are detected, we update the
-**upstreams/master** branch and push the changes to GitHub.
+**upstreams/6.0/stage** branch and push the changes to GitHub.
 
-The second step is to check if the **master** branch is up-to-date with
-**upstreams/master**. If it is already up-to-date, then we are done. If not,
-then we attempt merging **upstreams/master** into **master**.
+The second step is to check if the **6.0/stage** branch is up-to-date with
+**upstreams/6.0/stage**. If it is already up-to-date, then we are done. If not,
+then we attempt merging **upstreams/6.0/stage** into **6.0/stage**.
 
 If the merge is successful, then we push the changes to a staging branch on
-GitHub, called **projects/auto-update/master/merging**. The intent is for
+GitHub, called **projects/auto-update/6.0/stage/merging**. The intent is for
 a different system to fetch those changes, build them, and then launch tests.
 
 See [Scripts > sync-with-upstream.sh](#sync-with-upstreamsh) below.
 
 Once the merge has been tested, [Scripts > push-merge.sh](#push-mergesh) is
-called on the original VM to push the changes to the **master** branch on
+called on the original VM to push the changes to the **6.0/stage** branch on
 GitHub.
 
-Note that the example above targets the **master** branch, but the same
-workflow could apply to other branches, like **6.0/stage**, although it is
-not currently in use.
+Note that the example above targets the **6.0/stage** branch, but the same
+workflow could apply to other branches.
 
 ## Scripts
 
@@ -172,8 +171,8 @@ Usage:
 ```
 
 This checks if a package has updates in the upstream project that haven't been
-pulled into the **upstreams/master** branch, or if the **upstreams/master**
-branch has commits that haven't been merged into the **master** branch.
+pulled into the **upstreams/6.0/stage** branch, or if the **upstreams/6.0/stage**
+branch has commits that haven't been merged into the **6.0/stage** branch.
 
 If updates are available, the file `<WORKDIR>/update-available` will be created.
 
@@ -189,19 +188,19 @@ Usage:
 
 This script has 2 tasks:
 1. Check if the upstream project has updates that are not pulled into the
-**upstreams/master** branch of the package, and if so then update that branch
+**upstreams/6.0/stage** branch of the package, and if so then update that branch
 and push changes to GitHub.
-2. Merge **upstreams/master** into **master** and push the changes to a staging
-branch on GitHub, called **projects/auto-update/master/merging**. Another
+2. Merge **upstreams/6.0/stage** into **6.0/stage** and push the changes to a staging
+branch on GitHub, called **projects/auto-update/6.0/stage/merging**. Another
 system should use that branch to build the package, and then run the appropriate
 integration tests.
 
 After testing has been completed, `push-merge.sh <package>` should be called on
-the same system to push the merge to the **master** branch.
+the same system to push the merge to the **6.0/stage** branch.
 
 Note that the DRYRUN environment variable must be set when running this script.
 If DRYRUN is set to "true", then changes are not pushed to GitHub in step 1,
-and staged changes are pushed to **projects/auto-update/master/merging-dryrun**
+and staged changes are pushed to **projects/auto-update/6.0/stage/merging-dryrun**
 in step 2 instead of the non-dryrun branch. The intention is that when testing
 changes to the logic we want to be able to run most of the logic, but without
 affecting the production branches.
@@ -215,8 +214,8 @@ Usage:
 
 This must be called on a system that has previously called
 `sync-with-upstream.sh` for the same package. It will push the merge that was
-previously prepared by `sync-with-upstream.sh` to the production **master**
-branch, after checking that the **master** branch hasn't been modified since
+previously prepared by `sync-with-upstream.sh` to the production **6.0/stage**
+branch, after checking that the **6.0/stage** branch hasn't been modified since
 `sync-with-upstream.sh` was called.
 
 Like for `sync-with-upstream.sh`, the DRYRUN environment variable must be set
@@ -245,10 +244,10 @@ of some of the scripts defined above.
   revision defined. If not set, it will be auto-generated from the timestamp.
   Applies to [buildpkg.sh](#buildpkgsh).
 
-* **DEFAULT_GIT_BRANCH**: The product branch that is being built or updated is
-  typically stored in the file `branch.config`, however it can be overridden via
-  DEFAULT_GIT_BRANCH. It can either be set to a development branch, such as
-  "master" or "6.0/stage", or a release tag, such as "release/6.0.6.0".
+* **DEFAULT_GIT_BRANCH**: The product branch that is being built or updated
+  and must be set via DEFAULT_GIT_BRANCH. It can either be set to a
+  development branch, such as "6.0/stage", or a release tag, such as
+  "release/6.0.6.0".
   The product branch is used in multiple instances. When
   running [setup.sh](#setupsh), it will determine what linux-package-mirror
   link to use when fetching packages from apt (although those links can be
@@ -383,7 +382,7 @@ The `fetch()` hook is optional, as a default is provided and should be used. It
 is called when fetching the source code of the package to build or to update.
 The repository is cloned into `<WORKDIR>/repo` and checked out as
 branch **repo-HEAD**. If we are performing a package update, then we also
-fetch the **upstreams/master** branch into **upstream-HEAD**. The default
+fetch the **upstreams/6.0/stage** branch into **upstream-HEAD**. The default
 should only be overridden when not fetching the package source from git.
 
 #### Prepare (hook)
@@ -402,7 +401,7 @@ The `update_upstream()` hook should only be defined for third party packages
 that can be auto-updated. It is responsible for fetching the latest upstream
 source code on top of branch **upstream-HEAD** of our fetched repository in
 `<WORKDIR>/repo`. Note that any changes should be rebased on top of
-the **upstreams/master** branch. If changes are detected, file
+the **upstreams/6.0/stage** branch. If changes are detected, file
 `<WORKDIR>/upstream-updated` should be created.
 
 #### Merge With Upstream (hook)
@@ -552,8 +551,8 @@ UPSTREAM_GIT_BRANCH="<git branch>"
 
 Note that steps 3 to 5 are most useful when getting a third party package from
 an Ubuntu source package. When the third party package is fetched from git,
-you may simply fork the upstream repository and add an **upstreams/master**
-branch that points to the **master** branch; you can then update
+you may simply fork the upstream repository and add an **upstreams/6.0/stage**
+branch that points to the **6.0/stage** branch; you can then update
 `DEFAULT_PACKAGE_GIT_URL` in config.sh to your forked git repository and skip
 to step 6.
 
@@ -591,8 +590,8 @@ DEFAULT_PACKAGE_GIT_URL="https://github.com/<developer>/<package>"
 #### Step 5. Push to your developer repository
 
 Next step is to push the upstream code to the newly created repository to your
-developer repository. You should push the initial commit to both the **master**
-branch and the **upstreams/master** branch.
+developer repository. You should push the initial commit to both the **6.0/stage**
+branch and the **upstreams/6.0/stage** branch.
 
 #### Step 6. Build the package
 
@@ -618,7 +617,7 @@ build the package.
 
 For a package that doesn't have a `debian` metadata directory already defined in
 its source tree, you'll need to create it, and push the changes to the
-**master** branch of your developer repository. See
+**6.0/stage** branch of your developer repository. See
 [Common Steps > Creating debian metadirectory](#creating-debian-metadirectory)
 for more details.
 
@@ -729,9 +728,9 @@ is to create an official repository for it.
    [Delphix Open-Source Policy](https://docs.delphix.com/en/ip-strategy/outbound-open-source)
    if you haven't already, and provide the necessary info so that a
    `github.com/delphix/<package>` repository can be created for it. You'll need
-   to push the **master** branch from your developer repository, as well as the
-   **upstreams/master** branch if it is a third-party package. Note that if you
-   have modified **master** (i.e. it diverges from **upstreams/master**), you
+   to push the **6.0/stage** branch from your developer repository, as well as the
+   **upstreams/6.0/stage** branch if it is a third-party package. Note that if you
+   have modified **6.0/stage** (i.e. it diverges from **upstreams/6.0/stage**), you
    should submit your changes for review before pushing them.
 
 1. If this is a third-party package that is to be auto-updated by Delphix

buildpkg.sh

@@ -67,13 +67,13 @@ shift $((OPTIND - 1))
 PACKAGE=$1
 
 logmust check_package_exists "$PACKAGE"
+check_env DEFAULT_GIT_BRANCH
 
 #
 # DEFAULT_REVISION & DEFAULT_GIT_BRANCH will be set if called from buildlist.sh.
 # If the script is called manually, we set it here.
 #
 DEFAULT_REVISION="${DEFAULT_REVISION:-$(default_revision)}"
-logmust determine_default_git_branch
 
 echo ""
 echo_bold "===================================================================="

checkupdates.sh

@@ -53,7 +53,6 @@ logmust check_package_exists "$PACKAGE"
 # If the script is called manually, we set it here.
 #
 DEFAULT_REVISION="${DEFAULT_REVISION:-$(default_revision)}"
-logmust determine_default_git_branch
 
 logmust load_package_config "$PACKAGE"
 logmust create_workdir

default-package-config.sh

@@ -345,6 +345,7 @@ function kernel_update_upstream() {
 	local local_upstream_commit
 	local_upstream_commit=$(git rev-parse upstream-HEAD)
 	[[ -z "${local_upstream_commit}" ]] && die "could not find upstream-HEAD's commit"
+	check_env DEFAULT_GIT_BRANCH
 	echo "note: upstreams/${DEFAULT_GIT_BRANCH} commit: ${local_upstream_commit}"
 
 	if [[ "${upstream_tag_commit}" == "${local_upstream_commit}" ]]; then

lib/common.sh

@@ -144,32 +144,6 @@ function run_setup_if_needed() {
 	echo_bold "------------------------------------------------------------"
 }
 
-#
-# Determine DEFAULT_GIT_BRANCH. If it is unset, default to the branch set in
-# branch.config.
-#
-function determine_default_git_branch() {
-
-	[[ -n "$DEFAULT_GIT_BRANCH" ]] && return
-
-	echo "DEFAULT_GIT_BRANCH is not set."
-	if ! source "$TOP/branch.config" 2>/dev/null; then
-		die "No branch.config file found in repo root."
-	fi
-
-	if [[ -z "$DEFAULT_GIT_BRANCH" ]]; then
-		die "$DEFAULT_GIT_BRANCH parameter was not sourced" \
-			"from branch.config. Ensure branch.config is" \
-			"properly formatted with e.g." \
-			"DEFAULT_GIT_BRANCH='<upstream-product-branch>'"
-	fi
-
-	echo "Defaulting DEFAULT_GIT_BRANCH to branch" \
-		"$DEFAULT_GIT_BRANCH set in branch.config."
-
-	export DEFAULT_GIT_BRANCH
-}
-
 function is_release_branch() {
 	check_env DEFAULT_GIT_BRANCH
 	[[ "$DEFAULT_GIT_BRANCH" == release/* ]]

package-lists/build/main.pkgs

@@ -23,6 +23,7 @@ libkdumpfile
 make-jpkg
 makedumpfile
 masking
+misc-debs
 nfs-utils
 performance-diagnostics
 ptools

packages/adoptopenjdk/config.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 #
-# Copyright 2018, 2022 Delphix
+# Copyright 2018, 2023 Delphix
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -21,13 +21,13 @@ PACKAGE_DEPENDENCIES="make-jpkg"
 
 case $(dpkg-architecture -q DEB_HOST_ARCH 2>/dev/null || echo "none") in
 amd64)
-	_tarfile="OpenJDK8U-jdk_x64_linux_hotspot_8u345b01.tar.gz"
-	_tarfile_sha256="ed6c9db3719895584fb1fd69fc79c29240977675f26631911c5a1dbce07b7d58"
+	_tarfile="OpenJDK8U-jdk_x64_linux_hotspot_8u362b09.tar.gz"
+	_tarfile_sha256="1486a792fb224611ce0cd0e83d4aacd3503b56698549f8e9a9f0a6ebb83bdba1"
 	_jdk_path="/usr/lib/jvm/adoptopenjdk-java8-jdk-amd64"
 	;;
 arm64)
-	_tarfile="OpenJDK8U-jdk_aarch64_linux_hotspot_8u345b01.tar.gz"
-	_tarfile_sha256="c1965fb24dded7d7944e2da36cd902adf3b7b1d327aaa21ea507cff00a5a0090"
+	_tarfile="OpenJDK8U-jdk_aarch64_linux_hotspot_8u362b09.tar.gz"
+	_tarfile_sha256="9290a8beefd7a94f0eb030f62d402411a852100482b9c5b63714bacc57002c2a"
 	_jdk_path="/usr/lib/jvm/adoptopenjdk-java8-jdk-arm64"
 	;;
 *) ;;

packages/misc-debs/config.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+#
+# Copyright 2021, 2023 Delphix
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# shellcheck disable=SC2034
+
+DEFAULT_PACKAGE_GIT_URL=none
+SKIP_COPYRIGHTS_CHECK=true
+
+#
+# IMPORTANT NOTE
+# --------------
+#
+# Debian packages (debs) that are not built from source by linux-pkg can be
+# added to this "meta-package". As a general rule, pre-built debs should only
+# be added here when they have been fetched from a trusted third-party
+# package archive.
+#
+# Here are some valid reasons for adding new debs here:
+# - There are bugs with a recent version of a package provided by Ubuntu and
+#   we want to pin an older version of that package.
+# - Ubuntu provides a version of a package that is too old, and the package's
+#   maintainers provide a more recent version of the package. Note that in this
+#   case, you may also look into adding the maintainer's archive to the
+#   linux-package-mirror PPAs list.
+#
+# To add a new deb here, upload that deb to the linux-pkg/misc-debs directory
+# in artifcatory and note the deb's SHA256. Be explicit on where this deb
+# was fetched from and why it was added to this list.
+#
+# When removing debs from this list, you should not remove them from artifactory
+# as they would used when rebuilding older releases.
+#
+
+function fetch() {
+	logmust cd "$WORKDIR/artifacts"
+
+	local debs=(
+		"tar_1.30+dfsg-7ubuntu0.20.04.3_amd64.deb a3b1212570b10c192b3d49cffb68f909146d4b72a7104f213f4426c7e41d0e49"
+	)
+
+	local url="http://artifactory.delphix.com/artifactory/linux-pkg/misc-debs"
+
+	echo "Fetched debs:" >BUILD_INFO
+	local entry
+	for entry in "${debs[@]}"; do
+		local deb sha256
+		deb=$(echo "$entry" | awk '{print $1}')
+		sha256=$(echo "$entry" | awk '{print $2}')
+		[[ -n "$deb" && -n "$sha256" ]] || die "Invalid entry '$entry'"
+
+		logmust fetch_file_from_artifactory "$url/$deb" "$sha256"
+
+		echo "$entry" >>BUILD_INFO
+	done
+}
+
+function build() {
+	return
+	# Nothing to do, all the logic is done in fetch().
+}

push-merge.sh

@@ -18,6 +18,7 @@
 TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"
 source "$TOP/lib/common.sh"
 
+check_env DEFAULT_GIT_BRANCH
 logmust check_running_system
 logmust run_setup_if_needed
 
@@ -56,7 +57,6 @@ fi
 logmust check_package_exists "$PACKAGE"
 
 DEFAULT_REVISION="${DEFAULT_REVISION:-$(default_revision)}"
-logmust determine_default_git_branch
 logmust load_package_config "$PACKAGE"
 
 if [[ ! -d "$WORKDIR/repo" ]]; then

setup.sh

@@ -19,8 +19,7 @@ TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"
 source "$TOP/lib/common.sh"
 
 logmust check_running_system
-
-logmust determine_default_git_branch
+check_env DEFAULT_GIT_BRANCH
 
 #
 # Update the sources.list file to point to our internal package mirror. If no