Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Single release CMake file that contains all required links/hashes. #4631

Merged
merged 61 commits into from
Mar 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
8c02c87
Modify release.yml workflow
dudoslav Jan 31, 2024
94d0194
Don't build using superbuild
dudoslav Jan 31, 2024
442f9f1
Fix typo in binary package step
dudoslav Jan 31, 2024
4691c69
Change source package file name
dudoslav Jan 31, 2024
6cf7c9c
Fix artifact upload
dudoslav Jan 31, 2024
efe3772
Fix incorrect names for source and binary archive
dudoslav Jan 31, 2024
e1cb7ca
Use CPack in CI instead of CMake
dudoslav Jan 31, 2024
0884182
Try packaging in Superbuild
dudoslav Feb 1, 2024
ded384c
Build before packaging
dudoslav Feb 2, 2024
c276d08
Divide source and binary releases
dudoslav Feb 2, 2024
c445fb4
Fix Source builds
dudoslav Feb 2, 2024
307fea1
Fix typo
dudoslav Feb 2, 2024
26b6359
Properly name MacOS releases
dudoslav Feb 2, 2024
a9ac1f9
Also use proper macos variables
dudoslav Feb 2, 2024
918fc56
Fix check of empty string
dudoslav Feb 2, 2024
0845c5c
Remove redundant variable expansion
dudoslav Feb 2, 2024
d566647
Ignore files and also upload hash files
dudoslav Feb 2, 2024
c9ecc9d
Add testing append-release-cmake.yml
dudoslav Jan 11, 2024
7eb7606
Download release files from upstream
dudoslav Jan 11, 2024
6a7cff3
Add asset-output field
dudoslav Jan 11, 2024
eb2627e
Fix typo
dudoslav Jan 11, 2024
a79ee6c
Basic template
dudoslav Jan 15, 2024
15e3bf8
Fix release assets output
dudoslav Jan 15, 2024
ec984b8
Invalid asset_output
dudoslav Jan 15, 2024
42bf8ec
Change asset-outpu
dudoslav Jan 15, 2024
e5a5d24
Add make release directory
dudoslav Jan 15, 2024
482349e
Also upload template
dudoslav Jan 15, 2024
efdc830
Fix missing FindTileDB_EP.cmake asset
dudoslav Jan 15, 2024
2f8adf3
Debug
dudoslav Jan 15, 2024
dab1305
Fix typo
dudoslav Jan 15, 2024
048505e
Add another debug
dudoslav Jan 15, 2024
6dbd276
Add quotes
dudoslav Jan 15, 2024
95e45fa
Change output directory for script
dudoslav Jan 15, 2024
cd7f4e1
Change sed command
dudoslav Jan 15, 2024
0aea033
Also generate source hash/url
dudoslav Jan 15, 2024
1b05d49
Fix source hash
dudoslav Jan 15, 2024
f3d9dda
Remove find tiledb file and add tiledb package
dudoslav Jan 15, 2024
879e7e5
Fix typo
dudoslav Jan 15, 2024
bb29940
Produce release list
dudoslav Jan 16, 2024
77399ae
Change noavx column
dudoslav Jan 16, 2024
e43dfae
Modify CMake download tiledb logic
dudoslav Jan 16, 2024
6d30b6b
Also generate Module file
dudoslav Jan 16, 2024
b33324b
WIP: get_tiledb_source_url_and_hash
dudoslav Jan 26, 2024
705260d
Rebase and fix for CPack logic
dudoslav Feb 5, 2024
e8ac4cc
Ignore HASH files
dudoslav Feb 5, 2024
8bcf35e
Expand source file name
dudoslav Feb 5, 2024
448d0a8
Fix filename expansion
dudoslav Feb 5, 2024
04ed61f
Fix source file name path
dudoslav Feb 5, 2024
1fbcb02
Match release files and not source files
dudoslav Feb 5, 2024
43a877d
Fix matching version
dudoslav Feb 5, 2024
881d79c
Fix SHA256 HASH reading
dudoslav Feb 5, 2024
1160c31
Merge remote-tracking branch 'origin/dev' into db/sc-17775/release_cmake
dudoslav Feb 26, 2024
119107a
Merge remote-tracking branch 'origin/dev' into db/sc-17775/release_cmake
dudoslav Feb 27, 2024
1209aa0
Rework rellist format
dudoslav Feb 28, 2024
8c38ab0
Automatically call generate release-list in release
dudoslav Feb 28, 2024
e36fe6a
Fix workflow call
dudoslav Feb 28, 2024
d710704
Fix source file export
dudoslav Feb 28, 2024
2253c38
Add debugging urls
dudoslav Feb 28, 2024
6d12f0f
Fix artifact assets download url
dudoslav Feb 28, 2024
6d01260
Use fetch content in DownloadPrebuiltTileDB
dudoslav Mar 1, 2024
c224bda
Add ref_name to generate release list
dudoslav Mar 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions .github/workflows/append-release-cmake.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
name: Append Release CMake

on:
workflow_dispatch:
inputs:
ref:
description: 'Ref to be used as release'
default: 'latest'
required: true
type: string
workflow_call:
inputs:
ref:
description: 'Ref to be used as release'
default: 'latest'
required: true
type: string

jobs:
generate_cmake_files:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this should be part of the Release workflow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dudoslav will you do this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to release workflow as a reusable workflow

runs-on: ubuntu-latest
steps:
- name: Checkout TileDB
uses: actions/checkout@v3

- name: Make release and output directories
run: |
mkdir release output

- name: Github release data
id: release_data
uses: KevinRohn/[email protected]
with:
# repository: 'TileDB-Inc/TileDB'
version: ${{ inputs.ref }}
asset-file: '*.zip,*.tar.gz'
asset-output: './release/'

- name: Render template
run: |
PATTERN="tiledb-([^-]+)-([^-]+)(-noavx2)?-([^-]+).(tar.gz|zip)$"
RELLIST="output/releases.csv"
MODULE="output/DownloadPrebuiltTileDB.cmake"
cp cmake/inputs/DownloadPrebuiltTileDB.cmake $MODULE
echo "platform,url,sha256" > $RELLIST

for FILE in $(ls release)
do
if [[ $FILE =~ $PATTERN ]]
then
OS=${BASH_REMATCH[1]^^}
ARCH=${BASH_REMATCH[2]^^}
NOAVX2=${BASH_REMATCH[3]^^}
PLATFORM=${OS}-${ARCH}${NOAVX2}

URL="${{ github.server_url }}/${{ github.repository }}/releases/download/${{ inputs.ref }}/$FILE"
HASH=$(cat release/$FILE.sha256 | cut -d \t -f 1)

echo "${PLATFORM},${URL},${HASH}" >> $RELLIST
fi
done

SOURCE_FILE_NAME=$(ls release/tiledb-source-*.tar.gz)
URL_TILEDB_SOURCE="${{ github.server_url }}/${{ github.repository }}/releases/download/${{ inputs.ref }}/$(basename $SOURCE_FILE_NAME)"
HASH_TILEDB_SOURCE=$(cat $SOURCE_FILE_NAME.sha256 | cut -d \t -f 1)

echo "source,${URL_TILEDB_SOURCE},${HASH_TILEDB_SOURCE}" >> $RELLIST

HASH=$(sha256sum $RELLIST | cut -d " " -f 1)
echo $HASH > $RELLIST.sha256

cat $RELLIST

- name: Upload template to release
uses: svenstaro/upload-release-action@v2
with:
file: output/*
tag: ${{ steps.release_data.outputs.tag_name }}
overwrite: true
file_glob: true
8 changes: 7 additions & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ on:
- 'release-*'
tags:
- '*'
workflow_dispatch:

jobs:
Package-Source-Release:
Expand Down Expand Up @@ -175,6 +174,13 @@ jobs:
});
}

Generate-Release-List:
needs:
- Publish-Release
uses: ./.github/workflows/append-release-cmake.yml
with:
ref: ${{ github.ref_name }}

Create-Issue-On-Fail:
permissions:
issues: write
Expand Down
128 changes: 128 additions & 0 deletions cmake/inputs/DownloadPrebuiltTileDB.cmake
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
#
# FindTileDB_EP.cmake
#
#
# The MIT License
#
# Copyright (c) 2023 TileDB, Inc.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

include(FetchContent)

function(fetch_tiledb_release_list VERSION EXPECTED_HASH)
# Local constants
set(UPSTREAM_URL "https://github.com/TileDB-Inc/TileDB/releases/download")

if(NOT VERSION)
set(VERSION latest)
endif()

if(${EXPECTED_HASH})
file(DOWNLOAD
${UPSTREAM_URL}/${VERSION}/releases.csv
releases.csv
SHOW_PROGRESS
EXPECTED_HASH ${EXPECTED_HASH}
)
else()
message(WARNING "Downloading release list without SHA checksum!")
file(DOWNLOAD
${UPSTREAM_URL}/${VERSION}/releases.csv
releases.csv
SHOW_PROGRESS
)
endif()

file(STRINGS
${CMAKE_CURRENT_BINARY_DIR}/releases.csv
RELLIST
)

# Remove csv table headers
list(POP_FRONT RELLIST)

foreach(LINE ${RELLIST})
string(REPLACE "," ";" LINE ${LINE})
list(LENGTH LINE LENGTH)

list(GET LINE 0 PLATFORM)
list(GET LINE 1 URL)
list(GET LINE 2 SHA)

set(RELEASE_VAR TILEDB_${PLATFORM})
set(URL_${RELEASE_VAR} ${URL} PARENT_SCOPE)
set(HASH_${RELEASE_VAR} ${SHA} PARENT_SCOPE)
endforeach()
endfunction()

function(detect_artifact_name OUT_VAR)
if (WIN32) # Windows
SET(${OUT_VAR} TILEDB_WINDOWS-X86_64 PARENT_SCOPE)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should fail on other architectures. Same in Linux.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that CMAKE can only define these three variables: WIN32/APPLE/LINUX

elseif(APPLE) # OSX
if (DEFINED CMAKE_OSX_ARCHITECTURES)
set(ACTUAL_TARGET ${CMAKE_OSX_ARCHITECTURES})
else()
set(ACTUAL_TARGET ${CMAKE_SYSTEM_PROCESSOR})
endif()


if (ACTUAL_TARGET MATCHES "(x86_64)|(AMD64|amd64)|(^i.86$)")
SET(${OUT_VAR} TILEDB_MACOS-X86_64 PARENT_SCOPE)
elseif (ACTUAL_TARGET STREQUAL arm64 OR ACTUAL_TARGET MATCHES "^aarch64" OR CMAKE_SYSTEM_PROCESSOR MATCHES "^arm")
SET(${OUT_VAR} TILEDB_MACOS-ARM64 PARENT_SCOPE)
endif()
else() # Linux
SET(${OUT_VAR} TILEDB_LINUX-X86_64 PARENT_SCOPE)
endif()
endfunction()

function(fetch_prebuilt_tiledb)
# Arguments
set(options RELLIST_HASH)
set(oneValueArgs VERSION ARTIFACT_NAME)
set(multiValueArgs)
cmake_parse_arguments(
FETCH_PREBUILT_TILEDB
"${options}"
"${oneValueArgs}"
"${multiValueArgs}"
${ARGN}
)

fetch_tiledb_release_list(${FETCH_PREBUILT_TILEDB_VERSION} ${FETCH_PREBUILT_TILEDB_RELLIST_HASH})

if(NOT FETCH_PREBUILT_TILEDB_ARTIFACT_NAME)
detect_artifact_name(FETCH_PREBUILT_TILEDB_ARTIFACT_NAME)
endif()

string(STRIP ${HASH_${FETCH_PREBUILT_TILEDB_ARTIFACT_NAME}} HASH_${FETCH_PREBUILT_TILEDB_ARTIFACT_NAME})
FetchContent_Declare(
tiledb-prebuilt
URL ${URL_${FETCH_PREBUILT_TILEDB_ARTIFACT_NAME}}
URL_HASH SHA256=${HASH_${FETCH_PREBUILT_TILEDB_ARTIFACT_NAME}}
DOWNLOAD_EXTRACT_TIMESTAMP FALSE
)

FetchContent_MakeAvailable(
tiledb-prebuilt
)

set(TileDB_DIR "${tiledb-prebuilt_SOURCE_DIR}/lib/cmake/TileDB" PARENT_SCOPE)
endfunction()
Loading