Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Python versions and fix locking issues #48

Merged
merged 10 commits into from
Jan 2, 2025
Merged
37 changes: 14 additions & 23 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,14 @@ on:
env:
PYTHON_STANDALONE_VERSIONS: |
[
"20240415"
"20241219"
]
PYTHON_VERSIONS: |
[
"3.10.14",
"3.11.9",
"3.12.3"
"3.10.16",
"3.11.11",
"3.12.8",
"3.13.1",
]

jobs:
Expand Down Expand Up @@ -61,14 +62,6 @@ jobs:
uses: actions/setup-go@v5
with:
go-version: 1.19
- uses: actions/cache@v4
with:
path: |
~/go/pkg/mod
~/.cache/go-build
key: ${{ runner.os }}-${{ matrix.pythonStandaloneVersion }}-${{ matrix.pythonVersion }}-build-tag-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-${{ matrix.pythonStandaloneVersion }}-${{ matrix.pythonVersion }}-build-tag-
- name: build-tag
run: |
git config --global user.email "[email protected]"
Expand All @@ -79,11 +72,14 @@ jobs:
- name: git gc
run: |
git gc
- name: rename .git
run: |
mv .git git-dir
- uses: actions/upload-artifact@v4
with:
name: workdir-${{ matrix.pythonStandaloneVersion }} ${{ matrix.pythonVersion }}
path: |
.git
git-dir
build-num

tests:
Expand All @@ -94,8 +90,8 @@ jobs:
matrix:
os:
- ubuntu-22.04
- macos-12
- windows-2019
- macos-13
- windows-2022
pythonStandaloneVersion: ${{ fromJSON(needs.build-matrix.outputs.PYTHON_STANDALONE_VERSIONS) }}
pythonVersion: ${{ fromJSON(needs.build-matrix.outputs.PYTHON_VERSIONS) }}
fail-fast: false
Expand All @@ -105,17 +101,12 @@ jobs:
uses: actions/setup-go@v5
with:
go-version: 1.19
- uses: actions/cache@v4
with:
path: |
~/go/pkg/mod
~/.cache/go-build
key: ${{ runner.os }}-${{ matrix.pythonStandaloneVersion }}-${{ matrix.pythonVersion }}-tests-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-${{ matrix.pythonStandaloneVersion }}-${{ matrix.pythonVersion }}-tests-
- uses: actions/download-artifact@v4
with:
name: workdir-${{ matrix.pythonStandaloneVersion }} ${{ matrix.pythonVersion }}
- name: rename .git back
run: |
mv git-dir .git
- name: checkout tag
shell: bash
run: |
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,16 +45,16 @@ The following operating systems and architectures are supported:
Releases in this library are handled a bit different from what one might be used to. This library does currently not
follow a versioning schema comparable to sematic versioning. This might however change in the future.

Right now, every tagged release is compromised of the Python interpreter version, the [python-standalone](https://github.com/indygreg/python-build-standalone)
and a build number. For example, the release version `v0.0.0-3.11.6-20231002-2` belongs to Python version 3.11.6,
the [20231002](https://github.com/indygreg/python-build-standalone/releases/tag/20231002) version of python-standalone
Right now, every tagged release is compromised of the Python interpreter version, the [python-standalone](https://github.com/astral-sh/python-build-standalone)
and a build number. For example, the release version `v0.0.0-3.11.6-20241219-2` belongs to Python version 3.11.6,
the [20241219](https://github.com/astral-sh/python-build-standalone/releases/tag/20241219) version of python-standalone
and build number 2. The release version currently always has v0.0.0 as its own version.

The way versioning is handled might result in popular dependency management tools (e.g. dependabot) to not work as you
might require it. Please watch out to not accidentally upgrade your Python version!

## How it works
This library uses the standalone Python distributions found at https://github.com/indygreg/python-build-standalone as
This library uses the standalone Python distributions found at https://github.com/astral-sh/python-build-standalone as
the base.

The `./hack/build-tag.sh` script is used to invoke `python/generate` and `pip/generate`, which then downloads, extracts
Expand Down
5 changes: 3 additions & 2 deletions embed_util/embedded_files.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"bytes"
"compress/gzip"
"fmt"
"github.com/rogpeppe/go-internal/lockedfile"
"github.com/gofrs/flock"
"io"
"io/fs"
"os"
Expand Down Expand Up @@ -63,7 +63,8 @@ func (e *EmbeddedFiles) extract(embedFs fs.FS, withHashInDir bool) error {
return err
}

lock, err := lockedfile.Create(e.extractedPath + ".lock")
lock := flock.New(e.extractedPath + ".lock")
err = lock.Lock()
if err != nil {
return err
}
Expand Down
12 changes: 7 additions & 5 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,18 @@ go 1.19

require (
github.com/gobwas/glob v0.2.3
github.com/klauspost/compress v1.17.9
github.com/rogpeppe/go-internal v1.12.0
github.com/gofrs/flock v0.12.1
github.com/klauspost/compress v1.17.11
github.com/sirupsen/logrus v1.9.3
github.com/stretchr/testify v1.9.0
golang.org/x/sync v0.7.0
github.com/stretchr/testify v1.10.0
golang.org/x/sync v0.10.0
)

require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
golang.org/x/sys v0.21.0 // indirect
github.com/rogpeppe/go-internal v1.13.1 // indirect
golang.org/x/sys v0.28.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
38 changes: 19 additions & 19 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,35 +1,35 @@
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y=
github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8=
github.com/klauspost/compress v1.17.7 h1:ehO88t2UGzQK66LMdE8tibEd1ErmzZjNEqWkjLAKQQg=
github.com/klauspost/compress v1.17.7/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
github.com/klauspost/compress v1.17.8 h1:YcnTYrq7MikUT7k0Yb5eceMmALQPYBW/Xltxn0NAMnU=
github.com/klauspost/compress v1.17.8/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA=
github.com/klauspost/compress v1.17.9/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
github.com/gofrs/flock v0.12.1 h1:MTLVXXHf8ekldpJk3AKicLij9MdwOWkZ+a/jHHZby9E=
github.com/gofrs/flock v0.12.1/go.mod h1:9zxTsyu5xtJ9DK+1tFZyibEV7y3uwDxPPfbxeeHCoD0=
github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=
github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
golang.org/x/sync v0.6.0 h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0 h1:YsImfSBoP9QPYL0xyKJPq0gcaJdG3rInoqxTWbfQu9M=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.15.0 h1:h48lPFYpsTvQJZF4EKyI4aLHaev3CxivZmv7yZig9pc=
golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.21.0 h1:rF+pYz3DAGSQAxAu1CbC7catZg4ebC4UIeIhKxBZvws=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
golang.org/x/sys v0.28.0 h1:Fksou7UEQUWlKvIdsqzJmUmCX3cZuD2+P3XyyzwMhlA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
2 changes: 1 addition & 1 deletion pip/internal/requirements.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
pip==23.3.1
pip==24.3.1
4 changes: 2 additions & 2 deletions python/generate/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import (
)

var (
pythonStandaloneVersion = flag.String("python-standalone-version", "", "specify the python-standalone version. Check https://github.com/indygreg/python-build-standalone/releases/ for available options.")
pythonStandaloneVersion = flag.String("python-standalone-version", "", "specify the python-standalone version. Check https://github.com/astral-sh/python-build-standalone/releases/ for available options.")
pythonVersion = flag.String("python-version", "", "specify the python version.")
preparePath = flag.String("prepare-path", filepath.Join(os.TempDir(), "python-download"), "specify the path where the python executables are downloaded and prepared. automatically creates a temporary directory if unset")
runPrepare = flag.Bool("prepare", true, "if set, python executables will be downloaded and prepared for packing at the configured path")
Expand Down Expand Up @@ -173,7 +173,7 @@ func download(osName string, arch string, dist string) string {

downloadPath := generateDownloadPath(arch, dist)
fname := filepath.Base(downloadPath)
downloadUrl := fmt.Sprintf("https://github.com/indygreg/python-build-standalone/releases/download/%s/%s", *pythonStandaloneVersion, fname)
downloadUrl := fmt.Sprintf("https://github.com/astral-sh/python-build-standalone/releases/download/%s/%s", *pythonStandaloneVersion, fname)

if _, err := os.Stat(downloadPath); err == nil {
log.Infof("skipping download of %s", downloadUrl)
Expand Down
Loading