rsync --exclude-from=<(git ls-files -o --directory | awk '{print "/"$0}') is not working under windows

[ahoehma]

As already described in #1026.

It seems that the rsync command which is executed by gcl is not 100% working under windows gitbash.

I can run "gcl --list":

parsing and downloads finished in 4.04 s.
json schema validated in 482 ms
name                                           description  stage                  when        allow_failure  needs
threat_modeling                                             pre-build              manual      true
gitleaks                                                    pre-build              manual      true
trivy-fs                                                    pre-build              manual      true
commit-log                                                  pre-build              manual      true
build-develop                                               build                  on_success  false          []
sbom-fs                                                     post-build             on_success  true
authenticate-aws-tooling-ecr                                package-ecr            manual      true           []
package-ecr-jib-aws-tooling-ecr                             package-ecr            manual      true
sbom-ecr-jib-aws-tooling-ecr                                package-ecr            manual      true           [package-ecr-jib-aws-tooling-ecr]
scan-ecr-jib-aws-tooling-ecr                                package-ecr            manual      true           [package-ecr-jib-aws-tooling-ecr]
rollout-eks-dev                                             rollout-eks:dev        manual      true
unrollout-eks-dev                                           rollout-eks:dev        manual      false
scan-eks-dev                                                rollout-eks:dev        manual      false

But I can't run a build job "gcl --stage build":

parsing and downloads finished in 2.04 s.
json schema validated in 356 ms
Error: Command failed with exit code 11: rsync -a --delete-excluded --delete --exclude-from=<(git ls-files -o --directory | awk '{print "/"$0}') --exclude .gitlab-ci-local/ ./ .gitlab-ci-local/builds/.docker/
rsync: [client] failed to open exclude file /proc/1737/fd/63: No such file or directory (2)
rsync error: error in file IO (code 11) at exclude.c(1481) [client=3.3.0]
    at makeError (C:\ProgramData\nvm\v20.17.0\node_modules\gitlab-ci-local\node_modules\execa\lib\error.js:60:11)
    at handlePromise (C:\ProgramData\nvm\v20.17.0\node_modules\gitlab-ci-local\node_modules\execa\index.js:118:26)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Function.rsyncTrackedFiles (C:\ProgramData\nvm\v20.17.0\node_modules\gitlab-ci-local\src\utils.ts:306:9)
    at handler (C:\ProgramData\nvm\v20.17.0\node_modules\gitlab-ci-local\src\handler.ts:89:9)
    at Object.handler (C:\ProgramData\nvm\v20.17.0\node_modules\gitlab-ci-local\src\index.ts:37:21)

It would may be better to redirect the result of git ls-files -o --directory | awk '{print "/"$0}' into a temp file and use this as input for rsync "--exclude-from"?

I'm running gcl 4.54.0 under windows gitbash.

$bash --version
#GNU bash, version 5.2.37(1)-release (x86_64-pc-msys)

[ahoehma]

Just and idea...

gitlab-ci-local/src/utils.ts

Line 306 in 32176c6

await Utils.bash(`rsync -a --delete-excluded --delete --exclude-from=<(git ls-files -o --directory | awk '{print "/"$0}') --exclude ${stateDir}/ ./ ${stateDir}/builds/${target}/`, cwd);

What about replace

--exclude-from=<(git ls-files -o --directory | awk '{print "/"$0}')

with something like this:

--exclude-from=$(temp_file=$(mktemp) && git ls-files -o --directory | awk '{print "/"$0}' >> "$temp_file" && echo "$temp_file")

the only problem is that this temp_file is not cleaned up automatically 👎

[ANGkeith]

so you're saying that the process substituion

--exclude-from=<(git ls-files -o --directory | awk '{print "/"$0}')

is causing you issue in windows gitbash?

---

@ahoehma As i don't have a windows gitbash environment to test, could you test and see if the following fixes your issue ?

#1380

[ahoehma]

so you're saying that the process substituion

--exclude-from=<(git ls-files -o --directory | awk '{print "/"$0}')

is causing you issue in windows gitbash?

@ahoehma As i don't have a windows gitbash environment to test, could you test and see if the following fixes your issue ?

#1380

How can I try that draft?

[ANGkeith]

hmm, will need to clone the draft and build it locally from your machine...