Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make copying to efs using rsync (or similar) instead of cp #10

Closed
Hi-Fi opened this issue Jun 21, 2024 · 1 comment
Closed

Make copying to efs using rsync (or similar) instead of cp #10

Hi-Fi opened this issue Jun 21, 2024 · 1 comment

Comments

@Hi-Fi
Copy link
Owner

Hi-Fi commented Jun 21, 2024

Currently copying of externals take around 3 mintues, as runner copies full Node16 and Node20 also.

To save that time, either parallel copy (recommended by https://repost.aws/knowledge-center/efs-troubleshoot-slow-performance) or some better appoach (like shared externals) could be used.

Copying: https://github.com/Hi-Fi/runner-container-hooks/blob/aws/packages/ecs/src/hooks/prepare-job.ts#L208-L213

With those options shared externals doesn't help, as things will be overwritten according settings.

One option for rsync: https://www.npmjs.com/package/rsync2

@Hi-Fi
Copy link
Owner Author

Hi-Fi commented Sep 22, 2024

Relates to actions/runner-container-hooks#168.

Solved by building checksum of externals, and comparing it before doing copy. Also moved externals to own volume so those are easier to share between container.

See: https://github.com/Hi-Fi/runner-container-hooks/blob/azure/packages/aca/src/hooks/prepare-job.ts#L150-L169

@Hi-Fi Hi-Fi closed this as completed Sep 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant