Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lot3 - Load model data from object storage #904

Closed
sambles opened this issue Sep 26, 2023 · 1 comment · Fixed by #958 · May be fixed by #909
Closed

Lot3 - Load model data from object storage #904

sambles opened this issue Sep 26, 2023 · 1 comment · Fixed by #958 · May be fixed by #909
Assignees
@sambles
Labels
Enhancement Small improvement or refinement. kubernetes production

Comments

@sambles
Copy link
Contributor

sambles commented Sep 26, 2023 •
edited
Loading

Issue Description

Add an optional init step to model workers, this pre-loads the model data into temp SSD of a worker from a set directory from a given object store like
S3 or Blob-Storage

See: POC from https://github.com/OasisLMF/OasisPlatform/tree/performace-fixes
@sambles sambles self-assigned this Sep 26, 2023
@sambles sambles added Enhancement Small improvement or refinement. kubernetes production labels Sep 26, 2023
@sambles sambles moved this to Todo in Oasis Dev Team Tasks Sep 26, 2023
@sambles sambles linked a pull request Oct 3, 2023 that will close this issue
Feature/object store worker download #908
Closed
@sambles sambles moved this from Todo to In Progress in Oasis Dev Team Tasks Oct 3, 2023
@sambles sambles removed a link to a pull request Oct 4, 2023
Feature/object store worker download #908
Closed
@sambles sambles linked a pull request Oct 4, 2023 that will close this issue
Worker Object Store download #909
Draft
This was referenced Oct 5, 2023
Closed
Closed
@sambles
Copy link
Contributor Author

sambles commented Oct 6, 2023 •
edited
Loading

also need to switch chunk storage to blob upload instead of Azure Files. These can get pretty big (PiWind 500k, locNumber summary level)

oasis-worker(2.2.1) worker-oasislmf-piwind-1-666b887db6-z52d7 /tmp/run/analysis-1_losses-7b62212293be41ebb5d72d0594bf9759/run-data/work $ 
du -hd1
4.0K	./kat
19G	./gul_S1_summaryaalcalc
19G	.

@sambles sambles moved this from In Progress to Todo in Oasis Dev Team Tasks Jan 11, 2024
@sambles sambles mentioned this issue Jan 16, 2024
Merged
15 tasks
@sambles sambles changed the title Add worker option to pre-load model data from object storage Load model data from object storage (Lot3) Mar 3, 2024
@sambles sambles moved this from Todo to In Progress in Oasis Dev Team Tasks Mar 3, 2024
@sambles sambles changed the title Load model data from object storage (Lot3) Lot3 - Load model data from object storage May 2, 2024
@sambles sambles linked a pull request May 2, 2024 that will close this issue
Lot3 work #958
Merged
@sambles sambles closed this as completed May 2, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in Oasis Dev Team Tasks May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sambles sambles

Labels
Enhancement Small improvement or refinement. kubernetes production
Projects
Oasis Dev Team Tasks
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Lot3 work OasisLMF/OasisPlatform
Worker Object Store download OasisLMF/OasisPlatform
1 participant
@sambles