Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fault tolerance] Automatic resubmission #346

Merged
merged 45 commits into from
Jul 5, 2024
Merged

[Fault tolerance] Automatic resubmission #346

merged 45 commits into from
Jul 5, 2024

Conversation

maanug-nv
Copy link
Collaborator

Adds features:

  • automatic resubmission of Slurm jobs
  • launching with custom ft_launcher

jbieniusiewi and others added 30 commits November 2, 2023 07:55
@jbieniusiewi
Added fault tolerance config for gpt3 126m
28f3e43
@jbieniusiewi
inital auto-resume impl-WIP
7a955dd
@jbieniusiewi
Auto-resume loop-WIP
0b84305
@jbieniusiewi
Cleaning dbg code
733ac4c
Fixes after testing on EOS
adc5b7b
Updated fault tolerance-end of day version
cc4f54b
Read fault tolerance config from exp_manager section
2b0d228
@jbieniusiewi
Added autoresume after preemption
63dc927
@jbieniusiewi
Updated auto-resume params reading
2334995
@jbieniusiewi
Added FT to main branch
0aea955
Added SC2 config and run script
e50d388
Fixed launch cmd for Sc2
ca7ab16
Updated launcher cmd
74b2e42
increased timeouts and removed NVTE_APPLY_QK_LAYER_SCALING
b8d8a58
@jbieniusiewi
Added back NVTE_APPLY_QK_LAYER_SCALING=1 as error happens without it
0d9065e
@jbieniusiewi
Added fault tolerance unit tests
c6367c5
@jbieniusiewi
Use FT launcher, WIP
f1dfe54
@jbieniusiewi
Fix...
87e3386
@jbieniusiewi
Fix2
90fc006
@jbieniusiewi
Updating for FT launcher, wip...
89362b3
@jbieniusiewi
Updated FT params reading
b4ee9ed
@jbieniusiewi
Fixes after testing on DlCluster...
050ccf2
@jbieniusiewi
Update after 'create_fault_tolerance_callback' param was added
147024b
@jbieniusiewi
Improved auto-resume
5105a29
@jbieniusiewi
Improved auto-resume-cont
58741ae
@jbieniusiewi
Use hostname to get the rendezvous host
4442118
@jbieniusiewi
Added additional_ft_launcher_args
25cc521
@jbieniusiewi
Restored --kill-on-bad-exit=0, --wait=3600
874b651
@jbieniusiewi
Fixed comment
8316411
@jbieniusiewi
--kill-on-bad-exit=1
fb80ec0
@maanug-nv maanug-nv force-pushed the fault-tolerance-elastic branch from 93318d9 to 9a9accf Compare June 4, 2024 20:44
ericharper
ericharper approved these changes Jul 5, 2024
View reviewed changes
Copy link
Collaborator

@ericharper ericharper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@ericharper ericharper merged commit b6ad2a1 into main Jul 5, 2024
3 checks passed
@ericharper ericharper deleted the fault-tolerance-elastic branch July 5, 2024 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericharper ericharper ericharper approved these changes

@dimapihtar dimapihtar Awaiting requested review from dimapihtar

@yaoyu-33 yaoyu-33 Awaiting requested review from yaoyu-33

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@maanug-nv @ericharper @jbieniusiewi