add sharded data parallel all gather time estimation

[cli99]

When using fsdp shard_grad_op and full_shard, model weights are unsharded through allgather before being used for execution. This PR adds the time estimation of the all gather. If the all gather time > layer compute time, all gather time cannot be fully overlapped with compute. The time estimation relies on intra node and inter node memory bandwidth efficiency, which depends on the size of the data to transfer. 0.8 is a good estimation for data size > 128MB. for small data size, set the efficiency to a lower number.

[codecov-commenter]

Codecov Report

Attention: 6 lines in your changes are missing coverage. Please review.

Comparison is base (21d9fa3) 80.33% compared to head (60fc071) 80.64%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
llm_analysis/analysis.py	88.23%	6 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #15      +/-   ##
==========================================
+ Coverage   80.33%   80.64%   +0.30%     
==========================================
  Files           8        8              
  Lines        1083     1121      +38     
==========================================
+ Hits          870      904      +34     
- Misses        213      217       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.