Skip to content

Commit

Permalink
add parallel note
Browse files Browse the repository at this point in the history
  • Loading branch information
CodyCBakerPhD committed Aug 26, 2024
1 parent 46ea0c8 commit e003cbd
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ map_binned_s3_logs_to_dandisets \
--excluded_dandisets 000108
```

In the summer of 2024, this blobs process took less than 6 hours to complete (with caches; 8 hours without caches) with one worker.
In the summer of 2024, this blobs process took less than 8 hours to complete (with caches; 10 hours without caches) with one worker.

Some Dandisets may take disproportionately longer than others to process. For this reason, the command also accepts `--excluded_dandisets` and `--restrict_to_dandisets`.

Expand All @@ -167,6 +167,8 @@ map_binned_s3_logs_to_dandisets \

In the summer of 2024, this took ?? hours to complete.

The mapping process can theoretically be designed to work in parallel (and thus much faster), but this would take some effort to design. If interested, please open an issue to request this feature.



## Submit line decoding errors
Expand Down

0 comments on commit e003cbd

Please sign in to comment.