Augmentation SpeedUp #147
Augmentation SpeedUp #147
Conversation
…into fix/augmentation_speedup
This reverts commit 0d6d962.
|
I also conducted experiments to compare the performance of YOLOv6/YOLOv8 nano models with ours under the same settings and observed slightly longer timings. It took 18 minutes for our model and 21 minutes for the YOLOv6 and YOLOv8 models. Additionally, note that previously, an epoch without the polars library, load speedup, and augmentation fixes lasted ~5 hours on GCP. Reducing the time to ~15 minutes represents a very significant speedup. |
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified Files
|
| "keypoints_visibility_batch": keypoints_visibility_batch, | ||
| "keypoints_classes_batch": keypoints_classes_batch, | ||
| } | ||
| if return_mask: |
There was a problem hiding this comment.
Do we see any concerns with mask_batch missing downstream? Would it be safer to keep it in dict but have it as an empty value?
There was a problem hiding this comment.
No, the augmentations will still work if mask_batch is simply missing. I think it does not pose any threat.
Additionally, if we simply enter an empty value for masks instead of adding it when necessary, it fails.
It cannot be empty; it would have to contain zeros, but this is slow because augmentations are then applied to that dummy mask. This is what we were doing before, and it was slowing the code down
Moreover, using dummy keypoints (just one keypoint at x:0, y:0) or bounding boxes does not affect the time it takes to apply the augmentations.
Test Results 1 files 1 suites 19m 14s ⏱️ Results for commit 8124c29. |
Co-authored-by: Jernej Sabadin <[email protected]> Co-authored-by: Martin Kozlovsky <[email protected]>
This reverts commit d996b67.
This PR includes:
Performance Comparison
Measured on Quadro RTX4000 (8 GB GPU memory), CUDA 11.7, 8 CPU, 32GB RAM.
Params:
dataset: COCO train split
batch_size: 16
num_workers: 4 (available with Remove num_workers Hotfix)
augs: normalization + mosaic
model: yolov6n
img_size: 416x416