Time measurements for data augmentation steps

[dottipr]

Hello,

I would like to train the nnUNet on my dataset (xyt greyscale data series, just 1 modality) and comparing the runtime of my epochs with the benchmarks values I noticed that I have a bottleneck somewhere.

When training with the dummy loader the runtime values I get are in line with the ones given in the documentation, but with the other models I have drops in the GPU usage.

I've increased nnUNet_n_proc_DA to 16 but the CPU memory still fills up.

Therefore, I would like to print some time measurements for each data augmentation step, to see what is slowing down the training (I've been told that random rotations could be the cause).

Unfortunately, I don't have much experience with coding and I couldn't figure out how to do it (if I am not mistaken, the composition of the DA transformations is passed as argument to NonDetMultiThreadedAugmenter/MultiThreadedAugmenter, so I can't simply measure the runtime of each transformation).

Here is some additional information:

• the dataset contains 43 samples with 500-1900 slices, frame size is 64x512
• GPU is an NVIDIA GeForce RTX 3090
• CPU is i9-11900K with 8 cores

I am happy to provide more information if needed.

Any help would be truly appreciated!

[FabianIsensee]

Hi,
when you say the CPU memory fills up you mean the RAM? If so, then decreasing nnUNet_n_proc_DA might solve this (RAM usage is somewhat proportional to the number of processes you use for data augmentation).
Best,
Fabian