Pre-Processing: Disk usage and cores utilised at step 1 and 2

[lachlanem]

Apologies if these questions have been discussed elsewhere, I was unable to find answers between the issues & discussions pages.

1. Pre-processing requires 10x the disk space of raw_data, this seems quite substantial for disk usage? Could I unpack .npz plan files ,,online" temporarily rather than saving to disk?

Some of the Stage1 data plans are up to 1.3GB / case, making for large data storage requirements with large datasets.

2. The first step of pre-processing (checking case XXXXX) only utilizes a single core, and the second step only utilizes 8/16 cores?

I think the second step limitation is due to my ubuntu version [Ubuntu 22.04 LTS] not supporting alder lake efficiency cores -- apparently LTS Ubuntu uses the Linux kernel 5.15 by default which is unable to utilize the efficiency cores.

[FabianIsensee]

Hi,
you can train with --use_compressed_data. This will increase CPU usage by a lot but save disk space. Unfortunately there is no in-between. It's either or :-)
You seem to have found the answer for the other question already!? I have no experience with Alder Lake.
Best,
Fabian