chore(bench): improve heuristic to run throughput benchmarks

[soonum]

This changes the way we define the number of elements to load in the throughput pipeline.
Light operations needs more elements to saturate a backend. Conversely a heavy operations will require less elements to be able to run in a decent time.
That improvement will dramatically decrease benchmark total duration.

This has been tested with the following operations:

• bitand (light operation)
• add (mid operation)
• div_rem (heavy operation):

Saturation has been tested and successful on the following backends:

• CPU
• GPU

---

This change is 

[soonum]

I need a review of the design before exporting it to all other benchmark functions handling throughput variants.

[IceTDrinker]

what's the idea of the heuristic ?

Load depending on how many PBS are in an operation ?

[soonum]

what's the idea of the heuristic ?

Load depending on how many PBS are in an operation ?

Yes this is the idea. The load is computed based on the number of threads available divided by the number of PBS needed for one operation.
This value is then used a coefficient within the previous implementation. Typically the coefficient has a value greater than 1.0 for quick operations, if the machine is big enough, and a value below 1.0 for slow operations.
That way the number of elements to pass during the throughput benchmark is dynamically generated.

[IceTDrinker]

Yes this is the idea. The load is computed based on the number of threads available divided by the number of PBS needed for one operation. This value is then used a coefficient within the previous implementation. Typically the coefficient has a value greater than 1.0 for quick operations, if the machine is big enough, and a value below 1.0 for slow operations. That way the number of elements to pass during the throughput benchmark is dynamically generated.

Did you check the measured throughput with the new loading factor is similar to the old one ?

[soonum]

Yes I've just checked and they are the same 🎉

[soonum]

get_pbs_count() do not increment the atomic counter on GPU.
Thus I cannot test this new implementation on GPU for now.

[IceTDrinker]

get_pbs_count() do not increment the atomic counter on GPU. Thus I cannot test this new implementation on GPU for now.

use the CPU stats, they should be similar I would think

[soonum]

The PBS count for GPU is fixed. Ready for review now.

[agnesLeroy]

Hey! Thanks a lot @soonum! I only have some minor fixes 🙂 do you know how long the throughput benches take now?

[soonum]

Hey! Thanks a lot @soonum! I only have some minor fixes 🙂 do you know how long the throughput benches take now?

Yes, for Cuda benchmarks, we're now down to 46 mins for de-duplicated operations in 64 bits.
I'll launch a full precision on default ops today.

[IceTDrinker]

I saw I was still supposed to review some thing here ?

Btw @soonum just thought about something but enabling the pbs stats can have an impact on CPU performance as we update a single counter via many threads, so I guess there is a need (only for CPU) to do a first pass to measure PBSes for all ops and precision and relaunch throughput loading those data from file to avoid it having an adverse effect on precision I guess

we must not launch latency pbs with the pbs-stats feature

[soonum]

Yes you're right 100%. I was bothered about using the pbs-stats feature in the benchmarks.
So, how do you see this ? Performing a measurement dynamically and record a temporary files on disk and then trigger the actual benchmarks in another workflow step ?

[IceTDrinker]

Yes you're right 100%. I was bothered about using the pbs-stats feature in the benchmarks. So, how do you see this ? Performing a measurement dynamically and record a temporary files on disk and then trigger the actual benchmarks in another workflow step ?

not a different workflow step, can be the same step reading the file from the previous command

something like:

run:|
make measure_pbs_stats
make bench_integer

[soonum]

Just made some benchmarks. Turned out that pbs-stat feature has no significant impact on performance, either for latency or throughput.
I guess we can let the implementation as is.

[IceTDrinker]

A question, trying reviewable at the same time so some stuff may not work exactly as I expect

Reviewed 4 of 7 files at r1, all commit messages.
Reviewable status: 4 of 7 files reviewed, 11 unresolved discussions (waiting on @agnesLeroy and @soonum)

---

tfhe/benches/utilities.rs line 401 at r1 (raw file):

        // Some operations with a high count of PBS (e.g. division) would yield an operation
        // loading value so low that the number of elements in the end wouldn't be meaningful.
        let minimum_loading = if num_block < 64 { 0.2 } else { 0.1 };

This is a heuristic right? could it explain the lower throughput of large precisions?