Resource load

[jbbjarnason]

@jamwaffles what is normal load when you are running ethercrab binary with like 10 subdevices?

I have been testing it and without running any business logic just polling process data every 1 ms it is roughly 25% on Intel Ultra 7 165U.

For reference, I have a c++ binary using the soem library with same set of subdevices runs at 4% load, with 1 ms polling on same PC.

Resources,
my ethercrab binary test: https://github.com/centroid-is/tfc-exes/blob/0115d61e286999f3afc2744be75d87ce7550d8c6/src/bin/ethercat/main.rs#L119
the c++ binary: https://github.com/centroid-is/framework/blame/main/exes/ethercat/inc/public/tfc/ec.hpp#L236

[jamwaffles]

I don't have any data recorded for CPU load, but anecdotally 25% is pretty high. Can I ask what OS your code is running on? I'd also suggest not using tokio and instead use something like smol - what I have measured (although a long time ago so maybe this info isn't that relevant any more) is that tokio has a lot of overhead which may be contributing to the higher CPU load.

If you must use tokio and are using Linux, the tokio-timerfd crate helped reduce jitter and may help with CPU load, I'm not sure.

[jbbjarnason]

hmm okay, I will take a look ate timerfd crate.

I am currently running on arch linux, I have also ran the code on embedded intel pc with nixos with celeron J6412 and was roughly 40%

[jamwaffles]

I have an old i3-7100T test system which has roughly the same performance as the J6412. I'll do some benchmarking with that when I have a moment and see if my results match yours.

Also, do you have interrupt coalescing disabled? You may also want to look at using tx_rx_task_io_uring and see if that reduces load.

If you like, I'd be happy to look at a Wireshark capture to check for any timing issues on the networking side.

[jbbjarnason]

tokio timerfd did not have any difference on resource load.

I am not sure on how to manage interrupt coalescing and whether to configure it for the nvme disk or the ethernet interface.

Trying the io_uring on the laptop dropped the load from 25% down to 20%.

I attached wireshark logs, wireshark-logs.zip

And tried smol with io uring and got it down to 17-19%

[jamwaffles]

I fired up a test system with Debian 12.8, i3-7100T (similar in performance to the CPU in your system), 4GB of RAM and kernel Linux 6.1.0-27-rt-amd64 #1 SMP PREEMPT_RT Debian 6.1.115-1 (2024-11-01) x86_64 GNU/Linux.

On this setup with no tuning, I saw around 35% CPU load in htop which is similar to your result.

Some things to try:

• Make sure you're using the PREEMPT_RT realtime kernel
• Use the latency-performance profile with tuned-adm profile latency-performance (apt install tuned-utils for me). This pushed my CPU from 1600MHz to 3400Mhz, YMMV.
• Disable interrupt coalescing on the NIC with sudo ethtool -C <nic name here> tx-usecs 0 rx-usecs 0
• Make sure you're running the Rust code with --release
• Use io-uring (as you're already doing)

I don't know the Arch Linux equivalents but hopefully it's not too hard to find them!

After these changes on my test system, I'm now down under 10% CPU utilisation when running the io-uring example (edited for a 1ms cycle time) in the repo. Do note that this does some extra stuff like setting realtime thread priorities, core pinning, and running two simultaneous tasks along with timerfd and smol.

Also, I had a look at tokio-ethercat.pcapng and saw round trip times of 200-300us. This should be roughly an order of magnitude lower, which disabling interrupt coalescing usually helps with a lot. This may help with CPU loads as well.

[jbbjarnason]

Great information thanks!