Subdevice recovery

[jbbjarnason]

So I would like to implement recovery mechanism running the ethercat daemon.

The recovery mechanism would check the state of device or just the working counter of the group.

Example (non compilable)

let expected_working_counter = 21;
loop {
  let wc = group.tx_rx();
  if wc != expected_working_counter {
    for subdevice in group.iter() {
      if subdevice.status().0 != Operational { 
        subdevice.into_safe_op();
        subdevice.into_op();
      }
    }
  // ... normal run code
}

My intention is to disturb the network as little as possible and try to recover from any power outage or other problems in a smart way.

Is this possible with current api without reinitializing the whole group?

I have gone through the subdevice code and I can't seem to find if this is possible,

[jbbjarnason]

I think this would be possible if the function request_subdevice_state_nowait where public.

[jbbjarnason]

I have tried using this function and the state function, here are logs

            if wc != self.expected_working_counter {
                info!(target: &self.log_key, "Working counter mismatch, expected: {}, got: {}", self.expected_working_counter, wc);
                for subdevice in group.iter(&self.main_device) {
                    let instant = Instant::now();
                    let state = subdevice.state().await.map_err(|e| {
                        warn!(target: &self.log_key, "Failed to get subdevice state for {:#06x}: {}", subdevice.configured_address(), e);
                        e
                    });
                    if matches!(state, Err(ethercrab::error::Error::WorkingCounter { .. })) {
                        let _ = subdevice
                            .request_subdevice_state_nowait(ethercrab::SubDeviceState::Init)
                            .await.map_err(|e| {
                                warn!(target: &self.log_key, "Failed to transition subdevice {:#06x} to init: {}", subdevice.configured_address(), e);
                                e
                            });
                    }
                    let elapsed = instant.elapsed();
                    info!(target: &self.log_key, "Subdevice {:#06x} state: {:?}, elapsed: {:?}", subdevice.configured_address(), state, elapsed);
                    // subdevice
                    //     .request_subdevice_state_nowait(ethercrab::SubDeviceState::SafeOp)
                    //     .await?;
                }
            }

[2024-11-28T10:13:23Z WARN  ethercat] Failed to get subdevice state for 0x1008: working counter expected 1, got 0
[2024-11-28T10:13:23Z DEBUG ethercrab::subdevice] Set state Init for SubDevice address 0x1008
[2024-11-28T10:13:23Z WARN  ethercat] Failed to transition subdevice 0x1008 to init: working counter expected 1, got 0
[2024-11-28T10:13:23Z INFO  ethercat] Subdevice 0x1008 state: Err(WorkingCounter { expected: 1, received: 0 }), elapsed: 447.179µs

I am reproducing this by taking out breaker for Ek1100 unit which is slave number 2.

Not able to recover from this currently by using those methods.

[jbbjarnason]

For reference, here is how we implemented the recovery mechanism with soem,

  /**
   * Check the state of attached slaves.
   * If the slaves are no longer in operational mode. Attempt to
   * switch them to operational mode. And if the slaves have
   * been lost attempt to add them again.
   * @param group_index 0 for all groups
   */
  auto check_state(uint8_t group_index = 0) -> void {
    auto& grp = group_list_as_span()[group_index];
    while (running_) {
      if (grp.docheckstate == 1 || wkc_ < expected_wkc_) {
        grp.docheckstate = FALSE;
        ecx_readstate(&context_);
        auto slaves = slave_list_as_span();
        uint16_t slave_index = 1;
        for (ec_slave& slave : slaves) {
          if (slave.state != EC_STATE_OPERATIONAL) {
            grp.docheckstate = TRUE;
            if (slave.state == EC_STATE_SAFE_OP + EC_STATE_ERROR) {
              logger_.warn("Slave {}, {} is in SAFE_OP+ERROR, attempting ACK", slave_index, slave.name);
              slave.state = EC_STATE_SAFE_OP + EC_STATE_ACK;
              ecx_writestate(&context_, slave_index);
            } else if (slave.state == EC_STATE_SAFE_OP) {
              logger_.warn("Slave {}, {} is in SAFE_OP, change to OPERATIONAL", slave_index, slave.name);
              slave.state = EC_STATE_OPERATIONAL;
              ecx_writestate(&context_, slave_index);
            } else if (slave.state > EC_STATE_NONE) {
              if (ecx_reconfig_slave(&context_, slave_index, EC_TIMEOUTRET) != 0) {
                slave.islost = FALSE;
                logger_.warn("Slave {}, {} reconfigured", slave_index, slave.name);
              }
            } else if (slave.islost == 0) {
              ecx_statecheck(&context_, slave_index, EC_STATE_OPERATIONAL, EC_TIMEOUTRET);
              if (slave.state == EC_STATE_NONE) {
                slave.islost = TRUE;
                logger_.warn("Slave {}, {} lost", slave_index, slave.name);
              }
            }
          }
          if (slave.islost == 1) {
            if (slave.state != EC_STATE_NONE) {
              slave.islost = FALSE;
              logger_.info("Slave {}, {} found", slave_index, slave.name);
            } else if (ecx_recover_slave(&context_, slave_index, EC_TIMEOUTRET) != 0) {
              slave.islost = FALSE;
              logger_.info("Slave {}, {} recovered", slave_index, slave.name);
            }
          }
          slave_index++;
        }
        if (context_.grouplist->docheckstate == 0) {
          logger_.info("All slaves resumed OPERATIONAL");
        }
      }
      std::this_thread::sleep_for(milliseconds(10));
    }
  }

this ran in separate thread

I am trying to implement the same behavior with ethercrab.

[jbbjarnason]

My workaround currently is to bring down the whole group when working counter is not as expected, and try to initialize again until expected count of subdevices are discovered.