fix(deposits): Reduce deposit fetch retry timout

[shotes]

If a deposit is missed for whatever reason, the node needs to attempt to recover them as quickly as possible. IMO, the 20 second timeout for retrying deposit fetch is too long. This is reduced to check every 2 seconds. This could be made even shorter if needed.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 27.08%. Comparing base (419cd66) to head (576c695).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2406   +/-   ##
=======================================
  Coverage   27.08%   27.08%           
=======================================
  Files         351      351           
  Lines       15543    15543           
  Branches       20       20           
=======================================
  Hits         4210     4210           
  Misses      11130    11130           
  Partials      203      203           

Files with missing lines	Coverage Δ
beacon/blockchain/deposit.go	0.00% <ø> (ø)

[nidhi-singh02]

lgtm!

[abi87]

@shotes q: should we include some metrics so understand how long this can take?

[shotes]

This happens on a separate independent goroutine in depositCatchupFetcher(). That goroutine will just loop over and over every defaultRetryInterval seconds and check for failed blocks. If there are any, then it will attempt to fetch them.

What metric should we measure that might impact this value?

[abi87]

I am just wondering what happen if the time it takes to the EL to return deposits grows to be comparable with the new defaultRetryInterval.
Not sure if this is realistic but in that case wouldn't we clog the EL with continuous retries?
I was thinking a metric around the time it takes to fetch the deposits could help understanding this?

[shotes]

Ah I see. That's a good point to bring up, my gut feeling is that this should not be a problem.

1. We are only sending retries if we are unable to fetch it the first time.
2. For each request, the EL client fetches the corresponding payload for the blocknum, filters through the events that are emitted during that payload, and then returns them.
3. The fetcher isn't sending async requests every defaultRetryInterval seconds. It is synchronously requesting to fetch the deposits, so there should never be more than 1 request at a time.

Under expected circumstances, that should take very little amount of time. Even now, we expect it to take a small enough time that we can run it at the very end of FinalizeBlock.

Will have to investigate more how this could impact the EL client while it is under heavy load, but I don't think the impact is high.