Add next-hop retry for direct-path relays

[benallfree]

This is a draft for visibility. Note: this intro section will stay updated, so it may not match comments below

This is a wire-compatible PR for both passive and active direct hop ACKs.

Right now, repeaters relay-and-forget. Any missed link in the path results in a timeout and the user has to retry the entire path.

With this enhancement, repeaters will listen for an echo from the next hop. If they don't hear it, they will resend. If the receiving repeater hears a resend after it has already processed a message, it will send a hard ACK to prevent further spurious repeats.

We also add several CLI settings, including a volatile ignore setting to force link repeats for testing:

Command	Type	Parameter	Range	Default	Persisted	Purpose
get hop.retry	read	—	—	2	yes	Extra retransmits if next hop echo or HOP_ACK is missed (0 = off)
set hop.retry <count>	write	count	0–5	2	yes	Up to count + 1 total TX (initial + retries)
get hop.retry.ms	read	—	—	1500	yes	Base listen window (ms) before each retry
set hop.retry.ms <ms>	write	milliseconds	200–10000	1500	yes	Base window; actual deadline also includes direct.txdelay and packet airtime
get hop.ignore	read	—	—	0	no	Remaining forward drops (test mode)
set hop.ignore <count>	write	count	0–255	0	no	Silently drop next N direct-forward opportunities (no forward, no HOP_ACK, no dedup mark); resets on reboot

Prior art:

• For packets with a path set; auto try again if no echo was heard. High Availability Link for Overwhelmed Meshes - Project HALO. Stop the flood #2367
• Keymind - Keep Ensuring Your Message Is Network Delivered #2501
• Feature: Repeated sending for DIRECT-route packets with echo-cancellation (improves delivery, reduces floods) #2670

[usrflo]

Hi @benallfree ,

> From what I can tell, these listen for the message echo.

that's right, at least #2670 and #2367 do.

> That was my first instinct too, but after more thinking I feel this may actually make the problem worse on the links that need it most. It is good because it costs zero extra airtime, but it is risky on weak/problematic/busy links because the airtime needed to hear the echo dramatically increases the chance of collisions and errors.

The approach of PR2670 is to listen as long as possible and keep deafness low (prevent TX, prevent calibration that blocks RX) on the sending node. If a node is stressed with TX it may not hear a downstream package repetition so it will resend its package which is unnecessary. So yes, in busy environments the error correction remains but the package frequency becomes higher. I tried to reduce the probability of collisions by checking for noise before repeating a package but yes, the more traffic the worse the quality gets. But still better for deliverability than with common MeshCore without the duplication.

> In contrast, a hard ack is about 12 bytes regardless of the original message size. So for 12 extra bytes of airtime, you get the most reliable ack's possible.

That means to at least double the amount of TX for a direct message plus possible repetitions if the ACK was not received by the sender. Plus special collision problems when the ACK packages cannot be heard and the sender repeats its package.
In contrast to that I think "less is more": within PR2670 the regular package sent by a downstream repeater is considered as the ACK you're focusing on; in my view a duplicate confirmation does not change the situation but increases TX. Do I miss something?

Side note: package resending like with PR2670 is downwards compatible; so it works without updating all other nodes.
According to your explanation IMHO a repeater with your draft reacts on "no ACK received?" with a re-sending: this would multiply all direct packages as long as the surrounding repeaters are not updated, isn't it?

[benallfree]

Hi @usrflo :)

The approach of PR2670 is to listen as long as possible and keep deafness low (prevent TX, prevent calibration that blocks RX) on the sending node.

I like this! Very smart 🔥

That means to at least double the amount of TX for a direct message plus possible repetitions if the ACK was not received by the sender. Plus special collision problems when the ACK packages cannot be heard and the sender repeats its package.

Maybe we can both think about this, I'd like to hear your opinion.

You have probably noticed like I have that smaller packets are delivered far more reliably than larger packets. Hard ACK takes advantage of this idea and is 14 bytes:

10:57:30 - 15/5/2024 U: TX, len=14 (type=11, route=D, payload_len=12)

If the sender can't hear a 14 byte ACK, it seems even less likely that it would hear a >14 byte echo of the original message. A separate 14 byte ACK stays small even with large 200-byte messages. For 14 extra bytes of airtime, we buy the most reliable ACK possible.

At first, it does seem like increased TX compounds the problem. However, I think spending some airtime on the most reliable ACK possible will ultimately lead to the fewest resends between hops and also the fewest full path resends from the user (the worst problem), which far outweigh 15 byte ACKs.

I imagine we both agree that best ACK = fewest resends. So the question is passive vs active ACK. Maybe it could be a setting, like hop.ack.mode: passive|active.

this would multiply all direct packages as long as the surrounding repeaters are not updated, isn't it?

Sorry I didn't mention it, but this PR will not attempt resends until the first ACK is heard from the receiver. Graceful upgrade. I added the note.

Thanks for the interesting conversation. Any way we improve or solve this is a win for mesh 🙏

[benallfree]

@usrflo I have thought about your points more and made an update.

Now we start with passive echo listening. If A does not hear an echo, it will resend. If B detects a resend, it knows A did not hear the original echo and will send a hard ACK back to hopefully prevent A from spamming more retries when B has already relayed the message.

So now hard ack is reserved for only situations where passive echo fails.

Do you like it?