Subject: Re: Protocol for TCP heartbeats?

From: Martin Sustrik <[email-masked]>
Date: Wed, 14 Jul 2010 19:42:40 +0200

Ted,

>> The obvious problem is that heartbeats can thus sit in transmit buffer 
>> waiting to be delivered. They can even be retransmitted. Etc. In any 
>> case the functionality they are supposed to provide is pretty heavily 
>> distorted.
> 
> FWIW, I don't think it matters if the keepalives are stuck in a TCP
> transmit buffer or in a multi-continent routing loop.  If the
> application needs to hear from its peer every N seconds and it doesn't,
> they're disconnected.

Yes. That's true for dumb keepalive algorithm as described in the 
previous email.

However, if you want something more sensible (presumably something like 
SCTP's heartbeats) you need to take current RTO into account. That's 
something you can't do on top of TCP.

> If TCP can't meet an application's requirements, the developers need to use
> something else.

Exactly. Moving to SCTP would solve the problem. However, what you see 
in the wild is thousands of applications implementing dumb (and pretty 
faulty) heartbeat strategies on top of TCP rather than moving to SCTP.

That's why I thought there may have been an attempt to standardise the 
heartbeat algorithm in the past.

Martin

---

From: Martin Sustrik <[email-masked]>
Date: Wed, 14 Jul 2010 21:23:15 +0200

Ted,

> You're missing my point, perhaps because I'm being unclear.  Let me try
> it this way:
> 
> If an application needs a heartbeat, it almost always needs to be an
> application to application (layer 7 to layer 7) heartbeat.
> 
> Imagine you have a perfect TCP heartbeat algorithm: detects the
> existence of a running TCP instance on both endendpoints perfectly
> accurately at the timescales you care about.  Now one of your
> application endpoints deadlocks itself - it hangs, spins, whatever - but
> the process is alive and the TCP connections are open.  The application
> is not responding.  The TCP timeout won't help you at all; the TCP
> connection is fine.
> 
> Of course the way to deal with that is a layer 7 heartbeat.
> 
> My point is that if you need that layer 7 heartbeat, the layer 4 (TCP)
> one doesn't help much.  I can't think of an application that needs the
> TCP heartbeat and not the application heartbeat.  (There probably is
> one; my point is that needing both is the common case.)

Right. Layer 7 heartbeats are definitely needed to detect the whether 
application is hung up.

However, detecting application hangup is a problem orthogonal to 
detecting the unavailability of network peer. Being able to detect 
network unavialability is valuable in itself (i.e. application can start 
failover procedure in a timely fashion).

Those that need hangup detection can obviously implement heartbeats on 
layer 7, but that's beside the point here.

> So, TCP designers could create a highly parameterized heartbeat timer
> (every application has its own idea what a timeout is) and put all that
> complexity into the TCP protocol.

No complexity is needed IMO. Consider the following:

1. The keepalives are already defined in rfc 1122 (4.2.3.6)

2. There are no interoperability issues. With SCTP-like heartbeat 
mechanism each peer manages its failure detection mechanism itself and 
no extra effort on behalf of the other side is needed. Thus 
implementations with failure detection would work perfectly well with 
implementations with no failure detection.

3. There are no congestion control issues. The keepalives are data and 
thus they should adhere to TCP congestion control mechanism. When the 
peer is unreachable keepalives would back off in a decent manner.

AFAICS the only thing preventing specification of optional TCP heartbeat 
mechanism are the artificial restrictions in rfc 1122 4.2.3.6, such as 
"no less than two hours" rule.

It's interesting to look at the rationale in rfc 1122:

"The TCP specification does not include a keep-alive mechanism because 
it could:  (1) cause perfectly good connections to break during 
transient Internet failures; (2) consume unnecessary bandwidth ("if no 
one is using the connection, who cares if it is still good?"); and (3) 
cost money for an Internet path that charges for packets."

(1) Is exactly what you require for high-availability solutions.
(2) High-availability solution does care.
(3) True, but those that need it are happy to pay the extra cost.

Martin

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 16 Jul 2010 07:36:40 +0200

Ted Faber writes:
> ...
> TCP heartbeat and not the application heartbeat.

I can think of several whose L7 heartbeat needs TCP data in order to 
avoid false alarms.

It's really difficult to write an L7 heartbeat which works well with 
fast connections (ie. detects death soon after it occurs), also works 
with slow connections (ie. makes few false alarms), and makes no use of 
TCP data.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 16 Jul 2010 09:38:14 +0200

Put differently: The proper job of TCP heartbeats isn't to proclaim a 
connction dead, it is to proclaim it alive and working, so that L7 
heartbeats don't have to be so upgefucked.

IMO, a TCP keepalive API needs contain only three functions. Two to 
answer the questions "are any acks overdue, and if so, by how long?" and 
"what are the current RTT and bandwidth estimates?" and one to provoke 
the peer into sending an ack.

Arnt

---

From: Martin Sustrik <[email-masked]>
Date: Mon, 19 Jul 2010 08:48:58 +0200

> with slow connections (ie. makes few false alarms), and makes no use of 

It's even more complex IMO. The L7 heartbeat is to detect application 
hang-ups, right?

It's either implemented is a dedicated heartbeating thread in witch case 
it says nothing about whether the real worker thread is hung up.

Or there's some kind of general-purpose hang-up diagnostic algorithm in 
your application. Which is more or less impossible to implement (a 
problem unsolvable by Turning machine).

Thus the L4 heartbeats (detecting no response from the peer's TCP stack) 
are the only ones we are left with.

Shrug.

Martin

---

From: Martin Sustrik <[email-masked]>
Date: Mon, 19 Jul 2010 08:50:44 +0200

Ted Faber wrote:

> Perhaps.  I find the end-to-end violation more compelling.  YMMV.
> 
> Good luck in tcpm.

Thanks for the interesting disucssion!

Martin

---

From: Martin Sustrik <[email-masked]>
Date: Mon, 19 Jul 2010 09:00:27 +0200

> the peer into sending an ack.

I would say that RFC1122 already defines keepalives (unwillingly, but it 
does). Thus what's needed is an RFC saying "if you want to implement 
keepalives in a sane manner, do it like this".

Martin