Subject: Protocol for TCP heartbeats?

From: Martin Sustrik <[email-masked]>
Date: Fri, 25 Jun 2010 19:46:45 +0200

Hi all,

I haven't been able to find it but maybe someone knows here: Have there 
been a protocol defined for checking whether TCP peer is alive or not? 
(I mean one that plays well with networks with various latencies and 
throughputs and won't congest the network even if used on a wide scale.)

Thanks in advance!
Martin

---

From: "Rémi Denis-Courmont" <[email-masked]>
Date: Fri, 25 Jun 2010 21:03:31 +0300

> throughputs and won't congest the network even if used on a wide scale.)

On most OSes, you can enable TCP keep-alives. Then your TCP socket will return 
a time out error if the other end "dies". So yes, there is a way to do this at 
the TCP protocol level.

Unfortunately, there is no standard API to use and configure this feature of 
TCP. On Linux, you can adjust all parameter on a per-socket basis (refer to
'man 7 tcp' and look for TCP_KEEP for details) though.

There is also no programmatic way to know that the other peer is using keep-
alives or not (should you need to know that anyway).

-- 
Rémi Denis-Courmont
Nokia Devices R&D, Maemo Software, Helsinki

---

From: chadwick allison <[email-masked]>
Date: Fri, 25 Jun 2010 18:08:24 +0000

You could do packet captures with wireshark.  I would off load it if you are trying to research on a production machine though.  I think that you can do a defined capture just for TCP keepalives.  Cisco is a span-port command if you want to see from source to destination and reverse.

Just a thought

Chadwick
 
> Date: Fri, 25 Jun 2010 19:46:45 +0200
> From: [email-masked]
> To: [email-masked]
> Subject: Protocol for TCP heartbeats?
> 
> Hi all,
> 
> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf

---

From: Martin Sustrik <[email-masked]>
Date: Fri, 25 Jun 2010 20:10:45 +0200

> alives or not (should you need to know that anyway).

This is a dumb keepalive option (send heartbeat each N seconds, N being 
defined by user), right?

What I had in mind whether there ever been an attempt to define dynamic 
keepalive algorithm that adjusts keepalive intervals according to the 
observed throughput and roundtrip latency figures (dynamic in the same 
way as CC dynamically adjusts throughput).

Any ideas?
Martin

---

From: Bob Braden <[email-masked]>
Date: Fri, 25 Jun 2010 11:13:46 -0700

I trust you are familiar with section 4.2.3.6 of RFC 1122.

Bob Braden

---

From: "Rémi Denis-Courmont" <[email-masked]>
Date: Fri, 25 Jun 2010 21:24:18 +0300

On Friday 25 June 2010 21:10:45 ext Martin Sustrik, you wrote:
> > On most OSes, you can enable TCP keep-alives. Then your TCP socket will
> > return a time out error if the other end "dies". So yes, there is a way
> > to do this at the TCP protocol level.
> > 
> > Unfortunately, there is no standard API to use and configure this feature
> > of TCP. On Linux, you can adjust all parameter on a per-socket basis
> > (refer to 'man 7 tcp' and look for TCP_KEEP for details) though.
> > 
> > There is also no programmatic way to know that the other peer is using
> > keep- alives or not (should you need to know that anyway).
> 

You can probably change the values. But I don't really see the statistical 
correlation of throughput and round trip time to the probability that the peer 
will fail within a certain time interval.

I mean, you can probably establish a weak correlation between them, or at 
least between the variation of the bandwidth and RTT to the probability of 
failure. But I somehow doubt it will be sufficiently strong a correlation that 
that a very clever algorithm would be significantly better than a plain dumb 
ping-pong at constant or exponential backoff interval.

---

From: Martin Sustrik <[email-masked]>
Date: Fri, 25 Jun 2010 20:32:15 +0200

> I trust you are familiar with section 4.2.3.6 of RFC 1122.

Yes. I am aware of it.

I was just interested in whether the behaviour have been defined for 
those who need early failure detection (systems with failover 
capabilities) and are willing to pay for the additional bandwidth used 
(financial sector).

The wide range of proprietary implementations of the mechanism suggest 
that it wasn't but I wanted to make sure about it.

Martin

---

From: Martin Sustrik <[email-masked]>
Date: Fri, 25 Jun 2010 20:37:19 +0200

> ping-pong at constant or exponential backoff interval.

What I meant was that long timespan between consecutive packets can mean 
either:

1. Sender is sending data sparsely.
2. There's congestion on the network and packet loss causes decrease in 
throughput.

In both cases you want to increase the heartbeat interval as none of 
them means the peer is dead.

Dead peer or unreachable peer is rather identified by sharp decrease in 
throughput -- actually no packets from the peer for unexpectedly long time.

Martin

---

From: Martin Sustrik <[email-masked]>
Date: Fri, 25 Jun 2010 23:12:56 +0200

Hi Ted,

> The mechanism's not in TCP because the definition of a failed peer is
> application dependent.  A chemical plant automation application probably
> has different requirements about when to treat a peer as failed than an
> application that updates cute cat pictures, or one that keeps internet
> routing tables up to date.  The definition of peer failure is
> independent of whether the peers communicate over TCP, DCCP, SCTP or
> carrier pidgeon, so at some point the application's going to need to be
> responsible for detecting that failure.  Rather than provide a generic
> mechanism that would partially solve many instances of the problem, the
> TCP designers got TCP out of the way and let each application designer
> do it themself.

Thanks for detailed reply!

It's now clear why the raw TCP doesn't define heartbeats.

Now let me reiterate my question: Have anyone attempted to standardise 
the heartbeats since then (say as a layer on top of TCP) ?

Martin

---

From: Michael Dillon <[email-masked]>
Date: Fri, 25 Jun 2010 23:32:19 +0100

> I was just interested in whether the behaviour have been defined for those
> who need early failure detection (systems with failover capabilities) and
> are willing to pay for the additional bandwidth used (financial sector).

Why didn't you say so in the first place?

I believe that the common practice is to send two timestamped copies
of every packet which are routed over two distinct network paths, i.e.
not sharing circuits or routers, and then the receiver waits for both
copies to arrive. If the delay between the two identical packets is
too large, then the network is at risk, and you should assume the
worst, i.e. the information is out of date and should be discarded.
The timestamps are there to show you the variation in latency of the
first packet in the pair to arrive.

A couple of ways of sending two copies via two diverse paths are to
use MPLS where you can set up two LSPs over different topological
paths.or to use two different multicast trees which are routed
differently by your cooperating network provider.

Either SFTI or NYSE had published something about this which I
downloaded and read about 6 years ago. Can't remember all the details
but I remember that the intro talked about the transition from bisync
to IP. You really should be asking this kind of question elsewhere
where the financial types hang out. Latency is so important that even
the guys who are primarily software developers tend to know a lot
about how to do this.

--Michael Dillon

---

From: Martin Sustrik <[email-masked]>
Date: Sun, 27 Jun 2010 07:57:27 +0200

Michael,

> the guys who are primarily software developers tend to know a lot

Thanks for the details!

Now let me ask once again: Given there are thousands (or tens of 
thousands) ad hoc heartbeat implementations out there, have there ever 
been an effort to standardise it within IETF?

Martin

---

From: Joel Jaeggli <[email-masked]>
Date: Sun, 27 Jun 2010 10:07:55 -0700

> been an effort to standardise it within IETF?

The thing you seem to be missing is that applications or protocols have
different requirements for measuring liveness or availability (to pick a
couple of examples, http, imap, ospfv2 and bgp all have different keep
alive semantics, the puprose and mechanism of which varies with the
availability requirements.

TCP keepalive is detailed way in rfc 1122, bfd is rfc 5880-5884

as a community we tend to favor the use of in-band signaling such that
the application becomes aware of availability or lack-thereof rather
than on bolt-on mechanisms. on the context of networks in-band
signalling goes a long way towards avoiding the sorts of failures that
can occur when there is not congreunce between the forwarding path and
the control plane.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 28 Jun 2010 15:45:43 +0200

A lot of the application code I've seen could be described as 
"second-guess one or more TCP timers, add pepper and salt, serve as 
desired". The second half of that is obviously not amenable to 
standardisation. The TCP stack cannot take any action. But the first 
part seems more... reasonable. I think the TCP stack can inform the 
application of its state, better than it does via the APIs I know.

Of course it's a local matter, not really IETFish. Where is the boundary 
these days? Didn't some RFC extend the Berkeley sockets API for v6?

Arnt

---

From: Martin Sustrik <[email-masked]>
Date: Mon, 28 Jun 2010 16:14:03 +0200

> application of its state, better than it does via the APIs I know.

The most implementations I've seen are "send a couple of bytes each N 
seconds" on one side and "make sure there's no more 4N seconds elapse 
between two heartbeats".

The obvious problem is that heartbeats can thus sit in transmit buffer 
waiting to be delivered. They can even be retransmitted. Etc. In any 
case the functionality they are supposed to provide is pretty heavily 
distorted.

> these days? Didn't some RFC extend the Berkeley sockets API for v6?

SCTP extends BSD sockets API IIRC :)

Martin

---

From: Mark Andrews <[email-masked]>
Date: Tue, 29 Jun 2010 10:03:07 +1000

> these days? Didn't some RFC extend the Berkeley sockets API for v6?

Actually it was a pair of RFCs (manditory and optional parts of the
(singular) API).  The Open Group then botched the incorporation of
them into POSIX by only adding the manditory parts when they were
ment to be added as a pair.  Linux developers then componded Open
Group's error by hiding the optional parts of the API behind #ifdef's.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]