Subject: Re: Last Call: draft-ietf-syslog-protocol (The syslog Protocol) to Proposed Standard

From: Sam Hartman <[email-masked]>
Date: Thu, 01 Feb 2007 08:09:29 -0500

>>>>> "Mark" == Mark Andrews <[email-masked]> writes:

    >> - 'The syslog Protocol ' <draft-ietf-syslog-protocol-19.txt> as
    >> a Proposed Standard

    Mark> 	draft-ietf-syslog-protocol-19.txt recommends using a
    Mark> reliable protocol.  Existing implementations of syslog do
    Mark> this and deadlock with nameservers which are logging via
    Mark> syslog.

Please explain the deadlock in more detail.  One of the primary
reasons for the syslog working group is reliable syslog, so I think we
need to focus on how to avoid the deadlock in other ways rather than
avoiding reliability.

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "David W. Hankins" <[email-masked]>
Date: Thu, 1 Feb 2007 10:08:39 -0800

> need to focus on how to avoid the deadlock in other ways rather than

If you have 50,000 syslog lines to put out, and only enough
network/disk/something bandwidth for 5,000 within the same time
frame, that's a problem.

If you insist on keeping all 50,000 lines of output, there is no
solution to that problem.  If you block, that's a big problem as
it ultimatley totally disables the service attempting to log
information.  If you write to a growing backing store, well you'll
run out of space eventually (even disk is not infinite).
Compression can only get you so far.

Ultimately you have to drop something, and if it's not the
syslog output, it's the service's output.

Reliability is the problem, and the advice we give our users is
not to log reliably (or even to configure servers to log to a
local file, circumventing syslog entirely).

Note that reliable network transport of syslog information is
equally damaging as reliable local storage of syslog output (eg
by forcing disk synchronization of each line).  At least one
of today's syslog implementations insists that you prefix your
filenames with a "-" if you /don't/ want it to fsync() every
line.  This default cripples DHCP servers.

The current words in the draft;

   It may be desirable to use a transport with guaranteed delivery to
   mitigate congestion.

May be adequate to the point of suggesting that reliable delivery
might not be desirable.  But on the whole the draft doesn't read
that way, and it doesn't state the truth: reliable delivery of
syslog output is always harmful.  The point of bothering with
reliable syslog delivery, if there is one, is for those very
rare cases where losing the data is more harmful than harming
system services.

-- 
David W. Hankins	"If you don't do it right the first time,
Software Engineer		you'll just have to do it again."
Internet Systems Consortium, Inc.	-- Jack T. Hankins
_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Balazs Scheidler <[email-masked]>
Date: Thu, 01 Feb 2007 10:57:17 +0000

On Thu, 2007-02-01 at 08:31 +1100, Mark Andrews wrote:
> > - 'The syslog Protocol '
> >    <draft-ietf-syslog-protocol-19.txt> as a Proposed Standard
> 
> 	draft-ietf-syslog-protocol-19.txt recommends using a reliable
> 	protocol.  Existing implementations of syslog do this and
> 	deadlock with nameservers which are logging via syslog.
> 
> 	I'm very wary of this recommendation.

I think this is not the fault of the new standard, but an operational
issue, that probably has a fix/workaround in implementations. (for
example syslog-ng permits disabling DNS, or latest versions can use a
list of internal IP->hostname mappings)

-- 
Bazsi

---

From: Balazs Scheidler <[email-masked]>
Date: Thu, 01 Feb 2007 10:57:17 +0000

-- 
Bazsi

---

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 1 Feb 2007 14:19:28 +0100

I am thinking on how to come this by on the protocol basis, but so far I
have no good clue (except that Baszi is probably right and it is a
purely operational issue). However, I know some problems in this
relation and though it is not a deadlock issue, I would like to share
the scenario for sake of completeness:

It occurs on a typical *nix system that runs both syslogd and the dns
daemon. So it could potentially occur on almost all *nix DNS servers.
The problem exists if syslogd contains rules for remote forwarding AND
these rules do contain hostnames which need to be resolved by DNS. A
typical problem is that syslogd starts before the dns daemon. When it
intend to send a message to the remote host, it must do a dns lookup,
which fails because dns daemon is not yet active. The stock linux
syslogd contains some logic which simply defers the connection request,
but at the cost of some message loss. Other ways to avoid this problem
is to use IP addresses in syslog configuration, host files or, as Baszi
said, special configuration options for static address mapping in
syslgod.

I do not, however, know actual deadlock situations (that is neither
syslogd nor dnsd can proceed because they are waiting on each other). I,
too, would appreciate any details on that.

Rainer

> -----Original Message-----
> From: Sam Hartman [mailto:[email-masked]]
> Sent: Thursday, February 01, 2007 2:09 PM
> To: Mark Andrews
> Cc: [email-masked]; [email-masked]; [email-masked]
> Subject: [Syslog] Re: Last Call: draft-ietf-syslog-protocol (The
> syslogProtocol) to Proposed Standard
> 
> Syslog mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/syslog

---

From: "David W. Hankins" <[email-masked]>
Date: Thu, 1 Feb 2007 11:50:59 -0800

> network/disk/something bandwidth for 5,000 within the same time

s/5,000/49,999/

Apologies for the clerical error.

---

From: Pekka Savola <[email-masked]>
Date: Thu, 1 Feb 2007 22:30:17 +0200 (EET)

> Compression can only get you so far.

As an operator, I would find a reliable syslog useful.

However, I do not see this as a problem.  95% of the problem (from our 
perspective) is solved if no messages are lost _on the wire_ between 
the sender and the recipient of syslog messages.

It's acceptable for the syslog sender to replace overflowing lines of 
syslog (if some messages need to be dropped due to lack of resources) 
with a message about rate-limiting, messages being dropped, or 
whatever -- just the same way as messages might get dropped when 
syslogging to a local media.

As long as a) there is a message about syslogs getting dropped, and b) 
this is infrequent in a well operated system (i.e. the system log 
levels are set so that typically the amount of logging is OK), this 
should be no problem.

> rare cases where losing the data is more harmful than harming

IMHO, "reliable delivery of syslog output is always harmful" is very 
much an over-statement.

It seems your concern is limited to the corner case where the syslog 
sender would already have a full syslog buffer, and the sender would 
get even more syslog to send. While this is a serious problem when it 
occurs, it should be easily solvable: just drop the messages (with a 
suitable note in syslog or in the local log) that exceed the buffer 
size or prune messages from the buffer using some more advanced 
strategy.

As a particular example, we'd be very interested in getting reliable 
syslog from our routers to cover for messages lost due to network 
outages (fixed usually quickly with rerouting). We are talking of 
1-200 messages being lost within a 10 minute period -- a very 
reasonable packet rate in average.  A 1,000 or 10,000 line buffer in 
the router should be very reasonable in recent control processors. 
I'd rather the control processor reserve 0.1% of its memory (or CPU) 
to store and transmit these messages rather than try to use the last 
0.1% when it's already chugging at 99.9%; the last 0.1% would not make 
a meaningful difference anyway for whatever it's using almost all of 
its resources.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: "David W. Hankins" <[email-masked]>
Date: Thu, 1 Feb 2007 14:31:42 -0800

> syslogging to a local media.

Then the word that should be used here is not "reliable" without
any exceptions tagged on.

> levels are set so that typically the amount of logging is OK), this 

I should note that while this would be just fine, this is not in
fact what presently deployed syslog implementations that implement
"reliable" delivery (wether to network sockets or local files) do.

Today they block.

To use the world 'reliable' without illuminating clearly that
exceptions must be made is dangerous.

> IMHO, "reliable delivery of syslog output is always harmful" is very 

Well, I wasn't counting on shifting the definition of "reliable" to
"somewhat reliable".

Sure, "somewhat reliable delivery of syslog output is always harmful"
is very much an over-statement, I'll agree with that.

But I stand by what I said.

> size or prune messages from the buffer using some more advanced 

That would be fine.

But again, this is not what current implementations do, and that
is not "reliable" without exception.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 02 Feb 2007 10:01:09 +1100

> need to focus on how to avoid the deadlock in other ways rather than
 
	nameserver logs to syslog.  syslog trys to resolve a address
	which requires the nameserver to succeed.  syslog() uses a
	reliable transport to talk to syslogd.  This pipe fills up.
	syslog() then blocks waiting on syslogd which is waiting
	on the nameserver ....

	There are two way to break this.
	1. use a lossy transport
	2. don't attempt to resolve names/addresses in syslogd.

	You can reduce the problem by always logging using
	IP addresses.  However it doesn't remove the problem
	completely as you still need to look up addresses
	for forwarding purposes.

	Similarly if syslogd is using a reliable transport
	to talk to another syslogd.  That too can block which
	will eventualy lead to blockages to applications /
	memory exhaustion.
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Sam Hartman <[email-masked]>
Date: Thu, 01 Feb 2007 18:17:11 -0500

>> >> - 'The syslog Protocol ' <draft-ietf-syslog-protocol-19.txt>
    >> as >> a Proposed Standard
    >> 
    >> reasons for the syslog working group is reliable syslog, so I
    >> think we need to focus on how to avoid the deadlock in other
    >> ways rather than avoiding reliability.
 
    Mark> 	nameserver logs to syslog.  syslog trys to resolve a
    Mark> address which requires the nameserver to succeed.  syslog()
    Mark> uses a reliable transport to talk to syslogd.  This pipe
    Mark> fills up.  syslog() then blocks waiting on syslogd which is
    Mark> waiting on the nameserver ....

This is mostly out of scope for the current document which focuses on
syslog communication between systems not within a system.

Additional ways to break the deadlock:

1) Provide syslogd with at least one non-locally-logging nameserver

2) Provide enough buffering that the probability you run into this is acceptable.

3) Use an asynchronous DNS library in syslogd

I do think a note about deadlocks would be useful in the protocol
document.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 2 Feb 2007 08:31:49 +0100

On Thu, Feb 01, 2007 at 10:08:39AM -0800,
 David W. Hankins <[email-masked]> wrote 
 a message of 98 lines which said:

> If you block, that's a big problem as it ultimatley totally disables
> the service attempting to log information.

Wether it is a bug or a feature depends on your requirments. On some
high-security environments, people prefer to suspend the service
rather than not being able to log it. (Otherwise, an attacker could
easily attempt many attacks, fill in the hard disk and then perform
the real attack unlogged).

> reliable delivery of syslog output is always harmful.

Clearly, "always" is too much.

---

From: Eliot Lear <[email-masked]>
Date: Fri, 02 Feb 2007 15:04:26 +0100

I'd have to agree with Pekka.  We've just gone through this with ATIS 
and "reliable" accounting, which is quite a bit messier than this 
problem.  So long as the log indicates that something bad has happened, 
it seems to me that you're covered.  Of course, with reliability comes 
some amount of complexity.  If an implementation detects the problem, 
say, through the other side not ACKing data, it's probably a good idea 
to switch to one's backup log server, right?  I know of no syslogger 
that behaves in this manner today, although perhaps there are some out 
there.  That doesn't mean we should not provide the function.

Finally, it seems to me we are all arguing over non-normative text.  
Choice of level of reliability is generally a deployment concern.

Eliot

---

From: "David W. Hankins" <[email-masked]>
Date: Fri, 2 Feb 2007 09:59:45 -0800

> the real attack unlogged).

I'd just like to point out that you're choosing one bug over
another.  A DOS in preference to lack of observance of events.

In my opinion, that's a bad selection, but it's your selection to
make.

That kind of preference, that kind of choice, is a good thing to
have, but it would be unwise to apply to the general case a
systematic selection of DOS over observation.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sun, 04 Feb 2007 12:43:32 -0800

Daring to rush in without having read the documents....

it seems to me that somewhere one needs a NOTE, something along the lines 
of:

NOTE: In some situations, for instance when a destination disk is full or 
damaged, a syslog facility may be unable to process all messages, despite 
the message transport being reliable. In such a case, it is reasonable for 
the logger of a message to have the option of either not logging more 
messages or ceasing its own operation. This document does not specify which 
option to take.

Or words to that effect.

                  Harald

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 5 Feb 2007 11:32:42 +0100

On Fri, Feb 02, 2007 at 09:59:45AM -0800,
 David W. Hankins <[email-masked]> wrote 
 a message of 60 lines which said:

> another.

Not at all (a disk which is full is *not* a bug). I simply want to
emphasize that security is ALWAYS a compromise. And that the draft
should explain this compromise so the system administrator may make an
informed choice. (The text proposed by Harald seems OK.)

---

From: "Tom.Petch" <[email-masked]>
Date: Mon, 5 Feb 2007 14:06:03 +0100

<inline>
Tom Petch

----- Original Message -----
From: "Harald Tveit Alvestrand" <[email-masked]>
To: "David W. Hankins" <[email-masked]>; <[email-masked]>
Sent: Sunday, February 04, 2007 9:43 PM
Subject: Re: draft-ietf-syslog-protocol: "Reliable delivery considered harmful."

Harald

It might be worth reading the I-D:-)

I am not clear which piece of text in the I-D provoked the original comment.  I
do not see the I-D recommending reliable transport, with all the problems that
have been identified with that.  Rather, under security, the I-D points out that
with an unreliable transport, losing critical messages may adversely impact
operation.

It then goes on to say
" It may be desirable to use a transport with guaranteed delivery to mitigate
congestion.  It may also be desirable to include rate-limiting features in
syslog senders.  This can reduce potential congestion problems when message
bursts happen."

I find it hard to square this with the original statement that
'draft-ietf-syslog-protocol-19.txt recommends using a reliable protocol'

If we were to put in a comment about reliable transports introducing problems
with blocking, then I think that that should be in an I-D which specifies a
reliable transport, eg the soon to be completed one on TLS (there are no plans
for one with TCP).

Tom Petch

---

From: "Rainer Gerhards" <[email-masked]>
Date: Fri, 2 Feb 2007 15:24:25 +0100

> > >>>>> "Mark" == Mark Andrews <[email-masked]> writes:
[snip]

> 	will eventualy lead to blockages to applications /

*That* is a different beast, not yet discussed. I know that it exists
but it is not related to DNS. If it happens, it is really bad and
typically results in a complete system hang. There are some situations
where a lossy transport is preferable. For example, I have written a
syslog/TCP implementation which, if forced to run in single-threaded
mode, favours discarding messages over blocking as the later could
indeed lead to fatal problems on a typical linux system.

The root cause, however, is not reliable syslog per se. The root cause
is the implementation. A reliable syslogd actually needs to be
implemented with a non-blocking, de-coupled, buffered design. In almost
all cases, that means separate receiver threads, a to-be-processed
message queue and separate sender threads. Still, you have to decide
what to do if the queue size is exhausted. But that scenario is much
less likely. In that case, I'd still favour loosing some messages over
potentially loosing all AND the system the syslogd runs on. As far as I
see it, this is an app-layer issue out of scope for the underlying
protocol. The problem, of course, is that syslog is simplex and
hop-to-hop, so the original sender will never know that the message was
lost. Protocol-wise, I do not see any fix to that except integrating
some asynchronous notification mechanism. IMHO, this is outside the
scope of syslog.

It may be useful to add some hint to implementors pointing to this
blocking condition. But I am not sure if it is really justified.

Rainer