Subject: Re: IETF Service Outage

From: Eliot Lear <[email-masked]>
Date: Wed, 23 Mar 2016 13:50:58 +0100

Barney,

I'm curious- do you have sufficient data to perform an analysis to
determine the source of the service degradation?

Thanks,

Eliot

On 3/23/16 1:39 PM, Glen wrote:
> Greetings:
>
> Today, we experienced a degradation in network service on the primary
> IETF server.  Network service appears to have been impaired for just
> over an hour.
>
> Our automated systems did detect this issue, and we also received an
> emergency alert message from Stephen Farrell, and began an
> investigation immediately.  Before we could determine a cause, the
> degradation cleared, and network service was back to normal.
>
> In the meantime, this network issue did cause availability
> interruptions for the Datatracker, the mailing lists and related
> services; however, no data was lost, and no system damage occurred.
> Email sent during that interval might be delayed, but should be
> retried by sending systems over the next 1-4 hour period, and all
> other services are up and available as normal.
>
> We apologize for this inconvenience.  We will continue to monitor and
> address any further issues as they occur.
>
> Glen
> Glen Barney
> IT Director
> AMS (IETF Secretariat)
>
>

---

From: Glen <[email-masked]>
Date: Wed, 23 Mar 2016 06:13:12 -0700

> determine the source of the service degradation?

Hi Eliot -

We're still looking, but, at the moment, no, it does not appear that way.

Our servers run Linux, our switches are Cisco, and log levels on
everything are quite high.  Our logs are full of noise about all kinds
of other, unrelated, server activities; but, as often seems to be the
case, the logs are silent about things related to this issue.  To my
great frustration.

Our engineers are always looking at additional ways of monitoring
things, but whether this was some kind of denial-of-service attack
against a physical host, or an OS failure of some kind, or maybe just
a bad network cable, we can't yet tell.

When things like this happen unexpectedly, we only want to take a
minimal amount of time to try and perform testing mid-event.  Having
done a number of tests, checks, and localized reset procedures, we
were just about to do a reboot of the physical host when the network
just came back to normal, all by itself.  So, this time, so far, I am
unable to determine a source, which, of course, is probably the most
frustrating outcome possible.

Glen
Glen Barney
IT Director
AMS (IETF Secretariat)