Subject: Re: Update Re: IETF Website Degradation

From: Glen <[email-masked]>
Date: Mon, 3 Aug 2015 16:19:32 -0700

Job -

Thank you so much!  Yes, we are aware of those numbers,  but thank you so
much for making sure we have them!  And thanks also for making them aware
of our situation here!

Best regards,
Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

On Mon, Aug 3, 2015 at 3:50 PM, Job Snijders <[email-masked]> wrote:

> I pinged two cloudflare employees via facebook to make them aware of the
> fact you are trying to reach them.
>
> Did you try the following to reach cloudflare:
>
>     US callers: 1 (888) 99-FLARE
>     UK callers: +44 (0)20 3514 6970
>     International callers: +1 (650) 319-8930
>
> Kind regards,
>
> Job
>
> On Mon, Aug 03, 2015 at 03:24:08PM -0700, Glen wrote:
> > All -
> >
> > We have determined that the degradation was caused by a DDoS attack
> against
> > the www.ietf.org website.  The attack was a slowly-escalating attack,
> which
> > began several hours ago, and increased in load over the afternoon.  The
> > attack was directed at the Cloudflare servers, so we were not immediately
> > impacted.
> >
> > However, as time passed, the results of the attack started to spill over
> to
> > the actual IETF webservers, with the result that our webservers started
> to
> > slow.  We were alerted to this by our own monitoring systems, which is
> when
> > we did an initial check, and I then sent the initial report out.
> >
> > At this point, we have been unable to reach a human at Cloudflare,
> although
> > we are continuing to try.  We have therefore put our Cloudflare account
> > into "DDoS Mitigation Mode".
> >
> > In this mode, users will see a brief interstitial page when browsing the
> > IETF website.  This page allows Cloudflare to perform testing on each
> > browser to determine whether the request is part of an attack or not.
> You
> > may see this page as you approach the IETF website.  It is nothing to be
> > alarmed about, and is an expected side-effect of this protection mode.
> >
> > It is unknown, at this point, why Cloudflare did not automatically
> detect,
> > and block, the attack.
> >
> > It is unknown, at this point, why the attack caused Cloudflare to start
> > spilling requests over to us.
> >
> > It is unknown, at this point, why we are unable to reach a human there.
> :-)
> >
> > However, at this time, website service is restored, and, apart from the
> > interstitial page on the IETF website, everything is running as expected.
> > We will continue to reach out to Cloudflare to address these remaining
> > issues, and will get that check page deactivated as quickly as possible.
> >
> > Thank you for your patience during that happily brief degradation.
> >
> > Glen
> > Glen Barney
> > IT Director
> > AMS (IETF Secretariat)
>
>

---

From: Job Snijders <[email-masked]>
Date: Tue, 4 Aug 2015 00:50:30 +0200

Kind regards,

Job

On Mon, Aug 03, 2015 at 03:24:08PM -0700, Glen wrote:
> All -
> 
> We have determined that the degradation was caused by a DDoS attack against
> the www.ietf.org website.  The attack was a slowly-escalating attack, which
> However, as time passed, the results of the attack started to spill over to
> the actual IETF webservers, with the result that our webservers started to
> slow.  We were alerted to this by our own monitoring systems, which is when
> At this point, we have been unable to reach a human at Cloudflare, although
> browser to determine whether the request is part of an attack or not.  You
> It is unknown, at this point, why Cloudflare did not automatically detect,
> and block, the attack.
> 
> It is unknown, at this point, why we are unable to reach a human there. :-)
>

---

From: Glen <[email-masked]>
Date: Tue, 4 Aug 2015 09:07:34 -0700

Hi Paul -

There certainly has been an effort to learn the "From where" (a total of
seven different netblocks, all in China, which are now blocked)... but I
confess that it is beyond my ability, armed with that, to figure out who
was behind it, or what their motivations were (although I'm sure inferences
could be drawn, but that' is way beyond my scope or vision!)

Best regards,
Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

On Tue, Aug 4, 2015 at 9:03 AM, Paul Kyzivat <[email-masked]> wrote:

> Will there be an attempt to learn the "who" and "why" of the attack?
>
>         Thanks,
>         Paul
>
> On 8/3/15 6:24 PM, Glen wrote:
>
>> All -
>>
>> against the www.ietf.org <http://www.ietf.org> website.  The attack was
>> a slowly-escalating attack, which began several hours ago, and increased
>> in load over the afternoon.  The attack was directed at the Cloudflare
>> servers, so we were not immediately impacted.
>>
>> to the actual IETF webservers, with the result that our webservers
>> started to slow.  We were alerted to this by our own monitoring systems,
>> which is when we did an initial check, and I then sent the initial
>> report out.
>>
>> although we are continuing to try.  We have therefore put our Cloudflare
>> account into "DDoS Mitigation Mode".
>>
>> You may see this page as you approach the IETF website.  It is nothing
>> to be alarmed about, and is an expected side-effect of this protection
>> mode.
>>
>> detect, and block, the attack.
>>
>> interstitial page on the IETF website, everything is running as
>> expected.  We will continue to reach out to Cloudflare to address these
>> remaining issues, and will get that check page deactivated as quickly as
>> possible.
>>

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 04 Aug 2015 12:03:29 -0400

Thanks,
	Paul

On 8/3/15 6:24 PM, Glen wrote:
> to be alarmed about, and is an expected side-effect of this protection mode.
>

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 5 Aug 2015 11:52:35 -0400

> seven different netblocks, all in China, which are now blocked)... but I

it's a bit worrying that the IETF site is blocked from some parts of
the Internet, at least to me :( how long will that last? and were
these things that leaked around the cloudflare frontends?

What's the impact to the blocked networks? (ie: "No access to
www.ietf.org", "No access to http on www.ietf.org", "No https access
to www.ietf.org", "other")

> confess that it is beyond my ability, armed with that, to figure out who was
> behind it, or what their motivations were (although I'm sure inferences

almost all 'dos' events are summarized by purpose: "because"
generally there's no real use in speculating about the 'why', and most
often the 'who' is also opaque to the end site...

---

From: Glen <[email-masked]>
Date: Wed, 5 Aug 2015 09:50:26 -0700

On Wed, Aug 5, 2015 at 8:52 AM, Christopher Morrow <[email-masked]>
wrote:
> the Internet, at least to me :(

I understand.

> how long will that last? and were

It will last for an indeterminate period of time; however, I do intend to
periodically attempt unblocking to see if attacks continue.  But given the
public nature of the lists, I will not make a specific statement about
timing here.  I will simply say tha we (AMS) will proactively manage this
as we do all other such incidents, to ensure the maximum optimization of
global access to the IETF for the world, balanced against the need to
prevent disruptions from occurring.

> to www.ietf.org", "other")

At present, "no access to www.ietf.org".

Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

---

From: Randy Bush <[email-masked]>
Date: Thu, 06 Aug 2015 06:50:00 +0900

> Will there be an attempt to learn the "who" and "why" of the attack?

in the ops world, dos and ddos are a constant fact of life.  megabits
and gigabits of it.  you could throw lot of engineers at trying to
understand the human identities and motivations behind it and learn
nothing.  maybe throwing bunch of sociologists or lawyers would be a
better idea; never tried it.

hang in there glen/amsl.

randy

---

From: Glen <[email-masked]>
Date: Wed, 5 Aug 2015 17:20:00 -0700

> hang in there glen/amsl.

Thank you Randy!!!

The support from you and from all in the community is VERY much
appreciated!  :-)

I am very grateful!

Best regards,
Glen
Glen Barney
IT Director
AMS (IETF Secretariat)