Subject: Facebook DNS issue

From: Behcet Sarikaya <[email-masked]>
Date: Mon, 4 Oct 2021 15:30:17 -0500

Hi,

As you all know, Facebook can not be reached since quite some time today.
I heard that it is a DNS A and AAAA records problem.

I am not sure if this list is the right one to discuss or is there another
one?

Behcet

---

From: Matt Joras <[email-masked]>
Date: Mon, 4 Oct 2021 13:33:38 -0700

I don't see how this is relevant to a generic IETF list. I can assure you,
it is not relevant and there is nothing that can happen here but idle
speculation and pontification that will immediately go off topic.

Matt Joras

---

From: Töma Gavrichenkov <[email-masked]>
Date: Mon, 4 Oct 2021 23:35:17 +0300

Peace,

> I am not sure if this list is the right one to discuss or is there another

Most probably not, this is just an extraordinary operational (and
infrastructural) issue.

--
Töma

---

From: Ole Jacobsen <[email-masked]>
Date: Mon, 4 Oct 2021 13:37:52 -0700

(Affects Instagram too, follows a 60 minutes "exposure" last night).

DNS Ops list?

Ole

> On Oct 4, 2021, at 13:30, Behcet Sarikaya <[email-masked]> wrote:
> 
> Hi,
> 
> I am not sure if this list is the right one to discuss or is there another one?
> 
> Behcet

Ole J. Jacobsen
Editor and Publisher
The Internet Protocol Journal
Office: +1 415-550-9433
Cell:   +1 415-370-4628
Web: protocoljournal.org
E-mail: [email-masked]
E-mail: [email-masked]
Skype: organdemo

---

From: Ofer Inbar <[email-masked]>
Date: Mon, 4 Oct 2021 18:03:25 -0400

> I am not sure if this list is the right one to discuss or is there another

NANOG is probably the list where people were discussing this, as it
was caused by a botched BGP update, which is right in NANOG's alley.
  -- Cos

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 5 Oct 2021 11:54:18 +1300

> was caused by a botched BGP update, which is right in NANOG's alley.

Right, but there are a couple of points of interest for the IETF, I think.

I can't deduce from the NANOG thread whether the BGP4 mistake removed both IPv4 and IPv6 routes, but I did notice that their *authoritative* DNS servers are IPv4-only.
(https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20)

Is that good practice?

And then

"Someone on the Facebook recovery effort has explained that a routine BGP update went wrong, which in turn locked out those with remote access who could reverse the mistake. Those who do have physical access do not have authorization on the servers. Catch-22."
(https://twitter.com/SGgrc/status/1445116435731296256)

Thay makes a *very* strong case for what ANIMA has designed - an autonomic control plane that is absolutely independent of the normal data plane.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 5 Oct 2021 14:21:37 +1300

No, cancel what I said about the authoritative servers.

Regards
   Brian

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 4 Oct 2021 22:43:34 -0400

I don't see the specific Facebook issue as being an issue for the IETF in
itself. But it has drawn attention to issues that do inevitably involve the
IETF.

We are yet again reminded of the fragility of what is supposed to be a
critical infrastructure. For over 7 hours, people could not call or message
each other. That is a problem. But the even larger problem is that because
the design of OAUTH depends on four global identity providers, a billion
users were locked out of a wide swath of the non-Facebook internet.

Yes, DNS was involved but referring it to the DNS operations list
presupposes that it was a DNS operations issue and not an architecture
issue. The DNS query protocol is brittle. Maintaining the availability of
core DNS is ruinously expensive because registries and large services have
to survive DDoS attacks. Sure, Facebook flubbed a BGP update. But why did
they need to go there in the first place? When a BGP flub brings down DNS,
the most likely culprit is ANYCAST.

ANYCAST was a clever hack but it is only a hack. It allows the DNS to scale
massively for performance, but as we saw today, that ANYCAST address is a
single point of failure.

There are questions here that should be asked but won't be. Because there
are very few people left who are willing to look at the current
architecture with a critical eye and say 'that needs fixing'.

It would be nice if just once we could have discussions of alternative ways
of doing things without the first objection being 'nothing can ever change'.

Consider the fact that the DNS root zone has a couple of thousand TLDs. Why
do we have a hundred plus hosts providing real time query service for less
than a MB of data that only changes a few times a year?

I do not object to the cost. If people want to spend their own money
supporting that system, that is their lookout. But the real time query
requirement introduces real fragility into the system. If we had made
robustness the priority, that approach would have been changed long ago.

---

From: Lloyd W <[email-masked]>
Date: Tue, 5 Oct 2021 13:53:33 +1100

WAR IS PEACE
BGP IS DNS
IGNORANCE IS STRENGTH

L.

---

From: Donald Eastlake <[email-masked]>
Date: Mon, 4 Oct 2021 22:59:27 -0400

On Mon, Oct 4, 2021 at 10:44 PM Phillip Hallam-Baker
<[email-masked]> wrote:
>
>...
>
> Consider the fact that the DNS root zone has a couple of thousand TLDs. Why do we have a hundred plus hosts providing real time query service for less than a MB of data that only changes a few times a year?

Well, there is RFC 8606 on maintaining a local copy of root.

Thanks,
Donald
===============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 2386 Panoramic Circle, Apopka, FL 32703 USA
 [email-masked]

---

From: Matt Joras <[email-masked]>
Date: Mon, 4 Oct 2021 20:11:25 -0700

I will hop in here one more time. It was not a botched BGP update. The DNS
disappearance was an unfortunate but preventable side effect of a global
backbone issue. Had DNS been functioning everything still would have been
down: https://engineering.fb.com/2021/10/04/networking-traffic/outage/

Matt Joras

On Mon, Oct 4, 2021, 7:44 PM Phillip Hallam-Baker <[email-masked]>
wrote:

> ANYCAST was a clever hack but it is only a hack. It allows the DNS to
> scale massively for performance, but as we saw today, that ANYCAST address
> is a single point of failure.
>
>
> It would be nice if just once we could have discussions of alternative
> ways of doing things without the first objection being 'nothing can ever
> change'.
>
>
> Consider the fact that the DNS root zone has a couple of thousand TLDs.
> Why do we have a hundred plus hosts providing real time query service for
> less than a MB of data that only changes a few times a year?
>

---

From: "Theodore Ts'o" <[email-masked]>
Date: Tue, 5 Oct 2021 01:27:45 -0400

> down: https://engineering.fb.com/2021/10/04/networking-traffic/outage/

It was a botched BGP configuration issue, and when BGP advertisements
were updated across the global backbone, caused large portions of
Facebook's network to not be reachable.  The fact that this brought
down Facebook's DNS appears to have significantly increased its
downtime, since Facebook's engineers couldn't authenticate to the
servers needed to fix the problem.

There are certainly a large number of operational questions which this
brings up --- why didn't Facebook have their own internal backbone
networks, with their own internal split-view DNS?  Why didn't have
ways so their SRE's could get direct access to some of the servers in
their data centers which didn't depend on access via the public or
internal Internet backone networks?  These are however, out of scope
of the IETF, because it has to do with how a particular site
configures its networks.

> > the most likely culprit is ANYCAST.

Let's take a look at Facebook's nameservers:

facebook.com.		172786	IN	NS	b.ns.facebook.com.
facebook.com.		172786	IN	NS	d.ns.facebook.com.
facebook.com.		172786	IN	NS	c.ns.facebook.com.
facebook.com.		172786	IN	NS	a.ns.facebook.com.

b.ns.facebook.com.	143822	IN	A	129.134.31.12
b.ns.facebook.com.	163741	IN	AAAA	2a03:2880:f0fd:c:face:b00c:0:35
d.ns.facebook.com.	164470	IN	A	185.89.219.12
d.ns.facebook.com.	168652	IN	AAAA	2a03:2880:f1fd:c:face:b00c:0:35
c.ns.facebook.com.	164249	IN	A	185.89.218.12
c.ns.facebook.com.	167948	IN	AAAA	2a03:2880:f1fc:c:face:b00c:0:35
a.ns.facebook.com.	143821	IN	A	129.134.30.12
a.ns.facebook.com.	166558	IN	AAAA	2a03:2880:f0fc:c:face:b00c:0:35

These are any ANYCAST addresses.

Cheers,

						- Ted

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 5 Oct 2021 14:47:52 +0900

Theodore Ts'o wrote:

> It was a botched BGP configuration issue, and when BGP
> advertisements were updated across the global backbone, caused > large portions of Facebook's network to not be reachable.

That's all, having nothing to do with DNS.

> The fact that this brought down Facebook's DNS appears to have
> significantly increased its downtime, since Facebook's engineers
> couldn't authenticate to the servers needed to fix the problem.

That's totally unlikely.

BGP failure does not mean facebook-internal servers and clients
can't access facebook-internal name servers.

						Masataka Ohta

---

From: "Theodore Ts'o" <[email-masked]>
Date: Tue, 5 Oct 2021 02:01:19 -0400

> can't access facebook-internal name servers.

There is quite a bit of anecdotal reporting that internal servers and
clients were affected.  I agree that for a well-architected corporate
intranet and internal production network, this *should* be highly
unlikely.  But remember we're talking about a company that uses PHP as
its primary web langauge, and for which "move fast and break things"
is one of the founder's favorite sayings....

"Not only are Facebook's services and apps down for the public, its
internal tools and communications platforms, including Workplace, are
out as well. No one can do any work. Several people I've talked to
said this is the equivalent of a "snow day" at the company."

- https://twitter.com/RMac18/status/1445069187664293895

"Was just on phone with someone who works for FB who described
employees unable to enter buildings this morning to begin to evaluate
extent of outage because their badges weren’t working to access doors."

- https://twitter.com/sheeraf/status/1445099150316503057

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 5 Oct 2021 15:24:50 +0900

Theodore Ts'o wrote:

> There is quite a bit of anecdotal reporting that internal servers
> and clients were affected.

by the BGP failure, of course.

> internal tools and communications platforms, including Workplace,
> are out as well.

"out" should mean "not accessible from home", I'm afraid.

						Masataka Ohta

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 5 Oct 2021 09:59:48 -0400

> of the IETF, because it has to do with how a particular site

What Ted says above is entirely consistent with what Cloudflare and others
observed from BGP route advertisements.

But I do dispute the notion that this is not an issue for IETF.  Split
horizon DNS has been a common and necessary DNS configuration for 30+ years
now. But there is no support for split horizon in the DNS protocol and that
has consequences.

Or to look at the situation differently: Does the IETF want to be a part of
the solution to this particular set of problems?

If this was a unique and unprecedented incident, we could safely ignore it.
But it isn't. It is merely an incident that happened to have particularly
widespread effects.

Availability is a security issue.

30 years ago, this organization was built around the notion that the people
in it had a better idea of how to do communications than the telephone
companies. And 30 years later, that is still where most people seem to be
stuck.

We should be looking at how people are actually using the Internet in
practice and working out ways to make those uses work better.