Subject: www.ietf.org unresponsive over IPv6?

From: Tim Chown <[email-masked]>
Date: Fri, 1 Sep 2006 11:04:18 +0100

> 
>    The best way to handle this is to maintain the announced schedule,
>    INCLUDE in the published pool all those whose eligibility is
>    uncertain and to keep the published pool list numbering IMMUTABLE
>    after its publication.  If someone in the pool is later selected by
>    the algorithm and random input but it has been determined they are
>    ineligible, they can be skipped and the algorithm run further to make
>    an additional selection.  Thus the uncertainty only effects one
>    selection and in general no more than a maximum of U selections where
>    there are U uncertain pool members.
> 
>    Other courses of action are far worse.  Actual insertion or deletion
>    of entries in the pool after its publication changes the length of
>    the list and totally scrambles who is selected, possibly changing
>    every selection.  ...
> 
> The presence of ineligible persons in the list is no reason whatsoever
> to reset.
> 
> Donald
> 
> -----Original Message-----
> From: Ned Freed [mailto:[email-masked]] 
> Sent: Thursday, August 31, 2006 10:25 AM
> To: Brian E Carpenter
> Cc: [email-masked]; IETF-Discussion; Michael StJohns
> Subject: Re: Now there seems to be lack of communicaiton here...
> 
> ...
> 
> The correct thing to do now is to reject the reest and stick with the
> original list. The only case where a reset should be allowed is if the
> process produced a bogus result.
> 
> 
>>Full disclosure: My personal opinion, which I *did* give to Lynn and 
>>Andrew when I became aware of this glitch, is that a reset is the only
> 
> 
>>way to be certain that the selection process is unbiased.
> 
> 
> Well, I have to say I think you provided some extremely bad advice, and
> I sincerely hope that there isn't anyone on the first list that has an
> even slightly acrimonious public relationship with Andrew. We could be
> in very deep doo if there is.
> 
> 				Ned
> 
> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ietf
> 

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 01 Sep 2006 13:25:19 +0200

Tim Chown wrote:
> Hi,
> 
> While I can establish a fast telnet session to port 80:
> 
> $ telnet www.ietf.org 80
> Trying 2001:503:c779:b::d1ad:35b4...
> Connected to www.ietf.org.
> Escape character is '^]'.
> 
> Attempting to browse via MSIE leads to timeouts.
> 
> Connecting explictly to http://209.173.53.180 to assure IPv4 works fine.

"telnet -4 www.ietf.org" should do that trick too, but notez bien:

www.ietf.org            A       209.173.53.180
www.ietf.org            A       209.173.57.180
www.ietf.org            AAAA    2001:503:C779:B:0:0:D1AD:35B4

We don't know if those are 3 different boxes, or simply 1 box with 
multiple links/addresses. As IPv4 traceroutes die off one can't tell 
what really is happening behind AT&T (at least from my perspective).
Looking at routeviews it seems that both IPv4 /24's have somewhat 
different ASPaths, going over different upstreams.

> Perhaps some Apache issue or PMTU issue?

Most likely PMTU.

> Anyone else experiencing this?

Works from boxes behind my home DSL line (purgatory.unfix.org) and from 
my work box, which is behind SWITCH (2001:620::/32). Tested with telnet, 
  MSIE and firefox. See below for traces from the DSL.

Can you try tracepath6, which is available afaik only on Linux boxes 
though, to show the MTU path? (iputils-tracepath is the deb package)

Greets,
  Jeroen

--
jeroen@purgatory:~$ tracepath6 www.ietf.org
  1?: [LOCALHOST]                      pmtu 1480
  1:  2001:7b8:5:10:74::1                       32.572ms
  2:  i49.ge-0-1-0.jun1.kelvin.ipv6.network.bit.nl asymm  3  33.623ms
  3:  jun1.telecity.ipv6.network.bit.nl        asymm  4  34.609ms
  4:  zpr2.amt.cw.net                          asymm  5  35.522ms
  5:  so-5-2-0-dcr1.amd.cw.net                 asymm  6  34.597ms
  6:  as0-dcr2.amd.cw.net                      asymm  7  35.611ms
  7:  so-4-0-0-dcr1.tsd.cw.net                 asymm  8  41.612ms
  8:  so-1-0-0-zcr1.lnt.cw.net                 asymm  9  41.507ms
  9:  2001:7f8:4::31f9:1                       asymm  8 137.537ms
10:  v3323-mpd.cr1.ewr1.us.occaid.net         asymm  7 141.670ms
11:  ge-0-1-0.cr1.iad1.us.occaid.net          asymm  7 146.538ms
12:  unknown.occaid.net                       asymm  8 152.466ms

And then no responses, thus most likely filtered for this kind of trace.

But traceroute6 works fully:

traceroute to www.ietf.org (2001:503:c779:b::d1ad:35b4) from 
2001:7b8:20d:0:290:27ff:fe24:c19f, 30 hops max, 24 byte packets
  1  2001:7b8:5:10:74::1 (2001:7b8:5:10:74::1)  13.874 ms  9.645 ms 
10.963 ms
  2  i49.ge-0-1-0.jun1.kelvin.ipv6.network.bit.nl 
(2001:7b8:3:31:290:6900:31c6:d81f)  9.943 ms  15.581 ms  14.958 ms
  3  jun1.telecity.ipv6.network.bit.nl (2001:7b8::290:6900:1c6:7c1f) 
16.953 ms *  10.6 ms
  4  zpr2.amt.cw.net (2001:7f8:1::a500:1273:1)  10.972 ms  11.501 ms 
10.99 ms
  5  so-5-2-0-dcr1.amd.cw.net (2001:5000:0:12::1)  11.987 ms  11.614 ms 
  11.986 ms
  6  as0-dcr2.amd.cw.net (2001:5000:0:11::2)  11.989 ms  11.569 ms 
10.989 ms
  7  so-4-0-0-dcr1.tsd.cw.net (2001:5000:0:20::2)  17.988 ms  17.589 ms 
  17.989 ms
  8  so-1-0-0-zcr1.lnt.cw.net (2001:5000:0:61::2)  21.988 ms  17.505 ms 
  17.988 ms
  9  2001:7f8:4::31f9:1 (2001:7f8:4::31f9:1)  112.99 ms  112.727 ms 
113.984 ms
10  v3323-mpd.cr1.ewr1.us.occaid.net (2001:4830:fe:1010::2)  111.988 ms 
  112.501 ms  112.991 ms
11  ge-0-1-0.cr1.iad1.us.occaid.net (2001:4830:ff:f150::2)  117.969 ms 
118.47 ms  118.997 ms
12  unknown.occaid.net (2001:4830:e6:d::2)  121.972 ms  122.48 ms 
121.978 ms
13  stsc350a-eth3c0.va.neustar.com (2610:a0:c779::fe)  124.004 ms 
124.44 ms  124.975 ms
14  2001:503:c779:b::d1ad:35b4 (2001:503:c779:b::d1ad:35b4)  123.992 ms 
  122.722 ms  124.007 ms

Greets,
  Jeroen

---

From: Tim Chown <[email-masked]>
Date: Fri, 1 Sep 2006 13:16:24 +0100

> though, to show the MTU path? (iputils-tracepath is the deb package)

So I'm on RedHat, have tracepath6 installed (not used before, thanks for
the pointer :) and I see in comparison:

$ /usr/sbin/tracepath6 www.ietf.org
 1?: [LOCALHOST]                      pmtu 1500
 1:  2001:630:d0:f102::2                        1.191ms 
 2:  zaphod.6core.ecs.soton.ac.uk               1. 93ms 
 3:  ford.6core.ecs.soton.ac.uk                 1.915ms 
 4:  2001:630:c1:100::1                         2. 66ms 
 5:  2001:630:c1:10::1                          2.353ms 
 6:  no reply
 7:  no reply
 8:  no reply
 9:  po9-0.cosh-scr.ja.net                      8.440ms 
10:  po2-0.lond-scr.ja.net                      5.273ms 
11:  po0-0.lond-scr4.ja.net                     6.319ms 
12:  gi0-1.lond-isr4.ja.net                     6.238ms 
13:  2001:630:0:10::52                          8.138ms 
14:  ge-2-24.r01.londen03.uk.bb.gin.ntt.net    11.163ms 
15:  xe-0-1-0.r23.londen03.uk.bb.gin.ntt.net  asymm 16  11.525ms 
16:  xe-0-2-0.r22.londen03.uk.bb.gin.ntt.net  asymm 17  15.225ms 
17:  1d-uk6x.ipv6.btexact.com                  11.292ms 
18:  52.ge0-0.cr2.lhr1.uk.occaid.net           13.643ms 
19:  52.ge0-0.cr2.lhr1.uk.occaid.net          asymm 18  16. 63ms pmtu 1480
19:  v3323-mpd.cr1.ewr1.us.occaid.net          88.237ms 
20:  ge-0-1-0.cr1.iad1.us.occaid.net           92.623ms 
21:  unknown.occaid.net                       101.267ms 
22:  no reply
23:  no reply
24:  no reply
25:  no reply
26:  no reply
27:  no reply
28:  no reply
29:  no reply
30:  no reply
31:  no reply
     Too many hops: pmtu 1480
     Resume: pmtu 1480 

Hmm, 1500 vs 1480.

> 14  2001:503:c779:b::d1ad:35b4 (2001:503:c779:b::d1ad:35b4)  123.992 ms 

Likewise:

$ /usr/sbin/traceroute6 www.ietf.org
traceroute to www.ietf.org (2001:503:c779:b::d1ad:35b4) from 2001:630:d0:f102:230:48ff:fe23:58df, 30 hops max, 16 byte packets
 1  2001:630:d0:f102::2 (2001:630:d0:f102::2)  1.883 ms  2.719 ms  4.513 ms
 2  zaphod.6core.ecs.soton.ac.uk (2001:630:d0:f001::1)  4.462 ms  2.485 ms  2.886 ms
 3  ford.6core.ecs.soton.ac.uk (2001:630:d0:f000::1)  3.444 ms  0.648 ms  0.64 ms
 4  2001:630:c1:100::1 (2001:630:c1:100::1)  1.235 ms  1.104 ms  0.962 ms
 5  2001:630:c1:10::1 (2001:630:c1:10::1)  1.21 ms  1.138 ms  1.186 ms
 6  * * *
 7  2001:630:c1::1 (2001:630:c1::1)  4.982 ms  2.891 ms  2.138 ms
 8  2001:630:c1::1 (2001:630:c1::1)  2.562 ms  2.189 ms  2.662 ms
 9  po9-0.cosh-scr.ja.net (2001:630:0:10::85)  6.928 ms  5.495 ms  2.477 ms
10  po2-0.lond-scr.ja.net (2001:630:0:10::29)  4.459 ms  4.32 ms  4.166 ms
11  po0-0.lond-scr4.ja.net (2001:630:0:10::56)  4.739 ms  4.51 ms  4.492 ms
12  gi0-1.lond-isr4.ja.net (2001:630:0:10::12a)  4.739 ms  4.384 ms  4.721 ms
13  2001:630:0:10::52 (2001:630:0:10::52)  6.307 ms  5.528 ms  4.987 ms
14  ge-2-24.r01.londen03.uk.bb.gin.ntt.net (2001:728:1c00:5000::9)  5.8 ms  5.663 ms  6.625 ms
15  xe-0-1-0.r23.londen03.uk.bb.gin.ntt.net (2001:728:0:2000::21)  6.66 ms  6.656 ms  6.885 ms
16  xe-0-2-0.r22.londen03.uk.bb.gin.ntt.net (2001:418:0:2000::1c9)  6.114 ms  6.369 ms  6.042 ms
17  1d-uk6x.ipv6.btexact.com (2001:7f8:2:1::1d)  7.595 ms  6.708 ms  6.703 ms
18  52.ge0-0.cr2.lhr1.uk.occaid.net (2001:4830:fe:1100::2)  11.047 ms  6.983 ms  8.301 ms
19  v3323-mpd.cr1.ewr1.us.occaid.net (2001:4830:fe:1010::2)  81.249 ms  82.56 ms  80.423 ms
20  ge-0-1-0.cr1.iad1.us.occaid.net (2001:4830:ff:f150::2)  87.17 ms  87.134 ms  85.887 ms
21  unknown.occaid.net (2001:4830:e6:d::2)  90.419 ms  90.738 ms  91.462 ms
22  stsc350a-eth3c0.va.neustar.com (2610:a0:c779::fe)  92.066 ms  92.392 ms  92.732 ms
23  2001:503:c779:b::d1ad:35b4 (2001:503:c779:b::d1ad:35b4)  91.799 ms  92.663 ms  90.864 ms

-- 
Tim/::1

---

From: Jeroen Massar <[email-masked]>
Date: Fri, 01 Sep 2006 14:48:10 +0200

> On Fri, Sep 01, 2006 at 01:25:19PM +0200, Jeroen Massar wrote:
[..]

>>  1?: [LOCALHOST]                      pmtu 1480

This 1480 is actually from the pmtu cache, when it expires, or I flush 
it manually, it gives:

  1?: [LOCALHOST]                      pmtu 1500
  1:  2001:7b8:5:10:74::1                       33.631ms
  2:  i49.ge-0-1-0.jun1.kelvin.ipv6.network.bit.nl asymm  3  35.484ms
  3:  jun1.telecity.ipv6.network.bit.nl        asymm  4  34.532ms
  4:  zpr2.amt.cw.net                          asymm  5  36.527ms
  5:  so-5-2-0-dcr1.amd.cw.net                 asymm  6  36.572ms
  6:  as0-dcr2.amd.cw.net                      asymm  7  36.553ms
  7:  so-4-0-0-dcr1.tsd.cw.net                 asymm  8  42.392ms
  8:  so-1-0-0-zcr1.lnt.cw.net                 asymm  9  44.520ms
  9:  2001:7f8:4::31f9:1                       asymm  8 137.479ms
10:  2001:7f8:4::31f9:1                       asymm  8 137.774ms pmtu 1480
10:  v3323-mpd.cr1.ewr1.us.occaid.net         asymm  7 142.540ms
11:  ge-0-1-0.cr1.iad1.us.occaid.net          asymm  7 147.413ms
12:  unknown.occaid.net                       asymm  8 148.820ms

Long live native IPv6 over DSL :)

Hop 9/10 are a bit weird though, most likely a TTL bug.

> the pointer :) and I see in comparison:

Yes, tracepath6 is a *very* useful tool, but as I demonstrated myself 
above, it doesn't flush the cache, thus be careful there.
Tracepath6 uses some Linux specific tricks for getting certain 
information, which is why (afaik) it is not available on eg BSD's.

>  9:  po9-0.cosh-scr.ja.net                      8.440ms 

How sure are you these have a MTU of 1500? Best way to test is to do 
ping6's in the form of "ping6 -M do -s 1500 <target>" and decrementing 
per 10 or 20 till it doesn't respond anymore and then increasing again.

> 19:  52.ge0-0.cr2.lhr1.uk.occaid.net          asymm 18  16. 63ms pmtu 1480

Though from this hop you should have 1480 which is at least the correct 
value for the rest of the route. 1280 should always work of course, but 
then the link has to indicate this too. The problem with debugging this 
is that you can't do a traceroute6 from both sides, there might be an 
async route back to your network which might be causing this issue.
Assuming that http://www.sixxs.net/tools/traceroute/ indicates that the 
three SixXS PoPs inside the OCCAID network, over which the IETF seems to 
get connectivity at least pick Verio->NTT as an outbound route towards 
your network.

Maybe a traceroute6 interface on http://noc.ietf.org/ could help debug 
these issues easier? I'll send the secretariat a separate message about 
that, also that they might want to ask Neustar join up with GRH 
(http://www.sixxs.net/tools/grh/) to make debugging easier as then we at 
least have ASpaths also which might help here.

[..]
> Hmm, 1500 vs 1480.

Check your neighbor cache after the traceroute/path or even a full TCP 
connect has completed, you should have an entry similar to:

jeroen@purgatory:~$ ip -6 ro sho cache
2001:503:c779:b::d1ad:35b4 via 2001:7b8:5:10:74::1 dev eth1  metric 0
     cache  expires 429sec mtu 1480 advmss 1440 hoplimit 4294967295

The 1480 is noted here, check that that is the case.

>  8  2001:630:c1::1 (2001:630:c1::1)  2.562 ms  2.189 ms  2.662 ms

These hops are weird IMHO, 7 & 8 should not be duplicate, prolly a Cisco 
IOS located there as there are some releases which didn't decrement the 
TTL when going from Ethernet->ATM or some other weird interface changeover.

jeroen@purgatory:~$ tracepath6 2001:630:d0:f102::2
  1?: [LOCALHOST]                      pmtu 1500
  1:  2001:7b8:5:10:74::1                       33.450ms
  2:  i49.ge-0-1-0.jun1.kelvin.ipv6.network.bit.nl asymm  3  33.178ms
  3:  jun1.telecity.ipv6.network.bit.nl        asymm  4  34.478ms
  4:  ge-0.ams-ix.amstnl02.nl.bb.verio.net     asymm  5  34.595ms
  5:  xe-0-2-0.r22.amstnl02.nl.bb.gin.ntt.net   34.464ms
  6:  p64-2-0-0.r23.londen03.uk.bb.gin.ntt.net  42.452ms
  7:  xe-3-1.r01.londen03.uk.bb.gin.ntt.net    asymm  6  42.515ms
  8:  ge-0.ukerna.londen03.uk.bb.gin.ntt.net   asymm  7  42.596ms
  9:  fa1-1.lond-isr4.ja.net                   asymm  8  43.537ms
10:  gi4-0-1.lond-scr4.ja.net                 asymm  9  44.426ms
11:  po0-0.lond-scr.ja.net                    asymm 10  44.597ms
12:  po2-0.cosh-scr.ja.net                    asymm 11  46.558ms
13:  po0-0.cosham-bar.ja.net                  asymm 12  46.572ms
14:  lense.site.ja.net                        asymm 13  46.602ms
15:  no reply
16:  2001:630:c1:1::1                         asymm 15  47.266ms
17:  2001:630:c1:10::2                        asymm 16  48.727ms
18:  2001:630:c1:100::2                       asymm 17  48.722ms
19:  internal-router.6core.ecs.soton.ac.uk    asymm 18  48.714ms
20:  internal-router.6core.ecs.soton.ac.uk    asymm 18  48.597ms !H
      Resume: pmtu 1500

Seems also to think it is a MTU of 1500, though the traceroute doesn't 
finalize.

Greets,
  Jeroen

---

From: Tim Chown <[email-masked]>
Date: Fri, 1 Sep 2006 16:23:16 +0100

> >19:  52.ge0-0.cr2.lhr1.uk.occaid.net          asymm 18  16. 63ms pmtu 1480

These seem to be OK.

> least have ASpaths also which might help here.

I think that would be helpful.   I think IETF could support nice diagnostics
if showcasing v6 connectivity.
 
> The 1480 is noted here, check that that is the case.

Well, I can see www.ipv6tf.org fine, but www.ietf.org hangs in the
browser, but both seem to negotiate 1480 MTU:

For www.ipv6tf.org

$ /usr/sbin/tracepath6 www.ipv6tf.org
 1?: [LOCALHOST]                      pmtu 1500
 1:  2001:630:d0:f102::2                        1.231ms 
 2:  zaphod.6core.ecs.soton.ac.uk               4.443ms 
 3:  ford.6core.ecs.soton.ac.uk                 1.849ms 
 4:  2001:630:c1:100::1                         1.982ms 
 5:  2001:630:c1:10::1                          2.681ms 
 6:  no reply
 7:  no reply
 8:  no reply
 9:  po9-0.cosh-scr.ja.net                      7.729ms 
10:  po2-0.lond-scr.ja.net                      9.693ms 
11:  po0-0.lond-scr4.ja.net                    10.444ms 
12:  gi0-1.lond-isr4.ja.net                     5.880ms 
13:  2001:630:0:10::52                          6.636ms 
14:  uk6x.ipv6.btexact.com                      8.269ms 
15:  uk6x.ipv6.btexact.com                    asymm 14  11.370ms pmtu 1480
15:  v6-tunnel-consulintel.ipv6.btexact.com   asymm 16  68.662ms 
16:  no reply
17:  no reply
<snip>

Indicates 1480, and I get the 1480 MTU in my cache:

[tjc@login ~]$ /sbin/ip -6 ro sho cache
2001:630:d0:f102:20e:cff:fe09:e058 via 2001:630:d0:f102:20e:cff:fe09:e058 dev eth0  metric 0 
    cache  mtu 1500 rtt 27ms rttvar 20ms cwnd 3 advmss 1440
2001:7f9:1000:1::103 via fe80::214:1bff:fe3d:2c00 dev eth0  metric 0 
    cache  expires 557sec mtu 1480 advmss 1440

Then doing the same to www.ietf.org 

$ /usr/sbin/tracepath6 www.ietf.org
 1?: [LOCALHOST]                      pmtu 1500
 1:  2001:630:d0:f102::2                        1.297ms 
 2:  zaphod.6core.ecs.soton.ac.uk               1. 94ms 
 3:  ford.6core.ecs.soton.ac.uk                 1.984ms 
 4:  2001:630:c1:100::1                         2. 35ms 
 5:  2001:630:c1:10::1                          2.746ms 
 6:  no reply
 7:  no reply
 8:  no reply
 9:  po9-0.cosh-scr.ja.net                      3. 16ms 
10:  po2-0.lond-scr.ja.net                      6. 13ms 
11:  po0-0.lond-scr4.ja.net                     5.876ms 
12:  gi0-1.lond-isr4.ja.net                     8.529ms 
13:  2001:630:0:10::52                          8.918ms 
14:  ge-2-24.r01.londen03.uk.bb.gin.ntt.net     8.277ms 
15:  xe-0-1-0.r23.londen03.uk.bb.gin.ntt.net  asymm 16   8.742ms 
16:  xe-0-2-0.r22.londen03.uk.bb.gin.ntt.net  asymm 17   8.823ms 
17:  1d-uk6x.ipv6.btexact.com                  12. 38ms 
18:  52.ge0-0.cr2.lhr1.uk.occaid.net           16.152ms 
19:  52.ge0-0.cr2.lhr1.uk.occaid.net          asymm 18  12.347ms pmtu 1480
19:  v3323-mpd.cr1.ewr1.us.occaid.net          88.965ms 
20:  ge-0-1-0.cr1.iad1.us.occaid.net           93.819ms 
21:  unknown.occaid.net                       100.504ms 
22:  no reply
23:  no reply
<snip>

Suggests 1480, and that's in the cache:

[tjc@login ~]$ /sbin/ip -6 ro sho cache
2001:503:c779:b::d1ad:35b4 via fe80::214:1bff:fe3d:2c00 dev eth0  metric 0 
    cache  expires 591sec mtu 1480 advmss 1440
2001:630:d0:f102:20e:cff:fe09:e058 via 2001:630:d0:f102:20e:cff:fe09:e058 dev eth0  metric 0 
    cache  mtu 1500 rtt 27ms rttvar 20ms cwnd 3 advmss 1440
2001:7f9:1000:1::103 via fe80::214:1bff:fe3d:2c00 dev eth0  metric 0 
    cache  expires 329sec mtu 1480 advmss 1440

So both behave the same apparently in terms of PMTU discovery.

I just wondered if it might be an Apache thing because have run into
things like SENDFILE optimisation issues before.   But you've ruled
that out I think.

> >2001:630:d0:f102:230:48ff:fe23:58df, 30 hops max, 16 byte packets

Yeah, it's a Cisco MPLS network used by our regional academic network.
 
> Seems also to think it is a MTU of 1500, though the traceroute doesn't 

Try to www.ecs.soton.ac.uk maybe.
 
I guess we should take this offline or to the IPv6 ops list.

Thanks for the far end help :)    Whatever's happening changed quite
recently.  Just curious as to what... I believe we follow the ICMPv6
filtering recommendations text here, and the above suggests PMTU 
discovery is acting reasonably.

-- 
Tim/::1

---

From: YOSHIFUJI Hideaki / 吉藤英明<[email-masked]>
Date: Fri, 01 Sep 2006 19:46:09 +0900 (JST)

> Anyone else experiencing this?

Yes, I have similar issue.
If I set the interface MTU to 1280, it works fine.
PMTU issue, perhaps.

--yoshfuji