Subject: Re: FTP and file transfers

From: John C Klensin <[email-masked]>
Date: Wed, 04 Oct 2017 12:25:59 -0400

Sorry... hit "send" a bit too soon.  I meant to add that it
might be worth looking at some of those older protocols to see
whether, if their basic target functionality is still useful,
even if only for an identifiable niche, if known problems
(either in the original design or due to developments since they
were designed) can be fixed by new features.

Name-calling, e.g., "FTP is awful", doesn't help with improving
understanding.  I happen to find HTTP/HTTPS fairly awful when
all I need to do is transfer a file that I have no desire to
display or otherwise render in real time.  Some browsers
mitigate that particular awfulness somewhat by providing a
"download this URL" option, but, because the idea of
interpreting a location, finding an object, interpreting it, and
retrieving it for local interpretation and display seems
integral to the web design, that still doesn't do a good job of
addressing cases in which the desire is to precisely point to
something in a remote file system, retrieve it (optionally using
a network-defined canonical form), then then get it into the
local file system in appropriate form.    Those concepts are as
important to FTP --and native to it-- as the separate control
and data streams.  We could debate how often they are needed
today, but the need is clearly above zero.

Even with the two-stream model, I think it is important to
recognize what that does do, independent of aesthetic arguments
about separate or combined streams.  Later implementations and
even versions of the protocol notwithstanding, the two streams
were intended to be completely asynchronous, allowing not only
"passive" (including third host) transfers but checking status
or even changing or aborting transfers in progress without
tearing down the connection or leaving the host at one end
unsure whether the connection failed or the other one
deliberately aborted.  It allows starting (but still
controlling) several transfers from the same control connection.
That allows several things that we have never exploited or
included in the published protocol, but that doesn't mean the
possibilities are absent.  

Similarly, getting FTP to meet today's norms for authentication,
authorization, and privacy would not be straightforward.  Those
capabilities are certainly not part of the 1985 (or 1971)
specifications of the protocol.  But we certainly know enough to
be able to do those things and maybe even to exploit some
advantages of doing key exchange out of band wrt the data
stream.  IMO, it would be far more useful to have those
discussions rather than saying "FTP is awful" and, by
implication, "anyone who thinks they need or want the features
of protocols modeled like that is guilty of wrong thinking and
should, for the improvement of their souls, just lose out" (I am
deliberately exaggerating relative to anything that has been
said in this thread, but not much relative to things that have
been said in previous versions of this conversation).

    john

--On Wednesday, October 4, 2017 06:05 -0700 Joe Touch
<[email-masked]> wrote:

> On 9/29/2017 7:48 AM, Phillip Hallam-Baker wrote:
>> How strange, I have always found FTP to be an absolute dog of
>> a protocol precisely because it mandates idiot defaults and
>> implementations are required to perform heuristic hacks. I
>> always used to have to transmit files twice because the first
>> time the transfer would be corrupted by the default being
>> 'trash my data in case I am using an IBM machine'.
>> 
>> The separation of the command and data is a good one but
>> separating it over separate TCP/IP streams is an utter
>> disaster for reliability. None of the FTPS clients I have
>> found for Windows is able to reliably transmit a Web site
>> update.
> 
> HTTP ignored this at their own peril, and ended up creating
> its own multiplexing layer to avoid HOL blocking and support
> signalling and reinventing parts of FTP poorly, IMO.
> 
> Faults in the implementation of Windows software are far too
> easy to find to blame them on the protocol itself.

And that is a specific, and IMO helpful, example of the point I
was trying to make.   We would do well to look carefully at
protocols like FTP (and even, e.g., Archie, Gopher and WAIS) to
see what can be learned from them rather than saying "It, or
implementations of it, won't do XYZ well, therefore it is
hopelessly obsolete and should be abandoned or deprecated".
Similarly, rather than saying what amounts to "ABC (typically
'the web') has won and everything else is irrelevant" --a claim
I've even heard relative to lower-layer protocols not needed to
directly support HTTP/HTTPS-- we should be asking whether there
are problems, even niche problems, that can be better addressed
in other ways.  

We might even end up with a richer and more diverse and useful
Internet.

    john

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 04 Oct 2017 19:42:53 +0100

Name-calling doesn't help anyone's understanding of ftp's limitations,
although it does hint at the degree of frustration that the protocol has
caused and, thankfully, in increasingly rare cases, still causes.  It
was never fun to have to deal with ftp from an operational point of
view, and there is nothing about it that I miss.  It caused grief on the
internet, in no short supply.

We've moved in every respect, and have learned a good deal from ftp's
deficiencies.  We have protocols which will handle inline control
multiplexing to cater with both command and data streams; we have
encryption, command pipelining, relatively consistent name space
management, character set management, remote filesystem mirroring,
incremental transfers, remote filesystems and plenty more.  They don't
always work together in the same protocol suite, or when they do work,
they mightn't work as consistently as expected, but for sure they are a
vast improvement on what we once used.

There are plenty of utilities which will handle http/https file transfer
just fine.  In the rare cases where http doesn't work, it's usually due
to the data stream provider attempting to restrict access to their
content and succeeding equally at that and aggrieving their users.  Most
of the problems I've had with http downloads in the last many years have
been related to layer 9, not layer 4.

Inline data stream translation causes more problems than it fixes, e.g.
data integrity validation, and even if we might like the idea of ebcdic
or other things like vms structured records from an idealogical, or even
an archaeological point of view, they became roadkill on the highway to
device inter-compatibility.  This wasn't a bad thing either, although we
can still learn lessons about structured data streams and non ascii
encoding and use those ideas in places where they make sense.

Some people might even claim that we've ended up with a rich, diverse
and useful Internet, even if it didn't match what was originally
envisaged in the 1980s and 1990s :-)

All that said, this is a mighty fine rathole that we crawled into.
Unless someone wants to write an ID which formally deprecates FTP, which
I would enthusiastically support, a topic stack pop might be appropriate.

Nick

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Oct 2017 16:45:52 -0400

I think it is important to be clear. If someone is praising one thing above
another, then it is entirely reasonable and not name calling to point out
that the legacy protocol they are describing is 'awful'.

Seriously, you think 'awful' is name calling? No you do not. What is
upsetting is that I do not value what I consider obsolete technology. There
has always been an IETF faction that will argue that there are use cases
for UUCP following the belief that other countries must follow the same
technology progression as the West and deploy teletypes before people can
move on to mobiles. It doesn't work that way.

FTP is pretty much the only protocol we use today where the most up to date
version of the base RFC is still has a three digit number.
​It is 32 years old. The original spec is 1980 which is the year they
stopped making my MGB. Every time I work on that car, I say to myself 'they
don't make them like this any more - thank goodness'.

​Douglas Englebart's mother of all demos is still visionary even now. But
if someone told me I had to use the code he showed in his demo today I
would call it something much worse than awful.

If you are satisfied with the technology of 30 years ago, you probably
won't be much help designing the technology ​for the next 30 years.

​What was originally said was that FTP was superior to what has mostly
replaced it. Which is nonsense. And yes, I know FTP very well as I rewrote
the CERN Libwww implementation of FTP. It is less efficient than HTTP and
less reliable. The only thing going for it is that HTTP does not support
file upload to a directory, it only supports PUT and POST which are not
quite the same (and yes, I know those, I was the first person to use them
and spot the fact Tim's spec was utterly broken).

​SSH and rsync are much better tools for file upload. ​Unfortunately, 1&1
does not support them to a Windows host and I don't plan to switch to their
UNIX plan when I switch.

​If people want to indulge in nostalgia about the good old days, fine. Just
accept the fact that people are going to point out that they were not good
at all.​

---

From: Joe Touch <[email-masked]>
Date: Thu, 5 Oct 2017 05:19:10 -0700

On 10/4/2017 1:45 PM, Phillip Hallam-Baker wrote:
> FTP is pretty much the only protocol we use today where the most up to
> date version of the base RFC is still has a three digit number. 
> ​

TCP, IPv4, UDP, ARP, ICMPv4, telnet are also in that antiquated class.

Joe

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 5 Oct 2017 10:37:41 -0400

UDP and TCP are the only ones that are not obsolete.

IPv4 is replaced by IPv6. Telnet is obsoleted by SSH, etc. ICMPv4 to ICMPv6.

I think we should just get rid of ARP. People chatter about 'IP end to end'
yet the local network is unmanageable because there are two levels of
addressing and only one of them is visible.

I would like to replace my Ethernet hubs with IP switches that simply
ignore the ethernet MAC. I neither need nor want support for appletalk,
Novell or Chaosnet.

There is no need for an address at the link layer, it is a piece of wire.

Reducing the IEEE layer to the hardware and physical properties is quite
practical. All it would take is a little bit of co-ordination. It is
essentially STARTTLS at the link layer.

* Assign one MAC address for use as a version number.

* When a pure IP device plugs into a pure IP switch, it begins by
announcing itself with the reserved MAC address. If the switch returns the
reserved response, the device knows it can upgrade. Otherwise it defaults
back to legacy mode.

* A Pure IP session begins with a handshake, cryptographic authentication
and negotiation of the link layer encryption. Because you should always
encrypt when you can.

* Once the handshake is complete, packets moving from one pureIP device to
another across the switch are only limited by the IP maximum size, not the
1500 or 9000 byte limits of yore.

* The switch performs ingress filtering so that a device can only source
packets that it has been authorized so source. It also maintains logs of
recent activity so that the source of network issues can be identified
immediately. Current networks are a nightmare because one faulty process on
one machine can saturate a hub.

The advantages of this would be

* Higher throughput due to maxsize frames and no ethernet frame header

* Simplified administration. Packets only route via deterministic IP
tables. There are no heuristics, no broadcast packets mucking everything
up. The causes of network storms are logged and traceable (by an
authenticated, authorized admin).

* Security. Packets are encrypted at the link layer. This defeats traffic
analysis attacks on traffic only encrypted at the transport layer or higher.

---

From: Joe Touch <[email-masked]>
Date: Thu, 5 Oct 2017 08:49:29 -0700

> On Oct 5, 2017, at 7:37 AM, Phillip Hallam-Baker <[email-masked]> wrote:
> 
> IPv4 is replaced by IPv6.

IPv4 is not deprecated by IPv6. There are no “obsoleted by” marks on 791. 

> Telnet is obsoleted by SSH, etc. ICMPv4 to ICMPv6.

See above - same comment. These are both in active use and not deprecated or obsoleted by anything. 

> I think we should just get rid of ARP.

We can’t until we get rid of IPv4. 

> People chatter about 'IP end to end' yet the local network is unmanageable because there are two levels of addressing and only one of them is visible.
> 
> I would like to replace my Ethernet hubs with IP switches that simply ignore the ethernet MAC.

Yet we have TRILL, which is doing some of the opposite (pushing IP style forwarding support into Ethernet).

> I neither need nor want support for appletalk, Novell or Chaosnet.

Perhaps, but you do use Ethernet all the time right now or things that look very similar at the link layer (bluetooth, USB, etc). 

> There is no need for an address at the link layer, it is a piece of wire. 

Point to point wires don’t need addresses but shared links or subnets that emulate shared links do need addresses. 

And if you ditch the Ethernet, where do you get the low order bits of your IPv6 address?
> 
> 
> Reducing the IEEE layer to the hardware and physical properties is quite practical. All it would take is a little bit of co-ordination. It is essentially STARTTLS at the link layer.
> 
> 
> * When a pure IP device plugs into a pure IP switch, it begins by announcing itself with the reserved MAC address. If the switch returns the reserved response, the device knows it can upgrade. Otherwise it defaults back to legacy mode.
> 
> * A Pure IP session begins with a handshake, cryptographic authentication and negotiation of the link layer encryption. Because you should always encrypt when you can.
> 
> * Once the handshake is complete, packets moving from one pureIP device to another across the switch are only limited by the IP maximum size, not the 1500 or 9000 byte limits of yore. 
> 
> * The switch performs ingress filtering so that a device can only source packets that it has been authorized so source. It also maintains logs of recent activity so that the source of network issues can be identified immediately. Current networks are a nightmare because one faulty process on one machine can saturate a hub.
> 
> 
> * Simplified administration. Packets only route via deterministic IP tables. There are no heuristics, no broadcast packets mucking everything up. The causes of network storms are logged and traceable (by an authenticated, authorized admin).
> 
> * Security. Packets are encrypted at the link layer. This defeats traffic analysis attacks on traffic only encrypted at the transport layer or higher.

There are complexity costs to giving up on shared links or emulations thereof. We’re not there yet at all. 

Joe

>>> FTP is pretty much the only protocol we use today where the most up to date version of the base RFC is still has a three digit number.  ​
>>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 5 Oct 2017 13:20:58 -0400

> And if you ditch the Ethernet, where do you get the low order bits of your

​SHA-512 (Device_Public_Key)

As it should have been f​