Subject: [RFC 959] FTP in ASCII mode

From: "Sandeep Srivastava" <[email-masked]>
Date: Mon, 20 Feb 2006 18:54:16 +0530

Hi,

RFC 959, says that FTP supports two modes to transfer files -- ASCII and
Binary. Now, if I have a UTF-8 (or any other encoding) encoded text file,
containing, say Japanese characters, would it be correct to choose the
transfer mode as ASCII? And if I choose the transfer mode as ASCII, is there
a possibility that the content of the file on the receiving end is garbled?

In a nutshell, does ASCII mode means that only ASCII encoded text files are
supported?

Thanks,
Sandeep.
_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Peter Dambier <[email-masked]>
Date: Mon, 20 Feb 2006 17:25:14 +0100

Once upon a time the used to be computers speaking ASCII or EBCDIC. The
ASCII computers where unix mostly. The EBCDIC where mainframes. ASCII
mode meant that EBCDIC card decks with fixed lenght of mostly 80 characters
per line would be translated into variable lenght records terminated by
CR/LF before sending them. Also they would be translated from 8 bit
EBCDIC code to 7 bit ASCII code. Unix would have to replace their single
LF character by a CR/LF sequence before sending. The result could be
punched to 7 bit punched tapes automatically adding a parity bit to the
tape. Or you could print it directly on an ASR-33 terminal.

Binary means take it as it is. Dont touch the formatting or character
set.

As the ASR-33 terminal did not know UTF-8 it is not a good idea to use
ASCII mode for UTF-8. But you can send it only to systems who understand
UTF-8.

Regards
Peter and Karin Dambier

Sandeep Srivastava wrote:
> Binary. Now, if I have a UTF-8 (or any other encoding) encoded text 
> file, containing, say Japanese characters, would it be correct to choose 
> the transfer mode as ASCII? And if I choose the transfer mode as ASCII, 
> is there a possibility that the content of the file on the receiving end 
> is garbled?
> 
> In a nutshell, does ASCII mode means that only ASCII encoded text files 
> are supported?
> 
> Thanks,
> Sandeep.
> 
> 
> ------------------------------------------------------------------------
> 

-- 
Peter and Karin Dambier
The Public-Root Consortium
Graeffstrasse 14
D-64646 Heppenheim
+49(6252)671-788 (Telekom)
+49(179)108-3978 (O2 Genion)
+49(6252)750-308 (VoIP: sipgate.de)
mail: [email-masked]
mail: [email-masked]
http://iason.site.voila.fr/
https://sourceforge.net/projects/iason/

---

From: John C Klensin <[email-masked]>
Date: Mon, 20 Feb 2006 11:47:27 -0500

--On Monday, 20 February, 2006 18:54 +0530 Sandeep Srivastava
<[email-masked]> wrote:

> Hi,
> 
> RFC 959, says that FTP supports two modes to transfer files --
> ASCII and Binary. Now, if I have a UTF-8 (or any other
> encoding) encoded text file, containing, say Japanese
> characters, would it be correct to choose the transfer mode as
> ASCII? And if I choose the transfer mode as ASCII, is there a
> possibility that the content of the file on the receiving end
> is garbled?

Yes, it is quite likely.
 
> In a nutshell, does ASCII mode means that only ASCII encoded
> text files are supported?

Traditionally, there is a small ambiguity about ASCII mode.

If a server receives (and accepts) an ASCII mode request, RFC
959  requires it to be sure that its internal character set is
converted to ASCII and that the file be transmitted in NVT
(Network Virtual Terminal) form, i.e., with CRLF line endings
and 7bit ASCII characters right-justified in octets.   A strict
RFC 959 implementation might garble UTF-8 octets with the
high-order bit turned on but is arguably not required to do so.

You may want to take a look at RFC 2640 for a more extensive
discussion, and standards-track approach, to these issues.  I
don't know how widely it has been implemented; perhaps others
can help in that regard or, if you explore it, you can tell us.

     john

---

From: John C Klensin <[email-masked]>
Date: Mon, 20 Feb 2006 12:40:10 -0500

--On Monday, 20 February, 2006 17:25 +0100 Peter Dambier
<[email-masked]> wrote:

> Once upon a time the used to be computers speaking ASCII or
> EBCDIC. The ASCII computers where unix mostly.
>...

Actually, Peter, at the time FTP was designed, the "ASCII
computers" were mostly PDP-10s running Tenex or ITS, plus less
than a handful of Multics machines.  Unix wasn't really a major
presence on the network yet.   And that is important because the
PDP-10 and Multics machines were both 36-bit environments,
neither of which stored 7bit ASCII in octets.  The PDP-10
normally used five characters per word with the ASCII in 7 bits
(a very hard environment for UTF-8 or even "Latin-1") and
Multics normally stored ASCII right-justified in a nine-bit
field with the two leading bits set to zero).   So "convert to
network ASCII" was a non-trivial operation for almost everyone
until ASCII-native 32 bit machines started to become prominent
on the network -- among other things we needed it to get back
and forth between Multics, ITS, and Tenex systems all of which
were more or less ASCII-based.

You are correct about the EBCDIC character conversions.  I've
lose my memory about the state of "virtual card decks" at the
time, but my vague recollection is that text files of the type
that were likely to be transmitted over the network were at
least as likely to stored in files with variable length records
(lengths determined by counts, rather than character-delimiters)
than as fixed-length 80 (or 72) records.

> Or you could print it directly on an ASR-33 terminal.

My (also vague) recollection is that the ASR-33 terminal was
upper-case-only and hence could not fully support ASCII.  I do
remember (fondly except for the racket and speed) some KSR-38s
and maybe ASR-38s that were ASCII devices.
 
> As the ASR-33 terminal did not know UTF-8 it is not a good
> idea to use ASCII mode for UTF-8. But you can send it only to
> systems who understand UTF-8.

Sandeep's question raises another interesting issue.  I just
went back and reread RFC 2640.   It does not seem to address the
"TYPE A" issue at all.  It does say (Section 2, paragraph 1)
"Clients and servers are, however, under no obligation to
perform any conversion on the contents of a file for operations
such as STOR or RETR", which I would take to imply that it
anticipates I18N FTP operations to be entirely binary ("TYPE I")
although that is not explicit.

Whether the characters in use are UTF-8 or not, we've still got
that issue with line-endings.   

Based on the many things we have learned about
internationalization in the last half-dozen years --
demonstrated by the changes from Unicode 2.0 through 5.x,
introduction of IDNA, the "LTRU" work on language tagging, a
growing feeling in some quarters that RFC 2277 may have been a
bit naive in some ways-- it is probably time to revisit 2640.
Not only may some of its requirements not be quite right, but it
may be time to invent a "TYPE U" that would process and transmit
UTF-8 on the same basis that RFC 959's "TYPE A" does for ASCII:
mandatory conversion to that form if needed and CRLF
line-endings.

     john

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 21 Feb 2006 12:03:13 +0900

> although that is not explicit.

As for Japanese processing, UTF-8 is not visible by users and on
the network, because UTF-8 is not only useless but also harmful.

Instead, ISO-2022-JP, ShiftJIS and EUC are the major character sets.
Some ftp implementations does assume (sometimes depending on environment
 variables) network character code ShiftJIS or EUC and perform appropriate
conversions, which garbles UTF-8.

On the other hand, if you use ISO-2022-JP, which is 7 bit pure and ASCII
compatible (in a sense, it is pure ASCII), we can safely use ASCII mode
of vanilla ftp and there is no confusion as long as we are in ASCII
environment.

Similar encoding can be profiled using ISO 2022 to obtain a fully
internationalized, 7 bit pure, ASII compatible character encoding.

The only problem for RFC2460 was that it does not need MIME for
charset and 8bit extension that it makes it clear that MIME is
useless.

Note that long term state maintainance of full ISO 2022 is not
more complex than that of UTF-8. Note also that, carefully profiled
ISO 2022, such as ISO-2022-JP, requires state maintainance a lot
simpler than that of UTF-8.

> that issue with line-endings.   

Line-ending issues of any ISO 2022 based encoding are just as simple
as those of ASCII.

							Masataka Ohta

---

From: "Sandeep Srivastava" <[email-masked]>
Date: Tue, 21 Feb 2006 12:53:32 +0530

First of all thanks to everybody for the response.

I knew that a FTP transfer in ASCII mode does EOL and EOF conversions based
on the OS of the receiving system. And I very much expected my UTF-8 encoded
file to get garbled when I FTPied it in ASCII mode. But guess what, it was
not garbled on the receiving system. Maybe I was lucky, or maybe its because
UTF-8 is backward compatible with ASCII. But then, as ASCII is purely
7-bits, the FTP in ASCII mode should have corrupted the UTF-8 encoded file,
because UTF-8 is 8-bits.

Moreover, in ASCII code page, code point 13=CR and code point 10=LF, but
that might not be the case in every other code page. Hence the EOL
conversion (in FTP ASCII mode) might corrupt that text file if it is encoded
using a non-ASCII encoding. And what about handling the Unicode NewLine
characters? Anyway...

After reading all the wonderful replies, my conclusion is, even though my
FTP client/server handled the UTF-8 encoded text file (which BTW contained
Devanagri characters) correctly, there is a possibility that a text file,
encoded in an encoding other than ASCII runs a risk of being corrupted when
FTPied in ASCII mode. Therefore, always use ASCII mode to transfer only
ASCII encoded files, and Binary mode to transfer non-ASCII encoded files.

I was wondering why isn't there something like a "Text" mode for FTPing text
files, which could handle text files encoded using any encoding available in
this world, and then, the FTP client/server still does the EOL and EOF
conversions properly?

Thanks,
Sandeep.

---

From: John C Klensin <[email-masked]>
Date: Tue, 21 Feb 2006 03:34:27 -0500

--On Tuesday, 21 February, 2006 12:53 +0530 Sandeep Srivastava
<[email-masked]> wrote:

> I knew that a FTP transfer in ASCII mode does EOL and EOF
> conversions based on the OS of the receiving system.

No, it doesn't.  That was part of the point.  It does no EOF
conversions at all.   The command and data channels were
separated for several reasons, but the desire to stay out of the
EOF business was an important one.  And the server is required
to convert whatever line-end convention it uses to CRLF, and any
characters it uses to ASCII, and transmit that over the wire.
If the client then converts from CRLF and ASCII to some local
convention, that is its business, not that of the protocol.  In
other words, there are, at most, conversions to and from CRLF
and ASCII. There are no FTP-specified conversions based on the
properties of the receiving system.  

> And I
> very much expected my UTF-8 encoded file to get garbled when I
> FTPied it in ASCII mode. But guess what, it was not garbled on
> the receiving system. Maybe I was lucky, or maybe its because
> UTF-8 is backward compatible with ASCII. But then, as ASCII is
> purely 7-bits, the FTP in ASCII mode should have corrupted the
> UTF-8 encoded file, because UTF-8 is 8-bits.

"Should have corrupted" is what I referred to as an ambiguity in
my note.   First of all, because of the robustness principle,
you can never guarantee that bad things will happen when they
might -- proper implementation of protocols around her often
argues for never trashing a string because one can or because a
correct string wouldn't have the problem.

So, in practice, if an FTP server was implemented on an ASCII
system that used the "right justified in octets" model but with
LF as line-end, the authors might have well said "the character
codes don't need any conversion for ASCII mode, we just need to
implement conversion to CRLF".  If they had done that, and UTF-8
(or ISO 8859 Latin-1 or...) were added to the system, those CCSs
would go through nicely in ASCII mode, with the right
line-endings.  Substantially the same thing would occur, as
Ohta-san points out, with many of the ISO 2022-based encodings
of non-ASCII characters: completely safely with some of them and
at least as safely as UTF-8 with the others although, as with
UTF-8, the claim of strict ASCII would be technically false.
Now that wouldn't happen with a system that was natively EBCDIC,
or ASCII stored in seven bit chunks without padding, etc.: those
systems would need to do real conversions to get to network
ASCII and, if you thought you were getting UTF-8 over them, you
would be in big trouble.

> Moreover, in ASCII code page, code point 13=CR and code point
> 10=LF, but that might not be the case in every other code
> page. Hence the EOL conversion (in FTP ASCII mode) might
> corrupt that text file if it is encoded using a non-ASCII
> encoding. And what about handling the Unicode NewLine
> characters? Anyway...

Again, there is no conversion in the FTP protocol to local
character set, only to (and, outside the protocol but common in
client implementations) conversation to network ASCII with its
CRLF line endings.

> After reading all the wonderful replies, my conclusion is,
> even though my FTP client/server handled the UTF-8 encoded
> text file (which BTW contained Devanagri characters)
> correctly, there is a possibility that a text file, encoded in
> an encoding other than ASCII runs a risk of being corrupted
> when FTPied in ASCII mode. Therefore, always use ASCII mode to
> transfer only ASCII encoded files, and Binary mode to transfer
> non-ASCII encoded files.

Yes, that is probably wise guidance.  However, if you transfer
textual materials in binary (Image) mode, you also need to be
sure that you have programs available on the receiving host to
change line-end conventions from whatever the server uses
internally to whatever the client system uses.

> I was wondering why isn't there something like a "Text" mode
> for FTPing text files, which could handle text files encoded
> using any encoding available in this world, and then, the FTP
> client/server still does the EOL and EOF conversions properly?

For starters, because it would require that every FTP server
support at least the several thousand coded character sets in
the world.  Even for end of line, there are significantly more
different conventions than you seem to think there are.
"Convert from whatever we use as text here to a single standard
form, and then let the recipient sort out conversion from the
standard form to its preferred local form" is much more
plausible -- it requires the server to support one type of
conversion, not thousands, and the client to support one type of
conversion, not thousands.  In the early 1970s, the appropriate
standard form for transmission was network ASCII (including both
"right justified in eight bits" and CRLF).  Today, it is
probably UTF-8 with CRLF (although I sympathize with Ohta-san's
desire to be able to transmit 2022-based systems in canonical
form) and I think we should be considering that TYPE.  But ideas
about universal converters make both bad protocol design and bad
implementations.

    john

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 21 Feb 2006 10:41:13 +0100

John C Klensin wrote:

> I've lose my memory about the state of "virtual card decks"
> at the time

VM DEV PUN ?  All I recall about it is that I've (ab)used it
for some kind of "local mail" between different VMs (users).

Before this thread here it never occured to me that "ASCII"
in ftp could be about more than "translate from/to local EOL
convention".  So far I got away with "always use 'binary'
for ftp" disabling all attempts to be smart on the ftp-side,
and for a server option cp=NONE (don't assume a codepage for
file names) was what I wanted.  

Admittedly that's I18N from hell, "just don't", but IMO it's
not the time for worldwide upgrades and ftp-IRIs.  Let those
old ftp servers die in peace.  We have http and MIME for the
tricky cases.
                       Bye, Frank

---

From: "Sandeep Srivastava" <[email-masked]>
Date: Tue, 21 Feb 2006 16:28:02 +0530

Thanks John. Please see my response in-line...

On 2/21/06, John C Klensin <[email-masked]> wrote:
> EOF business was an important one.

Right. I understand the command/data channel part -- i.e. instead of sending
the EOF as a data, it is sent as a command, and the receiver can then use
the OS specific EOF. The overall affect to the end user is as if both EOL
and EOF are converted to the receiving OS defaults.

And the server is required
> characters it uses to ASCII, and transmit that over the wire.

I don't understand this point very well. Does it mean that as per the FTP
RFC the server reads 8-bits at a time, and sets the most significant bit to
zero (because ASCII is 7-bits) before transmitting it in ASCII mode? If this
is the case, then how did a UTF-8 encoded file containing Devanagri
characters (i.e. characters greater than 7F) got FTPied over (and back)
correctly in ASCII mode.

If not, -- i.e. it does not sets the msb to zero, then how does ASCII mode
differs from Binary mode?

Scenario:
I am using WS-FTP pro as the client on my windows 2000 machine, to FTP to
and back from a Solaris box (acting as FTP server).

Thanks,
Sandeep.

---

From: Peter Dambier <[email-masked]>
Date: Tue, 21 Feb 2006 14:04:50 +0100

Sandeep Srivastava wrote:
> On 2/21/06, *John C Klensin* <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
> 
> 
>     < [email-masked]
>     <mailto:[email-masked]>> wrote:
> 
> Right. I understand the command/data channel part -- i.e. instead of 
> sending the EOF as a data, it is sent as a command, and the receiver can 
> then use the OS specific EOF. The overall affect to the end user is as 
> if both EOL and EOF are converted to the receiving OS defaults.
> 
> 
> I don't understand this point very well. Does it mean that as per the 
> FTP RFC the server reads 8-bits at a time, and sets the most significant 
> bit to zero (because ASCII is 7-bits) before transmitting it in ASCII 
> mode? If this is the case, then how did a UTF-8 encoded file containing 
> Devanagri characters (i.e. characters greater than 7F) got FTPied over 
> (and back) correctly in ASCII mode.
> 
> If not, -- i.e. it does not sets the msb to zero, then how does ASCII 
> mode differs from Binary mode?
> 
> Scenario:
> I am using WS-FTP pro as the client on my windows 2000 machine, to FTP 
> to and back from a Solaris box (acting as FTP server).
> 
> Thanks,
> Sandeep.
> 

Sending ASCII from DOS/Windows means nothing to do for DOS, replace
CR/LF by LF only before storing on Solaris.

Receiving ASCII on DOS/Windows means for the Solaris box to replace
the lone LF by CR/LF before sending. If both DOS/Windows and Solaris
are lazy enough not to mask the 0x80 bit, then you are lucky.

Problems could arise when you have character sequences including
printable CRs or LFs. You will only find them when you least
suspect them :)

I remember having sent umlauts (ae=ä) (oe=ö) (ue=ü) between DOS and
Windoes. They do use different character sets. The result looked
nasty. If Solaris and Windows use the same character set, the same
printer? you could be lucky again.

Peter