Moin!

I am an assigned INT directorate reviewer for draft-koster-rep.
These comments were written primarily for the benefit of the Internet Area
Directors. Document editors and shepherd(s) should treat these comments just
like they would treat comments from any other IETF contributors and resolve
them along with any other Last Call comments that have been received. For more
details on the INT Directorate, see
https://datatracker.ietf.org/group/intdir/about/


While the document technically defines the content of the robots.txt files it could do a better job in describing with examples the semantic of how robots interpret them. Especially in 2.2.1 the notation of "Crawlers MUST find the group that matches the product token exactly" should be better explained. I assume it does not mean being fully equal but instead a substring match in the User-Agent Header, so in the example would also match a http user agent of ExampleBotnet/1.2. Is that understanding correct at least?

Also the examples in 5 seem a lot more arbirtary than what the ROBOTSTXT website has and it should explain all the outcomes, e.g in 5.1  it would allow access to all crawlers and all paths, but the foobot, barbot and bazbot /example/disallowed.gif. An example with a * group would be better and more realistic.

So long
-Ralf