SECDIR Review draft-koster-rep


Reviewer: Tirumaleswar Reddy
Review result: Ready with Issues



I have reviewed this document as part of the security directorate's

ongoing effort to review all IETF documents being processed by the

IESG..  Document editors and WG chairs should treat these comments

just like any other last call comments.



You may want to discuss the following security threats:



a) Revealing disallowed URIs will make its paths easily discoverable.
However, security by obscurity will not maintain or increase the security
of the content provider (you can refer to
https://datatracker.ietf.org/doc/html/rfc4949).

b) A malicious crawler will not honor the disallow rules and can try to
access the disallowed URIs, it should be mitigated by access control
restrictions. Discuss any other count-measures used to block such malicious
crawlers (like blocking the IP address).

c) Attacks possible on crawlers because of a malicious robots.txt file.


Cheers,

-Tiru