Robots.txt parsing fails when one rule line is invalid

[gyula-lakatos]

Bug report

Bug description:

The robots.txt parsing fails if one line is not parsable from a robots.txt file. I don't think this is valid behavior. Ideally, non-parsable/invalid lines should be skipped. The norobots-rfc says the same too: Implementors should pay particular attention to the robustness in parsing of the /robots.txt file..

  File "/usr/local/lib/python3.11/urllib/robotparser.py", line 123, in parse
    entry.rulelines.append(RuleLine(line[1], False))
                           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/urllib/robotparser.py", line 222, in __init__
    path = urllib.parse.urlunparse(urllib.parse.urlparse(path))
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/urllib/parse.py", line 395, in urlparse
    splitresult = urlsplit(url, scheme, allow_fragments)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/urllib/parse.py", line 500, in urlsplit
    _check_bracketed_host(bracketed_host)
  File "/usr/local/lib/python3.11/urllib/parse.py", line 446, in _check_bracketed_host
    ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ipaddress.py", line 54, in ip_address
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: '[routes.productDetail(product.sku, product.slug)' does not appear to be an IPv4 or IPv6 address

I know [routes.productDetail(product.sku, product.slug) is clearly not a valid URL, but I don't think the whole parsing should error out because of this one line.

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Linked PRs

• gh-111788: Don't treat path in robots.txt as URL in urllib.robotparser #113231
• gh-88375, gh-111788: Fix parsing errors and normalization in robotparser #138502

[sunmy2019]

The feasible solution would be to add a new flag for this function, telling whether the user wants to handle or ignore the error.

[gyula-lakatos]

I'm not sure about that to be honest. The crawl-delay and request-rate parameters are only processed if they are valid (no error handling there). I think the same logic should apply to urls. Also, lines are usually skipped if they are in a wrong order. I'm unsure why these errors specifically would demand a flag.

[aisk]

I think this is not caused by the rule being invalid, but robotparser's bug.

I assume that you are parsing a robot.txt with some thing like this:

Disallow: //[foo]/bar

This is invalid, because '[' is not quoted, but according to the RFC and what current codes do, robotparser is robust enough and can deal with unquoted pathes.

But this case is more special, the //[foo]/bar can be interpreted as an URL with invalid IPv6 host "[foo]" as well. And current codes passed it to urllib.parse.urlparse in c70a6ae, just to remove the empty query string "?" or empty fragment "#", so an exception occurs.

So I think it's better to using a different way to normalize the path.

[serhiy-storchaka]

The issue is more complicated. By applying quote(), unquote(), parse() and unparse() to the rule paths and the URL we lose the difference between the query separator and the percent-encoded ? and lose trailing ? (see #88375 which report the part of the issue). Also, parse() can fail or return incorrect result when trying to parse a path (this issue). #138502 fixes this.

The robotparse module was based on the old document http://www.robotstxt.org/orig.html. Now there is a new standard, RFC 9309 which specifies details. #138502 does not make the robotparse module completely conforming RFC 9309, because some details look unclear and self-contradictory.