some website requests User-Agent field in headers

[fuwaraido]

I found some website like "sony.co.jp" requests User-Agent field in headers when I try to fetch their robots.txt. Unless they returns 403 Forbidden status code and RobotTextParser fails to read robots.txt. Currently there is no way to specify headers so I propose add headers arguments in constructor.

[the-knights-who-say-ni]

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for your contribution, we look forward to reviewing it!

[corona10]

@fuwaraido
Please leave an issue on bugs.python.org.
Core developers will discuss your issue :)

[eamanu]

I think is a interest patch. I can open a issue on bpo.

@fuwaraido don't forget CLA

[eamanu]

IMO you should create the PR to master

[eamanu]

https://bugs.python.org/issue38067

[matrixise]

Hi @fuwaraido

You have not signed the CLA on the bug tracker, it's a requirement for us if we want to process your PR. Please, could you do the steps and re-open your PR.

Thank you,