Fixed the processing of headers.

[KOLANICH]

No description provided.

[florimondmanca]

@KOLANICH Hi, could you provide some context about this pull request?

[KOLANICH]

I tried to remove (not redefine, but remove completely) some default headers, but there was no such a feature in this lib. I have implemented it.

[florimondmanca]

@KOLANICH If you want to remove or otherwise modify headers before sending them over the wire, you can do this:

request = client.build_request(...)

# Modify `request.headers` as seems fit...
request.headers.pop("some-auto-header")

response = client.send(request)

See also: https://www.python-httpx.org/advanced/#request-instances

If I understood your use case correctly this should help you w/o requiring any modifications to HTTPX, so I'll close this tentatively. If that's not the case feel free to reach back. :-)

[KOLANICH]

This is different to requests and hyper. If the project positions itself as a compatible replacement to requests, it should just be compatible to it. If one sets headers as an argument, he shouldn't have to edit them manually.

My requirements about the headers are:

• I should be able to fully control the headers the lib sends.
• I should be able to fully control the order the lib sends headers in.

In requests and hyper I can set the order of all the headers by providing an OrderedDict of Nones (that can be filled later when doing requests using headers arg, but the order will still be enforced, headers with None values are ignored) and can delete an unneeded default header by providing get(..., headers={"User-Agent": None}) (BTW, WTF http libs leaks that I use them and their versions, is it to make it easier for a potentially malicious server to exploit any vulnrs in it and in python itself ?)

• the library mustn't create obstacles for me using it. Using a "manual" interface instead of an interface "for humans" is considered such.

Currently it is easier for me (and, I guess, not only for me) to just use hyper (their former maintainer claimed it is unmaintained and recommended using httpx instead, but it turns out that an "unmaintained" hyper (with TLS validation and working headers API and SOCKS proxy support implemented by me but not merged because it is unmaintained) is still better than "maintained" and "highest quality open-source software" httpx) + requests.

[StephenBrown2]

When creating this PR, it would have been incredibly helpful had you stated what your intention was (to increase requests compatibility), given a use-case (being able to omit None headers and merge them in a different way than is currently implemented, etc), and in general been more descriptive of your changes. A PR with no description is not something that the maintainers will want to review, and your tone has become hostile, thus lowering the motivation to assist you in your goals. That said, I will respond to some of your points below.

---

This is different to requests and hyper. If the project positions itself as a compatible replacement to requests, it should just be compatible to it. If one sets headers as an argument, he shouldn't have to edit them manually.

httpx aims to be broadly compatible with requests, and the high-level API certainly is, with some caveats, one of which you've found. Any known differences are documented here: https://www.python-httpx.org/compatibility/

headers with None values are ignored

This has been a point of discussion for several things including query parameters that has come up several times before, and the httpx developers have decided to not go with ignoring None for various reasons, including security, I believe.

(BTW, WTF http libs leaks that I use them and their versions, is it to make it easier for a potentially malicious server to exploit any vulnrs in it and in python itself ?)

requests does so by default as well. The default user agent is python-requests/{__version__} https://github.com/psf/requests/blob/589c4547338b592b1fb77c65663d8aa6fbb7e38b/requests/utils.py#L808-L814
hyper on the other hand, sends no headers at all by default, and so all header manipulation is "manual".

Currently it is easier for me... (with TLS validation and working headers API and SOCKS proxy support implemented by me but not merged because it is unmaintained) is still better than "maintained" and "highest quality open-source software" httpx) + requests.

Seems like you could simply use your own fork of httpx with this modification and you would have the same ease of use as your own fork of hyper.

[KOLANICH]

Any known differences are documented here: https://www.python-httpx.org/compatibility/

The issue with headers mentioned here is not.

hyper on the other hand, sends no headers at all by default, and so all header manipulation is "manual".

Not quite. There is a dict which is just merged to the provided dict. So I can set a dict with default headers and/or their order first, and second I can merge into it another dict.

requests does so by default as well. The default user agent is python-requests/{version}

It is not very wise to do that. I know, security through obscurity is an inherently flawed approach, but some websites send different content depending on a useragent in a non-ok way.

[sethmlarson]

It is not very wise to do that. I know, security through obscurity is an inherently flawed approach, but some websites send different content depending on a useragent in a non-ok way.

Dropping in to clarify that sending a User-Agent by default is following HTTP spec recommendations. From RFC 7321 Section 5.5.3:

A user agent SHOULD send a User-Agent field in each request unless specifically configured not to do so.

[KOLANICH]

Dropping in to clarify that sending a User-Agent by default is following HTTP spec recommendations. From RFC 7321 Section 5.5.3:

OK (though, IMHO, it is still not OK to follow the flawed specs.)