Unreserved characters are escaped when making a HTTP request

[szmarczak]

• Version: >=10.20.1
• Platform: all
• Subsystem: http

What steps will reproduce the bug?

const http = require('http');

const url = new URL('http://httpbin.org/anything?a=~');
url.search = url.searchParams.toString(); // trigger normalization

const request = http.get(url, response => {
    const chunks = [];

    response.on('data', chunk => {
        chunks.push(chunk);
    });

    response.once('end', () => {
        console.log(request._header);
        // Actual:
        // GET /anything?a=%7E
        //
        // Expected:
        // GET /anything?a=~
    });
})

How often does it reproduce? Is there a required condition?

Always.

What is the expected behavior?

GET /anything?a=~

What do you see instead?

GET /anything?a=%7E

Additional information

According to tools.ietf.org/html/rfc3986#section-2.3 tilde is unreserved and should not encoded by URI producers.

sindresorhus/got#1180 (comment)

[bnoordhuis]

The url.searchParams.toString() call is what changes ~ to %7E. I'm not 100% sure but it looks according to spec to me. Firefox behaves the same way, FWIW.

The spec says the application/x-www-form-urlencoded serializer should be applied:

The stringification behavior must return the serialization of the URLSearchParams object’s list.

@himself65 I see you tagged this with confirmed-bug. Can you elaborate?

[himself65]

oh, sry for I didn't look up the spec🤦‍.

and just a little confused me because

[Hamper]

Isn't that bug in URLSearchParams? Because in spec https://tools.ietf.org/html/rfc3986#section-2.3 we see that ~ is unreserved character and

   For consistency, percent-encoded octets in the ranges of ALPHA
   (%41-%5A and %61-%7A), DIGIT (%30-%39), hyphen (%2D), period (%2E),
   underscore (%5F), or tilde (%7E) should not be created by URI
   producers and, when found in a URI, should be decoded to their
   corresponding unreserved characters by URI normalizers.

Also in https://url.spec.whatwg.org/ we see that percent encoding must be used only for code points greater than U+007E (~) but not equal this symbol

[bnoordhuis]

@Hamper https://url.spec.whatwg.org/#concept-urlencoded-byte-serializer is the mapping for individual bytes. ~ must be percent-encoded according to that.

[jasnell]

The URL parser uses the WHATWG URL Standard as the normative reference and not rfc3986 and the two definitely differ with regards to some of the escaping rules.

That said, I do think this is a bug... not necessarily in our impl but in the URL Standard spec itself. Definitely warrants more investigation.

Specifically, look at section https://url.spec.whatwg.org/#url-parsing and search for "query state" ...

[jasnell]

@bnoordhuis ... the section you reference deals specifically with application/x-www-form-urlencoded serialization, which is different from both URL and URLSearchParams. While the URL Standard does provide a specification for application/x-www-form-urlencoded parsing and serialization, it does not require or even recommend it's use for URL parsing and handling of URLSearchParams. They are separate things. The issue opened here is in regards to URL parsing and not application/x-www-form-urlencoded

[bnoordhuis]

@jasnell I think you overlooked the line that says url.search = url.searchParams.toString() - it's the .toString() that encodes to application/x-www-form-urlencoded and that's per spec.

[jasnell]

Ah, yep, there it is. As I said, I don't think this is a bug in our implementation but I do think it's a bug in the spec itself because given the rules defined elsewhere in the doc ~ is handled differently. Let's keep this issue open and get clarification from the URL Standard folks on what the correct behavior should be.

/cc @annevk ... when you get a moment, could you help us resolve this issue? :-)