Missing query params in url when params option is set

[yanyongyu]

In version 0.28.0, query params are never merged.

import httpx

client = httpx.Client()
request = client.build_request(
    "GET", "https://httpbin.org/get?param=default", params={"foo": "bar"}
)
print(request.url)  # This will be `https://httpbin.org/get?foo=bar`

The query string in the request url will be ignored. It seems this is caused by the pr #3364. Params are replaced instead of merged when parsing.

httpx/httpx/_urlparse.py

Line 301 in 15e21e9

query = kwargs.get("query", url_dict["query"])

Related discussion #3428

[tomchristie]

Params are replaced instead of merged when parsing.

This is exactly the behavior I would expect.
As I see it, the simplification in #3364 introduced a bugfix that we haven't documented in the release.

[yanyongyu]

I know providing both params and query in url is a bad practice. But this can be counterintuitive in some situations like:

import httpx

client = httpx.Client(params={"foo": "bar"})
request = client.build_request(
    "GET", "https://httpbin.org/get?param=default"
)
print(request.url)  # This will be `https://httpbin.org/get?foo=bar`

The params are provided in the client options. This will also replace the query in the url.

May be a warning can be raised when the replaced query is not empty?

httpx/httpx/_urlparse.py

Line 301 in 15e21e9

query = kwargs.get("query", url_dict["query"])

[tomchristie]

Related issue... Client(params=...) is a not-good idea.

But this can be counterintuitive in some situations like

That is surprising behaviour, yes. I'd consider that a bug.

To the extent that we do currently support Client(params=...) we should probably? be merging those with any params=... passed by the user. Tho it's a bad pattern in the first place that we ideally could have avoided.

[yanyongyu]

Currently, as a workaround, we merge all the params by ourselves before passing to httpx. The code is clearer logically 😂 .

[mcgfeller]

Broke quite some of my code, so at least a warning and a hint on how best to merge parameters should be issued.

[eliasdorneles]

Beware, this breaks compatibility with Requests, which does merge the parameters:

[zanieb]

It seems like it may be worth restoring the previous behavior for parity with requests? Is there a way we can discourage the "bad pattern" while still merging?

[jhominal]

Personally, I believe that the "merge" behaviour is more intuitive for someone who specifies both a url with query parameters and a params dictionary.

I often encounter the case of a GET web service, that can be implemented by many different actors, who will give me a URL string that I will stick in a configuration.
And then, in order to call that web service, I will need to add some parameters that are specific to the call (a user ID, search parameters, etc.)
With the replace behaviour being the default, this means that, in order to handle the case where the original URL contains a query string, I need to do one of these things:

1. parse the URL and put everything in params so that I do not lose anything (taking in account that httpx query params handling is not necessarily round-tripping for every query string, which can sometimes be important);
2. parse the URL and use the copy_with_merge_params manually;
3. detect whether the URL contains a ? and add a string suffix to the URL based on the parameters I want to add;

No matter the solution we recommend for merging, the params argument seems like a trap: params works only if I can be sure that my URL never has a query string - otherwise I need to do the merging dance systematically.

It would be a more honest API, in that case, if the params argument was removed and we said "you need to add the parameters to the URL in order for them to work". However, we should not do that because that would be a significant usability regression.

[urschrei]

The new behaviour also makes it more difficult to work with link URLs (e.g. prev, next, last) in http responses – I'm surprised it hasn't been brought up before. Now, extra logic is required to convert the query parameters in each link to a dict and then merge that with any other query parameters that might be required. We aren't short of tools to do that in the stdlib, but adding new logic like this requires extensive testing to ensure correctness because there are several corner cases.

[danieldotnl]

I run into this issue as well. Query params were removed from the url, just because I provided an empty dictionary: params = {}

[mcgfeller]

As a grateful user (and not in a position to contribute) to httpx, I can understand any decision made by maintainers.
Nevertheless, I'd like to ask @tomchristie to please explain the decision to close this, given that:

• it changes the behavior from previous releases and breaks code (as shown in the links to issues in other projects), without a warning in the release notes.
• it deviates from requests
• some people consider it a practical (albeit maybe special case) feature (along the lines of Special cases aren't special enough to break the rules. Although practicality beats purity. in the Zen of Python).
• makes it more difficult to provide link headers commonly used in HATEOAS (see @urschrei 's comment above)
• workarounds are somewhat clumsy.

Thanks again for the great httpx and all the effort that go into it.
Martin

[tomchristie]

Simply put...

The behavior of params=... matches browser behavior when submitting an HTML <form ...> element.

The existing docs don't yet describe this clearly.

[jhominal]

I have taken a bit of time to look at HTTP client libraries in Java, .NET, Javascript, Ruby, Python and Go (you can find the raw results in an attachment), and:

• Among 31 HTTP client APIs that were looked at, there are 3 main ways that these client libraries allow for setting the HTTP request URL and query parameters:

  1. Only have a parameter for a URL, generally as a string object (12 libraries, including all client APIs that are part of a standard library);
  2. Allow setting the URL and request query parameters using a Builder design pattern (7 libraries, only in statically typed languages);
  3. Have parameters for both URL and params objects (generally specified using the equivalent of a dict) (11 libraries, mostly in dynamically typed languages);

• httpx has the same API pattern as the third category. However, all of the 11 libraries that follow that API pattern implement a merge of the query parameters. In other words, I want to insist that the choice at issue here is really unconventional, not only among Python HTTP libraries, but among all dynamically typed languages.

http-libraries.md

[urschrei]

submitting an HTML <form ...> element.

Apart from anything else, an HTTP request library is not a web browser, and isn't subject to the same constraints. I don't recall an argument that their behaviour should be consistent in this respect ever having been made, and I can't see how it would be justified.