Bad behavior when requesting crate pages with non-browser

[AMDmi3]

When try to fetch crates.io page which works perfectly via browser (say, https://crates.io/crates/bincode), with tools (say, curl), the server returns 404:

% curl -I https://crates.io/crates/bincode
HTTP/1.1 404 Not Found
Connection: keep-alive
Server: nginx
Date: Mon, 20 Nov 2017 19:20:16 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 35
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'self'; connect-src 'self' https://docs.rs https://d19xqa3lc3clo8.cloudfront.net; script-src 'self' 'unsafe-eval' https://www.google-analytics.com https://www.google.com; style-src 'self' https://www.google.com https://ajax.googleapis.com; img-src *; object-src 'none'
Set-Cookie: cargo_session=sJIiNcfM9yvCHoGNENQaO8JrPoTF1c7xuZ6xe/LTieY=; HttpOnly; Secure; Path=/
X-Xss-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=31536000
Via: 1.1 vegur

I've actually hit this problem with repology.org link checker (based on python requests module), which reports all links to crates.io as dead because of this problem. I think the server could behave more correct and return 200 somehow.

Related issue on adding crates.io support to repology: repology/repology-updater#390

[carols10cents]

Closing as a duplicate of #788. If you can change the request headers, passing Accept: text/html should work around this problem.

[AMDmi3]

#788 says it'll always return 200, so it's not a solution, not mentioning that I'd prefer not to add repository specific quirks.

[carols10cents]

I didn't say it was a solution, I said it was a workaround.