Unable to access the crates.io website through cURL or pragmatically without extra HTTP headers

[JeffResc]

Current Behavior

Unable to access the crates.io website through cURL or pragmatically without extra HTTP headers. Returns a 404 error, for example:

# curl https://crates.io/crates/rand
{"errors":[{"detail":"Not Found"}]}

Expected Behavior

I expect the request to respond with status 200 and the page contents

Steps To Reproduce

Run the following command on the command line:

curl https://crates.io/crates/rand

Environment

• Browser: curl 7.68.0
• OS: 20.04.4 LTS (Focal Fossa)

Anything else?

Please see Kovah/LinkAce#465 for more details.

[carols10cents]

Duplicate of #788. Please search existing issues to see if your problem has been reported before. Thanks!