api v1 starts to require Accept HTTP header for downloading crates

[semarie]

While using ftp tool from OpenBSD, I found that recently, https://crates.io/api/v1/crates behave differently if Accept: HTTP header is present in the request or not. The ftp tool (which is able to download document from http server) doesn't provide the optional Accept header.

I suspect c9e1c37 to be responsible of this behaviour.

I will use curl, which is more usual tool in Linux world for demonstrate the problem.

A request without Accept: */* serves an HTTP document:

$ curl -v -H 'Accept:' -L https://crates.io/api/v1/crates/aho-corasick/0.5.2/download | file -
*   Trying 50.19.254.21...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to crates.io (50.19.254.21) port 443 (#0)
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [89 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3453 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Alex Crichton; CN=crates.io
*  start date: Nov  7 00:00:00 2016 GMT
*  expire date: Nov 15 12:00:00 2017 GMT
*  subjectAltName: host "crates.io" matched cert's "crates.io"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> GET /api/v1/crates/aho-corasick/0.5.2/download HTTP/1.1
> Host: crates.io
> User-Agent: curl/7.51.0
> 
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0< HTTP/1.1 200 OK
< Connection: keep-alive
< Server: nginx
< Date: Sat, 12 Nov 2016 06:03:58 GMT
< Content-Type: text/html
< Content-Length: 2521
< Last-Modified: Sun, 18 Jan 1970 02:48:20 GMT
< Set-Cookie: cargo_session=--3Mz45cH3itfVkmY41+nVjWRm9+0%3D; HttpOnly; Secure; Path=/
< Strict-Transport-Security: max-age=31536000
< Via: 1.1 vegur
< 
{ [2521 bytes data]
* Curl_http_done: called premature == 0
100  2521  100  2521    0     0    877      0  0:00:02  0:00:02 --:--:--   878
* Connection #0 to host crates.io left intact
/dev/stdin: HTML document text

Whereas with the HTTP header, it is the crate that is served:

$ curl -v -L https://crates.io/api/v1/crates/aho-corasick/0.5.2/download | file -
*   Trying 50.19.254.21...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to crates.io (50.19.254.21) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [89 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3453 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Alex Crichton; CN=crates.io
*  start date: Nov  7 00:00:00 2016 GMT
*  expire date: Nov 15 12:00:00 2017 GMT
*  subjectAltName: host "crates.io" matched cert's "crates.io"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> GET /api/v1/crates/aho-corasick/0.5.2/download HTTP/1.1
> Host: crates.io
> User-Agent: curl/7.51.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Connection: keep-alive
< Server: nginx
< Date: Sat, 12 Nov 2016 06:05:59 GMT
< Transfer-Encoding: chunked
< Set-Cookie: cargo_session=--3Mz45cH3itfVkmY41+nVjWRm9+0%3D; HttpOnly; Secure; Path=/
< Location: https://crates-io.s3-us-west-1.amazonaws.com/crates/aho-corasick/aho-corasick-0.5.2.crate
< Strict-Transport-Security: max-age=31536000
< Via: 1.1 vegur
< 
* Ignoring the response-body
{ [5 bytes data]
* Curl_http_done: called premature == 0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host crates.io left intact
* Issue another request to this URL: 'https://crates-io.s3-us-west-1.amazonaws.com/crates/aho-corasick/aho-corasick-0.5.2.crate'
*   Trying 54.231.236.8...
* TCP_NODELAY set
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to crates-io.s3-us-west-1.amazonaws.com (54.231.236.8) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [89 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2703 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=*.s3-us-west-1.amazonaws.com
*  start date: Jul 18 00:00:00 2016 GMT
*  expire date: Oct 26 12:00:00 2017 GMT
*  subjectAltName: host "crates-io.s3-us-west-1.amazonaws.com" matched cert's "*.s3-us-west-1.amazonaws.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2
*  SSL certificate verify ok.
> GET /crates/aho-corasick/aho-corasick-0.5.2.crate HTTP/1.1
> Host: crates-io.s3-us-west-1.amazonaws.com
> User-Agent: curl/7.51.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< x-amz-id-2: i6GgYeY6/Gx0HJrpKieEm/M3BsxjTm/4H9HEHn0pvB1PqnPvAh+lV90OOcvg35q7o1qr4bSP45g=
< x-amz-request-id: 70AD6942561E94EC
< Date: Sat, 12 Nov 2016 06:06:01 GMT
< Last-Modified: Sun, 01 May 2016 03:49:13 GMT
< ETag: "08d6e33813055cfa3b6c9bf6c380a74d"
< Accept-Ranges: bytes
< Content-Type: application/x-tar
< Content-Length: 542932
< Server: AmazonS3
< 
{ [8633 bytes data]
  1  530k    1  8633    0     0   5048      0  0:01:47  0:00:01  0:01:46  7862 14  530k   14 78265    0     0  29018      0  0:00:18  0:00:02  0:00:16 37537 27  530k   27  144k    0     0  40168      0  0:00:13  0:00:03  0:00:10 48174 43  530k   43  228k    0     0  50173      0  0:00:10  0:00:04  0:00:06 57756
* Failed writing body (0 != 16384)
* Curl_http_done: called premature == 1
* stopped the pause stream!
 52  530k   52  279k    0     0  54067      0  0:00:10  0:00:05  0:00:05 61140
* Closing connection 1
* TLSv1.2 (OUT), TLS alert, Client hello (1):
} [2 bytes data]
curl: (23) Failed writing body (0 != 16384)
/dev/stdin: gzip compressed data, was "aho-corasick-0.5.2.crate", max compression, from Unix

If I directly point the URL in my browser (firefox), I obtain the HTML document which is "Oops, that route doesn't exist!".

[semarie]

cc: @tamird

[onur]

This also started affecting docs.rs. All requests made by hyper returning app/index.html instead of proper API response.

[semarie]

According to RFC 7231 (section 5.3.2. Accept):

A request without any Accept header field implies that the user agent
will accept any media type in response.

So if the request doesn't provide Accept HTTP header, it should be traited as equivalent to Accept: */*. I tend to think that #464 (assuming html if no Accept) was wrong: the problem wasn't in crates.io.

[alexcrichton]

Reverted #464 and we can figure out a separate solution for #163 later