Skip to content

api v1 starts to require Accept HTTP header for downloading crates #466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
semarie opened this issue Nov 12, 2016 · 4 comments
Closed

api v1 starts to require Accept HTTP header for downloading crates #466

semarie opened this issue Nov 12, 2016 · 4 comments

Comments

@semarie
Copy link

semarie commented Nov 12, 2016

While using ftp tool from OpenBSD, I found that recently, https://crates.io/api/v1/crates behave differently if Accept: HTTP header is present in the request or not. The ftp tool (which is able to download document from http server) doesn't provide the optional Accept header.

I suspect c9e1c37 to be responsible of this behaviour.

I will use curl, which is more usual tool in Linux world for demonstrate the problem.

A request without Accept: */* serves an HTTP document:

$ curl -v -H 'Accept:' -L https://crates.io/api/v1/crates/aho-corasick/0.5.2/download | file -
*   Trying 50.19.254.21...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to crates.io (50.19.254.21) port 443 (#0)
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [89 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3453 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Alex Crichton; CN=crates.io
*  start date: Nov  7 00:00:00 2016 GMT
*  expire date: Nov 15 12:00:00 2017 GMT
*  subjectAltName: host "crates.io" matched cert's "crates.io"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> GET /api/v1/crates/aho-corasick/0.5.2/download HTTP/1.1
> Host: crates.io
> User-Agent: curl/7.51.0
> 
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0< HTTP/1.1 200 OK
< Connection: keep-alive
< Server: nginx
< Date: Sat, 12 Nov 2016 06:03:58 GMT
< Content-Type: text/html
< Content-Length: 2521
< Last-Modified: Sun, 18 Jan 1970 02:48:20 GMT
< Set-Cookie: cargo_session=--3Mz45cH3itfVkmY41+nVjWRm9+0%3D; HttpOnly; Secure; Path=/
< Strict-Transport-Security: max-age=31536000
< Via: 1.1 vegur
< 
{ [2521 bytes data]
* Curl_http_done: called premature == 0
100  2521  100  2521    0     0    877      0  0:00:02  0:00:02 --:--:--   878
* Connection #0 to host crates.io left intact
/dev/stdin: HTML document text

Whereas with the HTTP header, it is the crate that is served:

$ curl -v -L https://crates.io/api/v1/crates/aho-corasick/0.5.2/download | file -
*   Trying 50.19.254.21...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to crates.io (50.19.254.21) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [89 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3453 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Alex Crichton; CN=crates.io
*  start date: Nov  7 00:00:00 2016 GMT
*  expire date: Nov 15 12:00:00 2017 GMT
*  subjectAltName: host "crates.io" matched cert's "crates.io"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> GET /api/v1/crates/aho-corasick/0.5.2/download HTTP/1.1
> Host: crates.io
> User-Agent: curl/7.51.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Connection: keep-alive
< Server: nginx
< Date: Sat, 12 Nov 2016 06:05:59 GMT
< Transfer-Encoding: chunked
< Set-Cookie: cargo_session=--3Mz45cH3itfVkmY41+nVjWRm9+0%3D; HttpOnly; Secure; Path=/
< Location: https://crates-io.s3-us-west-1.amazonaws.com/crates/aho-corasick/aho-corasick-0.5.2.crate
< Strict-Transport-Security: max-age=31536000
< Via: 1.1 vegur
< 
* Ignoring the response-body
{ [5 bytes data]
* Curl_http_done: called premature == 0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host crates.io left intact
* Issue another request to this URL: 'https://crates-io.s3-us-west-1.amazonaws.com/crates/aho-corasick/aho-corasick-0.5.2.crate'
*   Trying 54.231.236.8...
* TCP_NODELAY set
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to crates-io.s3-us-west-1.amazonaws.com (54.231.236.8) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [89 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2703 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=*.s3-us-west-1.amazonaws.com
*  start date: Jul 18 00:00:00 2016 GMT
*  expire date: Oct 26 12:00:00 2017 GMT
*  subjectAltName: host "crates-io.s3-us-west-1.amazonaws.com" matched cert's "*.s3-us-west-1.amazonaws.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2
*  SSL certificate verify ok.
> GET /crates/aho-corasick/aho-corasick-0.5.2.crate HTTP/1.1
> Host: crates-io.s3-us-west-1.amazonaws.com
> User-Agent: curl/7.51.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< x-amz-id-2: i6GgYeY6/Gx0HJrpKieEm/M3BsxjTm/4H9HEHn0pvB1PqnPvAh+lV90OOcvg35q7o1qr4bSP45g=
< x-amz-request-id: 70AD6942561E94EC
< Date: Sat, 12 Nov 2016 06:06:01 GMT
< Last-Modified: Sun, 01 May 2016 03:49:13 GMT
< ETag: "08d6e33813055cfa3b6c9bf6c380a74d"
< Accept-Ranges: bytes
< Content-Type: application/x-tar
< Content-Length: 542932
< Server: AmazonS3
< 
{ [8633 bytes data]
  1  530k    1  8633    0     0   5048      0  0:01:47  0:00:01  0:01:46  7862 14  530k   14 78265    0     0  29018      0  0:00:18  0:00:02  0:00:16 37537 27  530k   27  144k    0     0  40168      0  0:00:13  0:00:03  0:00:10 48174 43  530k   43  228k    0     0  50173      0  0:00:10  0:00:04  0:00:06 57756
* Failed writing body (0 != 16384)
* Curl_http_done: called premature == 1
* stopped the pause stream!
 52  530k   52  279k    0     0  54067      0  0:00:10  0:00:05  0:00:05 61140
* Closing connection 1
* TLSv1.2 (OUT), TLS alert, Client hello (1):
} [2 bytes data]
curl: (23) Failed writing body (0 != 16384)
/dev/stdin: gzip compressed data, was "aho-corasick-0.5.2.crate", max compression, from Unix

If I directly point the URL in my browser (firefox), I obtain the HTML document which is "Oops, that route doesn't exist!".
@semarie
Copy link
Author

semarie commented Nov 12, 2016

cc: @tamird

@semarie semarie changed the title api v1 started to required Accept HTTP header for downloading crates api v1 starts to require Accept HTTP header for downloading crates Nov 12, 2016
@onur
Copy link
Member

onur commented Nov 12, 2016 •
edited
Loading

This also started affecting docs.rs. All requests made by hyper returning app/index.html instead of proper API response.

@onur onur mentioned this issue Nov 12, 2016
Closed
onur added a commit to rust-lang/docs.rs that referenced this issue Nov 12, 2016
@onur
Set Accept header for crates.io API requests
6b61f44 
Fixes: #81
Ref: rust-lang/crates.io#466
Ref: rust-lang/crates.io#464
bob-beck pushed a commit to openbsd/ports that referenced this issue Nov 12, 2016
@landryb
Mark temporarly BROKEN, crates.io requires Accept: HTTP header, cf ru…
2b20b7d 
…st-lang/crates.io#466
@semarie
Copy link
Author

semarie commented Nov 12, 2016

According to RFC 7231 (section 5.3.2. Accept):

A request without any Accept header field implies that the user agent
will accept any media type in response.

So if the request doesn't provide Accept HTTP header, it should be traited as equivalent to Accept: */*. I tend to think that #464 (assuming html if no Accept) was wrong: the problem wasn't in crates.io.

@alexcrichton
Copy link
Member

Reverted #464 and we can figure out a separate solution for #163 later

hadronized pushed a commit to hadronized/docs.rs that referenced this issue Nov 30, 2016
@onur
Set Accept header for crates.io API requests
b68687c 
Fixes: rust-lang#81
Ref: rust-lang/crates.io#466
Ref: rust-lang/crates.io#464
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@alexcrichton @onur @semarie