w3link's cache didn't handle accept: application/vnd.ipld.raw correctly

[Xuanwo]

Hi, I'm the maintainer of OpenDAL: A rust lib for accessing data freely, painlessly, and efficiently. OpenDAL added ipfs support recently. During our test, we found that w3link didn't cache the correct data if the accept header is set.

For example:

QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ is a directory of opendal's test data: https://ipfs.io/ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ

• Without accept: application/vnd.ipld.raw: gateway should return the generated DirIndex.
• With accept: application/vnd.ipld.raw: gateway should return the raw block.

Background: OpenDAL use accept: application/vnd.ipld.raw to get the raw block of the directory for all child links. In this way, we don't need to parse the returning HTML.

However, w3s.link returns raw block even I didn't set accept header:

Shell Log

:( curl -L https://w3s.link/ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ | cat
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   488  100   488    0     0    824      0 --:--:-- --:--:-- --:--:--     0
───────┬───────────────────────────────────────────────────────────────────────────────
       │ STDIN
───────┼───────────────────────────────────────────────────────────────────────────────
   1   │ ^R2
   2   │ "^R �T�xj^^ :^Z+c�$���^?!�ș�bs^Fh,͗��c^R
   3   │ normal_dir^X<^R5
   4   │ "^R ߀^Q���^Xe�h��@{�+<)!^P��N�Ie L�c�^B^R^Knormal_file^X��^P^R4
   5   │ "^R )�~�b$^P��zv��'1^Y^QQ�^L^C^B�^Z��γ�b^R
   6   │ ontime.csv^X��^E^R7
   7   │ "^R ��^@^]?�ҹh?�2dD<�IY�~d�d��G�qJ^R^Nontime.csv.bz2^X�5^R6
   8   │ "^R �5�����/0X�l��Sc   -�C^Bڋh�tr��O^R^Montime.csv.gz^X�A^R7
   9   │ "^R �!��Ec��Qj�J7�^R�^Q^M�1]��S^XA^G�o�l�^R^Nontime.csv.zst^X�C^RI
  10   │ "^R �T�xj^^ :^Z+c�$���^?!�ș�bs^Fh,͗��c^R!special_dir  !@#$%^&*()_+-=;'><,?^X<^RL
  11   │ "^R ߀^Q���^Xe�h��@{�+<)!^P��N�Ie L�c�^B^R"special_file  !@#$%^&*()_+-=;'><,?^X��^P
  12   │ ^B^H^A
───────┴───────────────────────────────────────────────────────────────────────────────

[vasco-santos]

@Xuanwo w3link does not support yet format raw and format car queries. We will look into supporting this soon

[Xuanwo]

I'm not familiar with js. Maybe we can add &format=raw in query or accept header if input request contains accept: application/vnd.ipld.raw header while redirecting? I'm willing to contribute to this patch.

[Xuanwo]

The problem I know is Cloudflare's IPFS gateway doesn't support format raw.

[vasco-santos]

Hey @Xuanwo

Thanks for your interest. We probably want to whitelist the gateways that support this and change over time once they support.

If you would like to submit a patch to support this, the repo for it would be the reads pipeline:

• https://github.com/web3-storage/reads/blob/main/packages/edge-gateway/src/gateway.js#L84 here is our gateway racer
• probably recommend setting up a new racer for the gateways that support this for now https://github.com/web3-storage/reads/blob/main/packages/edge-gateway/src/env.js#L32 + https://github.com/web3-storage/reads/blob/main/packages/edge-gateway/wrangler.toml#L34