🐛 BUG: fetch in wrangler dev with sveltekit results in SSLV3_ALERT_HANDSHAKE_FAILURE

[maemigh]

Which Cloudflare product(s) does this pertain to?

Wrangler core

What version(s) of the tool(s) are you using?

3.14.0

What version of Node are you using?

v20.7.0

What operating system are you using?

Mac / Linux

Describe the Bug

Using fetch with wrangler dev results in errors. Attempted on both Mac and Linux.

const resp = await fetch(`https://stg.mul-bill.jp/`)

workerd/util/symbolizer.c++:98: warning: Not symbolizing stack traces because $LLVM_SYMBOLIZER is not set. To symbolize stack traces, set $LLVM_SYMBOLIZER to the location of the llvm-symbolizer binary. When running tests under bazel, use `--test_env=LLVM_SYMBOLIZER=<path>`.
workerd/jsg/util.c++:275: error: e = kj/compat/tls.c++:72: failed: OpenSSL error; message = error:10000410:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
error:1000009a:SSL routines:OPENSSL_internal:HANDSHAKE_FAILURE_ON_CLIENT_HELLO
stack: 10160890c 101610940 1027a0294 1027a06dc 10172893c 10172922c 10172a8f4 101707e30 10170e9f0 101724db4 100e6792c 1009b762c 100ed3ce4; sentryErrorContext = jsgInternalError

Please provide a link to a minimal reproduction

No response

Please provide any relevant error logs

No response

[stewarthsoj]

I want to add a similar issue is also happening for me when running Wrangler for a Remix application.

App setup

bun create cloudflare@latest my-remix-app -- --framework=remix

Operating system
macOS Monterey
Version 12.5

Node
20.9.0

Wrangler
3.10.1

[admah]

I believe this is the same issue users are seeing with cloudflare/workers-sdk#3631. These warnings are coming from the workerd runtime in dev mode. We're working to hide those since they are not actionable by users.

[maemigh]

I'm not sure this is the same thing as just hiding the warnings. The ssl errors are causing the rest of the function to fail, so I can only test after deploying to workers where it doesn't fail. Wrangler dev is failing with one of Japan's largest payment processors..

[admah]

@maemigh do you have a repo or some code that we could try and reproduce from? Using basic worker and the fetch you have above, I'm at least able to get a 200 response with no errors in dev console.

wrangler v3.15.0
node v20.7.0

[admah]

@JoshStwrt your issue looks to be a little different. It looks like you might need to specify to --inspector-port to be 57198 as that's not default (https://developers.cloudflare.com/workers/wrangler/commands/#dev)

[maemigh]

@admah I just created a repo showing the error. https://github.com/maemigh/cloudflare-test
Run the build and then
npx wrangler pages dev .svelte-kit/cloudflare, navigate to http://localhost:8788/ja/apply and press the submit button. The fetch will cause the internal server error.

[Nufflee]

I hit this issue myself and spent some time digging into the code to see what's going on.
It turns out the problem is in the TLS cipher list used when running worked locally through wrangler. It appears that it uses the default kj cipher list which only includes ciphers which use ECDH for key exchange. Neither the website I was trying to access nor the aforementioned stg.mul-bill.jp support any ECDHE ciphers (in the case of stg.mul-bill.jp the preferred cipher for TLS1.2 is AES256-GCM-SHA384).
Since my code (which is accessing a website using a DHE cipher) works when deployed to CF, but not locally, it leads me to conclude that there is a discrepancy between the wrangler workerd configuration and that which is used on CF infrastructure when it comes to TLS options, making this a wrangler bug.

cc @admah

EDIT: an absolute minimum reproducible example:

export default {
  async fetch(reques, env, ctx) {
    await fetch('https://stg.mul-bill.jp');
  }
}

Output from `sslscan stg.mul-bill.jp` showing available ciphers

Connected to 210.197.108.215

Testing SSL server stg.mul-bill.jp on port 443 using SNI name stg.mul-bill.jp

  SSL/TLS Protocols:
SSLv2     disabled
SSLv3     disabled
TLSv1.0   disabled
TLSv1.1   disabled
TLSv1.2   enabled
TLSv1.3   disabled

  TLS Fallback SCSV:
Server supports TLS Fallback SCSV

  TLS renegotiation:
Session renegotiation not supported

  TLS Compression:
Compression disabled

  Heartbleed:
TLSv1.2 not vulnerable to heartbleed

  Supported Server Cipher(s):
Preferred TLSv1.2  256 bits  AES256-GCM-SHA384
Accepted  TLSv1.2  128 bits  AES128-GCM-SHA256
Accepted  TLSv1.2  256 bits  AES256-SHA256
Accepted  TLSv1.2  128 bits  AES128-SHA256
Accepted  TLSv1.2  256 bits  AES256-SHA

  SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
RSA Key Strength:    2048

Subject:  stg.mul-bill.jp
Altnames: DNS:stg.mul-bill.jp
Issuer:   GlobalSign RSA OV SSL CA 2018

Not valid before: Sep 15 08:38:53 2023 GMT
Not valid after:  Oct 16 08:38:52 2024 GMT

[kentonv]

Is the problem that this server only support classic DH, as opposed to ECDH? KJ HTTP hasn't supported classic DH because of the need to provide dhparams, which is sort of a pain, but classic DH has been obsolete for quite a while now...

[Nufflee]

Is the problem that this server only support classic DH, as opposed to ECDH? KJ HTTP hasn't supported classic DH because of the need to provide dhparams, which is sort of a pain, but classic DH has been obsolete for quite a while now...

Yeah, precisely. But the Worker runtime running on CF infrastructure does support DH. If I deploy the above code using wrangler, it will work just fine.

What this issue is really about is the disparity that exists between the local runtime and the one used when the worker is deployed.

[maemigh]

I have wasted a lot of cloudflares servers time to test things because the local implementation is broken

[kentonv]

Yes, it seems like we need to figure out how to support classic DH. It's non-trivial due to the need to supply dhparams. We'll have to consult with our crypto team.

[Nufflee]

Yes, it seems like we need to figure out how to support classic DH. It's non-trivial due to the need to supply dhparams. We'll have to consult with our crypto team.

Unrelated to this issue but it would be interesting to know what the intersection between the code in the workerd repo and what's actually running on CF infra looks like. The README says that "most" of the code is the same, so it's curious that something as fundamental as making HTTP requests is completely different.

[kentonv]

@Nufflee The difference with Cloudflare infrastructure is that the runtime does not talk directly to the internet. Instead, it talks through Cloudflare's regular CDN proxy servers, both for incoming and outgoing traffic. Those proxy servers handle all TLS termination.