fix: fall back to GET when web reader HEAD is unsupported

[Jackkp0t]

Summary

Fixes #42237 by falling back to a GET request when the Web Reader's HEAD probe returns 405 (Method Not Allowed) or 501 (Not Implemented). The existing GET response is reused for HTML pages so the reader does not issue a duplicate request.

Changes

• Treat 405/501 from the HEAD probe as a signal to fetch the URL with GET.
• Preserve the existing content-type handling and 403 fallback behavior.
• Add regression coverage for both unsupported HEAD response codes.

Validation

• Python syntax compilation passed for the changed module and test file.
• Isolated behavior checks passed for both 405 and 501, asserting the request sequence is HEAD then GET and HTML content is returned.
• The repository's targeted pytest command could not run in this environment because pytest is not installed in the available runtime.

AI assistance

This change was prepared with Codex assistance. I reviewed the diff and the behavior-focused test coverage.

From Codex> [!IMPORTANT]

1. Make sure you have read our contribution guidelines
2. Ensure there is an associated issue
3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Screenshots

Before	After
...	...

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've verified the change and added or updated tests where meaningful regression risk justifies coverage.
• I've updated the documentation accordingly.
• I ran make lint && make type-check (backend) and vp staged (frontend) to appease the lint gods