[elhoim]

BLUF

• Priority: critical.
• Problem: sourcecache hands the attribute value straight to url_archiver. Enriching a hostile link or url attribute therefore makes the MISP server fetch any URL it can reach, such as loopback services or cloud metadata endpoints, and store the answer in the event as a page capture.
• Fix: the module fetches only http and https URLs whose host resolves solely to globally routable addresses. Anything else returns an error before url_archiver is called.
• Scope: this rebuilds the fix of the earlier PR on the current module, which now returns a page-capture object.

Details

misp_modules/modules/expansion/sourcecache.py gains an is_safe_url() check. It runs in handler() before the page is archived.

• The scheme must be http or https, and the URL must name a host.
• Every address the host resolves to must be globally routable (ipaddress.is_global). This refuses loopback, private, link-local (169.254.169.254) and reserved targets.
• A URL with a backslash, whitespace or a control character is refused outright. urlparse and the requests stack behind url_archiver split such a URL differently: http://127.0.0.1\@example.com/ would be checked as example.com and fetched from 127.0.0.1.
• A refused URL returns {"error": "Blocked URL ..."} and url_archiver is never called.

Known limits: url_archiver performs the request itself. A public URL that redirects to an internal one, or a host that is rebound between the check and the fetch, is not covered here. Closing those needs url_archiver to validate each hop, or the fetch to move into the module.

Tests

tests/test_sourcecache.py gains three cases. An internal address, a file:// URL and a backslash-userinfo URL each return an error without calling the archiver. All 12 tests in that file pass. flake8, with the repository's settings, is clean on the changed module.

Supersedes #873.

[adulau]

Thanks. Could you review it to include also IPv6 infrastructure?