Skip to content

Commit 5cc8c38

Browse files
adulauadulau
authored
Merge pull request #953 from MISP/codex/apivoid-v2
Migrate APIVoid expansion module to API v2
2 parents 79c0e49 + 372edfb commit 5cc8c38

7 files changed

Lines changed: 431 additions & 69 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ For further Information see the [license file](https://misp.github.io/misp-modul
5959
* [Abuse IPDB](https://misp.github.io/misp-modules/expansion/#abuse-ipdb) - AbuseIPDB MISP expansion module
6060
* [ANYRUN Sandbox Submit](https://misp.github.io/misp-modules/expansion/#anyrun-sandbox-submit) - A module designed to submit URLs or files to the ANY.RUN Sandbox for analysis and return the unique analysis link and ID.
6161
* [OSINT DigitalSide](https://misp.github.io/misp-modules/expansion/#osint-digitalside) - On demand query API for OSINT.digitalside.it project.
62-
* [APIVoid](https://misp.github.io/misp-modules/expansion/#apivoid) - Module to query APIVoid with some domain attributes.
62+
* [APIVoid](https://misp.github.io/misp-modules/expansion/#apivoid) - Query APIVoid v2 for DNS records, SSL certificates and email verification.
6363
* [AssemblyLine Query](https://misp.github.io/misp-modules/expansion/#assemblyline-query) - A module tu query the AssemblyLine API with a submission ID to get the submission report and parse it.
6464
* [AssemblyLine Submit](https://misp.github.io/misp-modules/expansion/#assemblyline-submit) - A module to submit samples and URLs to AssemblyLine for advanced analysis, and return the link of the submission.
6565
* [APIFreaks](https://misp.github.io/misp-modules/expansion/#apifreaks) - An expansion module for https://apifreaks.com/ that will provide WHOIS, DOMAIN and DNS information.

‎documentation/README.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -101,28 +101,29 @@ On demand query API for OSINT.digitalside.it project.
101101

102102
<img src=logos/apivoid.png height=60>
103103

104-
Module to query APIVoid with some domain attributes.
104+
Query APIVoid v2 for DNS records, SSL certificates and email verification.
105105
[[source code](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/apivoid.py)]
106106

107107
- **features**:
108-
>This module takes a domain name and queries API Void to get the related DNS records and the SSL certificates. It returns then those pieces of data as MISP objects that can be added to the event.
108+
>This module queries APIVoid v2 for DNS records and SSL certificates of domains and hostnames, or verifies email addresses. Results are returned as MISP objects.
109109
>
110-
>To make it work, a valid API key and enough credits to proceed 2 queries (0.06 + 0.07 credits) are required.
110+
>Configure `apikey` with a key from the APIVoid v2 dashboard (https://dash.apivoid.com/); v1 keys must be replaced. Each successful API call costs 1 credit: domain and hostname enrichment uses 2 calls, and email verification uses 1. Account overage settings apply.
111111
112112
- **config**:
113113
>apikey
114114
115115
- **input**:
116-
>A domain attribute.
116+
>A domain, hostname or email attribute.
117117
118118
- **output**:
119-
>DNS records and SSL certificates related to the domain.
119+
>DNS records, X.509 certificates or an APIVoid email verification object.
120120
121121
- **references**:
122-
>https://www.apivoid.com/
122+
> - https://www.apivoid.com/
123+
> - https://docs.apivoid.com/
123124
124125
- **requirements**:
125-
>A valid APIVoid API key with enough credits to proceed 2 queries
126+
>An APIVoid v2 API key with 2 credits for domain lookups or 1 credit for email verification
126127
127128
-----
128129

‎documentation/mkdocs/expansion.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -98,28 +98,29 @@ On demand query API for OSINT.digitalside.it project.
9898

9999
<img src=../logos/apivoid.png height=60>
100100

101-
Module to query APIVoid with some domain attributes.
101+
Query APIVoid v2 for DNS records, SSL certificates and email verification.
102102
[[source code](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/apivoid.py)]
103103

104104
- **features**:
105-
>This module takes a domain name and queries API Void to get the related DNS records and the SSL certificates. It returns then those pieces of data as MISP objects that can be added to the event.
105+
>This module queries APIVoid v2 for DNS records and SSL certificates of domains and hostnames, or verifies email addresses. Results are returned as MISP objects.
106106
>
107-
>To make it work, a valid API key and enough credits to proceed 2 queries (0.06 + 0.07 credits) are required.
107+
>Configure `apikey` with a key from the APIVoid v2 dashboard (https://dash.apivoid.com/); v1 keys must be replaced. Each successful API call costs 1 credit: domain and hostname enrichment uses 2 calls, and email verification uses 1. Account overage settings apply.
108108
109109
- **config**:
110110
>apikey
111111
112112
- **input**:
113-
>A domain attribute.
113+
>A domain, hostname or email attribute.
114114
115115
- **output**:
116-
>DNS records and SSL certificates related to the domain.
116+
>DNS records, X.509 certificates or an APIVoid email verification object.
117117
118118
- **references**:
119-
>https://www.apivoid.com/
119+
> - https://www.apivoid.com/
120+
> - https://docs.apivoid.com/
120121
121122
- **requirements**:
122-
>A valid APIVoid API key with enough credits to proceed 2 queries
123+
>An APIVoid v2 API key with 2 credits for domain lookups or 1 credit for email verification
123124
124125
-----
125126

‎documentation/mkdocs/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj
1818
* [Abuse IPDB](https://misp.github.io/misp-modules/expansion/#abuse-ipdb) - AbuseIPDB MISP expansion module
1919
* [ANYRUN Sandbox Submit](https://misp.github.io/misp-modules/expansion/#anyrun-sandbox-submit) - A module designed to submit URLs or files to the ANY.RUN Sandbox for analysis and return the unique analysis link and ID.
2020
* [OSINT DigitalSide](https://misp.github.io/misp-modules/expansion/#osint-digitalside) - On demand query API for OSINT.digitalside.it project.
21-
* [APIVoid](https://misp.github.io/misp-modules/expansion/#apivoid) - Module to query APIVoid with some domain attributes.
21+
* [APIVoid](https://misp.github.io/misp-modules/expansion/#apivoid) - Query APIVoid v2 for DNS records, SSL certificates and email verification.
2222
* [AssemblyLine Query](https://misp.github.io/misp-modules/expansion/#assemblyline-query) - A module tu query the AssemblyLine API with a submission ID to get the submission report and parse it.
2323
* [AssemblyLine Submit](https://misp.github.io/misp-modules/expansion/#assemblyline-submit) - A module to submit samples and URLs to AssemblyLine for advanced analysis, and return the link of the submission.
2424
* [APIFreaks](https://misp.github.io/misp-modules/expansion/#apifreaks) - An expansion module for https://apifreaks.com/ that will provide WHOIS, DOMAIN and DNS information.

‎misp_modules/modules/expansion/apivoid.py‎

Lines changed: 86 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -21,70 +21,98 @@
2121
"format": "misp_standard",
2222
}
2323
moduleinfo = {
24-
"version": "0.2",
24+
"version": "0.3",
2525
"author": "Christian Studer",
26-
"description": "Module to query APIVoid with some domain attributes.",
26+
"description": "Query APIVoid v2 for DNS records, SSL certificates and email verification.",
2727
"module-type": ["expansion", "hover"],
2828
"name": "APIVoid",
2929
"logo": "apivoid.png",
30-
"requirements": ["A valid APIVoid API key with enough credits to proceed 2 queries"],
30+
"requirements": ["An APIVoid v2 API key with 2 credits for domain lookups or 1 credit for email verification"],
3131
"features": (
32-
"This module takes a domain name and queries API Void to get the related DNS records and the SSL certificates."
33-
" It returns then those pieces of data as MISP objects that can be added to the event.\n\nTo make it work, a"
34-
" valid API key and enough credits to proceed 2 queries (0.06 + 0.07 credits) are required."
32+
"This module queries APIVoid v2 for DNS records and SSL certificates of domains and hostnames, or verifies"
33+
" email addresses. Results are returned as MISP objects.\n\nConfigure `apikey` with a key from the APIVoid v2"
34+
" dashboard (https://dash.apivoid.com/); v1 keys must be replaced. Each successful API call costs 1 credit:"
35+
" domain and hostname enrichment uses 2 calls, and email verification uses 1. Account overage settings apply."
3536
),
36-
"references": ["https://www.apivoid.com/"],
37-
"input": "A domain attribute.",
38-
"output": "DNS records and SSL certificates related to the domain.",
37+
"references": ["https://www.apivoid.com/", "https://docs.apivoid.com/"],
38+
"input": "A domain, hostname or email attribute.",
39+
"output": "DNS records, X.509 certificates or an APIVoid email verification object.",
3940
}
4041
moduleconfig = ["apikey"]
4142

4243

44+
class APIVoidError(Exception):
45+
"""An API failure that can be reported to the MISP user."""
46+
47+
4348
class APIVoidParser:
4449
def __init__(self, attribute):
4550
self.misp_event = MISPEvent()
4651
self.attribute = MISPAttribute()
4752
self.attribute.from_dict(**attribute)
48-
self.misp_event.add_attribute(**self.attribute)
49-
self.url = "https://endpoint.apivoid.com/{}/v1/pay-as-you-go/?key={}&"
53+
self.attribute = self.misp_event.add_attribute(**self.attribute)
54+
self.url = "https://api.apivoid.com/v2/{}"
5055

5156
def get_results(self):
52-
if hasattr(self, "result"):
53-
return self.result
5457
event = json.loads(self.misp_event.to_json())
55-
results = {key: event[key] for key in ("Attribute", "Object")}
58+
results = {key: event.get(key, []) for key in ("Attribute", "Object")}
5659
return {"results": results}
5760

61+
def _query(self, service, apikey, payload, timeout=30):
62+
try:
63+
response = requests.post(
64+
self.url.format(service),
65+
headers={"X-API-Key": apikey},
66+
json=payload,
67+
timeout=timeout,
68+
)
69+
except requests.Timeout:
70+
raise APIVoidError(f"APIVoid {service} request timed out.") from None
71+
except requests.RequestException:
72+
raise APIVoidError(f"Unable to contact APIVoid {service}.") from None
73+
try:
74+
data = response.json()
75+
except ValueError:
76+
raise APIVoidError(f"APIVoid {service} returned invalid JSON (HTTP {response.status_code}).") from None
77+
if not isinstance(data, dict):
78+
raise APIVoidError(f"APIVoid {service} returned an unexpected response format.")
79+
if response.status_code != 200 or "error" in data:
80+
message = str(data.get("error") or "Request failed").replace(apikey, "[redacted]")
81+
if response.status_code == 429 and response.headers.get("Retry-After"):
82+
message += f" Retry after {response.headers['Retry-After']} seconds."
83+
raise APIVoidError(f"APIVoid {service} error (HTTP {response.status_code}): {message}")
84+
return data
85+
5886
def parse_domain(self, apikey):
59-
feature = "dnslookup"
60-
if requests.get(f"{self.url.format(feature, apikey)}stats").json()["credits_remained"] < 0.13:
61-
self.result = {"error": "You do not have enough APIVoid credits to proceed your request."}
87+
mapping = {"a": "resolution-of", "aaaa": "resolution-of", "mx": "mail-server-of", "ns": "server-name-of"}
88+
dnslookup = self._query("dns-lookup", apikey, {"host": self.attribute.value, "dns_types": "A,AAAA,MX,NS"})
89+
records = dnslookup["records"]
90+
if not isinstance(records, dict):
91+
raise APIVoidError("APIVoid dns-lookup returned an unexpected response format.")
92+
for record_type, relationship in mapping.items():
93+
for item in records.get(record_type, []):
94+
self._handle_dns_record(dnslookup["host"], item, record_type, relationship)
95+
try:
96+
ssl = self._query("ssl-info", apikey, {"host": self.attribute.value}, timeout=60)
97+
except APIVoidError as error:
98+
if not self.misp_event.objects:
99+
raise
100+
# Keep DNS enrichment when only the SSL request fails, and make the partial result explicit.
101+
warning = str(error)
102+
comment = getattr(self.attribute, "comment", None)
103+
self.attribute.comment = f"{comment}\n{warning}" if comment else warning
62104
return
63-
mapping = {"A": "resolution-of", "MX": "mail-server-of", "NS": "server-name-of"}
64-
dnslookup = requests.get(f"{self.url.format(feature, apikey)}action=dns-any&host={self.attribute.value}").json()
65-
for item in dnslookup["data"]["records"]["items"]:
66-
record_type = item["type"]
67-
try:
68-
relationship = mapping[record_type]
69-
except KeyError:
70-
continue
71-
self._handle_dns_record(item, record_type, relationship)
72-
ssl = requests.get(f'{self.url.format("sslinfo", apikey)}host={self.attribute.value}').json()
73-
self._parse_ssl_certificate(ssl["data"]["certificate"])
105+
if ssl["certificate"]["found"]:
106+
self._parse_ssl_certificate(ssl["certificate"])
74107

75108
def handle_email(self, apikey):
76-
feature = "emailverify"
77-
if requests.get(f"{self.url.format(feature, apikey)}stats").json()["credits_remained"] < 0.06:
78-
self.result = {"error": "You do not have enough APIVoid credits to proceed your request."}
79-
return
80-
emaillookup = requests.get(f"{self.url.format(feature, apikey)}email={self.attribute.value}").json()
109+
emaillookup = self._query("email-verify", apikey, {"email": self.attribute.value})
81110
email_verification = MISPObject("apivoid-email-verification")
82111
boolean_attributes = [
83112
"valid_format",
84113
"suspicious_username",
85114
"suspicious_email",
86115
"dirty_words_username",
87-
"suspicious_email",
88116
"valid_tld",
89117
"disposable",
90118
"has_a_records",
@@ -108,31 +136,32 @@ def handle_email(self, apikey):
108136
for boolean_attribute in boolean_attributes:
109137
email_verification.add_attribute(
110138
boolean_attribute,
111-
**{"type": "boolean", "value": emaillookup["data"][boolean_attribute]},
139+
**{"type": "boolean", "value": emaillookup[boolean_attribute]},
112140
)
113-
email_verification.add_attribute("email", **{"type": "email", "value": emaillookup["data"]["email"]})
114-
email_verification.add_attribute("username", **{"type": "text", "value": emaillookup["data"]["username"]})
141+
email_verification.add_attribute("email", **{"type": "email", "value": emaillookup["email"]})
142+
email_verification.add_attribute("username", **{"type": "text", "value": emaillookup["username"]})
115143
email_verification.add_attribute(
116144
"role_address",
117-
**{"type": "boolean", "value": emaillookup["data"]["role_address"]},
145+
**{"type": "boolean", "value": emaillookup["role_address"]},
118146
)
119-
email_verification.add_attribute("domain", **{"type": "domain", "value": emaillookup["data"]["domain"]})
120-
email_verification.add_attribute("score", **{"type": "float", "value": emaillookup["data"]["score"]})
147+
email_verification.add_attribute("domain", **{"type": "domain", "value": emaillookup["domain"]})
148+
email_verification.add_attribute("score", **{"type": "float", "value": emaillookup["score"]})
121149
email_verification.add_reference(self.attribute["uuid"], "related-to")
122150
self.misp_event.add_object(email_verification)
123151

124-
def _handle_dns_record(self, item, record_type, relationship):
152+
def _handle_dns_record(self, host, item, record_type, relationship):
125153
dns_record = MISPObject("dns-record")
126-
dns_record.add_attribute("queried-domain", type="domain", value=item["host"])
127-
attribute_type, feature = ("ip-dst", "ip") if record_type == "A" else ("domain", "target")
128-
dns_record.add_attribute(f"{record_type.lower()}-record", type=attribute_type, value=item[feature])
154+
dns_record.add_attribute("queried-domain", type="domain", value=host)
155+
attribute_type = "ip-dst" if record_type in ("a", "aaaa") else "domain"
156+
dns_record.add_attribute(f"{record_type}-record", type=attribute_type, value=item["target"])
129157
dns_record.add_reference(self.attribute.uuid, relationship)
130-
self.misp_event.add_object(**dns_record)
158+
self.misp_event.add_object(dns_record)
131159

132160
def _parse_ssl_certificate(self, certificate):
133161
x509 = MISPObject("x509")
134-
fingerprint = "x509-fingerprint-sha1"
135-
x509.add_attribute(fingerprint, type=fingerprint, value=certificate["fingerprint"])
162+
for algorithm in ("sha1", "sha256"):
163+
fingerprint = f"x509-fingerprint-{algorithm}"
164+
x509.add_attribute(fingerprint, type=fingerprint, value=certificate[f"fingerprint_{algorithm}"])
136165
x509_mapping = {
137166
"subject": {"name": ("text", "subject")},
138167
"issuer": {"common_name": ("text", "issuer")},
@@ -152,7 +181,7 @@ def _parse_ssl_certificate(self, certificate):
152181
value=certificate[feature][subfeature],
153182
)
154183
x509.add_reference(self.attribute.uuid, "seen-by")
155-
self.misp_event.add_object(**x509)
184+
self.misp_event.add_object(x509)
156185

157186

158187
def handler(q=False):
@@ -168,10 +197,15 @@ def handler(q=False):
168197
return {"error": "Unsupported attribute type."}
169198
apikey = request["config"]["apikey"]
170199
apivoid_parser = APIVoidParser(attribute)
171-
if attribute["type"] in ["domain", "hostname"]:
172-
apivoid_parser.parse_domain(apikey)
173-
else:
174-
apivoid_parser.handle_email(apikey)
200+
try:
201+
if attribute["type"] in ["domain", "hostname"]:
202+
apivoid_parser.parse_domain(apikey)
203+
else:
204+
apivoid_parser.handle_email(apikey)
205+
except APIVoidError as error:
206+
return {"error": str(error)}
207+
except (KeyError, TypeError, ValueError):
208+
return {"error": "APIVoid returned an unexpected response format."}
175209
return apivoid_parser.get_results()
176210

177211

0 commit comments

Comments
 (0)