Fix CEF export serialization and standard compliance

documentation/README.md

@@ -3510,8 +3510,7 @@ Module to export a MISP event in CEF format.
 [[source code](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/cef_export.py)]
 
 - **features**:
->The module takes a MISP event in input, to look every attribute. Each attribute matching with some predefined types is then exported in Common Event Format.
->Thus, there is no particular feature concerning MISP Events since any event can be exported. However, 4 configuration parameters recognized by CEF format are required and should be provided by users before exporting data: the device vendor, product and version, as well as the default severity of data.
+>Exports supported event and object attributes as UTF-8 CEF file records, with standard escaping and field validation. IPv6 address records use CEF:1; other records use CEF:0. Device settings and severity have defaults. See the [usage guide](https://github.com/MISP/misp-modules/blob/main/documentation/export_mod/cef_export.md) for mappings and compatibility changes.
 
 - **config**:
 > - Default_Severity
@@ -3520,13 +3519,13 @@ Module to export a MISP event in CEF format.
 > - Device_Version
 
 - **input**:
->MISP Event attributes
+>MISP events and object attributes in standard, bare or legacy export format.
 
 - **output**:
 >Common Event Format file
 
 - **references**:
->https://community.softwaregrp.com/t5/ArcSight-Connectors/ArcSight-Common-Event-Format-CEF-Guide/ta-p/1589306?attachment-id=65537
+>https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors-24.3/pdfdoc/cef-implementation-standard/cef-implementation-standard.pdf
 
 -----
 

documentation/export_mod/cef_export.md

@@ -0,0 +1,69 @@
+# CEF export
+
+The exporter follows the [OpenText ArcSight CEF Implementation Standard, version 27](https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors-24.3/pdfdoc/cef-implementation-standard/cef-implementation-standard.pdf), especially the header, file output, character encoding and extension dictionary sections (pages 5–10).
+
+It produces a base64-encoded UTF-8 `.cef` file, with one line per supported attribute. It accepts a single event or a list of events: bare events, `{"Event": {...}}` wrappers, and legacy exports with `Attribute` or `Object` alongside `Event` metadata. Object attributes are included. As before, all supplied supported attributes are exported regardless of `to_ids`; unsupported types are ignored. Apply any desired filtering before export.
+
+## Configuration
+
+Missing configuration keys use these defaults. Explicit invalid values return a module error with no file.
+
+| Setting | Default | Accepted values |
+| --- | --- | --- |
+| `Device_Vendor` | `MISP` | Non-empty string, at most 63 characters |
+| `Device_Product` | `MISP` | Non-empty string, at most 63 characters |
+| `Device_Version` | `1` | Non-empty string or integer, at most 31 characters |
+| `Default_Severity` | `1` | Integer 0–10, numeric string 0–10, or `Unknown`, `Low`, `Medium`, `High`, `Very-High` |
+
+Integer device settings are converted to strings for compatibility with existing configuration. Header values cannot contain line breaks or other ASCII control characters. Overlong fields are rejected rather than truncated. Length limits apply to the original field values before CEF escaping.
+
+## Record mapping
+
+The header is `CEF:Version|Device Vendor|Device Product|Device Version|Device Event Class ID|Name|Severity|`.
+
+The event-class ID is the MISP attribute type, so `ip-src`, `ip-dst`, and each hash algorithm remain distinguishable. The name is the attribute category (at most 512 characters), falling back to the type when the category is absent. The category is also exported as `cat`.
+
+| MISP attribute type | CEF extension key | Limit |
+| --- | --- | --- |
+| `ip-src` | `src` | Single IPv4 or IPv6 address |
+| `ip-dst` | `dst` | Single IPv4 or IPv6 address |
+| `hostname`, `domain` | `dhost` | 1023 characters |
+| `md5`, `sha1`, `sha256` | `fileHash` | 255 characters |
+| `url` | `request` | 1023 characters |
+
+IPv4 and non-address records use `CEF:0`. IPv6 address records use `CEF:1`, since CEF specification 1.x permits IPv6 in the standard address fields. Address values are normalized; CIDR networks and scoped IPv6 addresses cannot be represented as a single CEF address and return errors. String values retain their content, including Unicode, spaces and extension pipes. Hash, hostname and URL syntax validation remains the responsibility of the input producer; the exporter validates their CEF string type and length.
+
+When an attribute has a `timestamp`, its non-negative Unix seconds are converted to milliseconds in `deviceCustomDate1`, labeled `MISP attribute timestamp` by `deviceCustomDate1Label`. This retains the year and avoids server timezone and locale dependence. A MISP attribute modification timestamp does not establish the time of an observed incident, so it is not mapped to `start`, `end`, or `rt`. Missing timestamps omit these two fields.
+
+Header backslashes and pipes are escaped. Extension backslashes and equals signs are escaped, and CR/LF become literal `\r`/`\n` sequences. Pipes and spaces in extensions remain literal, as required by CEF. Invalid requests, control characters, unsupported header values, malformed addresses and overlong fields return `{"error": "..."}` without a partial file.
+
+## Example
+
+Submit this JSON to the misp-modules `/query` endpoint:
+
+```json
+{
+  "module": "cef_export",
+  "config": {"Default_Severity": "Medium"},
+  "data": [{"Attribute": [{
+    "type": "url",
+    "category": "Network activity",
+    "timestamp": "1700000000",
+    "value": "https://example.org/?a=1&b=2"
+  }]}]
+}
+```
+
+The decoded `data` is:
+
+```text
+CEF:0|MISP|MISP|1|url|Network activity|Medium|request=https://example.org/?a\=1&b\=2 cat=Network activity deviceCustomDate1=1700000000000 deviceCustomDate1Label=MISP attribute timestamp
+```
+
+## Compatibility changes
+
+Version 2 removes the fabricated `MMM dd HH:mm:ss host` syslog prefix. The standard's file export procedure starts each line with `CEF:` (page 8); an actual syslog sender should supply its own transport prefix. Collectors configured for the old prefix must be updated.
+
+Rules matching the event-class ID must use the attribute type instead of the category. Rules matching the name can continue using the category. Consumers of IPv6 address records need CEF 1.x support. Invalid severity, header, address or string lengths now fail the export explicitly instead of producing malformed output. The `.cef` extension and base64 response envelope are retained; the advertised MIME type is corrected to `text/plain`.
+
+These checks cover file serialization against the published specification. They do not constitute ArcSight certification or a live collector integration test.

documentation/mkdocs/export_mod.md

@@ -5,8 +5,7 @@ Module to export a MISP event in CEF format.
 [[source code](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/cef_export.py)]
 
 - **features**:
->The module takes a MISP event in input, to look every attribute. Each attribute matching with some predefined types is then exported in Common Event Format.
->Thus, there is no particular feature concerning MISP Events since any event can be exported. However, 4 configuration parameters recognized by CEF format are required and should be provided by users before exporting data: the device vendor, product and version, as well as the default severity of data.
+>Exports supported event and object attributes as UTF-8 CEF file records, with standard escaping and field validation. IPv6 address records use CEF:1; other records use CEF:0. Device settings and severity have defaults. See the [usage guide](https://github.com/MISP/misp-modules/blob/main/documentation/export_mod/cef_export.md) for mappings and compatibility changes.
 
 - **config**:
 > - Default_Severity
@@ -15,13 +14,13 @@ Module to export a MISP event in CEF format.
 > - Device_Version
 
 - **input**:
->MISP Event attributes
+>MISP events and object attributes in standard, bare or legacy export format.
 
 - **output**:
 >Common Event Format file
 
 - **references**:
->https://community.softwaregrp.com/t5/ArcSight-Connectors/ArcSight-Common-Event-Format-CEF-Guide/ta-p/1589306?attachment-id=65537
+>https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors-24.3/pdfdoc/cef-implementation-standard/cef-implementation-standard.pdf
 
 -----
 

misp_modules/modules/export_mod/cef_export.py

@@ -1,29 +1,29 @@
 import base64
-import datetime
+import ipaddress
 import json
 
 misperrors = {"error": "Error"}
 
 # possible module-types: 'expansion', 'hover' or both
 moduleinfo = {
-    "version": "1",
+    "version": "2",
     "author": "Hannah Ward",
     "description": "Module to export a MISP event in CEF format.",
     "module-type": ["export"],
     "name": "CEF Export",
     "logo": "",
     "requirements": [],
     "features": (
-        "The module takes a MISP event in input, to look every attribute. Each attribute matching with some predefined"
-        " types is then exported in Common Event Format.\nThus, there is no particular feature concerning MISP Events"
-        " since any event can be exported. However, 4 configuration parameters recognized by CEF format are required"
-        " and should be provided by users before exporting data: the device vendor, product and version, as well as the"
-        " default severity of data."
+        "Exports supported event and object attributes as UTF-8 CEF file records, with standard escaping and field"
+        " validation. IPv6 address records use CEF:1; other records use CEF:0. Device settings and severity have"
+        " defaults. See the [usage"
+        " guide](https://github.com/MISP/misp-modules/blob/main/documentation/export_mod/cef_export.md) for mappings"
+        " and compatibility changes."
     ),
     "references": [
-        "https://community.softwaregrp.com/t5/ArcSight-Connectors/ArcSight-Common-Event-Format-CEF-Guide/ta-p/1589306?attachment-id=65537"
+        "https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors-24.3/pdfdoc/cef-implementation-standard/cef-implementation-standard.pdf"
     ],
-    "input": "MISP Event attributes",
+    "input": "MISP events and object attributes in standard, bare or legacy export format.",
     "output": "Common Event Format file",
 }
 
@@ -43,71 +43,132 @@
 
 mispattributes = {"input": list(cefmapping.keys())}
 outputFileExtension = "cef"
-responseType = "application/txt"
+responseType = "text/plain"
+inputSource = ["event", "event-collection"]
+DEFAULT_CONFIG = {
+    "Default_Severity": 1,
+    "Device_Vendor": "MISP",
+    "Device_Product": "MISP",
+    "Device_Version": 1,
+}
+
+
+def _text(value, name, limit, header=False):
+    if not isinstance(value, str) or not value.strip() or len(value) > limit:
+        raise ValueError(f"{name} must be a non-empty string of at most {limit} characters.")
+    value.encode("utf-8")
+    if any(ord(char) < 32 and (header or char not in "\r\n\t") or ord(char) == 127 for char in value):
+        raise ValueError(f"{name} contains an unsupported control character.")
+    return value
+
+
+def _header(value, name, limit):
+    if type(value) is int:
+        value = str(value)
+    value = _text(value, name, limit, header=True)
+    return value.replace("\\", "\\\\").replace("|", "\\|")
+
+
+def _extension(value):
+    return value.replace("\\", "\\\\").replace("=", "\\=").replace("\n", "\\n").replace("\r", "\\r")
+
+
+def _severity(value):
+    if type(value) is int and 0 <= value <= 10:
+        return str(value)
+    if isinstance(value, str):
+        if value in ("Unknown", "Low", "Medium", "High", "Very-High"):
+            return value
+        if value.isascii() and value.isdecimal() and 0 <= int(value) <= 10:
+            return str(int(value))
+    raise ValueError("Default_Severity must be an integer from 0 to 10 or Unknown, Low, Medium, High, Very-High.")
+
+
+def _records(container, key):
+    records = container.get(key, [])
+    if not isinstance(records, list) or any(not isinstance(record, dict) for record in records):
+        raise ValueError(f"{key} must be a list of objects.")
+    return records
+
+
+def _attributes(data):
+    if isinstance(data, dict):
+        data = [data]
+    if not isinstance(data, list) or any(not isinstance(event, dict) for event in data):
+        raise ValueError("data must be a MISP event or a list of events.")
+    for envelope in data:
+        event = envelope.get("Event", envelope)
+        if not isinstance(event, dict):
+            raise ValueError("Event must be an object.")
+        # Legacy exports place collections alongside the Event metadata.
+        collections = dict(event, **envelope)
+        yield from _records(collections, "Attribute")
+        for obj in _records(collections, "Object"):
+            yield from _records(obj, "Attribute")
+
+
+def _record(attr, device, severity):
+    kind = attr.get("type")
+    if not isinstance(kind, str) or kind not in cefmapping:
+        return None
+    key = cefmapping[kind]
+    value = _text(attr.get("value"), kind, 255 if key == "fileHash" else 1023)
+    cef_version = 0
+    if key in ("src", "dst"):
+        address = ipaddress.ip_address(value)
+        if "%" in value:
+            raise ValueError("Scoped IPv6 addresses cannot be represented in a CEF address field.")
+        value = str(address)
+        cef_version = int(address.version == 6)
+    category = _text(attr.get("category", kind), "category", 512, header=True)
+    name = _header(category, "category", 512)
+    extension = [(key, value), ("cat", category)]
+    if "timestamp" in attr:
+        timestamp = attr["timestamp"]
+        if not (type(timestamp) is int and timestamp >= 0) and not (
+            isinstance(timestamp, str) and timestamp.isascii() and timestamp.isdecimal()
+        ):
+            raise ValueError("Attribute timestamp must be non-negative Unix seconds.")
+        extension.extend(
+            [("deviceCustomDate1", str(int(timestamp) * 1000)), ("deviceCustomDate1Label", "MISP attribute timestamp")]
+        )
+    fields = [f"CEF:{cef_version}", *device, kind, name, severity]
+    return "|".join(fields) + "|" + " ".join(f"{key}={_extension(value)}" for key, value in extension)
 
 
 def handler(q=False):
     if q is False:
         return False
-    request = json.loads(q)
-    if "config" in request:
-        config = request["config"]
-    else:
-        config = {
-            "Default_Severity": 1,
-            "Device_Vendor": "MISP",
-            "Device_Product": "MISP",
-            "Device_Version": 1,
-        }
-
-    data = request["data"]
-    response = ""
-    for ev in data:
-        event = ev["Attribute"]
-        for attr in event:
-            if attr["type"] in cefmapping:
-                response += "{} host CEF:0|{}|{}|{}|{}|{}|{}|{}={}\n".format(
-                    datetime.datetime.fromtimestamp(int(attr["timestamp"])).strftime("%b %d %H:%M:%S"),
-                    config["Device_Vendor"],
-                    config["Device_Product"],
-                    config["Device_Version"],
-                    attr["category"],
-                    attr["category"],
-                    config["Default_Severity"],
-                    cefmapping[attr["type"]],
-                    attr["value"],
-                )
-
-    r = {
-        "response": [],
-        "data": str(base64.b64encode(bytes(response, "utf-8")), "utf-8"),
-    }
-    return r
+    try:
+        request = json.loads(q)
+        if not isinstance(request, dict) or "data" not in request:
+            raise ValueError("A request object with data is required.")
+        supplied_config = request.get("config", {})
+        if not isinstance(supplied_config, dict):
+            raise ValueError("config must be an object.")
+        config = {**DEFAULT_CONFIG, **supplied_config}
+        severity = _severity(config["Default_Severity"])
+        device = [
+            _header(config[key], key, limit)
+            for key, limit in (("Device_Vendor", 63), ("Device_Product", 63), ("Device_Version", 31))
+        ]
+        lines = []
+        for attr in _attributes(request["data"]):
+            line = _record(attr, device, severity)
+            if line is not None:
+                lines.append(line + "\n")
+        return {"response": [], "data": base64.b64encode("".join(lines).encode("utf-8")).decode("ascii")}
+    except (TypeError, ValueError) as error:
+        return {"error": f"Unable to export CEF: {error}"}
 
 
 def introspection():
-    modulesetup = {}
-    try:
-        responseType
-        modulesetup["responseType"] = responseType
-    except NameError:
-        pass
-    try:
-        userConfig
-        modulesetup["userConfig"] = userConfig
-    except NameError:
-        pass
-    try:
-        outputFileExtension
-        modulesetup["outputFileExtension"] = outputFileExtension
-    except NameError:
-        pass
-    try:
-        inputSource
-        modulesetup["inputSource"] = inputSource
-    except NameError:
-        pass
-    return modulesetup
+    return {
+        **mispattributes,
+        "responseType": responseType,
+        "outputFileExtension": outputFileExtension,
+        "inputSource": inputSource,
+    }
 
 
 def version():