Repository navigation
Add Elastic ECS threat indicator Bulk NDJSON export - #945
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 80f705fdf1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "template": { | ||
| "mappings": { | ||
| "dynamic": false, |
There was a problem hiding this comment.
When a valid MISP comment, tag, or raw provenance value exceeds Lucene's 32,766-byte term limit, this template's keyword, wildcard, and flattened fields have no ignore_above, so Bulk rejects the entire indicator instead of merely leaving the oversized field unindexed. Elasticsearch documents this behavior and recommends a limit of 8,191 or less; add an index-wide index.mapping.ignore_above setting or equivalent field-level limits so large MISP metadata does not make otherwise valid indicators unimportable.
Useful? React with 👍 / 👎.
| or not parsed.hostname | ||
| or parsed.query | ||
| or parsed.fragment | ||
| or parsed.username |
There was a problem hiding this comment.
When misp_url is shaped like https://:secret@misp.example, urlsplit().username is an empty, falsy string while .password is set, so this condition accepts credentials despite the documented prohibition. Reference construction then copies the complete URL into every exported document, exposing the password in Elasticsearch; reject when either parsed.username or parsed.password is not None.
Useful? React with 👍 / 👎.
MISP events and objects can now be downloaded through
elastic_ecs_exportas Elasticsearch Bulk NDJSON containing ECS 9.2threat.indicator.*documents. The module reads every supplied event in standard, bare or legacy format and supports IPs, domains/hostnames, URLs, emails, filenames, file hashes, AS numbers and composite indicators.Bulk
indexactions have stable IDs derived from source event/object/attribute identifiers and component position. Reimporting updates existing sources while keeping distinct provenance separate. Onlyto_idsattributes are exported by default;include_non_idsenables contextual attributes. Deleted data is excluded. Unsupported or malformed indicators and invalid optional timestamps are counted in API warnings.The exporter preserves original MISP values and source context under a custom
mispnamespace, combines scoped tags, selects the most restrictive recognized TLP marking, and retains source modification/observation timestamps without inventing confidence or expiry. IPs and hashes are normalized for matching; composite file names/hashes remain in one indicator, while domain/IP composites produce two typed documents.Includes a minimal index template and usage guide covering Bulk import, item-level errors, indicator-match fields, timestamp semantics, and lifecycle limitations. The starter template bounds keyword, wildcard and flattened values with
ignore_above=4096, preserving oversized values in_sourcewithout indexing oversized terms. MISP reference URLs reject all userinfo, including password-only and empty credentials. The exporter requires no new dependencies or remote credentials. Existingdomain-ip,fileandemailtemplates work without changes, so no misp-objects PR is needed.Validation:
tests/test_elastic_ecs_export.py, including real/modulesand/queryhandlers, existing strict PyMISP objects, legacy fixture data, composites, source stability, scoped tags/TLP, dates, filtering and malformed input.git diff --checkpass; generated module documentation matches metadata.No live Elasticsearch cluster was used for validation. This targets regular indices via Bulk
indexactions; it does not upload data, install detection rules, or reconcile expiry/deletions. The guide calls out cleanup requirements and that re-exporting refreshes@timestamp(export time).