Skip to content

OCSF pipeline for Nginx #21894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
+542 −189
Open

OCSF pipeline for Nginx #21894

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
c8ba329
OCSF pipeline for nginx
jbfeldman-dd Nov 17, 2025
649815d
revert name
jbfeldman-dd Nov 17, 2025
67cee7a
convert to camelcase
jbfeldman-dd Nov 17, 2025
9e7728e
fixed class name
jbfeldman-dd Nov 17, 2025
bfc17ec
add sample logs
jbfeldman-dd Nov 17, 2025
03af30e
put status in category-mapper
jbfeldman-dd Nov 18, 2025
1199dd4
fixed status mapping
jbfeldman-dd Nov 19, 2025
c1a608d
mapping + vendor name
jbfeldman-dd Nov 19, 2025
731aab2
fix test output
jbfeldman-dd Nov 19, 2025
6e99f32
targetformat for status code
jbfeldman-dd Nov 24, 2025
bbc9750
cast to string in test output
jbfeldman-dd Nov 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
231 changes: 225 additions & 6 deletions nginx/assets/logs/nginx.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,13 +117,13 @@ pipeline:
samples:
- HEAD http://174.138.82.103:80/sql/sql-admin/ HTTP/1.1
- type: url-parser
name: ''
name: ""
enabled: true
sources:
- http.url
target: http.url_details
- type: user-agent-parser
name: ''
name: ""
enabled: true
sources:
- http.useragent
Expand All @@ -139,16 +139,16 @@ pipeline:
enabled: true
categories:
- filter:
query: '@http.status_code:[200 TO 299]'
query: "@http.status_code:[200 TO 299]"
name: OK
- filter:
query: '@http.status_code:[300 TO 399]'
query: "@http.status_code:[300 TO 399]"
name: notice
- filter:
query: '@http.status_code:[400 TO 499]'
query: "@http.status_code:[400 TO 499]"
name: warning
- filter:
query: '@http.status_code:[500 TO 599]'
query: "@http.status_code:[500 TO 599]"
name: error
target: http.status_category
- type: status-remapper
Expand All @@ -157,3 +157,222 @@ pipeline:
sources:
- http.status_category
- level
- type: pipeline
name: OCSF sub pipeline for HTTP Activity [4002]
enabled: true
ocsf:
isOcsf: true
filter:
query: "@http.method:*"
processors:
- type: string-builder-processor
name: Add ocsf.metadata.product.name
enabled: true
template: "nginx"
target: ocsf.metadata.product.name
replaceMissing: false
- type: string-builder-processor
name: Add ocsf.metadata.product.vendor_name
enabled: true
template: "F5"
target: ocsf.metadata.product.name
replaceMissing: false

- type: schema-processor
name: Apply OCSF schema for 4002
enabled: true
mappers:
- type: schema-category-mapper
name: ocsf.activity_id
categories:
- filter:
query: "@http.method:CONNECT"
name: Connect
id: 1
- filter:
query: "@http.method:DELETE"
name: Delete
id: 2
- filter:
query: "@http.method:GET"
name: Get
id: 3
- filter:
query: "@http.method:HEAD"
name: Head
id: 4
- filter:
query: "@http.method:OPTIONS"
name: Options
id: 5
- filter:
query: "@http.method:POST"
name: Post
id: 6
- filter:
query: "@http.method:PUT"
name: Put
id: 7
- filter:
query: "@http.method:TRACE"
name: Trace
id: 8
- filter:
query: "@http.method:PATCH"
name: Patch
id: 9
- filter:
query: "@http.method:*"
name: Other
id: 99
targets:
name: ocsf.activity_name
id: ocsf.activity_id
- type: schema-category-mapper
name: ocsf.severity_id
categories:
- filter:
query: "@http.method:*"
name: Informational
id: 1
targets:
name: ocsf.severity
id: ocsf.severity_id
- type: schema-category-mapper
name: ocsf.status_id
categories:
- filter:
query: "@http.status_code:[200 TO 399]"
name: Success
id: 1
- filter:
query: "@http.status_code:[400 TO 599]"
name: Failure
id: 2
- filter:
query: "*"
name: Unknown
id: 99
targets:
name: ocsf.status
id: ocsf.status_id
- type: schema-remapper
name: Map `host` to `ocsf.dst_endpoint.ip`
sources:
- host
target: ocsf.dst_endpoint.ip
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `http.method` to `ocsf.http_request.http_method`
sources:
- http.method
target: ocsf.http_request.http_method
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `http.referer` to `ocsf.http_request.referrer`
sources:
- http.referer
target: ocsf.http_request.referrer
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `http.url_details.path` to `ocsf.http_request.url.path`
sources:
- http.url_details.path
target: ocsf.http_request.url.path
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `http.url_details.port` to `ocsf.http_request.url.port`
sources:
- http.url_details.port
target: ocsf.http_request.url.port
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `http.useragent` to `ocsf.http_request.user_agent`
sources:
- http.useragent
target: ocsf.http_request.user_agent
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `http.status_code` to `ocsf.http_response.code`
sources:
- http.status_code
target: ocsf.http_response.code
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `http.status_category` to `ocsf.http_response.status`
sources:
- http.status_category
target: ocsf.http_response.status
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `http.method` to `ocsf.metadata.event_code`
sources:
- http.method
target: ocsf.metadata.event_code
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `network.client.ip` to `ocsf.src_endpoint.ip`
sources:
- network.client.ip
target: ocsf.src_endpoint.ip
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `http.os.family` to `ocsf.src_endpoint.os`
sources:
- http.os.family
target: ocsf.src_endpoint.os
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `http.status_code` to `ocsf.status_code`
Copy link
Contributor

@orenma-dd orenma-dd Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to add string transformation
image

Copy link
Contributor Author

@jbfeldman-dd jbfeldman-dd Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should happen automatically without explicit mapping, I escalated to the OCSF team https://dd.slack.com/archives/C07FNKFD7RS/p1763582334628809

Copy link
Contributor Author

@jbfeldman-dd jbfeldman-dd Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Temporarily explicitly mapped it using targetFormat, should be resolved now. In the future this will automatically

sources:
- http.status_code
target: ocsf.status_code
preserveSource: true
targetFormat: string
overrideOnConflict: true
- type: schema-remapper
name: Map `duration` to `ocsf.duration`
sources:
- duration
target: ocsf.duration
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `ocsf.metadata.product.name` to `ocsf.metadata.product.name`
sources:
- ocsf.metadata.product.name
target: ocsf.metadata.product.name
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `ocsf.metadata.product.vendor_name` to `ocsf.metadata.product.vendor_name`
sources:
- ocsf.metadata.product.vendor_name
target: ocsf.metadata.product.vendor_name
preserveSource: true
overrideOnConflict: true
- type: schema-remapper
name: Map `date_access` to `ocsf.time`
sources:
- date_access
target: ocsf.time
preserveSource: true
overrideOnConflict: true
schema:
schemaType: ocsf
version: 1.5.0
className: HTTP Activity
classUid: 4002
extensions: []
profiles: []
Loading
Loading