Skip to content

[DOCS-9171] Q4 Obs Pipelines components#26921

Merged
dd-mergequeue[bot] merged 67 commits intomasterfrom
may/2024-q4-obs-pipelines
Jan 30, 2025
Merged

[DOCS-9171] Q4 Obs Pipelines components#26921
dd-mergequeue[bot] merged 67 commits intomasterfrom
may/2024-q4-obs-pipelines

Conversation

@maycmlee
Copy link
Copy Markdown
Contributor

@maycmlee maycmlee commented Dec 30, 2024

What does this PR do? What is the motivation?

Publish docs for Obs Pipelines new components and features:

  • Sources: Amazon S3 and Kafka
  • Processor: Remap to OCSF (Preview)
  • Destinations: Microsoft Sentinel, SentinelOne, Archiving to cloud storage for all use cases
  • Other: Dynamic destinations, dynamic indexes and archives

Parent PR of: #27299

Merge instructions

Merge readiness:

  • Ready for merge

Merge queue is enabled in this repo. To have it automatically merged after it receives the required reviews, create the PR (from a branch that follows the <yourname>/description naming convention) and then add the following PR comment:

/merge

Additional notes

@maycmlee maycmlee added the WORK IN PROGRESS No review needed, it's a wip ;) label Dec 30, 2024
@maycmlee maycmlee requested a review from a team as a code owner December 30, 2024 16:45
@github-actions github-actions Bot added the Architecture Everything related to the Doc backend label Dec 30, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Dec 30, 2024

Preview links (active after the build_preview check completes)

New or renamed files

Modified Files

@github-actions github-actions Bot added the Images Images are added/removed with this PR label Dec 31, 2024

| Max Events | Max Bytes | Timeout (seconds) |
|----------------|-----------------|---------------------|
| TKTK | TKTK | TKTK |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like there are some placeholders here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@p-parekh I know this has been pushed out to Feb, but did you get the batch settings numbers? It'd be good to just have this ready for Feb.

Comment thread content/en/observability_pipelines/processors/remap_ocsf.md
Comment thread content/en/observability_pipelines/set_up_pipelines/archive_logs/kafka.md Outdated
Comment thread layouts/shortcodes/observability_pipelines/processors/sds_library_rules.md Outdated
Comment thread layouts/shortcodes/observability_pipelines/processors/sds_library_rules.md Outdated
Comment thread layouts/shortcodes/observability_pipelines/source_settings/kafka.md Outdated
Comment thread layouts/shortcodes/observability_pipelines/source_settings/kafka.md Outdated
@@ -0,0 +1,4 @@
- Data collection endpoint (DCE)
- Stored as the environment variable: `DD_OP_DESTINATION_MICROSOFT_SENTINEL_DCE_URI`
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like I have both versions in the docs....I'll open a separate PR later to make it consistent.

- Data collection endpoint (DCE)
- Stored as the environment variable: `DD_OP_DESTINATION_MICROSOFT_SENTINEL_DCE_URI`
- Client secret
- Stored as the environment variable: `DD_OP_DESTINATION_MICROSOFT_SENTINEL_CLIENT_SECRET`
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as previous. Seems like I have both versions in the docs....I'll open a separate PR later to make it consistent.

@@ -0,0 +1,2 @@
- SentinelOne write access token:
- Stored as the environment variable: `DD_OP_DESTINATION_SENTINEL_ONE_TOKEN`
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as previous.

@@ -0,0 +1,9 @@
- Amazon S3 SQS URL
- The URL of the SQS queue to which the S3 bucket sends the notification events.
- Stored as the environment variable: `DD_OP_SOURCE_AWS_S3_SQS_URL`
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out! I'll open a separate PR later to fix it for all instances.

@@ -0,0 +1 @@
REUSE INSTRUCTIONS
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, this actually isn't needed so will remove.

Comment thread layouts/shortcodes/observability_pipelines/prerequisites/amazon_s3.md Outdated
@@ -0,0 +1,4 @@
To use Observability Pipelines's Kafka source, you need the following information available:
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I must have copy/pasta'd it! I'll make a batch update.

"c": "c value"
},
"d": "d value"
}
Copy link
Copy Markdown
Contributor Author

@maycmlee maycmlee Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is used in multiple places so will fix all in one commit.

Copy link
Copy Markdown
Contributor

@jhgilbert jhgilbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I flagged one page that isn't configured correctly in the TOC, but it was already like that, just caught it in the preview. Otherwise, looks good, let me know if you need anything else!

Comment thread config/_default/menus/main.en.yaml Outdated
maycmlee and others added 4 commits January 24, 2025 16:22
* add unresolvable behavior

* Update content/en/observability_pipelines/destinations/_index.md

Co-authored-by: Sandra (neko) <[email protected]>

* Apply suggestions from code review

---------

Co-authored-by: Sandra (neko) <[email protected]>
@maycmlee
Copy link
Copy Markdown
Contributor Author

/merge

@dd-devflow
Copy link
Copy Markdown

dd-devflow Bot commented Jan 30, 2025

Devflow running: /merge

View all feedbacks in Devflow UI.


2025-01-30 15:32:17 UTC ℹ️ MergeQueue: pull request added to the queue

The median merge time in master is 7m.


2025-01-30 15:39:02 UTC ℹ️ MergeQueue: This merge request was merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Architecture Everything related to the Doc backend Images Images are added/removed with this PR mergequeue-status: done WORK IN PROGRESS No review needed, it's a wip ;)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants