Skip to content

[Netskope] Add Events v2 data stream #14524

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[Netskope] Add Events v2 data stream #14524

wants to merge 3 commits into from
+5,042 −3

Conversation

moxarth-rathod
Copy link
Contributor

@moxarth-rathod moxarth-rathod commented Jul 14, 2025 •
edited
Loading

Proposed commit message

netskope: add events_v2 data stream in the integration

This PR introduces a new data stream, events_v2, along with its corresponding dashboards and 
ingest pipeline. Netskope Log Streaming sends all events and logs directly to customer-managed
cloud object storage buckets (such as Azure Blob Storage, Amazon S3, and Google Cloud Storage),
and the events_v2 data stream collects this data.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/netskope directory.
  • Run the following command to run tests.

elastic-package test -v

Related issues

Screenshots

netskope-event_v2-screenshot

@moxarth-rathod moxarth-rathod self-assigned this Jul 14, 2025
@moxarth-rathod moxarth-rathod requested a review from a team as a code owner July 14, 2025 05:52
@moxarth-rathod moxarth-rathod added enhancement New feature or request dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Integration:netskope Netskope Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Sit-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Jul 14, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@moxarth-rathod moxarth-rathod mentioned this pull request Jul 14, 2025
Merged
2 tasks
@moxarth-rathod
Copy link
Contributor Author

Currently successful CI for this PR is blocked by: elastic/package-spec#925

andrewkroh
andrewkroh reviewed Jul 15, 2025
View reviewed changes
andrewkroh
andrewkroh reviewed Jul 16, 2025
View reviewed changes
packages/netskope/data_stream/event_v2/_dev/deploy/tf/main.tf
resource "aws_s3_object" "object" {
bucket = aws_s3_bucket.bucket.id
key = "event.csv.gz"
source = "./files/event.csv.gz"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use the base64gzip function to avoid adding an opaque gzip file to the repo, which makes it hard to review changes via diffs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, let me use base64gzip function.

@andrewkroh andrewkroh requested a review from a team July 16, 2025 13:34
kcreddy
kcreddy reviewed Jul 17, 2025
View reviewed changes
packages/netskope/data_stream/event_v2/fields/base-fields.yml
- name: event.dataset
type: constant_keyword
external: ecs
value: netskope.event_v2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue description and PR commit says the name should be events_v2 instead of event_v2

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#14443 also has alerts_v2.

packages/netskope/changelog.yml
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.1.0"
changes:
- description: Add support for Events v2 data stream.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add necessary steps to README for setting up ingestion of this new data format from Netskope side and also Elastic side.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvm, I see you are going to first merge #14443 which takes care of docs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that was my plan to merge #14443 first and then i'll make changes on top of that.

packages/netskope/data_stream/event_v2/agent/stream/abs.yml.hbs
Comment on lines +5 to +8
auth.oauth2:
client_id: {{client_id}}
client_secret: {{client_secret}}
tenant_id: {{tenant_id}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update README with required permissions as per #14443 (comment)

packages/netskope/data_stream/event_v2/_dev/test/pipeline/test-event.log-expected.json
]
},
"server": {
"bytes": 67997
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy server.* into destination.*.

packages/netskope/data_stream/event_v2/_dev/test/pipeline/test-event.log-expected.json
Comment on lines +5 to +7
"client": {
"bytes": 3613917
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy client.* into source.*

@efd6
Copy link
Contributor

efd6 commented Jul 17, 2025

/test

@elasticmachine
Copy link

elasticmachine commented Jul 17, 2025 •
edited
Loading

💔 Build Failed

Failed CI Steps

History

cc @moxarth-rathod

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kcreddy kcreddy kcreddy left review comments

@andrewkroh andrewkroh Awaiting requested review from andrewkroh

At least 1 approving review is required to merge this pull request.

Assignees

@moxarth-rathod moxarth-rathod

Labels
dashboard Relates to a Kibana dashboard bug, enhancement, or modification. enhancement New feature or request Integration:netskope Netskope Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Sit-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Netskope] Add support for new data stream - Events V2
5 participants
@moxarth-rathod @elasticmachine @efd6 @andrewkroh @kcreddy