[Test] [history server] [collector] Ensure event type coverage #4343

JiangJiaWei1103 · 2026-01-05T15:48:01Z

Why are these changes needed?

This PR improves the robustness of collector e2e tests by verifying that all 10 potential event types are present in the aggregated node_events/ and job_events/ directories. Previously, tests only confirmed the existence of non-empty files.

NOTE: This PR follows up on #4342, which fixed getJobID for job event collection.

Key Changes

Event type coverage: Ensure all 10 defined Ray event types are collected across node_events/ and job_events/
- Events from these 2 directories are aggregated for verification due to their non-deterministic nature.
Improved testCollectorSeparatesFilesBySession: Force kill the ray-worker container, which triggers event flushing instead of relying on automatic deletion
Add sleep(5) in the RayJob manifest entrypoint
- We need to wait for the core worker to send events to the aggregator agent via async gRPC. Without this delay, the event transmission may be cut off prematurely.

Related issue number

N/A

Related PR

#4342

Test Results

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: Future-Outlier <[email protected]>

Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Jia-Wei Jiang <[email protected]>

Signed-off-by: Future-Outlier <[email protected]>

Signed-off-by: JiangJiaWei1103 <[email protected]>

JiangJiaWei1103 · 2026-01-06T14:54:49Z

historyserver/test/e2e/collector_test.go

+type rayEvent struct {
+	EventID   string `json:"eventId"`
+	EventType string `json:"eventType"`
+}
+


We define this custom type (with specific fields) to represent the decoded Ray event for future extensibility. For example, we might want to do more fine-grained verification after the history server becomes stable.

Signed-off-by: JiangJiaWei1103 <[email protected]>

historyserver/test/e2e/collector_test.go

Signed-off-by: JiangJiaWei1103 <[email protected]>

historyserver/test/e2e/collector_test.go

Signed-off-by: JiangJiaWei1103 <[email protected]>

win5923 · 2026-01-11T17:44:50Z

historyserver/test/e2e/collector_test.go

 func verifyS3SessionDirs(test Test, g *WithT, s3Client *s3.S3, sessionPrefix string, nodeID string) {
-	dirs := []string{"logs", "node_events"}
+	// TODO(jwj): Separate verification for logs and events.
+	dirs := []string{"logs"}


Is it possible to parse the --ray-root-dir argument? If not, it would be helpful to add a comment explain.

kuberay/historyserver/config/raycluster.yaml

Line 101 in 79b5c30

- --ray-root-dir=log

Hi @win5923,

Thanks for the review! We have another PR that focuses on log file verification: #4351. I’ll make the clarification there instead.

win5923 · 2026-01-11T17:52:22Z

historyserver/test/e2e/collector_test.go

+	LogWithTimestamp(test.T(), "Verifying all %d event types are covered: %v", len(rayEventTypes), rayEventTypes)
+	g.Eventually(func(gg Gomega) {
+		uploadedEvents := []rayEvent{}
+		for _, dir := range []string{"node_events", "job_events/AgAAAA==", "job_events/AQAAAA=="} {


Is it deterministic?

Good catch! Not in this case. We've fixed this in 9f70a21 by listing all subdirectories under job_events/.

Signed-off-by: JiangJiaWei1103 <[email protected]>

historyserver/test/e2e/collector_test.go

Signed-off-by: JiangJiaWei1103 <[email protected]>

historyserver/test/e2e/collector_test.go

Signed-off-by: JiangJiaWei1103 <[email protected]>

historyserver/test/e2e/collector_test.go

Future-Outlier · 2026-01-14T11:30:38Z

combined to 1 PR.
it's actually better to separate, but due to limited review bandwidth, let's combine to 1
#4351

Signed-off-by: JiangJiaWei1103 <[email protected]>

historyserver/test/e2e/collector_test.go

Signed-off-by: JiangJiaWei1103 <[email protected]>

JiangJiaWei1103 · 2026-01-14T12:36:38Z

Hi @Future-Outlier,

Please do a final pass! We've merged the master branch and I think it's ready for merge.

Local E2E Test

Future-Outlier · 2026-01-15T02:05:20Z

historyserver/config/rayjob.yaml

+
+    time.sleep(5)


can you help me try ray.shutdown(), maybe it will work

Unfortunately, it doesn't work either:

Maybe the error is on the Ray side, let's keep tracking this issue!

Created an issue here
ray-project/ray#60218

Signed-off-by: JiangJiaWei1103 <[email protected]>

Future-Outlier

LGTM, cc @rueian to merge

win5923 · 2026-01-16T15:59:03Z

historyserver/test/e2e/collector_test.go

+// rayEventTypes includes all potential event types defined in Ray:
+// https://github.com/ray-project/ray/blob/3b41c97fa90c58b0b72c0026f57005b92310160d/src/ray/protobuf/public/events_base_event.proto#L49-L61
+var rayEventTypes = []string{
+	"ACTOR_DEFINITION_EVENT",
+	"ACTOR_LIFECYCLE_EVENT",
+	"ACTOR_TASK_DEFINITION_EVENT",
+	"DRIVER_JOB_DEFINITION_EVENT",
+	"DRIVER_JOB_LIFECYCLE_EVENT",
+	"TASK_DEFINITION_EVENT",
+	"TASK_LIFECYCLE_EVENT",
+	"TASK_PROFILE_EVENT",
+	"NODE_DEFINITION_EVENT",
+	"NODE_LIFECYCLE_EVENT",
+}


Can we define in the event.go? Similar to how AllJobStatuses is defined here:

kuberay/ray-operator/apis/ray/v1/rayjob_types.go

Lines 17 to 33 in 910223a

const (

JobStatusNew JobStatus = ""

JobStatusPending JobStatus = "PENDING"

JobStatusRunning JobStatus = "RUNNING"

JobStatusStopped JobStatus = "STOPPED"

JobStatusSucceeded JobStatus = "SUCCEEDED"

JobStatusFailed JobStatus = "FAILED"

)

var AllJobStatuses = []JobStatus{

JobStatusNew,

JobStatusPending,

JobStatusRunning,

JobStatusStopped,

JobStatusSucceeded,

JobStatusFailed,

}

Hi @win5923, thanks for reviewing! Fixed at d94d02a.

Signed-off-by: JiangJiaWei1103 <[email protected]>

Future-Outlier and others added 8 commits January 5, 2026 21:10

[historyserver] Fix getJobID for job event collection

b0172bc

Signed-off-by: Future-Outlier <[email protected]>

add jia-wei as co-author, since he debug with me together

eeb7a7f

Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Jia-Wei Jiang <[email protected]>

remove unused code

0ff5bcf

Signed-off-by: Future-Outlier <[email protected]>

update rueian's advice

5aea305

Signed-off-by: Future-Outlier <[email protected]>

add task profile event example

5f547a2

Signed-off-by: Future-Outlier <[email protected]>

revert back oneof solution

0177160

Signed-off-by: Future-Outlier <[email protected]>

add task profile event

420ce9a

Signed-off-by: Future-Outlier <[email protected]>

test: Test event type coverage in happy path

0621ba4

Signed-off-by: JiangJiaWei1103 <[email protected]>

JiangJiaWei1103 moved this to Work in progress in My Kuberay & Ray Jan 5, 2026

JiangJiaWei1103 added this to My Kuberay & Ray Jan 5, 2026

JiangJiaWei1103 added 3 commits January 6, 2026 19:51

Merge branch 'my-master' into epic-4274/e2e-test-coverage-of-event-types

07403f7

Signed-off-by: JiangJiaWei1103 <[email protected]>

refactor: Remove redundant code

de153d1

Signed-off-by: JiangJiaWei1103 <[email protected]>

test: Verify event type coverage of aggregated node and job events

d1c2b18

Signed-off-by: JiangJiaWei1103 <[email protected]>

JiangJiaWei1103 commented Jan 6, 2026

View reviewed changes

test: Force kill worker container and verify event coverage

14a9b52

Signed-off-by: JiangJiaWei1103 <[email protected]>

JiangJiaWei1103 commented Jan 6, 2026

View reviewed changes

historyserver/test/e2e/collector_test.go Show resolved Hide resolved

JiangJiaWei1103 marked this pull request as ready for review January 6, 2026 15:47

JiangJiaWei1103 changed the title ~~[history server] [collector] Ensure event type coverage~~ [Test] [history server] [collector] Ensure event type coverage Jan 7, 2026

refactor: Create an WorkerPods adapter and remove redundancy

598dbfd

Signed-off-by: JiangJiaWei1103 <[email protected]>

cursor bot reviewed Jan 7, 2026

View reviewed changes

historyserver/test/e2e/collector_test.go Outdated Show resolved Hide resolved

historyserver/test/e2e/collector_test.go Show resolved Hide resolved

refactor: Use eventually to wrap coverage check

9611a42

Signed-off-by: JiangJiaWei1103 <[email protected]>

win5923 reviewed Jan 11, 2026

View reviewed changes

JiangJiaWei1103 mentioned this pull request Jan 12, 2026

[Test] [historyserver] [collector] Enhance log file coverage test for both head and worker pods #4351

Open

4 tasks

JiangJiaWei1103 moved this from Work in progress to In review in My Kuberay & Ray Jan 13, 2026

JiangJiaWei1103 added 2 commits January 13, 2026 23:27

refactor: List subdirs of job_events rather than hardcoding

9f70a21

Signed-off-by: JiangJiaWei1103 <[email protected]>

fix: Wait for async job events flushing on worker

84519bc

Signed-off-by: JiangJiaWei1103 <[email protected]>

cursor bot reviewed Jan 13, 2026

View reviewed changes

historyserver/test/e2e/collector_test.go Outdated Show resolved Hide resolved

Future-Outlier self-assigned this Jan 13, 2026

test: Consolidate tests by checking non-empty list

c796bfc

Signed-off-by: JiangJiaWei1103 <[email protected]>

cursor bot reviewed Jan 13, 2026

View reviewed changes

historyserver/test/e2e/collector_test.go Outdated Show resolved Hide resolved

JiangJiaWei1103 added 2 commits January 14, 2026 00:29

fix: Aggregate all event files not just the first file obj

c3f73b1

Signed-off-by: JiangJiaWei1103 <[email protected]>

fix: Avoid redundant appends

302d903

Signed-off-by: JiangJiaWei1103 <[email protected]>

cursor bot reviewed Jan 13, 2026

View reviewed changes

historyserver/test/e2e/collector_test.go Outdated Show resolved Hide resolved

JiangJiaWei1103 added 2 commits January 14, 2026 20:07

fix: Explicitly close content body to avoid resource leaks

5c8fc59

Signed-off-by: JiangJiaWei1103 <[email protected]>

docs: Remove redundant notes

10237ec

Signed-off-by: JiangJiaWei1103 <[email protected]>

cursor bot reviewed Jan 14, 2026

View reviewed changes

historyserver/test/e2e/collector_test.go Outdated Show resolved Hide resolved

historyserver/test/e2e/collector_test.go Show resolved Hide resolved

JiangJiaWei1103 added 2 commits January 14, 2026 20:26

fix: Close content on failure to prevent rsc leak

0e28fa2

Signed-off-by: JiangJiaWei1103 <[email protected]>

docs: Update helper usage

48681ab

Signed-off-by: JiangJiaWei1103 <[email protected]>

Merge branch 'my-master' into epic-4274/e2e-test-coverage-of-event-types

5c5c878

Future-Outlier reviewed Jan 15, 2026

View reviewed changes

docs: State why we use sleep

6f47105

Signed-off-by: JiangJiaWei1103 <[email protected]>

Future-Outlier approved these changes Jan 15, 2026

View reviewed changes

Future-Outlier moved this to can be merged in @Future-Outlier's kuberay project Jan 15, 2026

Future-Outlier added this to @Future-Outlier's kuberay project Jan 15, 2026

win5923 reviewed Jan 16, 2026

View reviewed changes

win5923 mentioned this pull request Jan 16, 2026

[Core] Ray Events are lost during ray.shutdown() when using Event Export to external collector ray-project/ray#60218

Open

win5923 approved these changes Jan 16, 2026

View reviewed changes

JiangJiaWei1103 added 2 commits January 17, 2026 17:09

Merge branch 'my-master' into epic-4274/e2e-test-coverage-of-event-types

926aee4

Signed-off-by: JiangJiaWei1103 <[email protected]>

refactor: Define AllEventTypes for reusability and maintainability

d94d02a

Signed-off-by: JiangJiaWei1103 <[email protected]>

	const (
	JobStatusNew JobStatus = ""
	JobStatusPending JobStatus = "PENDING"
	JobStatusRunning JobStatus = "RUNNING"
	JobStatusStopped JobStatus = "STOPPED"
	JobStatusSucceeded JobStatus = "SUCCEEDED"
	JobStatusFailed JobStatus = "FAILED"
	)

	var AllJobStatuses = []JobStatus{
	JobStatusNew,
	JobStatusPending,
	JobStatusRunning,
	JobStatusStopped,
	JobStatusSucceeded,
	JobStatusFailed,
	}

[Test] [history server] [collector] Ensure event type coverage #4343

Are you sure you want to change the base?

[Test] [history server] [collector] Ensure event type coverage #4343

Conversation

JiangJiaWei1103 commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Key Changes

Related issue number

Related PR

Test Results

Checks

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

win5923 Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Future-Outlier commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JiangJiaWei1103 commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Local E2E Test

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

win5923 Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JiangJiaWei1103 commented Jan 5, 2026 •

edited

Loading

win5923 Jan 11, 2026 •

edited

Loading

Future-Outlier commented Jan 14, 2026 •

edited

Loading

JiangJiaWei1103 commented Jan 14, 2026 •

edited

Loading

win5923 Jan 16, 2026 •

edited

Loading