core: limit job GC batch size to match other GC batches

[tgross]

In the core scheduler we have several object types where we can delete them by ID. We batch up to 7281 UUIDs because this works out to be about 256 KiB per request, which is well below the maximum Raft log entry size we want to have. When we GC jobs we use this same constant to size the batch, but the request body is not a list of UUIDs but instead a map of namespaced job IDs to a pointer to a struct. This pushes the batch size into 746 KiB (assuming UUID-sized job names), which is going to impact performance if GC happens during large volumes of short-lived dispatch work where users may be GC'ing jobs frequently.

Limit the batch size for JobBatchDeregisterRequest to roughly the same size as the requests that are lists of UUIDs.

Ref: https://hashicorp.atlassian.net/browse/NMD-1041

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
  • see core: limit job GC batch size to match other GC batches #26974 (comment)
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad website documentation to reflect this. Refer to
  the website README for docs guidelines. Please also consider whether the
  change requires notes within the upgrade guide.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

[tgross]

It doesn't make sense to have a test we run on every commit to determine the size of an object, but here's how I determined the sizes (which have a lot of reference objects like strings to account for):

nomad/core_sched_test.go

func TestMaximumGCRequestSizes(t *testing.T) {

	check := func(batchSize int) {
		largestSize := 0
		req := &structs.JobBatchDeregisterRequest{
			Jobs: map[structs.NamespacedID]*structs.JobDeregisterOptions{},
			WriteRequest: structs.WriteRequest{
				Region:    "global",
				AuthToken: uuid.Generate(),
			},
		}

		for range batchSize {
			id := structs.NewNamespacedID(uuid.Generate(), uuid.Generate())
			opts := structs.JobDeregisterOptions{
				Purge: true,
			}
			largestSize += len(id.ID)
			largestSize += len(id.Namespace)
			largestSize += int(unsafe.Sizeof(id))
			largestSize += int(unsafe.Sizeof(opts))
			req.Jobs[id] = &opts
		}

		largestSize += int(unsafe.Sizeof(*req))
		largestSize += int(unsafe.Sizeof(req.Jobs))
		largestSize += int(unsafe.Sizeof(req.WriteRequest.AuthToken))
		largestSize += int(unsafe.Sizeof(req.WriteRequest.Region))
		fmt.Printf("size of JobBatchDeregisterRequest with batch size of %d jobs: %d KiB\n", batchSize, largestSize/1024)

	}

	check(structs.MaxUUIDsPerWriteRequest)
	check(2048)

	largestSize := 0
	evalReq := &structs.EvalReapRequest{
		Evals:  []string{},
		Allocs: []string{},
		WriteRequest: structs.WriteRequest{
			Region:    "global",
			AuthToken: uuid.Generate(),
		},
	}
	for range structs.MaxUUIDsPerWriteRequest {
		id := uuid.Generate()
		largestSize += len(id)
		evalReq.Evals = append(evalReq.Evals, id)
	}
	largestSize += int(unsafe.Sizeof(*evalReq))
	largestSize += int(unsafe.Sizeof(evalReq.Evals))
	largestSize += int(unsafe.Sizeof(evalReq.Allocs))
	largestSize += int(unsafe.Sizeof(evalReq.WriteRequest.AuthToken))
	largestSize += int(unsafe.Sizeof(evalReq.WriteRequest.Region))
	fmt.Printf("size of EvalReapRequest with 7281 UUIDs: %d KiB\n", largestSize/1024)

}

$ go test -v -count=1 ./nomad -run TestMaximumGCRequestSize
=== RUN   TestMaximumGCRequestSizes
size of JobBatchDeregisterRequest with batch size of 7281 jobs: 746 KiB
size of JobBatchDeregisterRequest with batch size of 2048 jobs: 210 KiB
size of EvalReapRequest with 7281 UUIDs: 256 KiB
--- PASS: TestMaximumGCRequestSizes (0.01s)
PASS
ok      github.com/hashicorp/nomad/nomad        0.033s

[tgross]

By the way, I think we'll want to revisit this limit if we adopt raft-wal in Nomad 1.12.0, but let's cross that bridge when we get to it

[gulducat]

I don't really have a suggestion here, just a note/question.

UUIDs are a known length, but namespaces are variable. Your test uses structs.NewNamespacedID(uuid.Generate(), uuid.Generate()) with a UUID as the namespace as well as the ID. I imagine most namespaces are not as long as UUIDs, so is it possible that this might be giving (significantly?) more headroom than we'd like?

[tgross]

UUIDs are a known length, but namespaces are variable. Your test uses structs.NewNamespacedID(uuid.Generate(), uuid.Generate()) with a UUID as the namespace as well as the ID. I imagine most namespaces are not as long as UUIDs, so is it possible that this might be giving (significantly?) more headroom than we'd like?

That's a very reasonable question! But I've also been really surprised at how long namespace and job IDs get at some of our users who are running very large/busy clusters (who are also the only folks for whom this issue is really going to matter). And we add 29 characters to the length of every dispatch job ID!

[gulducat]

LGTM!

[jrasell]

LGTM!