E2E: dynamic host volume tests for sticky volumes

[tgross]

DRAFT

Add tests for dynamic host volumes where the claiming jobs have volume.sticky = true. Includes a test for forced rescheduling and a test for node drain.

This changeset includes a new e2e/v3-style package for creating dynamic host volumes, so we can reuse that across other tests.

---

This is currently failing the node drain portion of the test, which suggests there's a flaw in how we're dealing with blocked evals. An example run against a Nomad cluster with 2 nodes:

$ NOMAD_E2E_VERBOSE=1 go test -v -count=1 ./e2e/dynamic_host_volumes -run TestDynamicHostVolumes_StickyVolumes
=== RUN   TestDynamicHostVolumes_StickyVolumes
    dynamic_host_volumes_test.go:163: [69.181463ms] volume "167f0c30-0564-6a17-228c-31f138375315" created
    dynamic_host_volumes_test.go:163: [5.364112673s] volume "167f0c30-0564-6a17-228c-31f138375315" is "ready" on node "51954736-79b3-3ac0-26c9-bc540f27d432"
    dynamic_host_volumes_test.go:167: [39.428681ms] volume "1bad9672-9791-d2e4-ea2b-4fac99e30f43" created
    dynamic_host_volumes_test.go:167: [9.199357954s] volume "1bad9672-9791-d2e4-ea2b-4fac99e30f43" is "ready" on node "a7e4ed2e-2bab-255e-0da9-564d219afbfa"
    dynamic_host_volumes_test.go:171: [14.695503299s] submitting sticky volume mounter job
    jobs3.go:321: register (service) job: "example-751"
    jobs3.go:369: checking eval: cdaf10c9-c0e3-5043-d84b-3259a6d93f47, status: pending
    jobs3.go:369: checking eval: cdaf10c9-c0e3-5043-d84b-3259a6d93f47, status: complete
    jobs3.go:429: checking deployment: 997acbde-9528-b5c8-5878-3d9bf793441e, status: running
    jobs3.go:429: checking deployment: 997acbde-9528-b5c8-5878-3d9bf793441e, status: running
    jobs3.go:429: checking deployment: 997acbde-9528-b5c8-5878-3d9bf793441e, status: successful
    jobs3.go:445: deployment 997acbde-9528-b5c8-5878-3d9bf793441e was a success
    dynamic_host_volumes_test.go:182: [17.737897686s] volume "1bad9672-9791-d2e4-ea2b-4fac99e30f43" on node "a7e4ed2e-2bab-255e-0da9-564d219afbfa" was selected
    dynamic_host_volumes_test.go:190: [17.757009941s] stopped allocation "84a4c39c-a0f7-25ed-d2ab-fdc01ec56428"
    dynamic_host_volumes_test.go:219: [24.181992728s] replacement alloc "c09a7d71-42ca-4681-4fce-05aeb0eff6d1" is running
    dynamic_host_volumes_test.go:223: [24.182027975s] draining node "a7e4ed2e-2bab-255e-0da9-564d219afbfa"
    dynamic_host_volumes_test.go:254: [40.647869918s] undraining node "a7e4ed2e-2bab-255e-0da9-564d219afbfa"
    dynamic_host_volumes_test.go:281:
        dynamic_host_volumes_test.go:281: expected equality via cmp.Equal function
        ↪ Assertion | differential ↷
          []string{
                strings.Join({
                        "1",
        -               "bad9672-9791-d2e4-ea2b-4fac99e30f43",
        +               "67f0c30-0564-6a17-228c-31f138375315",
                }, ""),
          }
    jobs3.go:218: deregister job "example-751"
    jobs3.go:226: system gc
    jobs3.go:642: eval for tg 'web' failed; constraints: map[missing compatible host volumes:1]
    host3.go:166: deleting volume "1bad9672-9791-d2e4-ea2b-4fac99e30f43"
    host3.go:166: deleting volume "167f0c30-0564-6a17-228c-31f138375315"
--- FAIL: TestDynamicHostVolumes_StickyVolumes (42.03s)
FAIL
FAIL    github.com/hashicorp/nomad/e2e/dynamic_host_volumes     42.045s
FAIL