mongo-tools evergreen for mongodump passthrough tests #792

rcownie · 2025-04-14T17:55:22Z

Evergreen tasks and functions are ported from mongosync/evergreen into mongo-tools/mongodump_passthrough
Various changes are needed to match the different directory structure and evergreen environment
Slight changes to mongodump/mongorestore to help with debugging passthrough failures
New buildvariant "rhel80" used for mongodump passthrough testing only

rcownie · 2025-04-15T00:45:44Z

The check-sbom-lite task is failing apparently because of a disagreement between "go1.23.7" and "go1.23.8". I don't know where these versions are coming from, so I'd welcome any suggestions for fixing this. But it seems unrelated to the new functionality.

shyam12860 · 2025-04-15T17:11:43Z

mongodump_passthrough/README.md

@@ -0,0 +1,9 @@
+mongo-tools/mongodump_passthrough contains evergreen .yml files to support resmoke passthrough


Considering this is for both mongodump and mongorestore, would it be better to call this directory just passthrough rather than mongodump_passthrough?

Tricky one. In the scope document it was called "mongodump/mongorestore passthrough, in the technical design I shortened that to "mongodump passthrough". I'm inclined to stick with that because the mongo-tools repo contains other tools, e.g. mongoimport/mongoexport.

Reasonable. I was going to suggest putting them in mongodump/passthrough instead, but these tests are for dump + restore both, so this is fine in that case.

shyam12860 · 2025-04-15T17:13:55Z

mongodump_passthrough/functions.yml

+#   or:     "Changed for mongodump_passthrough"
+#
+# The tests are run by loading the mongosync repo as an evergreen module under src/mongosync.
+# The sources for mongodump-suite-gen, mongodump-task-gen, js tests (with slight modifications


Could you tell me why the code for these lives in the mongosync directory? I don't have the context

Sure. The mongosync repo has a fairly elaborate multi-language stack of JS code for generating traffic on the source db and verifying consistency between src and dst clusters; python code for "fixtures" which set up the various components of the test infrastructure - the two clusters, the driver for cluster-to-cluster-replication, and the particular method of replication (mongosync, multi-mongosync, and now mongodump+mongorestore); the suite.yml files which configure the fixtures to run a suite of tests; the suite generator which generates the suites; and the task-generator which tells evergreen about the suites.

And that code has to evolve to cope with changes in the server resmoke infrastructure.

So it's good to share as much of that code as possible, andonly have one copy to be maintained.

It also makes it easier to eyeball the code and see which parts (not very many) have different behavior for mongodump/mongorestore vs mongosync.

So the design choice was to leave that code in the mongosync repo and make minimal change to it, but
fetch it from the mongo-tools repo as an evergreen module.

shyam12860 · 2025-04-15T17:23:14Z

common.yml

@@ -673,6 +695,11 @@ post:
          set -v
        rm -rf /data/db/*
        exit 0
+  # Extra post steps needed for mongodump passthrough.


Considering all the commands added to pre / post / timeout are passthrough specific, is there a way to run them conditionally, only for passthrough?

Can running it for every task cause errors or slowdowns?

It would be nice if we could make them conditional - perhaps conditional on the buildvariant. But that's hard with the way they're written - within a shell_exec script, you could use ${build_variant} to select conditional behavior, but this is written as several evergreen function calls, and the whole "not-quite-a-programming-language" flavor of evergreen makes it hard.

I was definitely worried that it might cause errors on the existing tests, thought I'd just try it and see, and it seems the tests pass.

My wild-ass-guess was that it wouldn't cause significant slowdown because it's mostly to do with trying to pick up files produced by the tests, and if it tries to pick up files that weren't created, that doesn't waste a noticeable amount of time. Could be wrong though.

The meta-problem here is that evergreen configurations don't have the usual composability properties of a real programming language. So when you try to glue together two separate chunks of evergreen stuff, there are rough edges in dealing with the global namespace of tasks and functions, and the global behavior of pre and post.

Fair enough. I don't see a way to use tags to do this either because you need them to run in a specific order.

I'm okay with leaving them here if they're quick enough and don't cause failures.

Yeah, there's no way to do this. It's a bit annoying.

autarch

This looks good % some questions I had about specific changes. I also have a few general comments:

It'd be nice to separate out the Go version changes into a separate PR, since it's totally unrelated to this work.
It seems like you copied a lot of tasks & functions that aren't actually used. It'd be good to remove those. Maybe just leave a comment in where you deleted it if you're concerned about keeping up to date with future changes in the mongosync config.

autarch · 2025-04-17T16:35:55Z

common.yml

@@ -673,6 +695,11 @@ post:
          set -v
        rm -rf /data/db/*
        exit 0
+  # Extra post steps needed for mongodump passthrough.


Yeah, there's no way to do this. It's a bit annoying.

common.yml

mongodump/oplog_dump.go

autarch · 2025-04-17T16:40:37Z

mongodump_passthrough/empty.go

You can just add an exclude = "mongodump_passthrough/**" to the golangci-lint config in this repo's precious.toml` file.

The mongo-tools/common.yml has a "vet" task which doesn't go through precious.

name: vet
commands:

func: "fetch source"

command: shell.exec
type: test
params:
working_dir: src/github.com/mongodb/mongo-tools
script: |
${_set_shell_env}
set -x
set -v
set -e
go vet -composites=false ./bsondump ./mongo*

Ah, right. I forgot about that.

I guess you could just move that to precious too. It'd be fairly trivial.

mongodump_passthrough/functions.yml

autarch · 2025-04-17T16:43:21Z

mongodump_passthrough/functions.yml

+          ${PREPARE_SHELL}
+          PATH=$PATH:$HOME
+          echo $PATH 
+          MISE_INSTALL_PATH='${workdir}/.local/bin/mise' sh ${workdir}/src/mongosync/etc/mise.run.sh


Suggested change

MISE_INSTALL_PATH='${workdir}/.local/bin/mise' sh ${workdir}/src/mongosync/etc/mise.run.sh

MISE_INSTALL_PATH='${workdir}/.local/bin/mise' sh ${workdir}/src/mongosync/etc/mise.run.sh

Michael recently made a change to add retrying when this fails. It'd be good to get that into this as well.

autarch · 2025-04-17T16:45:11Z

mongodump_passthrough/functions.yml

+        cd mongodump-suite-config
+        tar cvf ../suite-config/a.tar .
+        cd ../suite-config
+        tar xvf a.tar


Why use tar like this instead of cp -r?

I'm more confident that I understand the behavior of tar.

"Historic versions of the cp utility had a -r option. This implementation supports that option, however,
its behavior is different from historical FreeBSD behavior. Use of this option is strongly discouraged as
the behavior is implementation-dependent."

We only run this stuff on Linux, which has a cp -r that works as you'd expect. Using tar seems unnecessarily baroque.

Baroque is my era. I'm easily confused by the way cp puts things inside a destination directory if it already exists.

I think this should be changed. This is a very idiosyncratic thing that we don't do elsewhere, AFAIK. It's time to move onto the Classical era, if not Romantic.

mongodump_passthrough/functions.yml

rcownie · 2025-04-17T19:16:41Z

This looks good % some questions I had about specific changes. I also have a few general comments:

It'd be nice to separate out the Go version changes into a separate PR, since it's totally unrelated to this work.

Done

It seems like you copied a lot of tasks & functions that aren't actually used. It'd be good to remove those. Maybe just leave a comment in where you deleted it if you're concerned about keeping up to date with future changes in the mongosync config.

I pruned it fairly heavily: the tasks+functions here are 892 lines, compared to 1826 lines in mongosync. Might have missed one or two unused functions, IDK, but "evergreen validate common.yml" seems mostly happy.

autarch · 2025-04-17T19:38:33Z

It seems like you copied a lot of tasks & functions that aren't actually used. It'd be good to remove those. Maybe just leave a comment in where you deleted it if you're concerned about keeping up to date with future changes in the mongosync config.

I pruned it fairly heavily: the tasks+functions here are 892 lines, compared to 1826 lines in mongosync. Might have missed one or two unused functions, IDK, but "evergreen validate common.yml" seems mostly happy.

I'm pretty sure validate won't complain about unused functions and tasks. I think it'd be good to audit for those and not include them. Otherwise, it's a burden on whoever has to maintain this in the future. They're forced to try figure out why some seemingly unused function is included.

rcownie · 2025-04-17T21:48:37Z

It seems like you copied a lot of tasks & functions that aren't actually used. It'd be good to remove those. Maybe just leave a comment in where you deleted it if you're concerned about keeping up to date with future changes in the mongosync config.

I pruned it fairly heavily: the tasks+functions here are 892 lines, compared to 1826 lines in mongosync. Might have missed one or two unused functions, IDK, but "evergreen validate common.yml" seems mostly happy.

I'm pretty sure validate won't complain about unused functions and tasks. I think it'd be good to audit for those and not include them. Otherwise, it's a burden on whoever has to maintain this in the future. They're forced to try figure out why some seemingly unused function is included.

It does complain about seemingly-unused tasks (except it can't tell if dynamically-generated tasks will have them as dependencies).

I made another pass through to find unused functions, that reduces the size to 775 lines.

autarch

LGTM % please do change the tar thing. I think it is likely to confuse basically most future developers, who have not ever seen this idiom. Keep in mind that nowadays new devs often have much less Unix CLI experience than people from our era.

autarch · 2025-04-18T18:47:15Z

mongodump_passthrough/functions.yml

+        cd mongodump-suite-config
+        tar cvf ../suite-config/a.tar .
+        cd ../suite-config
+        tar xvf a.tar


I think this should be changed. This is a very idiosyncratic thing that we don't do elsewhere, AFAIK. It's time to move onto the Classical era, if not Romantic.

autarch · 2025-04-18T19:30:30Z

@rcownie I think you missed my comment asking you to change the tar thing before you merged this. It's a common practice on our team to do an "LGTM but before you merge please do this one thing" approval.

rcownie force-pushed the rcownie/passthru3 branch from a189cba to 3706d62 Compare April 15, 2025 00:27

rcownie marked this pull request as ready for review April 15, 2025 00:43

rcownie requested review from autarch and shyam12860 April 15, 2025 00:43

shyam12860 approved these changes Apr 15, 2025

View reviewed changes

rcownie requested review from FGasper and autarch and removed request for autarch and FGasper April 17, 2025 01:43

autarch requested changes Apr 17, 2025

View reviewed changes

rcownie added 2 commits April 17, 2025 15:06

snapshot

072cead

review fixes

fc073cc

rcownie force-pushed the rcownie/passthru3 branch from d5c99bc to fc073cc Compare April 17, 2025 19:06

fixes for review issues

bb3909e

rcownie added 2 commits April 17, 2025 17:49

fix typo in comment

55ee65c

fixes for review issues

5d96fa0

rcownie requested a review from autarch April 18, 2025 10:51

autarch approved these changes Apr 18, 2025

View reviewed changes

rcownie merged commit 7214f6a into master Apr 18, 2025
38 checks passed

		@@ -0,0 +1,9 @@
		mongo-tools/mongodump_passthrough contains evergreen .yml files to support resmoke passthrough

	MISE_INSTALL_PATH='${workdir}/.local/bin/mise' sh ${workdir}/src/mongosync/etc/mise.run.sh
	MISE_INSTALL_PATH='${workdir}/.local/bin/mise' sh ${workdir}/src/mongosync/etc/mise.run.sh

mongo-tools evergreen for mongodump passthrough tests #792

mongo-tools evergreen for mongodump passthrough tests #792

Uh oh!

Conversation

rcownie commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rcownie commented Apr 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rcownie Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

autarch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rcownie commented Apr 17, 2025

Uh oh!

autarch commented Apr 17, 2025

Uh oh!

rcownie commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

autarch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

autarch commented Apr 18, 2025

Uh oh!

Uh oh!

rcownie commented Apr 14, 2025 •

edited

Loading

rcownie Apr 16, 2025 •

edited

Loading

rcownie commented Apr 17, 2025 •

edited

Loading