AssertionError: Starting restore 2024-05-14T11:37:59.412827365Z to point-in-time 2024-05-14T11:36:30 from '2024-05-14T11:35:25Z'...Started logical restore. Waiting to finish...Error: operation failed with: reply oplog: replay chunk 1715686534.1715686588: apply oplog for chunk: applying an entry: op: {"Timestamp":{"T":1715686559,"I":3},"Term":1,"Hash":null,"Version":2,"Operation":"i","Namespace":"config.databases","Object":[{"Key":"_id","Value":"test"},{"Key":"primary","Value":"rs1"},{"Key":"partitioned","Value":false},{"Key":"version","Value":[{"Key":"uuid","Value":{"Subtype":4,"Data":"MoGLn/QOTS+qy9pfihC+tw=="}},{"Key":"lastMod","Value":1}]},{"Key":"lastMovedTimestamp","Value":{"T":1715686559,"I":1}}],"Query":null,"UI":{"Subtype":4,"Data":"YzPwLZM/QyuMYfSVZ+9ZZQ=="},"LSID":null,"TxnNumber":null,"PrevOpTime":null} | merr <nil>: applyOps: (NamespaceNotFound) cannot apply insert or update operation on a non-existent namespace config.databases: { ts: Timestamp(1715686559, 3), t: 1, v: 2, op: "i", ns: "config.databases", o: { _id: "test", primary: "rs1", partitioned: false, version: { uuid: UUID("32818b9f-f40e-4d2f-aacb-da5f8a10beb7"), lastMod: 1 }, lastMovedTimestamp: Timestamp(1715686559, 1) }, ui: UUID("6333f02d-933f-432b-8c61-f49567ef5965") }

AssertionError: Error: backup '2024-05-14T11:41:37Z' didn't finish successfully

AssertionError: Starting restore 2024-05-14T11:37:58.694202867Z to point-in-time 2024-05-14T11:36:25 from '2024-05-14T11:35:19Z'...Started logical restore. Waiting to finish...Error: operation failed with: reply oplog: replay chunk 1715686528.1715686583: apply oplog for chunk: applying a transaction entry: apply txn: { "Timestamp": { "T": 1715686554, "I": 2 }, "Term": 1, "Hash": null, "Version": 2, "Operation": "c", "Namespace": "admin.$cmd", "Object": [ { "Key": "applyOps", "Value": [ [ { "Key": "op", "Value": "c" }, { "Key": "ns", "Value": "config.$cmd" }, { "Key": "ui", "Value": { "Subtype": 4, "Data": "bupKEYAPQaK8mXos5HBa9w==" } }, { "Key": "o", "Value": [ { "Key": "create", "Value": "databases" }, { "Key": "idIndex", "Value": [ { "Key": "v", "Value": 2 }, { "Key": "key", "Value": [ { "Key": "_id", "Value": 1 } ] }, { "Key": "name", "Value": "_id_" } ] } ] } ], [ { "Key": "op", "Value": "i" }, { "Key": "ns", "Value": "config.databases" }, { "Key": "ui", "Value": { "Subtype": 4, "Data": "bupKEYAPQaK8mXos5HBa9w==" } }, { "Key": "o", "Value": [ { "Key": "_id", "Value": "test" }, { "Key": "primary", "Value": "rs2" }, { "Key": "partitioned", "Value": false }, { "Key": "version", "Value": [ { "Key": "uuid", "Value": { "Subtype": 4, "Data": "JhfYtYSxRgyybRK0onMx4Q==" } }, { "Key": "timestamp", "Value": { "T": 1715686553, "I": 15 } }, { "Key": "lastMod", "Value": 1 } ] } ] }, { "Key": "o2", "Value": [ { "Key": "_id", "Value": "test" } ] } ], [ { "Key": "op", "Value": "i" }, { "Key": "ns", "Value": "config.placementHistory" }, { "Key": "ui", "Value": { "Subtype": 4, "Data": "fOZizkOfSMeDeCk+yvJUAw==" } }, { "Key": "o", "Value": [ { "Key": "_id", "Value": "66434c9a43e8cadbebd228cd" }, { "Key": "nss", "Value": "test" }, { "Key": "timestamp", "Value": { "T": 1715686553, "I": 15 } }, { "Key": "shards", "Value": [ "rs2" ] } ] }, { "Key": "o2", "Value": [ { "Key": "_id", "Value": "66434c9a43e8cadbebd228cd" } ] } ] ] } ], "Query": null, "UI": null, "LSID": "ZgAAAAVpZAAQAAAABM/+YxZwm0m/m72kxTeLpE4FdWlkACAAAAAAY5mrDaxi8gv8RmdTsQ+1j7fmkr7JUsabhNmXAheU0fgFdHhuVVVJRAAQAAAABEKLIxAg/ENFhJJNgdm2FEcA", "TxnNumber": 0, "PrevOpTime": "HAAAABF0cwAAAAAAAAAAABJ0AP//////////AA==" }: applying transaction op: op: {"Timestamp":{"T":1715686554,"I":2},"Term":1,"Hash":null,"Version":0,"Operation":"i","Namespace":"config.databases","Object":[{"Key":"_id","Value":"test"},{"Key":"primary","Value":"rs2"},{"Key":"partitioned","Value":false},{"Key":"version","Value":[{"Key":"uuid","Value":{"Subtype":4,"Data":"JhfYtYSxRgyybRK0onMx4Q=="}},{"Key":"timestamp","Value":{"T":1715686553,"I":15}},{"Key":"lastMod","Value":1}]}],"Query":[{"Key":"_id","Value":"test"}],"UI":{"Subtype":4,"Data":"bupKEYAPQaK8mXos5HBa9w=="},"LSID":null,"TxnNumber":null,"PrevOpTime":null} | merr <nil>: applyOps: (UnknownError) applyOps had one or more errors applying ops

AssertionError: Error: backup '2024-05-14T11:41:20Z' didn't finish successfully

AssertionError: Starting restore 2024-05-14T12:01:18.845946545Z to point-in-time 2024-05-14T12:00:48 from '2024-05-14T12:00:00Z'...Started logical restore. Waiting to finish...Error: operation failed with: reply oplog: replay chunk 1715688046.1715688076: apply oplog for chunk: applying an entry: op: {"Timestamp":{"T":1715688047,"I":3},"Term":2,"Hash":null,"Version":2,"Operation":"i","Namespace":"test.system.views","Object":[{"Key":"_id","Value":"test.test2"},{"Key":"viewOn","Value":"system.buckets.test2"},{"Key":"pipeline","Value":[[{"Key":"$_internalUnpackBucket","Value":[{"Key":"timeField","Value":"timestamp"},{"Key":"metaField","Value":"data"},{"Key":"bucketMaxSpanSeconds","Value":3600}]}]]}],"Query":[{"Key":"_id","Value":"test.test2"}],"UI":{"Subtype":4,"Data":"vvOyqutcRSaEanLtUHhN2g=="},"LSID":null,"TxnNumber":null,"PrevOpTime":null} | merr <nil>: applyOps: (UnknownError) applyOps had one or more errors applying ops

AssertionError: Starting restore 2024-05-14T12:10:47.926559823Z from '2024-05-14T12:10:18Z'...Started logical restore. Waiting to finish...Error: operation failed with: reply oplog: replay chunk 1715688618.1715688628: apply oplog for chunk: applying an entry: op: {"Timestamp":{"T":1715688618,"I":10},"Term":1,"Hash":null,"Version":2,"Operation":"i","Namespace":"test.system.buckets.test2","Object":[{"Key":"_id","Value":"664354981751f90b50a3a380"},{"Key":"control","Value":[{"Key":"version","Value":1},{"Key":"min","Value":[{"Key":"_id","Value":"664354aa8600ccc4e863ed24"},{"Key":"timestamp","Value":"2024-05-14T12:10:00Z"}]},{"Key":"max","Value":[{"Key":"_id","Value":"664354aa8600ccc4e863ed24"},{"Key":"timestamp","Value":"2024-05-14T12:10:18.53Z"}]}]},{"Key":"meta","Value":9},{"Key":"data","Value":[{"Key":"_id","Value":[{"Key":"0","Value":"664354aa8600ccc4e863ed24"}]},{"Key":"timestamp","Value":[{"Key":"0","Value":"2024-05-14T12:10:18.53Z"}]}]}],"Query":[{"Key":"meta","Value":9},{"Key":"_id","Value":"664354981751f90b50a3a380"}],"UI":null,"LSID":"SAAAAAVpZAAQAAAABDMHA8P+BEIMnCweieE6ruQFdWlkACAAAAAAY5mrDaxi8gv8RmdTsQ+1j7fmkr7JUsabhNmXAheU0fgA","TxnNumber":1,"PrevOpTime":"HAAAABF0cwAAAAAAAAAAABJ0AP//////////AA=="} | merr <nil>: applyOps: (NamespaceNotFound) cannot apply insert or update operation on a non-existent namespace test.system.buckets.test2: { ts: Timestamp(1715688618, 10), t: 1, v: 2, op: "i", ns: "test.system.buckets.test2", o: { _id: ObjectId('664354981751f90b50a3a380'), control: { version: 1, min: { _id: ObjectId('664354aa8600ccc4e863ed24'), timestamp: new Date(1715688600000) }, max: { _id: ObjectId('664354aa8600ccc4e863ed24'), timestamp: new Date(1715688618530) } }, meta: 9, data: { _id: { 0: ObjectId('664354aa8600ccc4e863ed24') }, timestamp: { 0: new Date(1715688618530) } } }, o2: { meta: 9, _id: ObjectId('664354981751f90b50a3a380') }, lsid: { id: UUID("330703c3-fe04-420c-9c2c-1e89e13aaee4"), uid: BinData(0, 6399AB0DAC62F20BFC466753B10FB58FB7E692BEC952C69B84D997021794D1F8) }, txnNumber: 1, prevOpTime: { ts: Timestamp(0, 0), t: -1 } }

pymongo.errors.OperationFailure: got stale shardVersion response from shard rs1 at host rs101:27017 :: caused by :: timestamp mismatch detected for test.test2, full error: {'ok': 0.0, 'errmsg': 'got stale shardVersion response from shard rs1 at host rs101:27017 :: caused by :: timestamp mismatch detected for test.test2', 'code': 13388, 'codeName': 'StaleConfig', 'ns': 'test.test2', 'vReceived': {'e': ObjectId('664354eb8c46bccacf564e46'), 't': Timestamp(1715688683, 51), 'v': Timestamp(1, 3)}, 'vWanted': {'e': ObjectId('000000000000000000000000'), 't': Timestamp(0, 0), 'v': Timestamp(0, 0)}, 'shardId': 'rs1', '$clusterTime': {'clusterTime': Timestamp(1715688726, 26), 'signature': {'hash': b'\x90\xc5\x0f\xae\xe6\x88\x05\xd5hl\xcb\x91\xb2\xff\x18\x82b\xe2\x8d\xdb', 'keyId': 7368826693407997975}}, 'operationTime': Timestamp(1715688726, 26)}

AssertionError: Starting restore 2024-05-14T11:38:09.800453019Z to point-in-time 2024-05-14T11:36:33 from '2024-05-14T11:35:28Z'...Started logical restore. Waiting to finish...Error: operation failed with: reply oplog: replay chunk 1715686537.1715686591: apply oplog for chunk: applying an entry: op: {"Timestamp":{"T":1715686562,"I":2},"Term":1,"Hash":null,"Version":2,"Operation":"i","Namespace":"config.databases","Object":[{"Key":"_id","Value":"test"},{"Key":"primary","Value":"rs1"},{"Key":"partitioned","Value":false},{"Key":"version","Value":[{"Key":"uuid","Value":{"Subtype":4,"Data":"/DtLiEcoQNa8HjFLLrxZtA=="}},{"Key":"timestamp","Value":{"T":1715686561,"I":15}},{"Key":"lastMod","Value":1}]}],"Query":[{"Key":"_id","Value":"test"}],"UI":{"Subtype":4,"Data":"7jXvmpcqTx+xzo9cucbbXg=="},"LSID":null,"TxnNumber":null,"PrevOpTime":null} | merr <nil>: applyOps: (NamespaceNotFound) cannot apply insert or update operation on a non-existent namespace config.databases: { ts: Timestamp(1715686562, 2), t: 1, v: 2, op: "i", ns: "config.databases", o: { _id: "test", primary: "rs1", partitioned: false, version: { uuid: UUID("fc3b4b88-4728-40d6-bc1e-314b2ebc59b4"), timestamp: Timestamp(1715686561, 15), lastMod: 1 } }, o2: { _id: "test" }, ui: UUID("ee35ef9a-972a-4f1f-b1ce-8f5cb9c6db5e") }

AssertionError: Starting restore 2024-05-14T12:03:30.548984043Z to point-in-time 2024-05-14T12:03:00 from '2024-05-14T12:02:10Z'...Started logical restore. Waiting to finish...Error: operation failed with: reply oplog: replay chunk 1715688177.1715688207: apply oplog for chunk: applying an entry: op: {"Timestamp":{"T":1715688178,"I":2},"Term":1,"Hash":null,"Version":2,"Operation":"i","Namespace":"test.system.views","Object":[{"Key":"_id","Value":"test.test2"},{"Key":"viewOn","Value":"system.buckets.test2"},{"Key":"pipeline","Value":[[{"Key":"$_internalUnpackBucket","Value":[{"Key":"timeField","Value":"timestamp"},{"Key":"metaField","Value":"data"},{"Key":"bucketMaxSpanSeconds","Value":3600}]}]]}],"Query":[{"Key":"_id","Value":"test.test2"}],"UI":{"Subtype":4,"Data":"dPWcknJ8RjelZt+eTOUZcQ=="},"LSID":null,"TxnNumber":null,"PrevOpTime":null} | merr <nil>: applyOps: (NamespaceNotFound) Failed to apply operation due to missing collection (74f59c92-727c-4637-a566-df9e4ce51971): { ts: Timestamp(1715688178, 2), t: 1, v: 2, op: "i", ns: "test.system.views", o: { _id: "test.test2", viewOn: "system.buckets.test2", pipeline: [ { $_internalUnpackBucket: { timeField: "timestamp", metaField: "data", bucketMaxSpanSeconds: 3600 } } ] }, o2: { _id: "test.test2" }, ui: UUID("74f59c92-727c-4637-a566-df9e4ce51971"), h: 0, wall: new Date(0) }

AssertionError: Starting restore 2024-05-14T12:13:42.620726544Z from '2024-05-14T12:13:13Z'...Started logical restore. Waiting to finish....Error: operation failed with: waiting for dumpDone: cluster failed: waiting for dumpDone: cluster failed: reply oplog: replay chunk 1715688793.1715688802: apply oplog for chunk: applying an entry: op: {"Timestamp":{"T":1715688793,"I":6},"Term":1,"Hash":null,"Version":2,"Operation":"i","Namespace":"test.system.buckets.test2","Object":[{"Key":"_id","Value":"6643554cc50f9bf082fddbd0"},{"Key":"control","Value":[{"Key":"version","Value":1},{"Key":"min","Value":[{"Key":"_id","Value":"6643555908fe37af0e9e3d75"},{"Key":"timestamp","Value":"2024-05-14T12:13:00Z"}]},{"Key":"max","Value":[{"Key":"_id","Value":"6643555908fe37af0e9e3d75"},{"Key":"timestamp","Value":"2024-05-14T12:13:13.261Z"}]}]},{"Key":"meta","Value":9},{"Key":"data","Value":[{"Key":"_id","Value":[{"Key":"0","Value":"6643555908fe37af0e9e3d75"}]},{"Key":"timestamp","Value":[{"Key":"0","Value":"2024-05-14T12:13:13.261Z"}]}]}],"Query":[{"Key":"meta","Value":9},{"Key":"_id","Value":"6643554cc50f9bf082fddbd0"}],"UI":null,"LSID":"SAAAAAVpZAAQAAAABKNTHpeuD0vUuilzu/eleTEFdWlkACAAAAAAY5mrDaxi8gv8RmdTsQ+1j7fmkr7JUsabhNmXAheU0fgA","TxnNumber":1,"PrevOpTime":"HAAAABF0cwAAAAAAAAAAABJ0AP//////////AA=="} | merr <nil>: applyOps: (NamespaceNotFound) cannot apply insert or update operation on a non-existent namespace test.system.buckets.test2: { ts: Timestamp(1715688793, 6), t: 1, v: 2, op: "i", ns: "test.system.buckets.test2", o: { _id: ObjectId('6643554cc50f9bf082fddbd0'), control: { version: 1, min: { _id: ObjectId('6643555908fe37af0e9e3d75'), timestamp: new Date(1715688780000) }, max: { _id: ObjectId('6643555908fe37af0e9e3d75'), timestamp: new Date(1715688793261) } }, meta: 9, data: { _id: { 0: ObjectId('6643555908fe37af0e9e3d75') }, timestamp: { 0: new Date(1715688793261) } } }, o2: { meta: 9, _id: ObjectId('6643554cc50f9bf082fddbd0') }, lsid: { id: UUID("a3531e97-ae0f-4bd4-ba29-73bbf7a57931"), uid: BinData(0, 6399AB0DAC62F20BFC466753B10FB58FB7E692BEC952C69B84D997021794D1F8) }, txnNumber: 1, prevOpTime: { ts: Timestamp(0, 0), t: -1 } }

pymongo.errors.OperationFailure: got stale shardVersion response from shard rs1 at host rs101:27017 :: caused by :: timestamp mismatch detected for test.test2, full error: {'ok': 0.0, 'errmsg': 'got stale shardVersion response from shard rs1 at host rs101:27017 :: caused by :: timestamp mismatch detected for test.test2', 'code': 13388, 'codeName': 'StaleConfig', 'ns': 'test.test2', 'vReceived': {'t': Timestamp(1715688861, 53), 'e': ObjectId('6643559d050697c38729c2d9'), 'v': Timestamp(1, 1)}, 'vWanted': {'t': Timestamp(0, 0), 'e': ObjectId('000000000000000000000000'), 'v': Timestamp(0, 0)}, 'shardId': 'rs1', '$clusterTime': {'clusterTime': Timestamp(1715688911, 22), 'signature': {'hash': b'\x94]!\xd9[R\x13>i\t\x06S\xdb\xba\xdc\xb5\xfcq.n', 'keyId': 7368827449322242074}}, 'operationTime': Timestamp(1715688911, 22)}

Failed: Timeout >300.0s

AssertionError: Starting restore 2024-05-14T11:42:42.854624125Z to point-in-time 2024-05-14T11:41:11 from '2024-05-14T11:40:05Z'...Started logical restore. Waiting to finish...Error: operation failed with: reply oplog: replay chunk 1715686814.1715686868: apply oplog for chunk: applying an entry: op: {"Timestamp":{"T":1715686839,"I":2},"Term":1,"Hash":null,"Version":2,"Operation":"i","Namespace":"config.databases","Object":[{"Key":"_id","Value":"test"},{"Key":"primary","Value":"rs1"},{"Key":"partitioned","Value":true},{"Key":"version","Value":[{"Key":"uuid","Value":{"Subtype":4,"Data":"sr21Sf7QTRS1g2ahEcQiFw=="}},{"Key":"timestamp","Value":{"T":1715686838,"I":16}},{"Key":"lastMod","Value":1}]}],"Query":null,"UI":{"Subtype":4,"Data":"sCBSa3vSSp29A1RRLQbYmA=="},"LSID":null,"TxnNumber":null,"PrevOpTime":null} | merr <nil>: applyOps: (NamespaceNotFound) cannot apply insert or update operation on a non-existent namespace config.databases: { ts: Timestamp(1715686839, 2), t: 1, v: 2, op: "i", ns: "config.databases", o: { _id: "test", primary: "rs1", partitioned: true, version: { uuid: UUID("b2bdb549-fed0-4d14-b583-66a111c42217"), timestamp: Timestamp(1715686838, 16), lastMod: 1 } }, ui: UUID("b020526b-7bd2-4a9d-bd03-54512d06d898") }

AssertionError: Error: backup '2024-05-14T11:46:27Z' didn't finish successfully

AssertionError: Starting restore 2024-05-14T12:06:23.505312398Z to point-in-time 2024-05-14T12:05:53 from '2024-05-14T12:05:03Z'...Started logical restore. Waiting to finish....Error: operation failed with: reply oplog: replay chunk 1715688350.1715688380: apply oplog for chunk: applying an entry: op: {"Timestamp":{"T":1715688351,"I":2},"Term":1,"Hash":null,"Version":2,"Operation":"i","Namespace":"test.system.views","Object":[{"Key":"_id","Value":"test.test2"},{"Key":"viewOn","Value":"system.buckets.test2"},{"Key":"pipeline","Value":[[{"Key":"$_internalUnpackBucket","Value":[{"Key":"timeField","Value":"timestamp"},{"Key":"metaField","Value":"data"},{"Key":"bucketMaxSpanSeconds","Value":3600},{"Key":"exclude","Value":[]}]}]]}],"Query":null,"UI":{"Subtype":4,"Data":"/q89ihAiQPO+kLExrE1/Zg=="},"LSID":null,"TxnNumber":null,"PrevOpTime":null} | merr <nil>: applyOps: (NamespaceNotFound) Failed to apply operation due to missing collection (feaf3d8a-1022-40f3-be90-b131ac4d7f66): { ts: Timestamp(1715688351, 2), t: 1, v: 2, op: "i", ns: "test.system.views", o: { _id: "test.test2", viewOn: "system.buckets.test2", pipeline: [ { $_internalUnpackBucket: { timeField: "timestamp", metaField: "data", bucketMaxSpanSeconds: 3600, exclude: [] } } ] }, ui: UUID("feaf3d8a-1022-40f3-be90-b131ac4d7f66"), h: 0, wall: new Date(0) }

Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: github/codeql-action/upload-sarif@v2. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.