[Stretch cluster] test device replacement in stretch cluster #9548

mashetty330 · 2024-03-22T10:54:48Z

No description provided.

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-stja1
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.15
OCS VERSION: 4.15
tested against branch: master

Job UNSTABLE (some or all tests failed).

github-actions · 2024-07-02T20:04:49Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 30 days if no further activity occurs.

tests/disaster-recovery/sc_arbiter/test_device_replacement.py

ocs-ci

Unknown PR validation on existing cluster

Cluster Name: mashetty-sta27
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job state: ABORTED.

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-sta27
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-sta26
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job PASSED.

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-sta26
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-sta26
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-sta26
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job PASSED.

tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py

Akarsha-rai

Small comment. rest lgtm

openshift-ci · 2024-11-07T07:31:09Z

New changes are detected. LGTM label has been removed.

openshift-ci · 2024-11-07T07:31:12Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mashetty330

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2024-11-07T07:31:18Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mashetty330

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-st04
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-st11
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-st19
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-st19
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-stj22
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.18
OCS VERSION: 4.18
tested against branch: master

Job PASSED.

ebenahar · 2025-01-05T12:53:42Z

tests/functional/disaster-recovery/sc_arbiter/test_add_capacity.py

@@ -107,6 +112,13 @@ def test_cluster_expansion(
        )
        logger.info("All the workloads pods are successfully up and running")

+        # setup vm and write some data to the VM instance


this code block (til line 160) is quite generic (it repeats itself in https://github.com/red-hat-storage/ocs-ci/pull/9548/files#diff-46f753f4e560b98288b4f1da2258354d49a5ad5d15b1e3110e64e02829a06ed5R26) and could be used for other tests that include VM workloads. Suggest moving it to a VM related helper file.

As discussed offline, created issue #11186 to track. Will be addressing this in the next release (4.19)

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-stj22
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.18
OCS VERSION: 4.18
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-stj22
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py
Additional Test Params:
OCP VERSION: 4.18
OCS VERSION: 4.18
tested against branch: master

Job PASSED.

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-stj22
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_add_capacity.py
Additional Test Params:
OCP VERSION: 4.18
OCS VERSION: 4.18
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-stj22
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_add_capacity.py
Additional Test Params:
OCP VERSION: 4.18
OCS VERSION: 4.18
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-stj23
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_mon_osd_failures.py
Additional Test Params:
OCP VERSION: 4.18
OCS VERSION: 4.18
tested against branch: master

Job UNSTABLE (some or all tests failed).

Signed-off-by: Mahesh Shetty <[email protected]>

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-stj23
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_mon_osd_failures.py
Additional Test Params:
OCP VERSION: 4.18
OCS VERSION: 4.18
tested against branch: master

Job PASSED.

PrasadDesala · 2025-01-24T09:08:41Z

ocs_ci/ocs/resources/pv.py

+    Args:
+        storage_class (str): storage class
+        status (str): status of the pv
+    Returns:


add an empty line above this

PrasadDesala · 2025-01-24T09:12:41Z

tests/functional/disaster-recovery/sc_arbiter/test_add_capacity.py

@@ -123,6 +135,29 @@ def test_cluster_expansion(
        sc_obj.post_failure_checks(start_time, end_time, wait_for_read_completion=False)
        logger.info("Successfully verified with post failure checks for the workloads")

+        # check vm data written before the failure for integrity


this md5sum is captured after the failure. Please update the comment accordingly

PrasadDesala · 2025-01-24T09:14:12Z

tests/functional/disaster-recovery/sc_arbiter/test_add_capacity.py

+        # stop the VM
+        vm_obj.stop()
+        logger.info("Stoped the VM successfully")


what is the need to stop the VM?

this is included part of test to see if we can stop the vm post recovery

PrasadDesala · 2025-01-24T09:18:56Z

tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py

+        sc_obj.post_failure_checks(start_time, end_time, wait_for_read_completion=False)
+        logger.info("Successfully verified with post failure checks for the workloads")
+
+        # check vm data written before the failure for integrity


this md5sum is captured after the failure. Please update the comment accordingly

PrasadDesala · 2025-01-24T09:19:39Z

tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py

+        # stop the VM
+        vm_obj.stop()
+        logger.info("Stoped the VM successfully")


what is the need to stop the VM?

this is included part of test to see if we can stop the vm post recovery

PrasadDesala · 2025-01-24T09:23:30Z

tests/functional/disaster-recovery/sc_arbiter/test_add_capacity.py

@@ -107,6 +112,13 @@ def test_cluster_expansion(
        )
        logger.info("All the workloads pods are successfully up and running")

+        # setup vm and write some data to the VM instance


PrasadDesala · 2025-01-24T09:26:46Z

tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py

+            pod.wait_for_pod_delete(timeout=120)
+        logger.info("All old CephFS logreader pods are deleted")
+
+        # check for any data loss


Checking for data loss and data corruption is quite generic and can be used in other stretch cluster tests as well. I would suggest moving it to a common location outside the test and calling that function within the test

ack. will take care of it as part of #11186 in the next release

ocs-ci

PR validation on existing cluster

Cluster Name: mashetty-stj23
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/functional/disaster-recovery/sc_arbiter/test_add_capacity.py
Additional Test Params:
OCP VERSION: 4.18
OCS VERSION: 4.18
tested against branch: master

Job UNSTABLE (some or all tests failed).

mashetty330 self-assigned this Mar 22, 2024

mashetty330 requested a review from a team as a code owner March 22, 2024 10:54

pull-request-size bot added the size/M PR that changes 30-99 lines label Mar 22, 2024

ocs-ci reviewed Apr 3, 2024

View reviewed changes

github-actions bot added the lifecycle/stale No recent activity label Jul 2, 2024

hnallurv added the team/e2e E2E team related issues/PRs label Jul 10, 2024

PrasadDesala removed the lifecycle/stale No recent activity label Jul 25, 2024

PrasadDesala reviewed Jul 25, 2024

View reviewed changes

tests/disaster-recovery/sc_arbiter/test_device_replacement.py Outdated Show resolved Hide resolved

mashetty330 added the do-not-merge/work-in-progress label Jul 30, 2024

mashetty330 force-pushed the sc_device_replacement branch from 4b75979 to 66d7603 Compare August 1, 2024 06:26

mashetty330 requested a review from a team as a code owner August 1, 2024 06:26

ocs-ci reviewed Aug 8, 2024

View reviewed changes

vavuthu mentioned this pull request Aug 12, 2024

Device replacement test fails on LSO cluster with the error: ocs_ci.ocs.exceptions.VSLMNotFoundException #9898

Closed

ocs-ci reviewed Aug 29, 2024

View reviewed changes

mashetty330 force-pushed the sc_device_replacement branch from 66d7603 to 84a914b Compare August 29, 2024 11:18

pull-request-size bot added size/L PR that changes 100-499 lines and removed size/M PR that changes 30-99 lines labels Aug 29, 2024

mashetty330 removed the do-not-merge/work-in-progress label Aug 29, 2024

ocs-ci reviewed Aug 29, 2024

View reviewed changes

ocs-ci reviewed Aug 30, 2024

View reviewed changes

mashetty330 added the Verified Mark when PR was verified and log provided label Aug 30, 2024

mashetty330 requested a review from Akarsha-rai September 2, 2024 06:28

Akarsha-rai reviewed Sep 10, 2024

View reviewed changes

tests/functional/disaster-recovery/sc_arbiter/test_device_replacement.py Show resolved Hide resolved

Akarsha-rai previously approved these changes Sep 10, 2024

View reviewed changes

openshift-ci bot assigned Akarsha-rai Sep 10, 2024

openshift-ci bot added the lgtm label Sep 10, 2024

mashetty330 dismissed Akarsha-rai’s stale review via 64111a2 November 7, 2024 07:31

mashetty330 force-pushed the sc_device_replacement branch from 69b33a6 to 64111a2 Compare November 7, 2024 07:31

openshift-ci bot removed the lgtm label Nov 7, 2024

ocs-ci reviewed Nov 7, 2024

View reviewed changes

mashetty330 force-pushed the sc_device_replacement branch from 64111a2 to 37f3659 Compare November 11, 2024 11:47

ocs-ci reviewed Nov 11, 2024

View reviewed changes

mashetty330 force-pushed the sc_device_replacement branch from 37f3659 to 8567577 Compare November 21, 2024 10:29

ocs-ci reviewed Nov 21, 2024

View reviewed changes

mashetty330 force-pushed the sc_device_replacement branch from b2c6c0d to f6a39ca Compare January 3, 2025 11:52

ocs-ci reviewed Jan 3, 2025

View reviewed changes

ebenahar reviewed Jan 5, 2025

View reviewed changes

mashetty330 mentioned this pull request Jan 22, 2025

Create separate func for VM workloads setup and verification in stretch cluster tests to reduce the repetitive code #11186

Open

ocs-ci reviewed Jan 22, 2025

View reviewed changes

mashetty330 force-pushed the sc_device_replacement branch from 78ac207 to 66346f5 Compare January 22, 2025 05:47

ocs-ci reviewed Jan 22, 2025

View reviewed changes

ocs-ci reviewed Jan 23, 2025

View reviewed changes

mashetty330 and others added 5 commits January 23, 2025 15:03

device replacement in stretch cluster

d152252

Signed-off-by: Mahesh Shetty <[email protected]>

run workloads with device replacement

ddd260d

Signed-off-by: Mahesh Shetty <[email protected]>

add markers

8b00a27

Signed-off-by: Mahesh Shetty <[email protected]>

add cnv workloads to the add capacity and device replacement tests

a250e1d

Signed-off-by: Mahesh Shetty <[email protected]>

Rebase and fix some of the code

64a4d37

Signed-off-by: Mahesh Shetty <[email protected]>

mashetty330 force-pushed the sc_device_replacement branch from 66346f5 to 64a4d37 Compare January 23, 2025 09:34

ocs-ci reviewed Jan 23, 2025

View reviewed changes

PrasadDesala reviewed Jan 24, 2025

View reviewed changes

ocs-ci reviewed Jan 27, 2025

View reviewed changes

[Stretch cluster] test device replacement in stretch cluster #9548

Are you sure you want to change the base?

[Stretch cluster] test device replacement in stretch cluster #9548

Conversation

mashetty330 commented Mar 22, 2024

ocs-ci left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 2, 2024

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

Akarsha-rai left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Nov 7, 2024

openshift-ci bot commented Nov 7, 2024

openshift-ci bot commented Nov 7, 2024

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment