[QA] Cycle 2: Feature testing for reCAPCTHA

[sdzhepa]

Description and purposes

Code version information:

On the starting day we used next commits and branch

• Magento Open Source: branch 2.4-develop commit: magento/magento2@15dc874

• Magento Commerce(EE): branch 2.4-develop commit: https://github.com/magento/magento2ee/commit/e736a90af06591661ba1653436a300d7674b5218

• Magento security-package: branch 1.0-develop commit: 496055e

  NOTE: IT is possible that during execution codebase could be updated due to critical issues that block test execution. Or exceptional situation

Magento editions for test-case execution:

1. Magento Commerce(EE)
-- The main env for full testing
-- Prepared Cloud instance with sample data
-- Should be executed all test-cases that we have. (exception could be only for specific cases that not possible to execute due to Cloud Env)

2. Magento Open Source(CE)
-- Additional env for execution(on local machine). Install using Metapackage Installation Guide
-- Should contains only the most critical cases e.g. Checkout Flow etc
-- Here should be executed tests that not possible to do on EE due to limitation on Cloud env
-- All found issues on EE should be checked on CE before reporting

Workflow and testing process:

Should start in the next order:

1. Cycle related to Magento Commerce(EE) Cloud.
2. Cycle related to Magento Open Source(CE) Local
3. Cycle related to cases/user flows in different browser

Bug reporting:

• Each found bug should be reverified on CE local env. To be sure is it env/edition specific or not
• Each bug should be reported as an issue in this repository. It should have labels: 'Component: Google reCAPCTHA' and 'bug'

Failed/Skipped/Blocked test cases:

• Each FAILED test-case should have a link to the created issue in the "Execution" section and additional comment if required
• Each SKIPPED/BLOCKED test-case should have a clear comment with explanation in the "Execution" section

Bug triaging and fixing

• Each newly created bug should be triaged. Discuss and understand its priority/severity. Must be fixed now or can be postponed
• Only after triage and decision, devs can start progress on it.
• Each fixed bug must be tested in the scope of Pull Request.
• Based on the number of new bugs and specific will be decided do we need additional Test Run or not

Test Cycle information, links, and Additional details

Magento Commerce Cloud

Name: Cycle 2: reCAPCTHA testing on Magneto Cloud

Additional testing scope:

• [QA] Need to test reCaptcha functionality and be sure that unique per each session and not cached  #102
• Recaptcha initialization #191
• security-package: Changes for include Google API js file - Admin panel #183
• security-package/issues/170: Renamed error message. #174

Magento Open Source(CE) Local

Name: Cycle 2: reCAPCTHA testing on Magneto Open Source

Additional testing scope:

• upgrade security-package/issues/101: Add ReCaptchaMigration module #89
• cli commands to disable reCAPCTH for admin security-package/issues/143: Update for console commands security:recaptcha:disable-for-user-login #147
• security-package: Added try/catch to Magento\ReCaptchaUi\Model\RequesttHandler #175
• check system/exception log files during execution, is it has enny errors/warnigs/etc

Browser specific testing (Google Chrome, Mozilla Firefox, IE11/Edge)

Name: Cycle 2: reCAPCTHA testing on Magneto Open Source

Additional testing scope:

• security-package/issues/189: Updated CSS styles. #190 (comment)

[sdzhepa]

Testing report on Apr 10 2020

Start testing.
No new bugs found

Details

1. Cycle related to Magento Commerce(EE) Cloud.

• Total test cases: 70
• Passed: 20
• Failed: 0
• Skipped 0

2. Cycle related to Magento Open Source(CE) Local

• Total test cases: 38
• Passed: 11
• Failed: 0
• Skipped 0

3. Cycle related to cases/user flows in different browser

• Total test cases: 21
• Passed: 0
• Failed: 0
• Skipped 0

[sdzhepa]

Testing report on Apr 14 2020 (3rd day of testing)

Found Bugs

• Recaptch badge is not shown on PayPal PayflowPro payment form if 'Invisible Badge Position' is 'Bottom Left/Right' #204
• reCAPTCHA badge with several language is not shown fully  #203
• "Uncaught (in promise) null" error and stuck page if try to login in Admin with invalid API for "reCAPTCHA v2|v3 Invisible" keys for  #205

Details

1. Cycle related to Magento Commerce(EE) Cloud.

• Total test cases: 70
• Passed: 51
• Failed: 2
• Skipped: 5 List

2. Cycle related to Magento Open Source(CE) Local

• Total test cases: 38
• Passed: 21
• Failed: 0
• Skipped 0

3. Cycle related to cases/user flows in different browser

• Total test cases: 21 per each browser(FireFox, Chrome, IE11, Endge)

FireFox

• Passed: 0
• Failed: 0

Chrome

• Passed: 13
• Failed: 0

IE 11

• Passed: 0
• Failed: 0

Edge

• Passed: 0
• Failed: 0

[sdzhepa]

Testing report on Apr 15, 2020 (4rd day of testing)

No new bugs found

All Found Bugs

• Recaptch badge is not shown on PayPal PayflowPro payment form if 'Invisible Badge Position' is 'Bottom Left/Right' #204
• reCAPTCHA badge with several language is not shown fully  #203
• "Uncaught (in promise) null" error and stuck page if try to login in Admin with invalid API for "reCAPTCHA v2|v3 Invisible" keys for  #205

Details

1. Cycle related to Magento Commerce(EE) Cloud.

• Total test cases: 70
• Passed: 56
• Failed: 3
• Skipped: 7 List

2. DONE Cycle related to Magento Open Source(CE) Local

• Total test cases: 38
• Passed: 38
• Failed: 0
• Skipped 0

3. Cycle related to cases/user flows in different browser

• Total test cases: 21 per each browser(Firefox, Chrome, IE11, Edge)

FireFox

• Passed: 14
• Failed: 0

Chrome

• Passed: 14
• Failed: 0

IE 11

• Passed: 0
• Failed: 0

Edge

• Passed: 0
• Failed: 0

[engcom-Delta]

Results for addional testing scope:

Magento Commerce Cloud

Name: Cycle 2: reCAPCTHA testing on Magneto Cloud

Additional testing scope:

• [QA] Need to test reCaptcha functionality and be sure that unique per each session and not cached  #102 - Done
• Recaptcha initialization #191 - Related issue "Uncaught (in promise) null" error and stuck page if try to login in Admin with invalid API for "reCAPTCHA v2|v3 Invisible" keys for  #205 according comment
• security-package: Changes for include Google API js file - Admin panel #183 - Done
• security-package/issues/170: Renamed error message. #174 - no longer valid due to security-package: Added try/catch to Magento\ReCaptchaUi\Model\RequesttHandler #175

Magento Open Source(CE) Local

Name: Cycle 2: reCAPCTHA testing on Magneto Open Source

Additional testing scope:

• upgrade security-package/issues/101: Add ReCaptchaMigration module #89 - Done
• cli commands to disable reCAPCTH for admin security-package/issues/143: Update for console commands security:recaptcha:disable-for-user-login #147 - Done
• security-package: Added try/catch to Magento\ReCaptchaUi\Model\RequesttHandler #175 - Related issue "Uncaught (in promise) null" error and stuck page if try to login in Admin with invalid API for "reCAPTCHA v2|v3 Invisible" keys for  #205 for better error handler for v2-v3 Invisible: not fixed by Recaptcha initialization #191 (Note: Initial issue is fixed)
• check system/exception log files during execution, is it has enny errors/warnigs/etc - Done: no additional errors was found

Browser specific testing (Google Chrome, Mozilla Firefox, IE11/Edge)

Name: Cycle 2: reCAPCTHA testing on Magneto Open Source

Additional testing scope:

• security-package/issues/189: Updated CSS styles. #190 (comment) - Done

[sdzhepa]

Testing report on Apr 16, 2020 (5th day of testing)

All planned test cycles were executed.
Testing is done

All Found Bugs

• Recaptch badge is not shown on PayPal PayflowPro payment form if 'Invisible Badge Position' is 'Bottom Left/Right' #204
• reCAPTCHA badge with several language is not shown fully  #203
• "Uncaught (in promise) null" error and stuck page if try to login in Admin with invalid API for "reCAPTCHA v2|v3 Invisible" keys for  #205
• Not the best user experience for reCPACTHA with PayPal if user does not check reCAPTHCA #207 (not regression bug and not related to reCACPCTHA)

Details

1. DONE Cycle related to Magento Commerce(EE) Cloud.

• Total test cases: 70
• Passed: 60
• Failed: 3
• Skipped: 7 List

2. DONE Cycle related to Magento Open Source(CE) Local

• Total test cases: 38
• Passed: 38
• Failed: 0
• Skipped 0

3. Cycle related to cases/user flows in different browser

• Total test cases: 21 per each browser(Firefox, Chrome, IE11, Edge)

FireFox

• Passed: 14
• Failed: 0

Chrome

• Passed: 14
• Failed: 0

IE 11

• Passed: 14
• Failed: 0

Edge

• Passed: 14
• Failed: 0

4. DONE Addition testing scope

• [QA] Cycle 2: Feature testing for reCAPCTHA #201 (comment)