Unable to generate Test files

[kolaente]

Preconditions

• php 7.4
• mftf version 3.5.1
• magento version 2.4.3
• Selenium driver 3.141.59 with chromium 98.0.4758.9

Steps to reproduce

I have a test which looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<tests xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:mftf:Test/etc/testSchema.xsd">
    <test name="StorefrontGuestCheckoutHandballTest">
        <annotations>
            <features value="Checkout"/>
            <stories value="Checkout via Guest Checkout"/>
            <title value="Guest Checkout - guest should be able to place an order"/>
            <description value="Should be able to place an order as a Guest"/>
            <severity value="CRITICAL"/>
            <testCaseId value="internal-66062"/>
            <group value="checkout"/>
        </annotations>
        <amOnPage url="{{HandballFuturaProduct.url}}" stepKey="onProductPage"/>
        <waitForPageLoad stepKey="waitForPageLoad1"/>
        <click selector="button[form=product_addtocart_form]" stepKey="addToCart"/>
        <waitForElementVisible selector="{{StorefrontCategoryMainSection.SuccessMsg}}" time="30" stepKey="waitForProductAdded"/>
        <see selector="{{StorefrontCategoryMainSection.SuccessMsg}}" userInput="Sie haben {{HandballFuturaProduct.name}} zu ihrem Warenkorb hinzugefügt." stepKey="seeAddedToCartMessage"/>
        <see selector="{{StorefrontProductDetailSection.quantity}}" userInput="1" stepKey="seeCartQuantity"/>
    </test>
</tests>

The test is placed in dev/tests/acceptance/tests/functional/Magento/FunctionalTest/CompanyGeneral/Test/StorefrontGuestCheckoutHandballTest.xml.

And a test suite which contains it:

<?xml version="1.0" encoding="UTF-8"?>
<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd">
    <suite name="Company_Checkout">
        <before>
            <magentoCLI stepKey="disableAdminSecurityKeyInUrl" command="config:set admin/security/use_form_key 0" />
            <magentoCLI stepKey="enableAdminAccountSharing" command="config:set admin/security/admin_account_sharing 1" />
            <magentoCLI stepKey="disableWYSIWYGEditor" command="config:set cms/wysiwyg/enabled disabled" />
            <magentoCLI stepKey="disableRecaptcha" command="config:set msp_securitysuite_recaptcha/frontend/enabled 0"/>
            <magentoCLI stepKey="setIndexModeToRealtime" command="indexer:set-mode realtime" />
            <magentoCLI stepKey="flushCache1" command="cache:flush" />
        </before>
        <include>
            <test name="StorefrontGuestCheckoutHandballTest"/>
        </include>
        <after>
            <magentoCLI stepKey="enableAdminSecurityKeyInUrl" command="config:set admin/security/use_form_key 1" />
            <magentoCLI stepKey="disableAdminAccountSharing" command="config:set admin/security/admin_account_sharing 0" />
            <magentoCLI stepKey="hideWYSIWYGEditor" command="config:set cms/wysiwyg/enabled hidden" />
            <magentoCLI stepKey="enableRecaptcha" command="config:set msp_securitysuite_recaptcha/frontend/enabled 1"/>
            <magentoCLI stepKey="setIndexModeToSchedule" command="indexer:set-mode schedule design_config_grid customer_grid catalog_category_product catalog_product_category catalogrule_rule catalog_product_attribute cataloginventory_stock catalogsearch_fulltext" />
            <magentoCLI stepKey="flushCache2" command="cache:flush" />
        </after>
    </suite>
</suites>

The testsuite is saved in dev/tests/acceptance/tests/_suite/CompanyCheckoutSuite.xml.

I have MODULE_ALLOWLIST=Magento_Framework,CompanyGeneral set in my .env.

Trying to generate the suite does not work:

$ vendor/bin/mftf generate:suite Company_Checkout --remove

NOTICE: 287 Section name violations detected. See mftf.log for details.[]
NOTICE: 741 section element name violations detected. See mftf.log for details.[]
NOTICE: 372 data entity name violations detected. See mftf.log for details.[]
NOTICE: 11 data entity key violations detected. See mftf.log for details.[]
NOTICE: 441 action group name violations detected. See mftf.log for details.[]
NOTICE: 45 Page name violations detected. See mftf.log for details.[]
ERROR:  1 Suite(s) has(have) tests with annotation errors or some included tests missing. See mftf.log for details.
ERROR: 17 Test(s) failed to generate. See mftf.log for details.

Suites Generated (with errors)

There's so many log entries I don't know what to do about this. Pretty much all of them seem unrelated to my test suite or the test I created.
Here is the full log

Trying to run the suite runs a million other tests, but not the one test specified in the suite:

$ vendor/bin/mftf run:group Company_Checkout --remove

ERROR:  1 Suite(s) has(have) tests with annotation errors or some included tests missing. See mftf.log for details.
ERROR: 17 Test(s) failed to generate. See mftf.log for details.

Generate Tests Command Run (with errors)

Codeception PHP Testing Framework v4.1.28
Powered by PHPUnit 9.1.5 by Sebastian Bergmann and contributors.
[Groups] Kuebler_Checkout

Magento\FunctionalTestingFramework.functional Tests (3567) ---------------------
Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, \Magento\FunctionalTestingFramework\Module\MagentoActionProxies, Asserts, \Magento\FunctionalTestingFramework\Helper\HelperContainer
--------------------------------------------------------------------------------

.... more test runs omitted....

The logs only tell me it worked:

[2022-01-28 15:43:23] Magento\FunctionalTestingFramework\Suite\SuiteGenerator.INFO: suite generated {"suite":"Company_Checkout","relative_path":"_generated/Company_Checkout"} []

Full logs here.

Trying to run only the test (without the suite) doesn't work either:

$ vendor/bin/mftf run:test StorefrontGuestCheckoutHandballTest --remove

ERROR:  1 Suite(s) has(have) tests with annotation errors or some included tests missing. See mftf.log for details.
ERROR: 1 Test(s) failed to generate. See mftf.log for details.

Generate Tests Command Run (with errors)


In RunTestCommand.php line 152:

  StorefrontGuestCheckoutHandballTestCest.php is not available under /home/konrad/projects/company.de/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/default/


run:test [-k|--skip-generate] [-r|--remove] [-f|--force] [-a|--allow-skipped] [-d|--debug [DEBUG]] [--] <name>...

This time though, the logs say

[2022-01-28 15:46:58] Magento\FunctionalTestingFramework\Test\Handlers\TestObjectHandler.ERROR: Line 289: Test StorefrontGuestCheckoutHandballTest not defined in xml. [] []
[2022-01-28 15:46:58] Magento\FunctionalTestingFramework\Suite\Util\SuiteObjectExtractor.ERROR: Unable to find <test> reference StorefrontGuestCheckoutHandballTest for suite StorefrontGuestCheckoutHandballTest [] []

but the test does exist, the mftf just doesn't find it??

Full log.

Honestly, at this point I'm pretty frustrated. I've tried for a few hours to get this to work, tried a few different configurations and nothing worked.

Expected result

Tests run or at least tell me what's wrong.

Actual result

Tests don't run.

[m2-assistant]

Hi @kolaente. Thank you for your report.
To speed up processing of this issue, make sure that you provided sufficient information.

Add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[kolaente]

Maybe related to #387

[hopeseekr]

Same here: