TestNG skipping tests - why?

Question

I am testing a web app with testng and selenium. The tests mainly consist in opening several pages of the app, and do some activities specific for each page. So I have an abstract base class which performs the "open page" test, and that defines an abstract method which is used as data provider for that test. Then there are several extending classes which provide implementation for the data provider, and which have several different tests apart from that of the base class. I have a testsuite.xml where all the classes are included, which is what I run from my eclipse.

The problem is that when I launch the test execution, testng runs the test in the base class for all objects, but skips systematically every other tests in the extending classes. Does anyone know why? Any information will be very appreciated...

To complete information, here are some of the classes and the xml I use:

Base class:

public abstract class BaseWebAppPageTest {

    @Test(dataProvider="getMenuLink")
    public void testOpen(String menulink){
        GenericPageActions.openPage(TestingContext.getSelenium(), menulink);
    }

    protected abstract String[][] getMenuLink();

}

And some extending classes:

TestLanguages:

public class TestLanguages extends BaseWebAppPageTest{

    @Test(dependsOnMethods={"testOpen"}, dataProvider="getCreateData")
    public void testCreateCorrect(String code, String description, String textDirection, String flag){
        Selenium selenium = TestingContext.getSelenium();
        LanguagesManagementActions.create(selenium, code, description, textDirection, flag);
    Assert.assertTrue(selenium.isTextPresent("Successfully created language"));
    }

    @Test(dependsOnMethods={"testCreateCorrect"}, dataProvider="getCreateData")
    public void testFilter(String code, String description, String textDirection, String flag){
        Selenium selenium = TestingContext.getSelenium();
        LanguagesManagementActions.filterTable(selenium, 2, code, 30000);
    Assert.assertTrue(selenium.getXpathCount("xpath=//span[.='"+code+"']").intValue() == 1);
    }

    @Test(dependsOnMethods={"testCreateCorrect"}, dataProvider="getCreateData")
    public void testModify(String code, String description, String textDirection, String flag){
        Selenium selenium = TestingContext.getSelenium();
        LanguagesManagementActions.modify(TestingContext.getSelenium(), code, description, textDirection, flag);
        Assert.assertTrue(selenium.isTextPresent("Successfully updated language"));
    }

    @Override
    @DataProvider
    protected String[][] getMenuLink() {
        return(TestingContext.getDataReader().getTableArray("openviewpage", "MULTILINGUAL_LANGUAGES"));
    }

    @DataProvider
    protected String[][] getCreateData() {
        return(TestingContext.getDataReader().getTableArray("multilingualcreate", "LANGUAGES"));
    }

}

TestTranslations:

public class TestTranslations extends BaseWebAppPageTest{

@Test(dependsOnMethods={"testOpen"}, dataProvider="getCreateData")
public void createCorrect(String code, String targetLanguage, String translation){
    Selenium selenium = TestingContext.getSelenium();
    TranslationsManagementActions.create(selenium, code, targetLanguage, translation);
    Assert.assertTrue(selenium.isTextPresent("Successfully created translation"));
}

@Test(dependsOnMethods={"createCorrect"}, dataProvider="getCreateData")
public void update(String code, String targetLanguage, String translation){
    Selenium selenium = TestingContext.getSelenium();
    TranslationsManagementActions.update(selenium, code, targetLanguage, translation);
    Assert.assertTrue(selenium.isTextPresent("Successfully updated translation"));
}

@Test(dependsOnMethods={"createCorrect"}, dataProvider="getCreateData")
public void filter(String code, String targetLanguage, String translation){
    Selenium selenium = TestingContext.getSelenium();
    TranslationsManagementActions.filterTable(selenium, 2, code, 30000);
    Assert.assertTrue(selenium.isElementPresent("xpath=//span[.='"+translation+"']"));
}

@Override
@DataProvider
protected String[][] getMenuLink() {
    return(TestingContext.getDataReader().getTableArray("openviewpage", "MULTILINGUAL_TRANSLATIONS"));
}

@DataProvider
protected String[][] getCreateData() {
    return(TestingContext.getDataReader().getTableArray("multilingualcreate", "TRANSLATIONS"));
}

}

And finally, the TestSuite.xml:

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
    <suite name="WebAppSuiteTest" parallel="none">
      <parameter name="selenium.host" value="localhost" />
      <parameter name="selenium.port" value="5555" />
      <parameter name="selenium.browser" value="*firefox3 C:\\Documents and Settings\\dgarcia\\Local Settings\\Application Data\\Mozilla Firefox\\firefox.exe" />
      <parameter name="selenium.url" value="http://localhost:8080/standard-webapp-war/home.seam" />
      <parameter name="selenium.timeout" value="1000000" />
      <parameter name="test.data.filepath" value="src\\test\\resources\\datatest_.xls" />

  <test name="standard" preserve-order="true" >
    <classes>
        <class name="com.standard.webapp.common.TestingContext" />      
        <class name="com.standard.webapp.login.TestLogin"/>
        <class name="com.standard.webapp.TestLanguages"/>
        <class name="com.standard.webapp.TestTranslations"/>
        </class>
    </classes>
  </test>
</suite>

---

there is no exception nor any reason on the output to skip those tests. I am not really sure about the output you mention, so I will paste here the content of the generated "myWebAppTest.xml" with the results of the tests:

<testsuite hostname="SP2L0044" name="com.sicpa.standard.dms.webapp.selenium.common.BaseWebAppPageTest" tests="14" failures="0" timestamp="4 Mar 2011 08:45:57 GMT" time="27.141" errors="0">
  <testcase name="testLoginLoadHome" time="2.188" classname="com.sicpa.standard.dms.webapp.selenium.login.TestLogin"/>
  <testcase name="testCreateCorrect" time="0.0" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestCodes">
    <skipped/>
  </testcase>
  <testcase name="testFilter" time="0.0" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestCodes">
    <skipped/>
  </testcase>
  <testcase name="testUpdate" time="0.0" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestCodes">
    <skipped/>
  </testcase>
  <testcase name="testView" time="0.0" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestCodes">
    <skipped/>
  </testcase>
  <testcase name="testCreateCorrect" time="0.0" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestLanguages">
    <skipped/>
  </testcase>
  <testcase name="testFilter" time="0.0" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestLanguages">
    <skipped/>
  </testcase>
  <testcase name="testModify" time="0.0" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestLanguages">
    <skipped/>
  </testcase>
  <testcase name="createCorrect" time="0.0" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestTranslations">
    <skipped/>
  </testcase>
  <testcase name="filter" time="0.0" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestTranslations">
    <skipped/>
  </testcase>
  <testcase name="update" time="0.0" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestTranslations">
    <skipped/>
  </testcase>
  <testcase name="testOpen" time="2.297" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestLanguages"/>
  <testcase name="testOpen" time="12.61" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestCodes"/>
  <testcase name="testOpen" time="9.469" classname="com.sicpa.standard.dms.webapp.selenium.multilingual.TestTranslations"/>
</testsuite>

Answer 1

There can be multiple reason for this but after seeing your code I noticed one thing, just make sure you've set the browser exe correctly, like to execute your tests on Chrome you would need chromedriver.exe in your path.

 System.setProperty("webdriver.chrome.driver", System.getProperty("user.dir")+
                "\\src\\main\\resources\\chromedriver.exe");

Answer 2

Mostly it skips those tests that depend on the result of other tests or which have some timeout set or due to configurations.

If you have set @afterMethods or @afterClass, it would be worth commenting them out once and re-running the script.

What you can do is set priority with all your test and if all your tests depend on a certain test, then keep that in @beforeMethod or @beforeClass which ever fits you.

Answer 3

There are various reasons why TestNG is skipping the tests, the most common one is that a method you depend on (e.g. testOpen or createCorrect) failed in some way.

I suggest setting the verbose level to 10 and paste the output here or on the testng-users mailing-list.

If you are using an XML definition, set the verbose level like so: <suite name="sweet_suite" verbose="10">. The verbosity level can also be set through code configuration.

TestNG tng = new TestNG();
XmlSuite suite = new XmlSuite();

suite.setVerbose(10);

// other configuration, add tests to run

List<XmlSuite> suites = new ArrayList<XmlSuite>();
suites.add( suite );
tng.setXmlSuites(suites);
tng.run();

Answer 4

I had this problem in the past and I found out that when I have 2 annotations of @BeforeMethod, one inside the test.class and one inside a basetest.class. Both, by the way, did something different, still a collision happened - preventing the tests to be executed and eventually skip. Bottom line: Check that you don't have multiple @Before, @After of any kind.

Answer 5

I had the same issue, test ignored. After degrading to stable version of TestNG it started running the test.

Answer 6

If you would like to run the tests in extending classes irrespective of the pass/fail of the depending methods, you can always add 'alwaysRun = true' for the tests in extending classes like this:

@Test(dependsOnMethods={"testOpen"}, dataProvider="getCreateData", alwaysRun = true)