Beruflich Dokumente
Kultur Dokumente
2
What is TestNG?
• Automated testing framework
• NG = Next Generation
• Similar to JUnit (especially JUnit 4)
• Not a JUnit extension (but inspired by JUnit)
• Designed to be better than JUnit, especially
when testing integrated classes
• Created by Dr. Cédric Beust (of Google)
• Open source (http://testng.org)
3
History
• sUnit for Smalltalk (~1998, Kent Beck)
• JUnit (~2000, Kent Beck & Erich Gamma)
– Latest 3.x: 3.8.2 (March 2006)
• TestNG (~2004)
– Latest: 5.8 (March 2008)
5
Basic Testing Terms
6
“Rhythm” of an Automated Test
• Setup (most of it in a separate method)
7
Test class/method (JUnit 3 vs. 4)
// JUnit 3
import junit.framework.TestCase;
public class MyTestClass extends TestCase {
public void testSomething1() throws ... { ... }
public void testSomething2() throws ... { ... }
}
// JUnit 4
import org.junit.Test;
public class MyTestClass {
@Test public void aTest1() throws ... { ... }
@Test public void aTest2() throws ... { ... }
}
8
Test class/method (JUnit 4 vs. TestNG)
import org.junit.Test; // JUnit 4
... or
import org.testng.annotations.Test; // TestNG
9
TestNG Assertions
import static org.testng.Assert.*;
import org.testng.annotations.Test;
12
TestNG “Groups”
• Each test method is tagged with any # of groups.
– @Test // no groups
– @Test (groups = “group1”)
– @Test (groups = { “g1”, “g2”, ... })
• A group therefore contains any # of test methods.
• Can also annotate test class this way; test method groups are union
of any explicit test method groups, and any of its test class.
• Groups can span classes.
• Groups can also be externally defined (TestNG xml configuration
file).
• A group is identified by a unique string (don’t use white space).
– There are no pre-defined group names.
– E.g., “slow”, “db”, “ui”, “unit”, “integration”, “broken.unknownReason”,
“check-in”, “week-end”, “functional.billing”
13
TestNG Groups (continued)
• TestNG community suggests hierarchical
names from more general to less. E.g.:
– db.table.CUSTOMER
– state.broken.fix-in-progress
• Design group names so that you can
select them with prefix patterns.
• Groups complement other features (as
we’ll see).
14
Setup/Teardown (JUnit 3/4)
JUnit 3
• protected void setUp() throws Exception { ... }
• protected void tearDown() throws Exception { ... }
15
Setup/Teardown (TestNG)
@BeforeMethod
@BeforeClass (no need to be static)
@BeforeGroups ({“group1”, …})
@BeforeTest
@BeforeSuite
• And @After*
16
TestNG run-time concepts
• A org.testng.TestNG instance runs tests.
– Invoked by IDE, ant, or on command line.
– Run params specified as setters, or in xml.
• Hierarchy of suite/test/class/method.
17
testng.xml
• Controls test runs
• Optional
• File can have any name (.xml suffix advised)
– TestNG creates “testng-failed.xml”, for easy re-run.
• Root: <suite> (then <test>/<classes>/<method>)
• Suite can also point to a list of additional xml suite files.
• Some of the available specifications are described next.
18
testng.xml (continued)
• Which potential test methods to run: suite/test: list of
java package name patterns to include/exclude; test: list
of fully qualified class names to include; within a class,
optional (default = include all) list of method names to
include/exclude; test: list of group name patterns to
include/exclude (and can define new groups for this file
with a list of existing group names).
• Parameter name/value pairs: Suite/test: (primitive
values) for @Parameters (test values override suite
values).
• JUnit semantics: test-level flag (default: false);
migration aid.
19
Disable Tests
• TestNG
– @Test(enabled = false)
– Add to a group which is excluded
– Exclude in other ways in testng.xml
JUnit 4
• @Test(timeout = 1000)
21
The Costs of Testing
• Time, money, schedule, production scope.
• Doubled (very roughly) code base.
– Maintenance (production and test).
• Sloppily written tests??
• Is test code quality as good as production?
• Slow tests?
• True buy in from business sponsors/owners,
project managers, user reps?
22
“High View” of Test Benefits
• “High View”: using tests to best advantage
• “Do the thing right”
• “Do the right thing” (acceptance tests)
• Safety net for modifications/refactoring
• Defect localization (1 fail is more info than 100)
• Documentation/training
• Minimize costs of testing
• This view leads to better tests
23
Integration vs. Unit Tests
• Unit test: tests 1 invocation of 1 method of 1
isolated class with 1 thread
• Integration test: tests more than 1 method
invocation of an isolated class, often much more,
or more than 1 thread
• Distinction: unit test vs. a test written in JUnit
• Examples
– Test hits db
– Life-cycle of an object (e.g., a customer order)
– Testing units by testing integrated classes
24
Integration vs. Unit (continued)
• Should both be tested?
• Unit
– How big do you want the holes in your
refactoring safety net?
– How many permutations can integration tests
handle?
• Integration
– If the units work independently, will they work
together?
25
If Unit Tests are Really Integration Tests
• Slow tests.
• False sense of security for refactoring safety net.
• Permutations of business rules likely not
covered.
• Poor defect isolation: 100 tests failing is less
information than 1 test failing.
• High maintenance for test/production
modification.
• Counter to the goals of TDD and Agility.
26
Test Method Dependencies
• JUnit 3: Antithetical concept; each test method
gets its own test class instance; high isolation;
unit oriented by intentional design
• TestNG:
– @Test(dependsOnMethods = {“m1”, “m2”, ...})
– @Test(dependsOnGroups = {“g1”, “g2”, ...})
– E.g.: configuration dependencies; production lifecycle
• JUnit 4: tests ignored if “assumptions” (similar to
assertions) are violated: assumeTrue(bool),
assumeNoException(Throwable), assumeNotNull(refs
…), assumeThat(…) high-readability method
27
Parallel (multiple threads)
@Test(threadPoolSize = #,
invocationCount = #,
timeout = #)
testng.xml:
– <suite|test
parallel =“tests|methods|none”
thread-count=“#” (5); 1 if not parallel
time-out =“#” ms (default: 0=none)
...>
• Testing production code with multiple threads
(unless not supported); stress testing; speeding up
tests! 28
Test Method Parameters I
@Test
@Parameters({“name1”, “name2”})
public void aTest(String name1, int name2) {
...
}
@DataProvider(name = “myDataProvider”)
public Object[][] getSomeTestData() {
...
}
• Runs (possibly multiple: first []) parameterized tests.
• Multiple parameters: second []
• Return type can also be Iterator<Object[]>
• Data provider itself can take 0, 1, or 2 args: Method, and test context
30
Test Class Factories
@Factory
public static Object[] myTestFactory() {
// return instances of test classes
}
31
Migrating JUnit 3 => TestNG
• TestNG tool that modifies source code by
adding imports, @Test, @BeforeTest,
@AfterTest, and an optional list of desired group
names; tool accessible via command line, ant, or
Eclipse/IDEA.
• Can use testng.xml <test junit=“true”> for
tests that need further migration analysis.
• That’s not necessarily enough; for other
warnings and tips, see Appendix D of [TestNG].
32
More TestNG and JUnit 4
• TestNG and JUnit 4 each have other
features not mentioned here.
34