Home arrow Blog

JUnit vs TestNG a comparison

Since I launched this site, I have done a few tutorials on the two major unit testing frameworks in the Java space: JUnit and TestNG (you can view them in the Tutorials section of this site). Each of them are very competent frameworks that provide a robust set of features that allow developers to create complex test scenarios to adequately test just about any piece of code. But if you are starting a new project, which should you choose? If you already have a suite of test cases in one or the other, should you switch? Let’s find out.

Feature JUnit TestNG
User Defined Life Cycle check check
Test Organization (groups, etc) check
Distributed Test Execution check
Parallel Test Execution check
Data Driven Tests check
Dependency Testing check
IDE Integration check check
Ant Integration check check
Maven Integration check check
Domain Specific Extensions (Database, HTTP, etc) check
Active Community check check

The criteria I looked at, as you can see above was pretty diverse. User defined life cycle is the ability to tell the test runner what methods to execute in what order. Both JUnit and TestNG allow you to do this via annotations. The ability to organize my tests is another nice to have. TestNG allows you to group tests across test classes via annotations. This is a handy tool for crateing groups for different development tasks (checking in, continuous integration build, by functional area, etc) that JUnit doesn’t have. The most grouping you can do with JUnit is by grouping test methods together in a single class. On a large project the ability to run tests in either a distributed mode or in a parallel mode can definitely speed up execution. Yet another two points for TestNG. We have all come across very data intensive areas that require a large amount of data to truly test all scenarios. TestNG’s DataProvider method allows for the ability to build a data set for a single test scenario. The scenario is then executed over all of the data. Very powerful stuff. Dependency testing is also something that is important in a large system. I know these are both technically “unit test” frameworks, but they still should be able to handle all testing scenarios we throw at it. Given JUnit’s rigid enforcement of testing in isolation, it is difficult to test dependencies. Yet another point to TestNG.

Both frameworks integrate well into all major IDEs as well as the major build tools of the day. JUnit has a large collection of third party add ons that can help in domain specific areas like database testing and HTTP front end testing. However since TestNG can execute JUnit tests, you don’t loose the ability to use those tools by using TestNG. They both have an active community with a plethora of documentation available on the net (although at the time of this writing, JUnit.org has been down for a few days. Not sure what is going on there).

For me, the decision is clear. TestNG is a high caliber testing framework that can assist in just about any testing scenario you can think of. The guys over at TestNG did a good job. They originally intended to take JUnit and make it better. Given the easy learning curve and powerful features, it’s hard to argue against TestNG as the way to go in unit testing.

18 Responses to “JUnit vs TestNG a comparison”

  • kasper graversen responded:

    My understanding of TestNG is that it does not reload classes upon executing each test method. This was due to a lack of understanding of the Java language by the TestNG implementor. He even knew it was happening in Junit but apparently ignored that fact. And now so many people rely on the fact that classes won’t be reloaded, that he feels he can’t change testng…

    Stuff such as this concerns me to an extent that I’ve never picked up the framework. With regards to grouping tests with annotations, putting tests in different test folders have actually always sufficed for the stuff that we’ve been doing. such strategy doesn’t support multiple groupings of the same tests… but we’ve never felt the need for that.

    just my 2 cents
    kasper, http://firstclassthoughts.co.uk/

  • Hani Suleiman responded:

    I’m somewhat aghast that any Java developer would say that object should be reinstatiated on every method invocation, and that NOT having this behaviour is due to a misunderstanding of how Java works. Serious Kaspar, do you have a job doing Java stuff?

    Exactly how many people do this? Create an object, invoke a method on it, throw it away, create another instance, invoke a 2nd method, and so on?

  • Cedric responded:

    Hi Kasper,

    I don’t even know where to start :-)

    The motivations behind TestNG are very clearly explained in the documentation, in the multiple articles that have covered TestNG since its inception and, of course, in our book (”Next Generation Testing in Java”, check it out, you might find a few things that will challenge your thinking).

    And by the way, get your vocabulary straight: we are not talking about reloading classes but reinstantiating them.

    Let me give you a short summary of these reasons since you obviously didn’t bother researching any of this: keeping the values of the test objects between method invocations is not only intuitive to any Java developer (as Hani pointed above) but it’s also the only sane way to maintain expensive state across your tests (I’m betting you use static fields to achieve this?).

    Please check out some of the material I have written over the past years on the subject and I’ll be happy to discuss specific issues with you, as long as you refrain from using ad hominem attacks.


    Cedric

  • Wolter responded:

    I’m not sure that I understand the issue you guys are referring to.

    Are you saying that TestNG instantiates the objects to test only once, and then uses the same objects for all of the tests?

    If so, wouldn’t this cause unexpected dependencies between the current test and previous tests?

    I’m imagining some complex object that holds a number internally as part of its function. It’s a very magic number that gets changed when certain methods are called. This means that in the course of testing, the number will get changed depending on what tests are run and in what order. In a later test, you call addToNumber(int value), and then getCurrentNumber() to make sure it added correctly. However, if previous tests called methods that muck about with this number, how can you write an effective test for addToNumber short of reinstantiating the object?

  • Crias responded:

    ~kasper

    I don’t know the history of TestNG, so I can’t comment, but the fact that JUnit has moved in same direction should sufficiently demonstrate the testing world’s inclination toward this practice. And for good reason.

    Example. setUp() takes 0.99s and sets up some variables that I have no intention to change. Each test takes 0.01s. For 1 test, this cost is annoying… but I’ll accept it. For 100 tests, repaying this cost each time is unacceptable.

    The trade-off is that you’re assuming that after a test you can just throw away the test instance and move on to a new one. In the old way, that was true, and it saved you writing a proper teardown(). So the cost of saving huge amounts of test time by proper separation is a little more effort to do proper setup and teardown via @Before and @After

    ~Hani

    I believe you’re misunderstanding Kasper. He claimed TestNG’s creator had a lack of understanding of the Java language, not a lack of understanding of Java or OO paradigms.

    Essentially, he’s claiming the guy knew he “should” reinstantiate, but didn’t know how to. He doesn’t mean every object you ever create should only ever have a 1-function lifespan.

    (Note: I’m not agreeing with him, just clarifying his stance.)

    ~Cedric

    Statements like “since you obviously didn’t do your research” and “get your vocabulary straight” followed by “but don’t attack me” are clearly flaimbait. Trolling is lame dude, even if you do know your stuff.

    ~Wolter

    It depends on your testing methodology. There is a trade-off made during the switch.

    During JUnit 3.8 tests, the lifecycle is simplistic.
    foreach(Test) { Instantiate TestCase; setup(); test(); teardown(); Discard TestCase; }
    Because TestCase was being discarded, you could actually opt to skip teardown() completely in most cases.

    During JUnit 4 and TestNG tests, the lifecycle is different.
    { Instantiate; BeforeClass; foreach(test) { Before; Test; After; } AfterClass; Discard; }
    As a result, After (or teardown) becomes very important - you must clear out any dirty data. You must also make sure you put the right stuff into BeforeClass and Before - only put static unchanging setup into BeforeClass. However, as a result it makes the storage of expensive data a lot simpler.

  • http://room8brightwater.blogspot.co.nz/2013/08/term-3-writing-speeches.html responded:

    I think the admin of this web site is really working hard in
    support of his website, for the reason that here every material
    is quality based material.

  • Read the Full Article responded:

    I am regular visitor, how are you everybody? This post posted at this website is
    genuinely fastidious.

  • alturl.com responded:

    An impressive share! I have just forwarded this onto a colleague
    who had been doing a little homework on this.
    And he in fact bought me dinner due to the fact that I discovered
    it for him… lol. So allow me to reword this….
    Thank YOU for the meal!! But yeah, thanx for spending time to discuss this topic
    here on your web page.

  • choiceness N: choicenesses responded:

    I’d like to find out more? I’d want to find
    out some additional information.

  • http://wheretoshopseattle.com/?page_id=2 responded:

    Great blog you have here but I was curious about if you knew of any forums that cover the
    same topics discussed in this article? I’d really love
    to be a part of group where I can get responses
    from other experienced individuals that share the
    same interest. If you have any suggestions, please let me know.
    Kudos!

  • Jaunita responded:

    It’s fantastic that you are getting thoughts from thjis article as
    well as from our argument made at this time.

  • Shaunte responded:

    Nice post. I learn something totally new and challenging on websites I stumbleupon everyday.
    It will always be interesting to read content from other authors and use a little something from their sites.

  • Cosmetic responded:

    It’s fawntastic that you are getting thoughts from this paragraph as well as from our
    dialogue made here.

  • Google responded:

    * A sitemap to navigate your website is helpful for visitors to access main pages.
    These pre-computed numbers, hold on in a very giant information bank for millions or URLs on the net.
    But it seems Memorial Day wasn’t important enough to Google.

  • zapatos novia responded:

    Más bien iluminando agradezco, creo que sus suscriptores pueden seguramente querrá mucho
    mass revisiones a llo largo de estas líneas sigan el buen lugar de visita
    obligada.

  • mccarthyvdsz.jigsy.com responded:

    Hello, I want to subscribe for this web site to take hottest updates, so where can i do it please help.

  • bestcrib mattress 2014 responded:

    If you want to take much from this paragraph then you have to apply such
    techniques to your won weblog.

  • Perception Sport Swifty 9.5 Kayak Spray Skirt responded:

    Good post however I was wanting to know if you could write a litte more on this topic?

    I’d be very thankful if you could elaborate
    a little bit further. Thank you!

Add your own comment...

Tweets

    Twitter Button from twitbuttons.com