Unit Testing Frameworks: A comparative study

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 07 | July 2022 www.irjet.net p ISSN: 2395 0072

Unit Testing Frameworks: A comparative study

Gurubasava1 , Anusha L S2

1Student, RV college of Engineering, Karnataka India

2Professor, RV College of Engineering, Karnataka India ***

Abstract - Nowadaystheneedofsoftwaredevelopmentin every field increased exponentially, thereby increase the needtoperformintensesoftwaretesting.Unittestingplaysa majorroleforasuccessfulsoftwaredevelopmentprocess.To meettheincreasedneedforproducinghigh qualitysoftware morequickly,or"QualityatSpeed,"softwaretestingmustbe carriedoutmorequicklyandsuccessfully.Themaingoalof unit testing is to confirm that each piece of software code operates as intended. This paper is an effort towards comparativestudyofdifferentunittestingframeworksfor variousmodernlanguagessuchasJAVA,C++,andPYTHON.

Key Words: Unit testing, Google Test, Google Mock, TestNG, Junit, CppUnit, CppUTest, Pytest, unitttest, junit and jmock.

1.INTRODUCTION

Unit testing frameworks are software tools that help in writingandexecutingunittests.Providesaframeworkfor building tests as well as the ability to run the tests and reporttheresults.Frameworkscanbeusedasdevelopment tools on par with preprocessors and debuggers, therefore frameworksarenotjusttoolsfortesting.Theseframeworks arealmostcontributedateverystageofthedevelopmentof softwareincludingarchitectureanddesignofthesoftware, assuranceofsoftwarequalityandperformancetuning.

Unit testing is a component of the testing methodologies wherethetesterisawareoftheinternalimplementationof the proposed software which is being tested. The main purpose of unit tests is to verify the functionality of your code units. Both manual and automated unit testing are possible.In codemanual testsaretypicallyusedforsmall projects where the expense of establishing a testing framework is not required. Using testing frameworks, automated testing is carried out. Pytest is a Python programminglanguagepackageforautomatedunittesting.

Inthispaperacomparativestudyofthedifferentlanguages unittestingframeworkisdiscussed.

2. STUDY OF VARIOUS LANGUAGES UNIT TESTINGFRAMEWORKS

C++: Chosen some of the most used unit testing frameworks such as CppUnit, Google Test and gMock, CppUTestandUnityforC++unittesting.Andcomparative

study of each unit testing framework is discussed with differentpapersasshowninTableI.

TABLEI

Variouslanguagesunittestingframeworks

Languages Variousunit testing frameworks

C++

Google Test and Google Mock, Unity,CppUTestandCppUnit

JAVA JunitandJMock,TestNG, PYTHON Pytest,unitest,Doctest

CppUnit: OneofthemostusedC++unittestingframeworks. CppUnitusesthefixtures,suitesandmacrosalso.CppUnit includesa TestSuiteclassthatallowsprogrammerto run manynumbersof Testcasesconcurrently.If having a Test suite for theany code,CppUnit includes toolsfor defining and displaying the suite’s results. A static method suite whichreturnsatestsuitethatmakesthetestsuiteaccessible to a TestRunner programme. CppUnit uses For executing testcases first required to define the path with using ($TargetPath)inVisualStudioCodethereisnodirectwayto run it on any of the IDE’s. For checking the output of the function for which unit testing is wrote by programmer. CPPUNIT_ASSERTisusedforcheckingthefunctionoutputis comingexactlyasexpectedornot.

CppUnithasmanydisadvantagesoneofamongistohavethe own test case, must create a subclass [1]. Inheritance is a propertywhichconnectsthestrongbondingbetweentwo classes.Therequirementthatatest caseclassinheritfroma CPPUnit frameworkclassbindsthetwotogether

Google test (Gtest) and Google mock (gMock):

Gtest: Gtest is open source unit testing framework developed by google which is helpful for writing unit testcases for each function and running the tests for C++ codinglanguage.Intermsofcompatibility,itisusedinmany operatingsystemssuchasWindows,Mac,andLinuxandit supportsbothBazelandCMakeforbuilding[2].

Figure1aboveshownisthebasicmethodologytowritea unittestcase.

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 07 | July 2022 www.irjet.net p ISSN: 2395 0072

Startthetestcasewithmacro TEST.InsidethatTESTmacro writethecoderesponsibleforthetestexecution

Fig.

1.Unittestcasewritingmethodology

requirementssuchasinitializationofvariablesandobjects, settingthevaluestothosevariablesandcallingthatfunction need to be checked and check the results by using the macro’sEXPECTandASSERT.Themajordifferencebetween them is ASSERT stops the execution if the test is unsuccessful [2] whereas EXPECT macro continues its executiontillthecodeends.

gMock: gMockisalsoamockinglibrarydevelopedbygoogle itself.ThemaingoalofthegMockistoprovideamechanism forimplementingmockclassesthatmimicaportionofthe testedsystemwithingoogltesttests.Itsoperatingprinciple is based on the interaction between tested code and the mock objects. These objects have same interface as the system component that they replace. They use the same abstractclassastheactualcomponent.Figure2belowisthe basichighlightforthemockingthefunction.

HereEXPECT_CALLmacroalsoplaysamajorroleforsetting expectations to the functions such as how many times functiontobeinvoked,howmaytimethefunctionshould returnthevalue[2]etc

CppUTest: CppUTest is a test framework for unit testing primarilyusedforembeddedsystemsthatsupportsboththe C and C++ programming languages. This framework was written in C++, but test cases can be written in C without knowing C++. This makes it easier for C programmers to workwiththisframework.ItissimilartoUnityexceptthatit does not require a Test Runner. As a result, it overcomes Unity’s main disadvantage [3]. As a result, it reduces the amountofworkandtimerequiredtoupdatetestcasesinthe testrunner.Itsupports multipleoperatingsystemsand is primarily intended for use in embedded software development[4].

The setup () and teardown () routines must be specified insidetheTEST_GROUP(GroupName)intheCppUTestTest, even though each group includes several test cases, one setup, and one takedown. In terms of framework organization,CppUTestandUnitydivergealittle.Duetothe factthateverythingisdescribedinsidethegroup,itappears morecompactthanUnity.Thetestcaseformatisasfollows inFigure3.

Fig. 2.Functionmockingmethodology

MOCK_METHOD is a macroutilized inside the mock class. Therearethingstodeclaredinsidethemacroandtheyare

1. Nameoftherealmethodused

2. Returntype

3. Argumentsoftherealmethodnameusedinmockclass [2]

Fig. 3. Basiccodeexample

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 07 | July 2022 www.irjet.net p ISSN: 2395 0072

Comparison of C++ unit testing frameworks

ComparedtoGoogletestandCppUnit,forCppUTest,thetest runner, is not required, which saves a tonne of time and effort.Thisframeworkalsoadherestothefour phasetest pattern, which contributes to well organized code that is simpletoreadandcomprehend[4].Mockingisoptionisnot thereforbothCppUnitandCppUTestcomparedtoGmock and setting the expectations is also possible for gMock libraryunittestingframework[2].GoogleTestconsistsof various kind of macros options, and it consists of more featurestoexecutethetestresultseasilywhencomparedto bothCppUnitandCppUTest.

JAVA:

JUNIT and JMOCK:

JUnit: Junitisoneofthebestandmostwidelyunittesting framework used across the entire IT sector for java programming. It is a open source unit testing framework. Programmerstressanddebuggingtimearedecreasedasa resultoftheincreasedproductivityoftheprogrammerand thestabilityoftheprogrammecode.

JesterisusedastesterforrunningJUnittests;italtersthe source code in various ways and determines whether the tests are still passable after each change. Jester identifies possible code modifications that can be made without causingtheteststofail.Ifchangestothecodecanbemade withoutthetestsfailing,eitheratestismissing,orthecode isredundant[5].Jestercanbeusedtodeterminewhether thecurrenttestsaresufficientortoprovideinformationon theteststhatarelacking.

JUnit has design rules for writing and executing the unit testcases.In[6],explainedindetaillyabouthowtheclassof TestClass is derived from junit and collaborating with the targetclassexplained[6].

JMock: JMockoffersaneasy to useandexpressiveAPIfor mimickinginterfaces,defininganticipatedinvocations,and executinginvokedbehavior[7].

Writing tests that can be executed and read as documentationisauniquepurposeofjMock.Themajorityof thejMockAPIisdevotedtocreatingreadablesyntacticsugar for expectations definition. This ambition, which aims to developadomain specificembeddedlanguage[7]hostedin Java, has resulted in an API that is extremely unorthodox whencomparedtotypicalJavadesigns.

Even though jMock has a huge library of restrictions and matchingcriteria,itisnotabletoaccommodateeverycase thataprogrammerwouldrequire.Infact,addingconstraints thatareuniquetoyourproblemdomainmakesyourtests clearer, proving that matching rules and constraints are extendable.

Inordertomakeitsimpletodeterminewhatledtoatest failure,jMockismadetogivehelpfulmessages.Mockobjects have names that make it simple for the programmer to connecterrormessagestohowthetestwasexecutedand thetargetcode.Aclearfailuremessagecanbeproducedby combiningdescriptionsfromthecoreobjectsthatareused toexpressexpectations.

TestNG: TestNG is one of the most popular open source testingframeworksforautomationtestingforjava.TestNG framework is developed for any web application developmentalongwithusingseleniumwebdriver[8]

Fig.4.TestNGwithseleniumwebdriver

In[8],seleniumwebdriversupportsthewebapplicationand testsuiteswrittenforthewebapplicationarealsosupported bytheseleniumwebdriverwhichusesthetestsuiteswritten forthewebapplicationusingTestNGframeworkandwhich is automated by using selenium webdriver. So that its pipeline directly it will generate report based on the test suites written by the developer with selenium webdriver andathird partytoolproducesascreenshotofthereport [8].

Comparison of TestTNG and Junit

TestNGofferscertainnewfeaturesthatgiveitanedgeover JUnit in terms of power. All test types, including unit, functional,andintegrationtesting,arecoveredbyTestNG. JUnitcannotabletorunparalleltests,won’tsupportadvance annotations,groupingtestsisnotpossibleanddependency testsarealsonotexecutablewhereasallthesefeaturesare supportedbyTestNGframework.

Python Pytest and unittest

Python developers are majorly using pytest and unittest two unit test framework to test their python code. Nowadays python projects are migrating from unittest to pytesttherearesomespecific reasons whichdiscussedin paper[9].

Several reasons are there for migrating from unittest to pytest those are easier syntax defined, fixture reuse, easy maintenance and implicit mechanics of the pytest major reasonsofthemigration[9].

2022, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 07 | July 2022 www.irjet.net p ISSN: 2395 0072

Inpaper[9],surveyisconducteduponmigrationoftesting infromunittesttopytestwhichconcludesthat66%ofinitial developersrelyonunittestnowconvertedtopytestbecause ofvariousadvantagesoverunittest.

Migratingfromunittesttopytestinvolvesfollowingsteps

1. Removing the test defined inside the class and migratingtothenormalfunctions.

2. Updatingthenormalassertionstoassertsinpytest.

3. Replacingsetupfunctionoperationstofixture

Exampleofthesamplecodeinpytestandunittestasshown inFigure5

lacked the tool support for simple to use unit testing. Without tests, refactoring and streamlining C++ code is extremelychallenginganderrorprone.Sojavaunittesting unittestingframeworksarebetterincomparisonwithc++ unittestingframeworks.

Incaseofselenium,Nomatterthetechnology,itisusedto automate web application testing. Due to its simplicity of setup, Python may be the best choice for Selenium automatedtesting.PythonisfrequentlypreferredoverJava by start ups and medium sized businesses because of its straightforward programming syntax. Python makes it considerablysimplertodevelopSeleniumprogrammesthan Java does. In addition, the Python Selenium framework PyTest makes it the ideal choice to take advantage of the creationofsophisticatedfunctionaltests.

4.CONCLUSIONS AND FUTURE WORK

Fig. 5. Samplecodeofpytestandunittest

Migrationfromunittesttopytestisbecomequitecommon because the survey conducted by the [9] proves this. Migration to pytest also have many advantages and disadvantages over unitest and future work in this field covered [9]. To migrate the testing framework is also possibleformunittesttopytest[9].

Comparison of pytest and unittest

When compared to unittest, Pytest has many inbuilt capabilitiesthatneedlesscode.Forunittest,wefirstneedto importamodule,thenweneedtomakeaclass,andlastwe needtospecifytestingfunctionsinsidetheclass.However,in thecaseofPytest,wemustspecifythefunctionsandmake thecriteriaincludedwithinthemexplicit.Overall,theformat of unittest use cases is difficult, there is no compatibility, therearefewplug ins,andsecondarydevelopmentiseasy. Theusecaseformatisstraightforward,andPytestismore practicalandquicker.Itoffersstrongcompatibilityandcan run unittest style test cases without altering their source code.Pytestplug insaremanyandincludetheflashplug in, whichcanbeusedtoreruntestsintheeventofanerror,and thexdistplug in,whichallowsformoreproductiveparallel deviceexecution.

3.CROSS COMPARISON BETWEEN DIFFERENT LANGUAGES UNIT TESTING FRAMEWORKS

Whencomparedtojava,C++doesn’thavecertainofJava’s reflectionfeatures,wemustcreateourownmainfunction fortesting.Forstraightforwardsituations,wemerelycreate a runner object and allow our test to run. Despite the complexityofthelanguagecallingforit,C++programmers

We have discussed about various languages unit testing frameworkssuchasC++,JAVAandPYTHONetc.Hereifwe compare among the C++ available unit frameworks googletestandgooglemockarebestresourcesincetheyare opensourcealsoacquiredmanyfeatureswhencomparedto otherframeworks.WhereasincaseofJAVAJunitisthebest whencomparedtojmockandTestNG.IncaseofPYTHON, pytest is the best because in paper [9] we have seen the migrationfromunittesttopython.

Infuturework,wehavetoseeforsuchkindofframeworkor toolifwedumptheanylanguagesdevelopedcodefortesting means automatically tool should test everything without depending on any technology it should be technology independent.

ACKNOWLEDGEMENT

The authors thank ANUSHA L S for the collaboration of preparingthispaperandZebratechnologiesforproviding opportunitytoworkinthisarea.

REFERENCES

[1] Peter Sommerlad and Emanuel Graf. “Cute: C++ unit testingeasier”.In: Companiontothe22ndACMSIGPLAN conferenceonObject orientedprogrammingsystemsand applications companion.2007,pp.783 784.

[2] NenadPetrovicetal.“Model drivenautomatedgMock´ testgenerationforautomotivesoftwareindustry”.In: XV International SAUM (2021),pp.1 4.

[3] Lari MatiasOrjala.“Unittestingmethodsforinternetof things Mbed OS operating system”. In: University of Oulu, PhD diss (2019).

[4] Sandhya Mudduluru. Investigation of Test Driven Development based on Mock Objects for Non OO Languages.2012.

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

[5] IvanMoore.“Jester aJUnittesttester”.In: Proc. of 2nd XP (2001),pp.84 87.

[6] Michael Wick, Daniel Stevenson, and Paul Wagner. “UsingtestingandJUnitacrossthecurriculum”.In: ACM SIGCSE Bulletin 37.1(2005),pp.236 240.

[7] SteveFreemanetal.“jMock:supportingresponsibility baseddesignwithmockobjects”.In: Companion to the 19th annual ACM SIGPLAN conference on Object oriented programming systems, languages, and applications.2004,pp.4 5.

[8] SatishGojare,RahulJoshi,andDhanashreeGaigaware. “Analysis and design of selenium webdriver automationtestingframework”.In: Procedia Computer Science 50(2015),pp.341 346.

[9] Lıvia Barbosa and Andre Hora. “How and Why DevelopersMigratePythonTests”.In:().

Volume: 09 Issue: 07 | July 2022 www.irjet.net p ISSN: 2395 0072 © 2022, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal