Codedengjiagui

GUICat:GUITestingasaServiceLinCheng,JialiangChang,ZijiangYangDepartmentofComputerScienceWesternMichiganUniversityKalamazoo,MI,USAChaoWangDepartmentofComputerScienceUniversityofSouthernCaliforniaLosAngeles,CA,USAABSTRACTGUIsareevent-drivenapplicationswheretheowofthepro-gramisdeterminedbyuseractionssuchasmouseclicksandkeypresses.
GUItestingisachallengingtasknotonlybe-causeofthecombinatorialexplosioninthenumberofeventsequences,butalsobecauseofthedifcultytocoverthelargenumberofdatavalues.
WeproposeGUICat,therstcloud-basedGUItestingframeworkthatsimultaneouslygenerateseventsequencesanddatavalues.
Itisawhite-boxGUItest-ingtoolthataugmentstraditionalsequencegenerationtech-niqueswithconcolicexecution.
Wealsoproposeacloud-basedparallelalgorithmformitigatingbotheventsequenceexplosionanddatavalueexplosion,bydistributingthecon-colicexecutiontasksoverpubliccloudssuchasAmazonEC2.
WehaveevaluatedthetoolonstandardGUItestingbench-marksandshowedthatGUICatsignicantlyoutperformsstate-of-the-artGUItestingtools.
ThevideodemoURLishttps://youtu.
be/rfnnQOmZqj4.
CCSConceptsSoftwareanditsengineering→Softwarevericationandvalidation;Softwaretestinganddebugging;KeywordsSymbolicexecution,Testgeneration,GUItesting,Cloud1.
INTRODUCTIONGraphicalUserInterfaces(GUIs)provideaconvenientwayfortheusertointeractwiththecomputer.
Theyareevent-drivenapplicationswheretheowoftheprogramisdeter-minedbyuseractionssuchasmouseclicksandkeypresses.
Incontrasttoconsoleapplicationswhoseonlypointofin-teractionisatthebeginning,GUIshaveapotentiallylargenumberofinteractionpoints,eachofwhichmaybeassoci-atedwithadifferentstate.
Thesefeaturesoftenmaketra-ditionalsoftwaretestingtechniquesineffective.
Specically,Correspondingauthor(Email:zijiang.
yang@wmich.
edu)GUItestinghastwosignicantchallenges.
First,coveringallpossibleeventsequencesofaGUIapplicationisdifcultduetothecombinatorialexplosion,i.
e.
,thenumberofpossiblewaysofclickingkbuttonscanbeaslargeask!
.
Second,GUIbehaviorsdependnotonlyontheeventsequencebutalsoonthedatavaluesofwidgetssuchastext-boxes,edit-boxes,andcombo-boxes,thusleadingtoanextremelylargeinputspace.
Forexample,coveringallpossiblevaluesofak-characterin-putstringrequires26ktestcases.
AlthoughexistingGUItest-ingtools[9–11,13]haveaddressedthechallengeofgener-atinghigh-qualityeventsequences,theyhavenotaddressedthechallengeofsimultaneouslygeneratinghigh-qualitydatavalues.
Assuch,data-dependentGUIbehaviorsareoftenin-adequatelytested.
WeproposeGUICat,acloud-basedGUItestingframeworkthatgeneratesbothhigh-qualityeventsequencesandhigh-qualitydatavalues,byaugmentingstate-of-the-arteventse-quencegenerationtechniqueswithconcolicexecution.
Theresultisawhite-boxGUItestingtoolthatuniformlyexplorestheeventowaswellasthedataow.
Wealsoproposeaparallelconcolicexecutionalgorithmformitigatingthedatavalueexplosion,bydistributingthecomputationtasksoverworkersonprivateclustersaswellaspubliccloudssuchasAmazonEC2[3].
ItprovidesanillusionofrunningGUICatonapowerfulsupercomputerandthusallowsittohandlesignicantlylargerapplicationsthanpreviouslypossible.
WehaveimplementedGUICatbasedonanumberofopen-sourcetools,includingGUITAR[10]forgeneratingtheinitialeventsequences,ASM[1]forJavabytecodeinstrumentation,Catg[12]forconcolicexecution,andJaCoCo[2]forcomput-ingcodecoverage.
Unlikepriortechniques,GUICatisfullyautomatedinmodelingGUIwidgets.
Thatis,itdoesnotre-quiredeveloperstomanuallymodelthesewidgets.
Thisisimportantbecausemanualmodelingisnotonlylaborinten-siveanderrorpronebutalsohardtosustaininthelongrunduetofrequentwidgetupdates.
WehaveevaluatedGUICatonAmazonEC2forasetofGUItestingbenchmarks.
GUICatachievesscalabilitythroughthedistributionofsymbolicexecutiontasksTheremainderofthepaperisorganizedasfollows.
WeillustratethemainideabehindGUICatusingmotivatingex-amplesinSection2.
WepresentouralgorithminSection3,whichisfollowedbyourexperimentalresultsinSection4.
WediscusstherelatedworkinSection5.
Finally,wegiveourconclusionsinSection6.
2.
MOTIVATINGEXAMPLESFigure1showsaGUIexampleforcomputingticketfarebasedonuserinputsincludingName,AgeLevel,Distance,andCoupons.
OncetheBuybuttonisclicked,theapplica-Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprotorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationontherstpage.
CopyrightsforcomponentsofthisworkownedbyothersthanACMmustbehonored.
Abstractingwithcreditispermitted.
Tocopyotherwise,orrepublish,topostonserversortoredistributetolists,requirespriorspecicpermissionand/orafee.
RequestpermissionsfromPermissions@acm.
org.
ASE'16,September3–7,2016,Singapore,Singaporec2016ACM.
978-1-4503-3845-5/16/09.
.
.
$15.
00http://dx.
doi.
org/10.
1145/2970276.
2970294858name,JTextField,getText,String,supermandistance,JTextField,getText,int,100AgeLevel,JComboBox,getSelectedIndex,int,0:1$100,JComboBox,isSelected,int,0:1$200,JComboBox,isSelected,int,0:1$400,JComboBox,isSelected,int,0:1Figure1:AGUIexampleandGUICat'scongurationle.
OnClickComputePrice(){intcoupon=0;Stringage=(String)ageComboBox.
getSelectedItem();Stringsdistance=distanceTextField.
getText();intdistance=Integer.
parseInt(sdistance);if(d100CheckBox.
isSelected())coupon+=100;if(d200CheckBox.
isSelected())coupon+=200;if(d400CheckBox.
isSelected())coupon+=400;if(age.
equals(Child)){coeffienct=1;}else{coeffienct=2;}if(distance0);infoField.
setText(price);}Figure2:Codesnippetforcomputingtheticketprice.
tioncomputesandthendisplaysticketprice,usingacoef-cientassociatedwiththechosenagelevel.
ToallowGUICattogeneratetestcases,theusermustprovideacongurationlethatspeciesthenameandtypeofthesymbolicvariablesasshowninFigure1(right).
Eachentry(line)ofthecongu-rationleconsistsofthewidgetname,widgettype,methodforobtaininguserinput(e.
g.
,getText),typeofuserinput,andthedefaultvalue(e.
g.
,superman).
Here,0:1meansthedefaultvalueisofenumtypewithtwovalues0and1.
Figure2showsthecodeforcomputingtheticketprice.
Ithasabugthatcanleadtonegativeticketprices.
Forexample,ifauserhasthreecoupons,thenpurchasingachildticketforadistanceof60mileswouldresultinthepricebeing-40dol-lars.
However,sinceanegativepricerequiresaspeciccom-binationofwidgetvalues,suchbugisdifculttodetectusingstate-of-the-artGUItestingtoolssuchasGUITAR[10].
ThisisbecauseGUITARfocusesprimarilyongeneratingeventse-quencesasopposedtogeneratingadiversesetofwidgetval-ues.
OurnewtoolGUICat,incontrast,canquicklygenerateacombinationofeventsequencesandwidgetvaluestoexposethisassertionfailure.
GUICatgeneratesthetestcasesasfollows.
First,itusesGUITARtogeneratetheinitialsetofeventsequencesuptoaboundedlength.
Then,foreacheventsequence,itcreatesaninstrumentedGUIwheresomevariablesaremarkedassymbolicbasedonthecongurationle.
Next,itconductssymbolicexecutionoftheinstrumentedGUIoverthecloud.
Finally,itusesJaCoCotogeneratethecoveragereport.
Now,weexplaineachstepinmoredetail.
Step1.
WeuseGUITARtogenerateeventsequencesofaboundedlength.
Assumetheboundis2,andeventse1,e7denotenameTextField,distanceTextField,AgeComboBox,100Dol-larCheckbox,200DollarCheckbox,400DollarCheckbox,andBuy-Button.
AfterrunningGUITAR,wehavethefollowingsevenlength-2eventsequences:(e1,e7),(e2,e7),(e3,e7),(e4,e7),(e5,e7),(e6,e7),and(e7,e7).
Inthisexamplealltheboundedsequencesendine7becauseotherwisenoactioncanbetakenattheendoftheuserinteraction.
Consider(e2,e7)asanex-ample.
Itmeanstheuserspeciesadistance(e2)beforeclick-ingtheBuyButton(e7).
Althoughlogicallymeaningless,thisparticulareventsequenceisfeasible.
Step2.
ForeacheventsequenceproducedbyGUITAR,wegeneratemoresequencesbyenumeratethevaluesofwidgetswithenumtypes.
Forexample,in(e3,e7),weknowJCom-boBoxisassociatedwithe3andithasanenumtype.
There-fore,weenumerateallpossiblevaluesofJComboBoxtopro-ducethenewsequences(e03,e7)and(e13,e7),whereei3meansthevalueiischosenforevente3.
Wealsoenumeratethevaluesofotherwidgetswithenumtypes,includingAgeCom-boBox,100DollarCheckBox,200DollarCheckBox,400DollarCheck-Box.
Sincetheseselectablewidgetshavealimitednumberofdatavalues,enumeratingthemisoftenmoreefcientthangeneratingthevaluesusingconcolicexecution.
Step3.
ForeacheventsequencegeneratedinStep2,weinstrumenttheGUIapplicationbymarkingcertainvariablesassymbolic.
Considertheguicat-confleinFigure1(right),wherethelastfourwidgetsareofenumtypesbutthersttwoarenot.
Thus,wemarkthersttwowidgetsassym-bolic.
Thatis,weuseCATG.
readString("superman")tocreateasymbolicstringanduseCATG.
readString(100)tocreateasymbolicinteger.
Here"superman"and100arethedefaultconcretevaluesforthesymbolicvariables.
Step4.
WeuseCatgtoexecuteeachinstrumentedGUIap-plicationsymbolicallyfollowingthespeciceventsequence.
Consider(e13,e7)again,wheree7isassociatedwithbranchesofanifstatement.
Symbolicexecutionwillleadtothecre-ationofnewvaluesfornameanddistance,e.
g.
,name=supermananddistance=50.
Mappingthevaluesbacktotheeventse-quencewillresultin(esuperman1,e502,e13,e7),wherethersttwoeventsareaddedtosetthevaluesofthewidgets.
Step5.
Aftergeneratingalltheeventsequencesanddatavaluesusingconcolicexecution,weuseJaCoCotocomputethecoveragereport.
JaCoCoisanopen-sourcecodecover-agelibraryforJava,whoseoutputisformulatedasanHTMLpagetoshowthecoveragestatistics.
3.
ARCHITECTUREFigure3showsthearchitectureofGUICat.
GivenaGUIprogramPasinput,GUICatrstinvokesGUITARtogener-ateeventsequences.
Then,itinstrumentstheprogrambasedoneachsequenceandthesymbolicvariablesspeciedingui-cat-conf.
Next,itinvokesthedistributedalgorithmtoconductsymbolicexecutionoftheinstrumentedprogramonacloudnode.
Finally,thetestcasesgeneratedbyallinstrumentedprogramsarecollectedandthenusedbyJaCoCotocomputethecoveragereport.
Inthedistributedsymbolicexecutionalgorithm,N0istheloadbalancerandN1.
.
.
Nkarethekworkersonthecloud.
N0distributesthesetEofinstrumentedGUIprograms,onepereventsequence,overthekworkers.
Theworkersthenconductsymbolicexecutionontheirshareoftasks.
Initially,eachworkerreceivesroughlythesamenumberoftasks.
How-859guicat-confBytecodeInstrumentationeventsequencesEventSeq.
Gen.
(GUITAR)Program(P)instrumentedGUIDistributedSym.
Exec.
(Catg)testcasesCoverageGen.
(JaCoCo)coveragereportFigure3:ThearchitectureofourGUICattool.
Algorithm1SymbolicExecutiononaWorker.
1:E=T=;2:whiletruedo3:ifreceivingtasksetE′then4:E=E∪E′;5:send(N0,|E|);6:elseifrequestingtasksonbehalfofNithen7:send(Ni,E[0.
.
|E|/2]);8:send(N0,|E|);9:elseifcollectingtestcasesthen10:send(N0,T);11:terminate;12:endif13:foralle∈Edo14:Pe=instrument(P,e,cfg)15:Te=Catg'.
execute(Pe);16:send(N0,|E|);17:T=T∪Te;18:endfor19:endwhileever,sincethecostofsymbolicexecutionvariesforeacheventsequence,someworkersmaynishtheirsymbolicexecutiontaskssoonerthanothers.
N0detectssuchimbalanceandre-questsaworkerwiththelargestworkloadtoshareitstaskswiththeidleworker.
Afterallworkerscompletetheirtasks,theloadbalancerN0collectsthetestcases.
Algorithm1showsthesymbolicexecutionprocedureforeachindividualworker.
Initially,thesetEoftasksandtestcasesTarebothempty.
Then,theworkerkeepscheckingmessagesfromN0andconductinglocalsymbolicexecution.
IfitreceivesasetE′oftasks(fromN0oranotherworker),thenewtasksareaddedtothelocalset.
Sincethenumberoftasksischanged,itupdatesN0withitscurrentnumberoftasks.
ThisalsooccursatLines8and16.
Withsuchupdates,N0knowsthenumberoftaskstobeprocessedattheworkers.
IftheworkerreceivesamessagefromN0thatrequestsmoretasksonbehalfofanotherworkerNi,itsendshalfofitstaskstoNi(Line7).
AsignalofterminationisreceivedifN0asksforitstestcases,andinsuchcase,thecurrentworkersendsthelocallygeneratedtestcasesandthenterminates.
Symbolicexecutionisconductedforeachindividualeventsequencee∈E.
Essentially,itallowsustoexecutetheevent-drivenapplicationasifitisasequentialJavaprogram.
WeleveragetheCatgconcolicexecutiontool,whichmaintainstwoexecutionstacks:oneforconcreteexecutionandtheotherforsymbolicexecution.
WhenCatgexecutesanunknownmethod,forexample,Integer.
parseInt(),thesymbolicexecu-tionstackwouldnotbeupdated,wehavemodiedCatgtohandleunknownmethods.
LoadbalancerWorker1Worker2Worker3Worker4Worker5Figure4:ArchitectureofGUICat'sdistributedalgorithm.
forworkerin$(workers);doscp-r${AUTDIR}.
tar.
gz{worker}:~/gui/guicat/log/done.
.
.
forworkerin$(workers);doscp${worker}:~/gui/guicat/${AUTDIR}/report.
tar.
gz.
/${AUTDIR}/${worker}.
tar.
gzdoneFigure5:Communicationfromloadbalancertoworkers.
ThedistributedalgorithminGUICathasbeenimplementedonAmazonEC2asaCloudservice.
WedivideEC2instancesintotheloadbalancerandtheworkers.
Theloadbalancerisamulti-processorEC2instancethatgenerateseventsequences,distributestaskstotheworkers,andcollectsthetestcasesgeneratedbytheworkers.
Eachworkerisasingle-processorEC2instancethatsymbolicallyexecutesaneventsequencetogeneratetestcases.
GUICatallowstheusertocustomizetheCloudservice,suchasthenumberofworkersandtheircom-putationcapabilities,basedoncustomerrequirementssuchaswhetherabudget-rsttestingispreferredoveraspeed-rsttesting,orwhetherbranchcoverageispreferredoverinstructioncoverage.
Toalloweasycustomization,weimplementGUICatbyfollowingthestartopologyshowninFigure4,wheretheloadbalancergeneratesanddistributeseventsequencestotheworkers,andtheworkersconductconcolicexecutionwithrespecttotheeventsequenceinisolation,beforesendingtestcasesbacktotheloadbalancer.
Figure5illustrateshoweventsequencesaredistributedtotheworkersandhowtestcasesarecollectedfromtheworkers.
Fornow,distributedlesys-temlibrariesareusedtoimplementthetransferofeventse-quencesandtestcasesbetweentheloadbalancerandwork-ers.
Themainadvantageofthisarchitectureisefciencysincethereisnocommunicationbetweentheworkers.
4.
EXPERIMENTALEVALUATIONWehaveimplementedGUICatusinganumberofopen-sourcetools,includingGUITARforgeneratingtheinitialeventsequences,ASMforinstrumentingtheJavabytecode,Catgforimplementingthedistributedparallelconcolicexecution,andJaCoCoforcomputingthecodecoveragereport.
WehaveevaluatedGUICatonseveralGUItestingbench-marks.
Inallexperiments,wehaveusedtheAmazonEC2cloudcomputinginfrastructure,wheretheloadbalancerisdeployedasamulti-processorEC2instanceandeachworkerisdeployedasasingle-processorEC2instance.
Intheremainderofthissection,wereporttheresultsoftwocasestudies:aticketsellerandaworkoutgenerator.
Ineachcasestudy,ourexperimentconsistsofthefollowingsteps.
First,wecreateacongurationlefortheapplica-tionundertest.
Then,wegeneratetheeventsequencesusing860Figure6:TicketSellerGUITAR[10].
Next,wedistributetheeventsequencesfromtheloadbalancertoworkersonAmazonEC2.
TheinitialdistributionisstaticanddividesthetasksevenlytotheEC2instances.
Afterreceivingtheeventsequences,eachworkerconductsconcolicexecutionusingCatg;asaresult,testcasesaregeneratedfortheseeventsequences.
Whenallworkersnished,theloadbalancercollectstheirtestcasesandthenusesJaCoCotocomputethecoveragereport.
4.
1TheTicketSellerFigure6showstheuserinterfaceofamoresophisticatedticketsellerthantheoneshowninFigure1.
ItallowstheusertoprovidepassengerinformationsuchastheName,ID,startdistance(From),enddistance(To),AgeLevel,ClassLevel,andtheCoupon.
WhentheuserclickstheBuyTicketbutton,theapplicationstoresthepassengerinformationtoanobjectnamedTicketModel,checksforconsistencyusingthemethodcheckModel(),andcomputesthepriceusingthemethodcom-putePrice().
TherearevedifferenttypesofGUIwidgetsinFigure6:fourofJTextFieldtype,oneofJComboBoxtype,oneofJRa-dioButtontype,oneofJCheckBoxtype,andtwoofJButtontype.
ThersttwoJTextFieldwidgetscollectthevaluesofNameandIDbyinvokingJTextField.
getText(),thecombinationofwhichmayleadtobuggybehaviors.
Wemarkbotheldsassymbolic.
Thatis,whenloadingtherelatedJavaclass,weusethebytecoderewritingtoolASMtoinstrumenttheprogramonthey,toreplaceinvocationsofgetText()withinvocationsofsGetText(),amethodthatwedeveloptoreturnasymbolicvalue.
ThesymbolicvaluesforNameandIDareusedduringthesubsequentsymbolicexecutionstep.
ThenexttwoJTextFieldwidgetscollectthevaluesofFromandTobyrstinvokingJTextField.
getText()andtheninvokingInteger.
parseInttocastthestringsintointegers.
Sincethesym-bolicexecutionengineCatgdoesnotsupportsuchcasting,wehavemodiedCatgtoconvertthesestringsintointegersbeforeusingtheminthesubsequentlogic.
AsfortheselectablewidgetsJComboBox,andJCheckbox,weenumerateallpossiblevaluesoftheenumtypes.
Wechooseenumerationoversymbolicexecutionfortheseselectablewid-getsduetoefciencyandeaseofimplementation.
First,se-lectablewidgetsdonothavemanydifferentvalues.
Second,theexistingsymbolicexecutionengineoftenhastroublehan-dlingthem.
Forexample,JComboBoxhastwomethodsgetS-electedIndex()andgetSelectItem()thatreturnvaluesofacus-tomizedObjecttype,whichcannotbeeasilycasttostringsorintegersinsideCatg.
Tosummarize,weusedGUITARtogeneratetheinitialeventsequencestogetherwiththeinitialparameters/states.
Then,weaddtheenumeratedvaluesforselectablewidgets,be-foreconductingsymbolicexecutiontogeneratethevaluesforwidgetsofothertypes.
IftheinitialeventsequenceisTable1:Resultsontheticketseller(length=2,node=1).
ToolButtonBranchesCoverageInstructionsCoverageGUITARcheckModel633.
3%5653.
5%computePrice3423.
5%17230.
8%GUICatcheckModel6100%56100%computePrice34100%172100%Table2:Resultsonticketseller:thetestcasesgeneratedinadditiontoGUITARandtheuniquepathscovered.
ToollengthTestCase(TC)enumTCconcolicTCpathscoveredGUITAR211--43110--7GUICat211132869631101563212190tooshorttocontainallwidgetsofJTextFieldtypeneeded,weremovethestatelessJTextFieldeventandthenappendmorestatefulJTextFieldeventtothebeginningoftheeventsequence,thusincreasingthelengthofthesequence.
Weanalyzedtheticketsellerwithseveralcongurations.
Table1showstheresultsofgeneratingthelength-2testcases.
Columns1-2showthetoolnameandthenameofthebuttonclicked.
Columns3-4showthetotalnumberofbranchesandthepercentageofbranchescovered.
Columns5-6showthetotalnumberofinstructionsandthepercentageofinstruc-tionscovered.
TheresultsshowthatGUICatcanachievefullbranchandinstructioncoverageevenwithlength-2testcases,whereasGUITARcanonlyachieve33.
3%branchcov-erageand53.
5%instructioncoverageforcheckModel,23.
5%branchcoverageand30.
8%instructioncoverageforcompute-Price.
Table2comparesGUICatandGUITARintermsofthenum-berofpathscovered.
Column1showsthetoolname.
Col-umn2showsthelengthofthetestsequences.
Column3showsthetotalnumberoftestcasesgeneratedbyGUITAR.
Column4showsthenumberofadditionaltestcasesgener-atedbyGUICatafterenumeratingthevaluesofselectablewidgets.
Column5showsthenumberoftestcasesgener-atedbyGUICatafterconcolicexecution.
Column6showsthepathcoverage.
Toaccuratelycomputethenumberofpathscovered,wemanuallyaddedcodeintotheprogram.
Specically,weusedavectornamedpath,whereeachelementwasmappedtoanif-statement.
Forexample,theagecombo-boxcorrespondstoanif-statementwherewesetpath[0]=0inthethenbranchandpath[0]=1intheelsebranch.
Eachtimetheprogramter-minates,wewillobtainauniquevectorpaththatactsasthepathidentier.
ThesevectorsarestoredandthenusedtocomputethepathcoverageafterGUICatterminates.
Ourresultshowsthat,overall,GUICatachievedsigni-cantlyhigherpathcoveragethanGUITAR.
Forlength-2testsequences,inparticular,GUICathad96/4=24timeshigherpathcoverage,whereasforlength-3testsequences,GUICathad190/7=27timeshigherpathcoverage.
GUICatalsosuccessfullydetectedtwobugsinticketseller.
OnebugisaNullPointerExceptioncausedbytheracecondi-tionbetweenclickingofthesavebuttonandclickingofthebuybuttonasshowninFigures7and8.
Theotherbugisthefailureofanassertionduetothecomputedpricebeinglessthanzero.
4.
2TheWorkoutGeneratorFigure9showstheuserinterfaceoftheworkoutgenerator,whichistakenfromBarad[6].
Itgeneratesaworkoutplanbasedontheuserinput,includingtheGender,Metabolism,861buyButton.
addActionListener(newActionListener(){publicvoidactionPerformed(ActionEvente){ticketModel.
name=nameInput.
getText();if(ticketModel.
checkModel()){ticketModel.
computePrice();assertticketModel.
price>0:@Bug:price<=0!
;}}});saveButton.
addActionListener(newActionListener(){publicvoidactionPerformed(ActionEvente){.
.
.
ticketModel=null;}});Figure7:Thebuggycodesnippetinticketseller.
[AWT-EventQueue-0]ERRORUncaughtexceptionjava.
lang.
NullPointerExceptionatexamples.
ticket.
BaradTicket$1.
actionPerformed(BaradTicket.
java:177).
.
.
[AWT-EventQueue-0]ERRORUncaughtexceptionjava.
lang.
AssertionError:@Bug:price<=0!
atexamples.
ticket.
BaradTicket$1.
actionPerformed(BaradTicket.
java:235)Figure8:TheticketsellerbugdetectedbyGUICat.
Figure9:WorkoutGeneratorExperience,Age,Height,andWeight.
ThecomputationstartswhentheuserclickstheGeneratebutton.
Dependingontheuserinformation,thecomputationgoesthroughdifferentex-ecutionpathsthatusedifferentcardiocoefcients.
TherearethreeJTextFieldwidgetsandthreeJComboBoxwid-gets.
TheJTextFieldwidgetsreturnstringvaluesofAge,Height,andWeightusingthemethodJTextField.
getText(),beforecast-ingthemtointegersusingInteger.
parseInt().
Basedonthecongurationleprovidedbytheuser,GUICatcreatesthreesymbolicvariablesforthesethreewidgets,andreplacesget-Text()withsGetText(Object)soitcanreturnsymbolicvalues.
TheJComboBoxwidgetsreturnvaluesofGender,Metabolism,andExperience.
Sincetheyareselectablewidgets,weuseenu-merationtocreatethevaluecombinations.
Table3showstheresultsofapplyingGUICattoworkoutgeneratorwiththelengthoftestsequencessetto1and3.
Columns1-2showthetoolnameandthelengthofthetestsequences.
Columns3-4showthenumberofbranchescov-eredandthepercentageofbranchescovered.
Columns5-6showthenumberofinstructionscoveredandthepercentageofinstructionscovered.
Again,GUICatsignicantlyoutper-formedGUITARintermsofbothpathcoverageandinstruc-tioncoverage.
Neithertoolwasabletoreachfullpathandinstructioncoveragewithlength-2testsequences,becausetherearethreeselectablewidgets,whichrequiresthelengthoftestsequencesbeingtobelargerthan2.
Withlength-3testsequences,GUICatwasabletoachieve100%instructioncoverageand97.
4%branchcoverage.
Incontrast,GUITARdidnotshowsignicantimprovementinTable3:Resultsonworkoutgenerator(node=1).
ToolLengthBranchesCoverageInstructionsCoverageGUITAR215413.
0%242542.
3%315413.
0%242542.
3%GUICat215468.
9%242585.
7%315497.
4%2425100%Table4:Resultsonworkoutgenerator:thetestcasesgener-atedinadditiontoGUITARandtheuniquepathscovered.
ToollengthTestCase(TC)enumTCconcolicTCpathscoveredGUITAR29--1372--2GUICat2914692437218290956branch/instructioncoverage.
ThereasonisthatGUITARonlyusedthedefaultvaluesofthewidgets,andthuscannotex-plorealternativepathsevenwithalongereventsequence.
Incontrast,GUICatusedbothenumerationandsymbolicexe-cutiontodiversifythevaluesofthewidgets.
Table4comparesGUICatandGUITARintermsofthenum-berofpathscovered.
Column1showsthetoolname.
Col-umn2showsthelengthofthetestsequencesusedintheex-periments.
Column3showsthenumberoftestcasesgen-eratedbyGUITAR.
Column4showsthenumberofaddi-tionaltestcasesgeneratedbyGUICatafterenumeratingthevaluesofselectablewidgets.
Column5showsthenumberoftestcasesgeneratedbyGUICatafterconcolicexecution.
Column6showsthenumberofpathscovered.
Ourresultdemonstratesthat,overall,GUICatcanachievesignicantlyhigherpathcoveragethanGUITAR.
Forlength-2testsequences,inparticular,GUICatreached24/1=24timeshigherpathcov-erage,whereasforlength-3testsequences,GUICatreached56/2=28timeshigherpathcoverage.
WealsoobservedthatGUICatgeneratedmanymoretestcasesthanthepathscovered.
Forexample,withthelengthsetto3,GUICatgenerated909testcasestocover56uniquepaths,whichmeanssomeofthesetestcaseshaveledtothesamepaths.
Ifwecould,forexample,identifyandelimi-natetheseredundanttestcases,theperformanceofGUICatwouldbefurtherimproved.
However,weleavethepruningoftheseredundanttestcasesforfuturework.
4.
3EffectofCloudComputingFigure10showstheeffectivenessofdistributingthetest-ingoftheticketseller(a)andtheworkoutgenerator(b)overAmazonEC2.
Thex-axisdenotethenumberofworkers,rangingfrom1to16,andthey-axisdenotethetimeusageinsecond.
Thesolid,dashedanddottedlinesrepresenttheboundedlengthof2,3and4,respectively.
Duetothein-herentparallelisminsymbolicexecutionofdifferenteventsequences,thespeedupisalmostlinear.
5.
RELATEDWORKGUITAR[10]istherstframeworkcapableofperformingthewholeprocessoftestgeneration,execution,andresultas-sessmentforGUIs.
Sinceitsrstpublicationtherehavebeenmultipleimprovements(e.
g.
[14]).
Thisframeworkgeneratestestsaseventsequencesuptoagivenbound.
Foremulatinguserinputaspecicationbasedapproachisadopted,i.
e.
,us-ingvaluesfromaprelleddatabase.
SinceGUITARdoesnotprovideamechanismforreasoningaboutinputvaluesfordatawidgets,GUICatofferscomplementaryandmorecom-prehensivetesting.
862(a)ticketseller(b)workoutgeneratorFigure10:Timereductionachievedbycloudcomputing.
TheworkclosesttooursisBarad[6]thatalsoexploitssymbolicexecutiontocomputeinputvaluesfordatawid-gets.
ItmanuallycreatessymbolicmirrorofjavaGUIlibrary,soitsreleasedsourcecodecontainsalargesymbolicjavaGUIlibrary.
Manualmodelingiserrorproneandhardtosus-tain.
Infact,wedownloadedthetoolbutfailedtomakeitworkonourbenchmarks.
Weemployadifferenttestgener-ationalgorithmandsymbolicanalysismethodforobtaininginputs.
Anotherlineofworkistoapplymodelcheckingtech-niques.
Forexample,jfp-awt[8]isanextensionoftheJavaPathFinderforGUIapplications.
PerformanceenhancementofGUItestinghastraditionallyfocusedonminimizingeventsequences[9,13].
Baradgen-eratestestcasesaschainsofeventlistenermethodinvoca-tionsandmapsthesechainstoeventsequencesthatforcetheexecutionoftheseinvocations.
Suchapproachprunestheeventinputspacebecauseitdoesnotneedtoconsidereventswheretherearenoeventlisteners.
Morerecentworkstartstoapplyeventdependencyanalysis[4],programslicing[5]orpartialorderreduction[7]toimprovetheperformance.
Ourperformanceimprovementisobtainedbyexploitingmassivehardwareresourceavailableoncloud.
Thereforeourapproachisorthogonaltotheexistingalgorithmicapproaches.
6.
CONCLUSIONWehavepresentGUICat,therstcloud-basedGUItest-ingtoolforsimultaneouslygeneratinghigh-qualityeventse-quencesaswellashigh-qualitydatavalues.
Internally,GUICatleveragesGUITARtogeneratetheinitialsetofeventsequences,andthenusesacombinationofvalueenumerationandsym-bolicexecutiontogeneratedatavaluesofthewidgets.
GUICatalsoleveragesthecloudcomputationinfrastructuretospeedupthetestgeneration,bydistributingindependentconcolicexecutiontaskstoEC2nodes.
WehaveimplementedGUICatandevaluateditonasetofGUItestingbenchmarks.
OurexperimentsshowthatGUICatcansignicantlyoutperformGUITARonstandardGUItestingbenchmarksintermsofbothbranchcoverageandinstructioncoverage.
7.
ACKNOWLEDGMENTSThisworkwasprimarilysupportedbyNSFundergrantsDGE-1522883andCCF-1500365.
Partialsupportwaspro-videdbyCCF-1149454,CCF-1405697,andCCF-1500024.
8.
REFERENCES[1]ASM.
http://asm.
ow2.
org/.
[2]JaCoCo.
http://eclemma.
org/jacoco/.
[3]Amazon.
Amazonelasticcomputecloud.
http://aws.
amazon.
com/ec2/.
[4]S.
Arlt,A.
Podelski,C.
Bertolini,M.
Schf,I.
Banerjee,andA.
M.
Memon.
LightweightstaticanalysisforGUItesting.
InInternationalSymposiumonSoftwareReliabilityEngineering,2012.
[5]S.
Arlt,A.
Podelski,andM.
Wehrle.
ReducingGUItestsuitesviaprogramslicing.
InInternationalSymposiumonSoftwareTestingandAnalysis,2014.
[6]S.
Ganov,C.
Killmar,S.
Khurshid,andD.
E.
Perry.
EventlisteneranalysisandsymbolicexecutionfortestingGUIapplications.
FormalMethodsandSoftwareEngineering,2009.
[7]P.
Maiya,R.
Gupta,A.
Kanade,andR.
Majumdar.
Partialorderreductionforevent-drivenmulti-threadedprograms.
InToolsandAlgorithmsfortheConstructionandAnalysisofSystem,pages680–697,2016.
[8]P.
Mehlitz,O.
Tkachuk,andM.
Ujma.
JPF-AWT:ModelcheckingGUIapplications.
ASE,2011.
[9]A.
Memon,I.
Banerjee,N.
Hashmi,andA.
Nagarajan.
DART:Aframeworkforregressiontesting"nightly/-dailybuilds"ofGUIapplications.
InInternationalCon-ferenceonSoftwareMaintenance,2003.
[10]B.
N.
Nguyen,B.
Robbins,I.
Banerjee,andA.
Memon.
GUITAR:aninnovativetoolforautomatedtestingofGUI-drivensoftware.
AutomatedSoftwareEngineering,2014.
[11]R.
K.
ShehadyandD.
P.
Siewiorek.
Amethodtoau-tomateuserinterfacetestingusingvariablenitestatemachines.
InInternationalSymposiumonFault-TolerantComputing,1997.
[12]H.
Tanno,X.
Zhang,T.
Hoshino,andK.
Sen.
TesMaandCATG:automatedtestgenerationtoolsformodelsofenterpriseapplications.
InternationalConferenceonSoft-wareEngineering,2015.
[13]L.
WhiteandH.
Almezen.
GeneratingtestcasesforGUIresponsibilitiesusingcompleteinteractionsequences.
InInternationalSymposiumonSoftwareReliabilityEngi-neering,2000.
[14]Q.
XieandA.
M.
Memon.
UsingapilotstudytoderiveaGUImodelforautomatedtesting.
ACMTrans.
Softw.
Eng.
Methodol.
,18(2):7:1–7:35,Nov.
2008.
863