impactedm

SequenceMatters,ButHowDoIDiscoverHowTowardsaWorkflowforEvaluatingActivitySequencesfromDataShayanDoroudi1,KennethHolstein2,VincentAleven2,EmmaBrunskill11ComputerScienceDepartment,2Human-ComputerInteractionInstituteCarnegieMellonUniversity{shayand,kjholste,aleven,ebrun}@cs.
cmu.
eduABSTRACTHowshouldawidevarietyofeducationalactivitiesbesequencedinordertomaximizestudentlearningWerecentlyproposedtheSequencingConstraintViolationAnalysis(SCOVA)methodtohelpaddressthisquestion.
Inthispaper,weproposehowSCOVAcouldbetransformedintoaworkflowinLearnSpheresothatotherresearchersandpractitionerscanfindanswerstotheaforementionedquestionintheirowndatasets.
Wehopethatsuchaworkflowwillleadtomoreandbetterresearchintothisimportantquestion,aswellasinterestingnewfindingsforboththeeducationaldataminingandlearningsciencescommunities.
Keywordssequencing,ordering,IntelligentTutoringSystems,LearnSphere,DataShop,workflow.
1.
INTRODUCTIONHowtosequenceeducationalactivitiesisanimportantpedagogicalquestion[12].
Muchoftheexistingworkonsequencingactivitiesconsistsoftheoreticalanalyses[2,4,7]andempiricalstudies[1,13,5,11].
Whileempiricalstudiescanhelpaddressquestionsthatcomparetwoorthreedifferentwaystosequenceacurriculum(e.
g.
,whethertopicsshouldbeblockedorinterleaved),itcannoteffectivelyscaletoanalyzingthemyriadofpotentialsequencesthatcouldbeconsidered.
However,educationaldatamining(EDM)techniquescanenableonetosimultaneouslystudydifferenttypesofsequencesbasedonpastdata.
Werecentlyproposedonesuchmethod—SequencingConstraintViolationAnalysis(SCOVA)—forcomparingtheefficacyofdifferentsequencingconstraintsgivenadatasetthatisrichinthevarietyofsequencesitexplores[3].
SCOVAcanbeusedtoanalyzeawidevarietyofsequencingconstraints,suchasprerequisiterelationships,constraintsonwhendifferentlearningmechanismsshouldbeintroduced,blocking,interleaving,andspiraling.
SCOVAcanbothbeusedtobetterunderstandhowproblemsshouldbesequencedinspecificlearningenvironments,includingintelligenttutoringsystems(ITSs),aswellastofindsomegeneralizabletrendsthatmayinformthelearningsciencesliterature(e.
g.
,onwhetherblockingorinterleavingismoreeffectiveorinwhatorderlearningmechanismsshouldbesupported).
SCOVAcanalsobeusedtoinformthecreationofadaptivepoliciesforITSs.
However,SCOVAwillmostlikelynotbeusedforanyofthesepurposesifitjustremainsinapaperthatafewresearchersmight,atbest,readandcite.
Rather,itsbenefitwilllikelyonlyoutlivetheconfinesofaone-offEDMpaperifitisreleasedasaworkflowonaplatformlikeLearnSpherethatisusedbyresearchersandpractitioners.
Ifreleasedassuchaworkflow,SCOVAcanalsointroduceresearcherswhomaynothaveotherwiseconsideredthequestionofhowactivitiesshouldbesequencedintheirlearningenvironmentstofindanewfoundinterestinthisarea,whichwebelieveisbecomingincreasinglyimportanttoboththelearningsciencesandeducationaldataminingcommunities.
2.
WORKFLOWMETHOD2.
1DataInputsSCOVAisapplicabletodatasetswithsubstantialvariabilityinthetypesofactivitysequencesthatstudentscomplete.
Thisvariabilityistypicalofmanydatasets,includingonesthatincluderandomnessinhowproblemswerepresentedtostudents(e.
g.
,[9]),oneswhereadaptivepolicieswereusedforproblemselectionresultinginsequencesthatvaryfromstudenttostudent(e.
g.
,[10]),andoneswherestudentsareabletodochoosewhichproblemstoworkonthemselves(e.
g.
,[8]).
TheworkflowcanworkwithdatasetsinthePSLCDataShopformat.
GiventhatSCOVAisaverygeneral-purposemethod,whichcanbeusedtoanalyzehowawidevarietyofsequencingconstraintsimpactpotentiallydifferentmeasuresofstudentperformance(e.
g.
,within-tutorperformance,posttestscores,learninggains,timeontask,etc.
),itmaypotentiallyneedtoutilizeavarietyofthecolumnsinaDataShopdataset.
However,forsimplicitywewilldescribeaversionofSCOVAthatislimitedtoanalyzingsequencingconstraintsthatmayonlydependonwithin-tutorcorrectnessandpropertiesoftheactivitiespresentedtostudentsandcanonlymeasuretheimpactwithrespecttowithin-tutorperformanceandfunctionsofpretestandposttestscores(suchaslearninggains).
Infull,SCOVAneedsthreeinputfiles:1.
TheDataShoptransaction-levelfile.
Foreverystepinatransaction-leveldataset,SCOVAneedstoknowtheproblemnameandwhetherthestepwasansweredcorrectlyornot.
2.
Amappingofeveryproblemnametocategoriestowhichtheproblembelongs.
Forexample,whenusingSCOVAonourfractionsITS[3],welabeledeachproblemwithoneofthreetopiclabels(makingandnamingfractions,fractionequivalenceandordering,andfractionaddition)aswellasoneofthreeactivitytypescorrespondingtolearningmechanismsfromtheKnowledge-Learning-Instruction(KLI)framework(sense-making,inductionandrefinement,andfluency-building)[6].
Thesecategorylabelswillthenbeusedasthebuildingblocksofsequencingconstraints,asexplainedinSection2.
2.
3.
Afilethatgivesthepretestandposttestscoreforeachstudent.
2.
2WorkflowModelTheworkflowbeginswiththeresearcherselectingdifferentsetsofsequencingconstraintsthattheywanttoanalyze.
Eachsequencingconstraintcanbeselectedbyfirstchoosingacategory(e.
g.
,topicsoractivitytype)andthenselectingapatternthatcorrespondstothesequencingconstraint.
Thepatterncantakeononeofthreeforms:1.
Specifyingaparticularsequence(e.
g.
,ABCABCABC,whichmaycorrespondtointerleavingdifferentactivitytypesortopics).
2.
SpecifyingthatastudentshouldbeexposedtoaproblemwithlabelAbeforeaproblemoflabelB(e.
g.
,astudentshouldbeshownanumberlineproblembeforebeingshownafractionequivalenceproblem)3.
SpecifyingthatastudentshouldhavereachedsomeperformancethresholdonaproblemwithlabelAbeforeaproblemwithlabelB(e.
g.
,astudentshouldhave95%accuracyonfractionequivalenceproblemsbeforebeingexposedtofractionaddition)Theresearchercanselectasmanysequencingconstraintsofthethreeformsabove.
Thenforeachpossiblepermutationofcategorylabels(e.
g.
,A=fractionequivalence,B=fractionaddition,C=namingfractions),SCOVAcomputesascoreforhowwelleachstudent'ssequenceinthedatasetmatchesthegivensequencingconstraints.
Thescoreistheproportionofproblemsinthetrajectorywhereasequencingconstraintwasviolated.
SCOVAthenlearnsalinearregressionmodelthatusesthedegreetowhichastudentviolatesaparticularsetofsequencingconstraintstopredictsomechosenoutcomevariable(i.
e.
,somemeasureofwithin-tutorperformanceorsomefunctionoftheposttestandpretestscores).
Noticethatifthemodelhasanegativecorrelationthenthatimpliesthemoreastudentobeysaparticularsequencingconstraint,thebetterthatstudentlearns/performsinthetutoringsystem,i.
e.
negativecorrelationsareindicativeofbeneficialsequencingconstraints.
ThefinalstepofSCOVAistocomparethemodelfitsfordifferentsetsofsequencingconstraintstoguidethepractitioner/researchertowhichsequencingconstraintshavethelargestpositiveimpactonstudentlearning.
Formoredetailsonthemethodandparticularinstantiationsofsequencingconstraints,referto[3].
2.
3WorkflowOutputsTheprimaryoutputisatableofBICvaluesofmodelsforeverysetofsequencingconstraintsevaluated.
Thepractitionercanchoosefromasetofoptionshowtheywantthetableorganized.
Forexample,ifwewereevaluatingtheimpactofconstraintsoftheformtopicAshouldcomebeforetopicB,whichshouldcomebeforetopicCintandemwithconstraintsoftheformactivitytypeXshouldcomebeforeactivitytypeY,whichshouldcomebeforeactivitytypeZ,thiscouldberepresentedina6-by-6tablewheretherowscorrespondtothedifferentpermutationsovertopicsandthecolumnscorrespondtothedifferentpermutationsoveractivitytypes.
(Iftherewasathirdcategoryofinterestwiththreedifferentlabels,suchassaywhetherthedifficultyleveloftheproblemwaseasy,medium,orhard,thentheworkflowcoulddisplaysixdifferenttables,oneforeachpermutationofdifficultylevels.
)Foranexampleofsuchatable,seeTable3in[3].
InadditiontoshowingBICvalues,thetablewillhighlightthosecellswheretheviolationofsequencingconstraintscorrelatesnegativelywithperformance/learning(againanindicatorthatthesequencingconstraintisbeneficialforstudentsratherthanharmful),andwilldesignatethemodelwiththelowestBIC(i.
e.
,thebest-fittingmodel).
TherewillalsobeatoggletodisplayotherquantitiesofimportanceinplaceofBIC,suchasthecoefficientsofthepredictorsinthemodels.
Inthecaseofevaluatingsequencingconstraintsoverasinglecategory(e.
g.
,onlyhowactivitytypesshouldbesequenced),theusercanchoosetodisplaythescatterplotsusedtofiteachmodelandthebest-fitlinesthemselves.
Theusercanalsochoosetocolor-codeeachpointofthescatterplotswiththevalueofsomefeature(e.
g.
,howmanyproblemsthatstudentreceived).
Thiscolor-codingoftheplotscanhelpidentifypotentialconfounds(e.
g.
,studentswhodomoreproblemsmighttendtoviolatefewerofasequencingconstraintandalsodobettersimplybecausetheydidmoreproblems).
Finally,theworkflowwillallowdoingexploratoryanalysestodetectotherpotentialconfounds.
Forexample,ifthesequencesinthedataweregeneratedaccordingtoadaptivepolicies,onepotentialconfoundisthatastudent'sperformanceaffectsthedegreetowhichsequencingconstraintsareviolatedinadditiontotheintendedcausaldirectionofthedegreetowhichasequencingconstraintisviolatedinfluencingthestudent'sperformance.
Toanalyzethepresenceofsuchaconfound,modelscanbelearnedwheretheoutcomevariableisthestudent'spretestscore(ratherthansayposttestscore);sincethepretestscorecomesbeforethestudents'useofthetutor,weknowthattheonlyreasonitwouldcorrelatewithviolationsofcertainsequencingconstraintsisiftheadaptivepoliciesdiscriminatedbetweenstudentswithdifferentamountsofpriorknowledge.
InusingSCOVAonourfractionstutor,wefoundthatwhilethisreversecausaldirectiondidexist,itwasseeminglynegligibleandactuallybiasingagainsttheconclusionsthatourresultssupport[3].
SuchaworkflowshouldallowuserstheabilitytodoexploratoryanalysesbeforemakingfirmconclusionsusingSCOVA.
3.
DISCUSSIONHavingaworkflowforanalyzingtheimpactofdifferentsequencingconstraintscanhaveanumberofbenefitsforboththeEDMandlearningsciencecommunities.
SCOVAcanbothbeusedtobetterunderstandhowproblemsshouldbesequencedinspecificlearningenvironments,aswellastofindsomegeneralizabletrendsthatmayinformthelearningsciencesliterature(e.
g.
,onwhetherblockingorinterleavingismoreeffectiveorhowlearningmechanismsshouldbesequenced).
SCOVAcanalsobeusedtoinformthecreationofadaptivepoliciesforITSs.
However,forSCOVAtobeusedinsuchafashion,itwilllikelyhavetobereadilyavailableasaworkflowonaplatformlikeLearnSpherethatisusedbyresearchersandpractitioners.
Additionally,byhavingsuchaworkflowonLearnSphere,moreresearchersmaybeattractedtothequestionofhowtosequenceproblemsintheirlearningenvironmentofinterest.
Furthermore,ifLearnSpherealsoincludesworkflowsforothermethodsofanalyzingsequencingconstraintssuchas[9],moreresearchcanbedoneincomparingthesemethods.
Currentlywhensuchamethodispublisheditisnotwidelyadoptedeitherinpracticeorbyotherresearchers,anditisnotcomparedtomethodsthatsucceedit.
Byputtingallmethodsthatdosimilarstylesofanalysesononeplatform,LearnSpherecanleadtomoreproductiveresearch,includinghopefullybetterwaysofunderstandinghowweshouldsequenceeducationalactivitiesindifferentlearningenvironments.
4.
ACKNOWLEDGMENTSTheresearchreportedherewassupportedbytheInstituteofEducationSciences,U.
S.
DepartmentofEducation,throughGrantsR305A130215andR305B150008toCarnegieMellonUniversity.
TheopinionsexpressedarethoseoftheauthorsanddonotrepresentviewsoftheInstituteortheU.
S.
Dept.
ofEducation.
5.
REFERENCES[1]W.
Battig.
Intrataskinterferenceasasourceoffacilitationintransferandretention.
Topicsinlearningandperformance,pages131–159,1972.
[2]R.
E.
Clark,D.
Feldon,J.
J.
vanMerrienboer,K.
Yates,andS.
Early.
Cognitivetaskanalysis.
Handbookofresearchoneducationalcommunicationsandtechnology,3:577–593,2008.
[3]S.
Doroudi,K.
Holstein,V.
Aleven,andE.
Brunskill.
SequenceMatters,ButHowExactlyAMethodforEvaluatingActivitySequencesfromData.
InEDM,2016.
[4]J.
-C.
Falmagne,M.
Koppen,M.
Villano,J.
-P.
Doignon,andL.
Johannesen.
Introductiontoknowledgespaces:Howtobuild,test,andsearchthem.
PsychologicalReview,97(2):201,1990.
[5]S.
Kalyuga.
Expertisereversaleffectanditsimplicationsforlearner-tailoredinstruction.
EducationalPsychologyReview,19(4):509–539,2007.
[6]K.
Koedinger,A.
Corbett,andC.
Perfetti.
TheKnowledge-Learning-Instructionframework:Bridgingthescience-practicechasmtoenhancerobuststudentlearning.
CognitiveScience,36(5):757-798,2012.
[7]K.
Korossy.
Modelingknowledgeascompetenceandperformance.
Knowledgespaces:Theories,empiricalresearch,andapplications,pages103–132,1999.
[8]Y.
LongandV.
Aleven.
Supportingstudents'self-regulatedlearningwithanopenlearnermodelinalinearequationtutor.
InAIED,2013.
[9]Z.
A.
PardosandN.
T.
Heffernan.
Determiningthesignificanceofitemorderinrandomizedproblemsets.
2009.
[10]M.
A.
Rau,V.
Aleven,andN.
Rummel.
Complementaryeffectsofsense-makingandfluency-buildingsupportforconnectionmaking:AmatterofsequenceInAIED,2013.
[11]A.
RenklandR.
K.
Atkinson.
Structuringthetransitionfromexamplestudytoproblemsolvingincognitiveskillacquisition:Acognitiveloadperspective.
Educationalpsychologist,38(1):15–22,2003.
[12]F.
E.
Ritter,J.
Nerb,E.
Lehtinen,andT.
M.
O'Shea,editors.
Inordertolearn:howthesequenceoftopicsinfluenceslearning.
OxfordUniversityPress,2007.
[13]D.
RohrerandK.
Taylor.
Theshufflingofmathematicsproblemsimproveslearning.
InstructionalScience,35(6):481–498,2007.