includesvim

TheCognitiveComplexityofaProviderOrderEntryInterfaceJanHorsky,MA,DavidR.
Kaufman,PhD,VimlaL.
Patel,PhDLaboratoryofDecisionMakingandCognition,DepartmentofBiomedicalInformatics,ColumbiaUniversity,NewYork,NYABSTRACTComputer-basedproviderorderentry(POE)canreducethefrequencyofpreventablemedicalerrors.
However,overlycomplexinterfacesfrequentlyposeachallengetousersandimpedeclinicalefficacy.
WepresentacognitiveanalysisofclinicianinteractionwithacommercialPOEsystem.
Ourinvestigationwasinformedbythedistributedresourcesmodel,anovelapproachdesignedtodescribethedimensionsofuserinterfacesthatintroduceunnecessarycognitivecomplexity.
Thisapproachcharacterizestherelativedistributionofuser'sinternalrepresentationsandexternalrepresentationsembodiedinthesystemorenvironmentalartifacts.
Theresearchconsistedoftwocomponentanalyses:amodifiedcognitivewalkthroughevaluationandasimulatedclinicalorderingtaskperformedbysevenphysicians.
Theanalysisrevealedthattheconfigurationofresourcesplacedunnecessarilyheavycognitivedemandsontheuser,especiallythosewholackedarobustconceptualmodelofthesystem.
Theresourcesmodelwasalsousedtoaccountforpatternsoferrorsproducedbyclinicians.
INTRODUCTIONThereisagrowingrecognitionthatmanyerrorsareneithersolelyattributabletolapsesinhumanperformanceortoflawedtechnology,butdevelopasaproductoftheirinteraction1.
Cognitiveengineeringisaninterdisciplinaryapproachtounderstandingthecomplexityoftheintellectualpartnershipbetweenhumansandmachines2andisausefulapproachfortheinvestigationofinteractionerrors.
Errorsareroutineinmostareasofcomplexhumanperformanceandafractionwillhavedramaticconsequences3.
Newlyadoptedtechnologiestendtoalterworkhabitsandfamiliarpractices,andasaresultmayintroducenewsourcesoferror2.
Wepresentanapproachtotheanalysisofacomputer-basedproviderorder-entrysystem(POE),intendedtocharacterizethecognitivedemandsofinteractionwiththiscomplextechnology.
Orderentrysystemsweredevelopedinparttoeliminateerrorsassociatedwithhand-writtenorderingandtoincreasethespeedandqualityofcommunicationbetweenclinicians.
Thereisevidencetosuggestthatsystemscurrentlyinusehavealreadyreducedtherateofmedicationerrorsandimprovedpatientcare4.
However,POEimplementationhasproventobeasignificantchallenge5,oftenresultinginworkflowreorganizationinhospitals,andrequiringclinicianstolearnadvancedinformationapplications.
Recently,discontentedphysiciansassociatedwithalargeCaliforniahospitalnetworkevenforcedahalttotherolloutofPOE.
ComplexPOEinterfacesimposeasteeplearningcurveonthenoviceuserwhilethebenefitsofthesysteminspeedandaccuracymaynotbeattainedforsometime.
Thiscomplexitycanbepartlyattributedtothemultifacetednatureofclinicalmedicine.
However,apoorlydesignedPOEinterfacenotonlyslowsdowntheclinicianbutmayintroduceanewsourceofmedicalerrorsintotheorderingprocess,intrinsictohumaninteractionwithinformationtechnology6.
Theseobservationssuggesttheneedforcharacterizingdimensionsofuserinterfacestoidentifysourcesofunnecessarycognitivecomplexitythatincreasecognitiveloadanddivertattentionfromtheclinicaltaskathand.
Theoreticalframeworksandmethodsfromcognitiveengineeringcanproductivelyinformresearchontheevaluationofmedicalcomputerinterfaces7.
Acognitiveengineeringapproachhasbeenemployedtodiagnosethepatternoferrorsinvolvedintheuseofapatientcontrolledanalgesicdevice8,andtoinvestigatechangesindiagnosticreasoningstrategiesofcliniciansusinganelectronicmedicalrecordsystem7.
TheresearchpresentedinthispaperisinformedbyatheoreticalframeworkthatincorporatesextensionsofNorman'stheoryofaction9,inparticularthecognitivewalkthrough10,andrecentdevelopmentsindistributedcognitionmethodsofhuman-computerinteraction(HCI)analysis.
ItisusefultothinkofHCIasacontinuousprocessofcyclicalinteraction,recognizingchangesofstateintheenvironmentandactingontheenvironmenttobringaboutnewchangesinstate.
Themodeliscyclicalinthesensethatactionisinformedbytheconfigurationofresourcesrepresentedintheinteractionataparticulartime-eitherexternallyintheinterfaceorinternallyinthemindoftheuser11.
Cognitionisthenviewedasaprocessofcoordinatingdistributedinternalandexternalrepresentations.
Thedesignimplicationsofthisideahavelongbeenrecognized.
Forexample,Norman9arguedthatwell-designedartifactscouldreducetheneedforuserstorememberlargeamountsofinformation,whereaspoorlydesignedartifactsincreaseddemandsontheuser'sworkingmemory.
Mostcognitivetasksaredescribedashavinganinternalandexternalcomponent12.
Thereasoningprocesstheninvolvescoordinatingtheserepresentationstoderivenewinformation.
Differentbutfunctionallyequivalentdisplays(i.
e.
,supportingthesamesetoffunctions)canhavedissimilarrepresentationaleffects.
Forexample,certainexternalrepresentations(e.
g.
,pick-lists)canminimizethedifficultyofataskbysupportingrecognition-basedmemoryorperceptualjudgmentsratherthanfreerecall.
ThisisanalogoustothedifferencesbetweenGUIsandcommandlineinterfaces.
ThedistributedresourcesmodelproposedbyWrightetal11addressesthequestionof"whatinformationisrequiredtocarryoutataskandwhereshoulditbelocated,asaninterfaceobjectorassomethingthatismentallyrepresentedtotheuser.
"Inotherwords,theuserbringsasetofresourcestotheinteractionintheformofhisorherknowledgeandexperiences.
Similarly,"systemresources"suchasdialoguesboxes,buttons,andhelpfacilitiesguidetheinteractioninspecificways.
Thesecanbecategorizedandquantified.
Therelativedifferencesinthedistributionofrepresentations(internalandexternal)arecentralindeterminingtheefficacyofasystemdesignedtosupportacomplextask.
Thismodelincludesacharacterizationofabstractinformationstructures(i.
e.
,resourcetypes)thatcanbeusedtoanalyzeinteraction.
Howtheseinformationstructuresarerealizedininterfaceswillcriticallyaffectthequalityofuserinteraction.
Thismayenhanceorimpedeperformance.
Theauthors11proposesixabstractinformationstructuresPlans-resourcesforactionthatincludeasequenceofactionsandanticipatedstates.
Goals-statestheuserwantstoachieve,generatedinternallyoremergingfromsysteminteraction.
Affordances-links,buttons,ormenusthatsuggestpossiblenextactionsatagivenstateofthesystem.
History–thepartofaplanalreadyaccomplished(e.
g.
,alistofpreviouslyvisitedsitesinawebbrowser).
Action-effectrelations-indicatethecausalrelationshipbetweenanactionandtheeffectedchangeinstate.
State-thecurrentconfigurationofresources,asembodiedinthedisplayscreenatagivenpoint.
WeneededtoinduceadditionalinformationstructurestoadequatelydescribetheconsiderableinterfacecomplexityofthisPOE.
Biomedicalknowledgewasdifferentiatedintopatient-specific(e.
g.
,age,bloodpressure),generalmedical(admissionorderstructure)andinstitution-specific(formulary,locations).
Eachinformationstructurewasdividedintointernalandexternalrepresentation.
Forinstance,apatient-specificexternalresourcecouldbeabloodpressurereadingdisplayedonthescreenorinaclinicalnote,whereastherecallofthepatient'shistoryoflabilehypertensionfromthehospitalroundscouldbeaninternalresource.
Aconceptualmodelofthesystem(internalrepresentation)correspondstousers'understandingofhowthesystemworks.
Ourresearchobjectivewastoevaluateacomplexproviderorderentrysystemusingthedistributedresourcesframework.
Specifically,wewantedto1)analyzehowthesituationaldistributionofcognitiveresourcesmayresultinperformancevariationorthecreationofopportunitiesforerror,and2)usethismodeltoevaluatetheperformanceofcliniciansusingthePOEsysteminanexperimentaltask.
METHODSTheanalysisofthisPOEsystemconsistedoftwocomplementaryapproaches.
First,weperformedamodifiedversionofthecognitivewalkthroughinformedbythedistributedresourcesmodeltodescribeandquantifytherelativedistributionofcognitiveresourcesactiveduringclinicalordering.
Wethenconductedanexperimentaltaskinwhichsevenphysicianswereaskedtoenterappropriateordersforagivenclinicalscenario.
Thecombinationofthesetwomethods,thecognitiveanalysisandempiricaldatacollection,wasintendedtoa)characterizethecognitivedemandsoftheorderingtask,b)toevaluatehowwellthedemandsaresupportedbyavailableresources,andc)toidentifypossiblesourcesoferror.
Thefocusisonanin-depthqualitativeanalysisofperformance,thusnecessitatingfewersubjects.
AdevelopmentversionofcommerciallyavailablePOEsystemwasusedforboththewalkthroughanalysisandfordatacollection.
Ageneralpatientadmissionorderscenariowasdevelopedandusedbecauseitisreasonablywellstructured,largelyinvariantsetofconstituentorders,andthefactthatitdoesn'trequirespecializedmedicalexpertise.
Thescenarioispresentedbelow.
A65-year-oldmanwithamedicalhistoryofuntreatedlabilehypertensionandiodinesensitivityisadmittedtothehospitalbyhisprimaryphysician,Dr.
Lesion.
HehasanindwellingFoleycatheterinplaceandisadmittedearlyinthemorningforaTURPlaterthesameday.
Pre-operativetestingwasdoneasanoutpatienttwodayspriortoadmission,andthepatientcomeswithcopiesoftheresults.
Dr.
Lesioncallsandasksyoutoadmitthepatient,getanIVgoing,andputthelabsonthechartfortheurologistthatwillcomebylatertowritepre-operativeorders.
Writetheadmissionordersforthispatient.
Thetaskrequiredsubjectstodevelopaproblemrepresentationoftheclinicalscenarioand1)assessthepatientcondition,2)recordnoteworthyfindings,and3)enterordersasrequested.
Itwasimportanttonotethepatient'siodinesensitivityandthathehasanindwellingFoleycatheterthatnecessitatesanursingordernotincludedwiththeavailableorderentryset.
STATE:(6)AdmissionordersetindefaultstateGOAL:SelectasubsetofappropriateordersART:PatientscenariowithdataandfindingsAFF:40buttonswithtextlabels14visibleorderheaders22orderheadersscrolledoffscreen.
MED:Generaladmissionrequirements,IVfluidsadultdosing(2).
SPEC(I):Vitals,activity,nursing,diet(4)SPEC(E):Allergy,diagnosis(2)CSK:-Multimarkcheckboxenablestheselectionofmultipleorders.
-Checkboxinthefirstordertogglesvalueofalldisplayedorders.
-Someordersarevariantsofthesameorderwithdifferentdefaultvalues.
-Ordersmustbeactivatedbeforedefaultvaluescanbechanged(4)HSI:Ordervaluesnotvisibleonthelist(1)HSE:Selectedorderschecked(7)PLI:Select7individualorders,clickF9Activatebutton(2)Systemwalkthrough:Thisanalysiswasdesignedtosimulateanexpertcompletingthepatientadmissionorderentrytask.
ItwascompletedbytworesearcherswiththeassistanceofaphysicianwhowasalsoanexpertPOEuser.
Medicalordersappropriateforthegivenscenariowereenteredandtherelativedistributionofavailableresourceswasrecordedateverysystemstateandclassifiedaccordingtothenotationalmodeldescribedintheresultssection(Figure1providesanexample).
Opportunitiesforpotentialerrorsandtheirpossiblemedicallyadverseconsequenceswereidentifiedandnoted.
Orderentrybyclinicians:Seveninternalmedicinephysicianswithayearormoreofdailyorderentryexperienceandarangeof2-5yearsofclinicalexperienceweregivenawrittenclinicalscenarioandinstructedtoproceedwithenteringappropriatemedicalorderswhileverbalizingtheirthoughts(athink-aloudprotocol).
Thescreenvideosignalwascapturedandrecordedonavideotapesothatmousemovements,actionsandscreentransitionscouldbeanalyzed.
Thesubjectswerealsovideotapedastheyperformedthetask.
Eachsessiontookabout30minutes.
Subjects'verbalizationsweretranscribedandcodedforacognitivetaskanalysis7.
Figure1.
DistributedResourcesAnalysisofState6RESULTSSystemwalkthrough:.
TheGUIprovidesnumerousaffordances(e.
g.
,buttonsandactionableobjectsonthedisplay),buttheconfigurationofresources(forexample,thefacilitationofsuccessivesteps)islessthanoptimalforachievinggoalswithoutasignificantcognitiveeffort.
Thisisillustratedinthecontextoftheanalysisofasystemstate(screenconfigurationofaffordancesandresources)describedinFigure1whereusersselectasubsetofapplicableordersfromanadmissionorderset.
Inournotation,theSTATEservesasalabelforthecurrentconfigurationofinternalandexternalresources.
Eachscreentransitionconstitutesastatechange.
TheGOALisformedbytheuserbasedonthecurrentstateandhisorherconceptualmodelofthesystem.
Here,theuserneedstoselectasubsetofordersappropriatefortheclinicalscenariofromadefaultsetof36orders.
ARTisanavailableartifact,thatmaybeadrugdosingmanualinpaperorelectronicversion,oralistofnotes.
Inthiscaseitisthewrittenscenariocontainingpatientdata.
AFFsignifiesavailablesystemaffordancesandsuggestspossiblenextactions.
Althoughtheseareexternalrepresentations,thecomplexityofthescreen(40buttonswithtextuallabels)precludesthepossibilityforquickperceptualjudgmentsfor"less-than-expert"users.
MEDandSPECareinternal(I)andexternal(E)instancesofbiomedicalknowledge,asdescribedearlier.
Thewrittenscenarioconstitutedanexternalreferenceresourceofpatientfindingsanddatainthisstate,anddecisionsabouttheinclusionofordersinthesubsetweremostlysupportedbygeneralandpatient-specificknowledgeofthephysician.
CSKisaconceptualsystemknowledgeresource.
Thereareaboutasmanyinstancesofconceptualsystemknowledgeactiveduringthisstateasthereareinstancesofbiomedicalknowledge.
Theuser'sattentionneedstobedividedbetweentreatmentplanningandmanagingsystemoperations(e.
g.
,searchingforthenextorder).
HSIandHSEareinternalizedandexternalizedhistoryresources.
Althoughselectedordersareclearlymarked,thereareatotalof36orderspresentedtotheuser,eachcontainingsome14textualitemsin3linesoftext.
Theuserneedstoscrollthroughthreescreenstobrowseallavailableorders,withoutthepossibilityofasingleviewoftheselectedsubset.
PLIisaninternalizedactionplanthatreferstothesequenceofactionsthattheuserwillneedtoexecutetoaccomplishthegoalandadvancetothenextstate.
Thenextstepofactivatingtheselectedordersneedstoberecalledfrommemory.
Thisframe-basedtemplatewasusedtodescribeeachstate,withadditionalabstractinformationstructuresusedasnecessary.
Forexample,AEIorinternalaction-effectrelationsarepredicatedontheuser'sconceptualmodelofthesystemandspecificknowledgeofactionconsequences(i.
e,activatingordersbyclickingabutton).
AEE,theirexternalrepresentationssuchasexplicitlabelsorentriesinmanualsaffordtheuseranadditionalandexplicitsemanticmappingofactiontoconsequence.
Tocompletethetaskwithoptimalefficiencyandaccuracy,auserneedstonavigatethrough12systemstates.
Manyofthesestatesmakeconsiderabledemandsonusers'internalresources,inparticularonconceptualmodelsofthesystem.
SummaryresultsoftheanalysisarepresentedinTable1.
Internalandexternalresourcesaresubcategorizedaspatientandsystemcentered,dependingonwhichaspectoftheorderingtasktheysupport.
Thereismorethantwicethenumberofinternalresources(44to17)requiredforsystemoperationthenthereareforpatient-centeredclinicalreasoning.
Thisunfavorableratioindicatesthatusersmustdirectattentionawayfromtheclinicaltask.
Asimilarlyadverse2to1ratio(61to27)characterizestheinternal/externaldistributionofallavailableresources.
Awell-designedsystemminimizesthecognitiveoverheadofusersbyprovidingmoreresourcesasreflectedintheexternalrepresentationintheinterface.
Therelativedistributionofresourcesinthissystemplacesheavycognitivedemandonusersandrenderstheorderentrytaskasdifficult,especiallyintheabsenceofarobustconceptualmodel.
Fromthisanalysiswecaninferthatthesystemwillrequireanespeciallysteeplearningcurveandmayincreasethelikelihoodofusererrors.
Orderentrybyclinicians:Nosubjectproducedaflawlesssetofordersascomparedtoareferencemodel.
Theentrieswerecodedascorrect,partiallycorrect,incorrectandomitted,asshowninFigure2.
Errorsofomissionsweremadebyfivesubjects,rangingfromonetothreeitemsmissedoutofthepossibleninethatrequiredentries.
Fivesubjectsenteredseveralincorrectentries.
Theseerrorsmayhaveresultedindelaysorextrarequestsforclarificationbytheorderrecipient.
Twosubjectsrecordederroneousallergyinformationwithpotentiallyseriousmedicalconsequences("NKDA"insteadofthedocumentediodinesensitivity).
Thiserrorseemedtobetheresultofanoversightandprobablynotattributabletointerfacecomplexity.
Thenumberofbothtypesoferrors(omissionandcommission)persubjectrangedfromonetofive.
Thesystemwalkthroughidentifiedparticularstatesinwhichagivenconfigurationofresourceswerelikelytoposeproblemsfortheusers.
Thiswasevidencedbyusers'actionsanderrorpatterns.
Forexample,asubjectmistakenlyselectedaurologyInternalExternalPatientSystemPatientSystemGOALStatesSpecMedInstCSKHSIAEIPLISpecHSEAEEOpenchart111221Selectset421153411Selectsubset14241217Changedefaults4221421227Addorder1161211Reviewandsign1112111Total129622046147173TotalPatient/System1744720TotalInternal/External6127Resources:Patient–Patient-centeredreasoningsupport,System-System-centeredreasoningsupportKnowledge:Spec–Patient-specific,Med–Generalmedical,Inst–Institution-specificIncorrectTable1.
NumberofResourcesActiveDuringanOrderingTask02468101234567SubjectNumberofEnteredOrdersCorrectPartlyCorrectOmittedFigure2.
Accuracyandcompletenessoforderspost-operativetransferorderset.
Hesubsequentlyneededtorecognizeandeliminateinapplicableordersandtoreconstructtheadmissionsetbyenteringindividualorders.
Thiswasatime-consumingandlaboriousprocess.
Thiserrorwasprecipitatedbyalackofclarityinthepresentationofordersetsinthepicklist.
Theclinicianneededtorelyonspecificconceptualsystemknowledgetosuccessfullynavigatethehierarchicalmenuofordersets.
Inaddition,thesystemdoesnotaffordeasybacktrackingorerrorrecovery.
Thewalkthroughanalysisofthisparticularstateenabledustoexplainwhyinappropriateselectionsmayeventuateandthaterrorrecoverywouldbedifficultgiventhelimitedexternalnavigationresources.
CONCLUSIONProviderorderentryisaninherentlycomplexprocess,buttheconfigurationofsystemresourcescaneitherexacerbateorminimizeitscomplexity.
Thisresearchwaspredicatedonatwo-prongedapproachtothestudyofhumancomputerinteraction.
Thefirstcomponentinvolvedadistributedresourcestaskanalysiscarriedoutbytheteamofinvestigators.
Thesecondinvolvedusabilitytestingofcliniciansenteringclinicalordersintothesystem.
Thedistributedresourceanalysisenabledustoaccountforpatternsofuserbehavior.
Inturn,usabilitytestingallowedustorefineourintuitionsaboutthewaysinwhichconfigurationsofresourcescanfacilitateorder-entrytasks.
Thisresearchwasguidedbythebeliefthatcognitionisbestconstruedasadistributedprocessthatstretchesacrosshumansandartifacts.
Well-designedtechnologiesreducetheneedforuserstorememberlargeamountsofinformationandappropriateexternalrepresentationscanminimizethedifficultyofataskbysupportingrecognition-basedmemory.
Inourview,theresourcemodelisavaluabletoolforthestudyofcomplexmedicalinformationtechnologies.
Adistributedresourceanalysiscouldinformdesigndecisionsbymakingtaskdemandsmoretransparentandprovidingguidanceforexternalizingresourcesthatalleviatetheworkingmemoryburden.
Towardsthatend,adesignermayexaminetheratiobetweenexternalandinternalresourcesandalsodeterminehowtoreallocateuserandsystemresources.
Althoughtheapplicationofthismodeltoexplainuserperformanceisstillatanearlystage,itwasusefulinaccountingforcertainpatternsoferrorsandinteractivestrategies.
Theredistributionandreconfigurationofresourcesmaysuggestguidingprinciplesanddesignsolutionsinthedevelopmentofcomplexinteractivesystems.
REFERENCES1.
Woods,DDandRoth,EM.
Cognitiveengineering:Humanproblemsolvingwithtools.
HumanFactors198830(4):415-430.
2.
Roth,E,Patterson,E,Mumaw,R.
Cogntitiveengineering:Issuesinuser-centeredsystemdesign.
In:Marciniak,J,editor.
Encyclopediaofsoftwareengineering.
NewYork:Wiley;2002.
3.
Reason,JT.
Humanerror.
Cambridge,England:CambridgeUniversityPress;1990.
4.
Bates,DW.
,Leape,LL,Cullen,DJ,Laird,N,Petersen,LA,Teich,JM,Burdick,Eetal.
Effectofcomputerizedphysicianorderentryandateaminterventiononpreventionofseriousmedicationerrors.
JAMA1998280:1311-1316.
5.
Ash,JS,Gorman,PN.
,andHersh,WR.
PhysicianorderentryinU.
S.
hospitals.
ProcAMIAAnnuSymp1998:235-239.
6.
Bates,DW,Cohen,M,Leape,LL,Overhage,JM,Shabot,M,andSheridan,T.
ReducingtheFrequencyofErrorsinMedicineUsingInforma-tionTechnology.
JAMIA20018(4):299-308.
7.
Patel,V,Kushniruk,A,Yang,S,andYale,JF.
Impactofacomputer-basedpatientrecordsys-temondatacollection,knowledgeorganization,andreasoning.
JAMIA20007(6):569-585.
8.
Lin,L,Isla,R,Doniz,K,Harkness,H,Vicente,K,andDoyle,D.
Applyinghumanfactorstothedesignofmedicalequipment:patient-controlledanalgesia.
JCMon&Comp199814(4):253-263.
9.
Norman,DACognitiveengineering.
In:Norman,DAandDraper,SW,editors.
Usercenteredsystemdesign:Newperspectivesonhuman-computerinteraction.
Hillsdale,NJ:LawrenceErlbaumAssociates;1986.
p.
31-61.
10.
Polson,PG,Lewis,C,Rieman,J,andWharton,C.
Cognitivewalkthroughs:Amethodfortheory-basedevaluationofuserinterfaces.
IntJofMan-MachineStudies199236(5):741-773.
11.
Wright,PC,Fields,RE,andHarrison,MD.
Analyzinghuman-computerinter-actionasdistributedcognition:Theresourcesmodel.
Human-ComputerInt200015(1):1-41.
12.
Zhang,J,Patel,VL,Johnson,KA.
,andMalin,J.
Designinghuman-centereddistributedinformat-ionsystems.
IEEEIntelSystems200217:42-47.
ACKNOWLEDGEMENTSJanHorskyissupportedbyNationalLibraryofMedicineMedicalInformaticsTrainingGrantLM07079-09.
WethankMichaelI.
Oppenheim,MDandRandolphBarrows,MDfortheirhelpwithmedicalexpertiseandtoallsubjectsfortheirtime.