Improving Learning in Business Simulations with an Agent

©CopyrightJASSS
MárciaBaptista,CarlosRoqueMartinho,FranciscoLima,PedroA.SantosandHelmutPrendinger(2014)
ImprovingLearninginBusinessSimulationswithanAgent-BasedApproach
JournalofArtificialSocietiesandSocialSimulation 17(3)7
<http://jasss.soc.surrey.ac.uk/17/3/7.html>
Received:17-Apr-2013Accepted:23-Feb-2014Published:30-Jun-2014
Abstract
Artificialsocietysimulationsmayprovideunprecedentedinsightintotheintricatedynamicsofeconomicmarkets.Suchaninsightmayhelpsolvethewell-knownblack-boxdilemmaofbusinesssimulations,where
designersprefermodelconcealmentovermodeltransparency.Thecorecontributionofthisworkisanagent-basedbusinesssimulationthatmodelsthemarketplaceasanartificialsocietyofconsumers.Inthe
simulation,usersassumetheroleofastoreownerplayingagainstanartificialintelligencecompetitor.Thesimulationcanbeaccessedviaagraphicaluserinterfacethatanimatesthedecisionbehaviorof
consumers.Consumersaremodeledasagentswithconcretebeliefs,intentionsanddesiresthatacttomaximizetheirutilityandaccomplishtheirpurchaseplans.Weclaimthatunliketheclassicalequation-based
approach,thevisualizationofmarketdynamicsfacilitatedbyouragent-basedapproachcanprovideimportantinformationtotheuser.Wehypothesizethatsuchinformationiskeytounderstandingseveral
economicconcepts.Tovalidateourhypothesis,weconductedanexperimentwith30users,wherewecomparedtheeffectsofthegraphicalanimationofthemarket.Ourresultsindicatethattheagent-based
approachhasbetterlearningoutcomesbothatthelevelofusers'subjectiveself-assessmentandatthelevelofobjectiveperformancemetricsandknowledgeacquisitiontests.Asasecondarycontribution,we
demonstratebyexamplehowsimplecodificationrulesattheleveloftheutilityfunctionsofagentsallowtheemergenceofdiversemacroeconomicbehaviorofatwo-productduopoly.
Keywords:
Agent-BasedModeling,BusinessSimulation,ConsumerBehavior,LearningProcesses
Introduction
1.1
Businesssimulationswerecreatedtoserveasvirtualenvironmentswherethelearningofconcepts,theoriesandpracticesfromeconomicsandmanagementcouldoccurinasystematicexperimentalway.These
simulationsusuallyconsistofbusinessgameswheretheparticipanttakestheroleofafirmmanagerinacompetitivemarket,mostoftenagainstotherhumanplayers(Summers2004).
1.2
Sincethe1980smuchdebatehasbeenheldontheblack-boxnatureoftraditionalbusinesssimulations.Inblack-boxsimulations,theinternalstructureofthemodelsupportingthesimulationisnotdisclosedtothe
participants.Participantsoperatebytrialanderrorandbasetheirdecisionsonthestaticresultsthatconventionalequation-basedmodelsprovide.
1.3
AuthorssuchasMachuca(2000)andGrößleretal.(2000)proposethatbusinesssimulationsshouldbemadetransparenttopromotemoreeffectivelearning.Theyarguethatusersshouldhaveaccessto
informationregardingtheunderlyingmodelstructureandbeabletorelatethisinformationtotheobservedresults.Theauthorsclaimthattheadditionalprovisionofinformationmayimproveusers'performanceand
learningacquisition.
1.4
Despitetheongoingdiscussion,empiricalstudiessuchasKopainskyetal.(2011)showonlyaweaktomoderaterelationbetweensimulationtransparencyandlearningofusers.Thiscanbeexplainedbythe
limitedexplanatorycapabilitiesoftheequation-basedmodelsusedinthestudies.
1.5
Aparadigmnotyetextensivelyappliedtobusinesssimulationsistheagent-basedapproach,whoseexplanatorypotentialhasbeensubjecttomuchdebate(Grüne-Yanoff2009;Waldherr&Wijermans2013).With
theagent-basedapproach,theunderlyingoperationofthesimulationcanbedisclosedwithoutdiminishingitsstrategicvalue,asthistechniquecanprovideanaturaldescriptionofaneconomicenvironmentand
captureitsmacrobehaviorasanemergentphenomena.
1.6
Inthiswork,wepresentanagent-basedbusinesssimulationandconductanempiricalstudyontheimpactoftheinformationprovidedbytheagent-basedapproach.WestartinSection2withareviewofdemand
modelsinbusinesssimulations.Section3and4detailouragent-basedsimulationandunderlyingmodelofconsumerbehavior.InSection5,wedescribeourempiricalstudyanddemonstratethatinformation
exclusivelyprovidedbytheagent-basedapproachcanpromoteeffectivelearning.InSection6weconcludewithadiscussionofourresultsandareflectiononourwork.
Background
2.1
Theproblemofmodelingthemarketplaceisperhapsthemostfundamentalchallengeofabusinesssimulation.Infact,thealgorithmsresponsibleforcalculatingmarketandfirmleveldemandhavebeen
consideredthemostcomplexanddemandingalgorithmsofabusinesssimulation(Goosenetal.2001).Suchalgorithmsarecoreasfirms'abilitytocapturemarketshareformstheessenceofabusiness
simulation.
2.2
SinceGoosen(1981)firstproposedaformalalgorithmforbusinesssimulations,severaldemandmodelshavebeenpresented,withmucheffortbeingdevotedtoenhancetheirflexibilityandvalidity.Thesemodels
typicallyfallintothefollowingfourmajorcategories,whichwedescribeindetailinthesectionsbelow:
Equation-based:mathematicalfunctionsmodelmarketdemandandfirmdemand.
Interpolation-based:aninterpolationmethodderivesthegraphicsofmarketandfirmdemandfunctions.
Statistical-based:purchaseprobabilitydistributionsmeasuretheproportionofconsumerswhoconsumeagivenproduct.
Agent-based:thecomplexityofthemarketplaceiscapturedusingabottom-upapproachthatmodelsthebehavioralrulesofeachconsumer.Inthisway,themacrobehaviorofthesystememergesfromthe
interactionbetweenconsumersandfirms.
Equation-basedmodels
2.3
Anequation-baseddemandmodelusuallyconsistsoftwofunctions:
1. Afunctionofmarketdemand(Q)calculatedfromtheaveragevaluesofdemanddeterminantssuchasprice(P),advertisingandpromotionvariables(M)andproductqualityvariables(R):
Q =f(P,M,R) qi=wiQ wi=g( Pi,Mi,Ri) 2. Afunctionoffirmleveldemand( qi)usedtocalculatetheweightofeachfirm(wi)whenallocatingmarketshare:
Sincethefirstformalequation-basedmodel,severalmodelswhichallowthesimulationofdifferenteconomicphenomenahavebeenproposed.AmongthemostimportantcontributionsistheworkofTeach(1990),
withitsgravityflowmodel,laterextendedbyGold&Pray(1997)andCannonetal.(2012).
Interpolationapproach
2.4
Goosen&Kusel(1993)recognizedthattofindgeneralflexiblemathematicalfunctionstomodelmarketandfirmdemandwasanintricatetask.Hence,theauthorsproposedtheinterpolationapproachasamethod
ofimplementingself-designedfunctions.Themethodisagraphicalapproachwherethesimulationdesignerhastosketchthefunctiondesiredbetweentwovariablessuchaspriceandquantity,chooseanumber
ofimportantpoints(inflection,minimumandmaximumpoints),programaninterpolationfunctionandfinally,usingthatfunction,generatetheremainingpoints.
2.5
Theinterpolationmethodwascriticized(Gold1993)foritsdifficultiestomodelinteractiveeffectsbetweendemandvariablesandforbeingrathertime-consuming.Thismightexplainwhyitsadoptionneverbecame
aswidespreadastheequation-basedapproach.
Statisticalapproach
2.6
Carvalho(1995)proposedanotherapproachtomodelmarketdemand.Theauthorconsideredthatpreviousworkdidnotexplicitlymodelthecrucialelementofamarketplace:theconsumer.Hence,Carvalho
proposedamodelbasedontheso-called'equimarginalprinciple'usingagammaprobabilitydistributiontosimulatethepreferencesofconsumers.
2.7
Carvalho'smodelreceivednegativecriticsregardingtheinstabilityofitsdemandfunction,thelackoftheself-claimedindependenceofconsumersandsuppliers,andthedifficultiestocontrolthedistribution
http://jasss.soc.surrey.ac.uk/17/3/7.html
1
22/10/2015
parametervaluesdefiningthemarketplace(Gold&Pray1995).Similarlytotheinterpolationapproach,thistypeofmodelneverbecameaspopularasequation-basedmodels.
Agent-basedapproach
2.8
Agent-basedmodelingcanbedefinedasanapproachtosimulatecomplexadaptivesystemscomprisedofseveralautonomousandinteractingentities(Bandinietal.2009).Parunaketal.(1998)providean
extensivediscussionofthesimilaritiesanddifferencesbetweentheagent-basedapproachandtraditionalequation-basedmodeling.
2.9
Severalauthors(Yilmazetal.2006;Araietal.2005)havearguedinfavorofusingtheagent-basedapproachtogeneratebehaviorinbusinesssimulations.IndustrygamessuchasSimCity2013havestartedto
adoptmulti-agentsystemsastheirmainapproachtoeconomicenvironmentmodeling.Inacademiaalso,anumberofagent-basedgameshavebeenpresented,suchastheworkofUmedaetal.(2009).
2.10 Extensiveworkhasbeendoneinthegeneralareaofagent-basedmodelsofconsumerbehavior.AmongthemostimportantcontributionsistheworkofSaidetal.(2002)andDelreetal.(2010).Thesemodels
simulatethemarketplaceasamulti-agentsystemwhereconsumersareagentsthatperformcomplexinteractionssuchasimitationandsocialcomparison.Suchmodelshaveaconsiderabledegreeofcomplexity
sincetheyaredevisedtoexplainparticularmarketphenomenasuchaslock-inandinnovationdiffusion.
BusinessSimulation
Figure1:Interfaceofsimulationincludingthe(I.A)decisionspanel,(I.B)agent-basedpaneland(I.C)panelwithinformationoffirms.
3.1
Wedevelopedanagent-basedbusinesssimulationwheretheuserplaysagainstanartificialintelligence(AI)competitorinatwo-productduopoly.Thesimulationisaround-basedgame,asitistypicalinmost
businesssimulations(Kenworthy&Wong2005).Thesimulationgameconsistsofthefollowingelements:
Firmrepresentsthehumanplayer.Theuseractsasafirmandmanagesabakerystorewhereheorshecanselltwodistinctproducts(bread,cookies)tothefinal(artificial)consumer.
CompetitorFirmrepresentstheAIcompetitortotheuser.TheAIcompetitormanagesabakerystorethatcompeteswiththehumanplayertoselltwoproducts.Thefinalconsumerisagnosticregardingthetypeof
seller(humanplayerorartificialplayer).
Consumersrepresenttheautonomous(artificial)agentsofthesimulation.Ineachround,theconsumerhasagivenbudget(I)tospendonproductsfromthebakeries.Thegameconsidersapopulationof100
consumerswithanincomedistributionaccordingtoaPoissondistribution(λ =300).Todecideonwhichproductstopurchase,theconsumerusesinformationabouttheproductsforsaleandmaximizesanutility
function.Consumersattempttoaccomplishtheirintentionssequentiallyinarandomorder.
Productsrepresentthegoodssoldbythefirmstoconsumers.Inthesimulation,weconsidertwoproducts:breadandcookies.Weassumethatmarginalcostsareconstantandequalforbothfirms(USD10).The
twoproductshaveequalmarginalutilitiesastheutilityfunctionofconsumersisthefollowing:
u( x 1,x 2)=
(
) j+
)j (1)
x 1p1+x 2p2≤I (2)
(
wherex 1,x 2representthequantitiesofproducts1(bread),2(cookies)thattheconsumercanpurchaseatpricesp1,p2withabudgetI.
3.2
Fig.1depictsthesimulationinterfaceanditsthreefundamentalelements:(I.A)decisionspanel,(I.B)agent-basedsimulationpaneland(I.C)panelwithinformationaboutthefirms.
3.3
Ineachround,thehumanplayerhastodecideontwoissuesusingthedecisionspanelI.A:(1)quantityforsaleand(2)sellingpriceofproducts.Aftertheusersubmitshis/herdecisionsinpanelI.A,theAI
competitorsubmitsitsdecisions.ThehumanplayerandtheAIcompetitordecidesimultaneouslywithoutknowingeachotherdecisionsforthatround.
3.4
ThereasoningalgorithmoftheAIcompetitorisbasedonCournot'sadjustmentmodel,firstproposedinthecontextofaduopolymodelbyCournot(1838).TheAIcompetitorassumesthatthehumanplayer(1)will
setthepricesofproductsequaltothemarketpricesofthelastmarketquantitiessoldand(2)setthesamequantitiesforsaleasinthepreviousround.TheAIcompetitorsetsthepriceofproductsequaltotheprices
itpredictsthehumanplayerisgoingtosetandchoosesquantitiesforsaleasbestresponsestothequantitiesitassumesthehumanplayerisgoingtochose.
3.5
Theperformanceofthehumanplayer(andAIcompetitor)isevaluatedaccordingtotheircumulativeprofits.Theuserwinsthegameifs/heeither(i)hasmorecumulativeprofitsthanhisorherAIcompetitoratthe
endofthegame(determinedbyapre-setnumberturns)or(ii)isabletobankrupthisorherAIcompetitor.BothfirmsstartwithaninitialcapitalofUSD10,000.
http://jasss.soc.surrey.ac.uk/17/3/7.html
2
22/10/2015
Figure2:InterfaceofCAPSTONEsimulation(http://www.capsim.com/business-simulations/products/capstone.html).
3.6
Notethatexistingequation-basedsimulationsusuallydonotpresentresultstoparticipantswithreal-timeanimations,astheycanonlydisplaythefinaloutputofademandequationattheendofeachround.Hence,
resultsarepresentedstaticallythroughtheuseofspreadsheets.Fig.2showsanexampleofatraditionalequation-basedbusinesssimulationinterface.
(a)Startofround
(b)Middleofround
Figure3:Participantscananalyzeperformanceindicatorschanginginvirtualtimeasthe
simulationruns(PanelI.C-Informationoffirms).
3.7
Incontrast,inoursimulation,theresultsofaroundarenottheoutcomeofanequation.Instead,resultsemergefromthecomplexinteractionofseveralconsumeragentsduringaperiodoftime.Consequently,in
eachround,participantscananalyzeinvirtualtimethechangeofseveralindicatorsoffirms'performance,suchasquantitiessold,stockandprofit(seeFig.3).
3.8
Importantly,ouragent-basedapproachcanshowhowperformanceindicatorschangeinresponsetothebehaviorofconsumeragents.Theintentionofaconsumertopurchaseaproductisshownbyananimation
whereaniconrepresentingacustomermovesfromahouseicontoastoreicon(seeFig.4).Ifconsumerschangetheirpurchaseplans,theymayvisitthesamebakeryseveraltimesduringthesameround.
http://jasss.soc.surrey.ac.uk/17/3/7.html
3
22/10/2015
(a)Consumerstrytoaccomplishtheirpurchaseplans
(b)Consumersreturnhomeandreformulatetheirpurchaseplans
Figure4:Theagent-basedsimulationallowsanalysingthebehaviorofconsumersinvirtualtime.
3.9
Whenelaboratingtheirpurchaseplans,consumersareawareofproductpricesbutnotofproductavailabilityatstores.Accordingly,consumersmayfailtoaccomplishtheirintentionsifaproductisnotavailableat
aparticularstore.Insuchcase,theymaygotootherstoresorgiveupontheirpurchaseintentions.Consumersdisplaythelevelofaccomplishmentoftheirintentionsbyagraphicalchangeoftheirstate.Satisfied
consumers,i.e.,consumerswhoareabletopurchasealltheproductstheyintendedtogetatagivenstore,arerepresentedwithgreencolorwhiledissatisfiedconsumersarerepresentedwithredcolor(see
Fig.4b).
ConsumerModel
4.1
Whileseveralmodelsofconsumerbehaviorhavebeenpresented(Delreetal.2010;Leeetal.2013;Sengupta&Glavin2013),theseusuallyarenotsuitedforbusinesssimulations.Theyaimtoexplainparticular
phenomenaandarerathercomplex.Teach&Murff(2008)arguethatthetrendtodevelopmoreandmorecomplexmodelsmayinfacthindertheeducationaleffectivenessofbusinesssimulations.Amongother
arguments,theauthorschallengethepre-establishedviewthattheamountofcognitiveprocessingofusersrelatestocomplexityofsimulationmodel.Weagreedtothisassessmentandthuscreatedasimple
agent-basedmodelforourbusinesssimulation.
Asimpleagent-basedmodel
4.2
Inoursimulation,consumerbehaviorisbasedonthedeliberativeBeliefs,DesiresandIntentions(BDI)architecture(Bratman1987),perhapsthebestknownandmoststudiedmodelofpracticalreasoningagents.
Inourarchitecture,theinternalstateofthe(artificial)consumerconsistsofthefollowingelementsillustratedinFig.5:
Beliefsrepresenttheinformationtheconsumerhasaboutthecurrentstateoftheworld,i.e.,themarketplace,anditsowninternalstate.Twotypesofbeliefsareconsidered:
Beliefintheavailabilityofaparticularproductatastore.
Beliefthatgivenitsavailablebudget(I),theconsumerisabletopurchaseproducts.
Desiresrepresentthegoalsoftheconsumer.Weconsidertheexistenceofthedesiretoconsumeproducts.
Intentionsrepresentthepossiblecoursesofactionoftheconsumer.Purchaseintentionsaregeneratedduringthereasoningprocessoftheconsumer,eachforapossiblebundleofproductsgiventheagent's
budgetrestriction.Apurchaseintentionischaracterizedbyatuple(x 11,x 12,x 21,x 22)with x ijrepresentingthequantityofproductifromfirmjdesiredbytheagent.
Figure5:TherationalityelementsoftheBDIarchitectureofaconsumer.
4.3
Thereasoningprocessofaconsumerconsistsinthefollowingsteps:
Step1Theconsumerverifiesthreeconditions:(1)ifthedesiretoconsumegoodsisacurrentgoal,(2)theagentholdsthebeliefinitseconomiccapabilityand(3)atleastonebeliefintheavailabilityofproductsat
stores.Ifallthesethreeconditionshold,thereasoningprocessproceedstoStep2.Otherwise,thereasoningprocessends.
Step2Foreachaffordablecombinationofgoodstheconsumergeneratesapurchaseintention.Thegenerationofintentionsisgroundedinthebeliefsregardingtheavailabilityofaproductonthemarketandits
economiccapability.For illustrationpurposes,considerascenariowherethehumanplayer(Player1)setsprices p1=200andp2=200andtheAIcompetitor(Player2)setsp1=400andp2=400.Aconsumer
withabudgetof400believingthatstoreshaveallproductsavailablegeneratesfiveconsumptionintentionsatthisstep.Thegeneratedintentionscorrespondtoallpossiblecombinationsofproductsitcan
affordgivenitscurrentbudget:(1,1,0,0),(2,0,0,0),(0,2,0,0),(0,0,1,0)and(0,0,0,1).
Step3Theconsumerselectsapreferredintentionfromthelistofpurchaseintentionsaccordingtothemaximizationofanutilityfunction.Incaseofidenticalutilityvalues,arandomfactorisused.Utilityfunction(1)
isusedtosimulatethepreferencesoftheconsumer.
Step4Theconsumerattemptstoaccomplishitspreferredpurchaseintentionasaneffectivepurchaseplan.Atthispoint,allthesimulatedconsumersattempttopurchasetheirdesiredproductsinsequenceina
randomassortment.Theaccomplishmentofanagent'spurchaseplanthusdependsonitsindirectinteractionwithitspeers.
http://jasss.soc.surrey.ac.uk/17/3/7.html
4
22/10/2015
Step5Theconsumerupdatesitsbeliefsanddesiresasfollows:
Economiccapabilitybelief:accordingtotheamountspentonsuccessfulpurchases,theagentupdatesitsavailablebudgetandthebeliefinitseconomiccapabilityifitstillhasmoneylefttobuy
anotherunitofanavailableproduct
Productavailabilitybelief:theagentabandonsitsbeliefintheavailabilityofaproductatastoreifitsattempttopurchaseagivenproductatastorewasunsuccessful
Desiretoconsume:ifthelistofintentionsgeneratedinStep2isemptytheagentdropsitsdesiretoconsume
Afterthisrevision,thereasoningprocessoftheagentreturnstoStep1forareformulationofintentions.
4.4
Whileourbusinesssimulationsimulatesatwo-productduopolywithsingularmarketdynamics,ourmodelofconsumerbehaviorwasdesignedtobeabletoaccommodatethesimulationofdiversemarket
structures.TheuseofdistinctutilityfunctionsinStep3canleadtoamyriadofemergenteconomicphenomena.Onthisdiversityliesmuchofthepotentialoftheagent-basedapproachasopposedtotheequationbasedapproach.Forillustrativepurpose,weprovidespecificexamplesofutilityfunctionsthatcancoverthemajoreconomicphenomenaaddressedinintroductorymicroeconomicstextbookssuchasFrank&
Glass(1997).
Simulationofindependentgoods
4.5
Independentgoodscanbesimulatedinourmodelbyintroducingdifferentbudgetrestrictionsforeachgood.Thefollowingequationsillustrateamarketoftwoindependentgoods.
U( x 1,x 2)=
b1( j)+
b2( j) (3)
x 2p2=I2,x 1p1=I1 (4)
I1+I2≤I (5)
Simulationofeffectsofchangesinpriceandincome
Itispossibletosimulateordinaryaswellasinferiorandnormalgoodsusingourmodelandthefollowingconstruction.Thisconstructionalsoimplementstheconceptofdiminishingmarginalutilitybytheuseofa
decreasingpreferencefunction( bi( j))(7).WiththissetofequationswecanalsosimulateVeblengoodsbyintroducingthepriceofthegoodinthedefinitionofthepreferencefunction(bi( j,pi)).Astheprice
increases,theconsumercanperceivethisincreaseasanevidenceofhighquality.Asimilarpreferencefunction( bi( j,ai)),varyingwiththeexposureoftheconsumertoadvertising(ai),canmodeladvertising
effects.
U( x 1,...,x n)=
b1( j)+...+
bn( j) (6)
bi( j+1)≤bi( j) (7)
x 1p1+x 2p2+...+x npn≤I (8)
Simulationofsubstituteandcomplementarygoods
Itispossibletosimulatesubstitutioneffectsinourmodel.Forinstance,Equation9allowssimulatingperfectsubstitutegoodswhileequation10allowssimulatingsubstitutegoodswithdifferentmarginalutilityrates.
U( x 1,x 2)=
U( x 1,x 2)=
b1( j) (9)
b2( j) (10)
x 1p1+x 2p2≤I (11)
b1( j)+
Complementarygoodscanbesimulatedbyintroducingavirtualcomplementaryproducttotheconsumer'sutilityfunctionwithr 1andr 2representingtheratioofproducts1and2whichcomposethe''virtual''
complementaryproduct12:
U( x 1,x 2)=
b1( j)+
b2( j)+
b12( j) (12)
x 1p1+x 2p2≤I (13)
Experiment
5.1
Weconductedanexperimentwithuserstoinvestigatehowtheinformationprovidedbytheagent-basedapproachinfluenceslearningwithbusinesssimulations.Toassesslearningweusedtheclassicalfour-level
frameworkofKirkpatrick(1959).ToapplythisframeworkwefollowedtheguidelinesofSchumannetal.(2001)onhowtoadaptKirkpatrick'smodeltobusinesssimulations.
5.2
WedecidedtoassesstwoofthefourlevelsofKirkpatrick'smodel:reactionandlearning.Thereactionlevelassesseshowthelearnerperceivesthesimulationexperience.Atthislevel,wewereinterestedinhow
humanplayersperceivetheirlearning.Thelearninglevelevaluatestheextenttowhichparticipantschangetheirattitudes,improvetheirknowledgeorincreasetheirskillwithinthecontextofasimulationsession.At
thislevelwewereinterestedintheimprovementofperformanceinplayingthebusinesssimulationandtheacquisitionofbasicconceptsrelatedwithbehavioroffirmsandmarkets.Basedonthesetwolevelswe
formulatedthreemainhypotheses:
ReactionHypothesis(ReactionLevel):Humanplayerssubjectivelyperceivethattheylearnmorewhenagent-basedinformationispresent.
EconomicConceptAcquisitionHypothesis(LearningLevel):Humanplayers'performanceontestsabouteconomicconceptsisbetterwhenagent-basedinformationispresent.
In-SimulationPerformanceHypothesis(LearningLevel):Humanplayers'performanceinthesimulationisbetterwhenagent-basedinformationispresent.
5.3
Beforeproceeding,webrieflyexplainwhywedidnotconsidertheothertwolevelsofKirckpatrick'sframework.Thebehaviorlevelrequiresassessinghowparticipantsareabletoapplythenewlyacquired
knowledgeandskillstonewsettings.Giventhedifficultiesassociatedwithsuchassessment,thislevelwasnotconsidered.Wealsodidnotevaluatetheresultslevelwhichconsistsoftheevaluationofthereturn
oninvestmentofthelearningexperience.Thislevelwasnotapplicableinthecontextofourexperiment.
Method
SubjectsandDesign
5.4
ThirtysubjectsfromtheNationalInstituteofInformaticsparticipatedinthestudy.Therewere6femalesand24malesbetween22and39yearsofage(average27years).Allsubjectswerestudentsorstaffatthe
institute,recruitedthroughflyers.25subjects(83%)hadpreviousexperiencewithcomputersimulations,7subjects(23%)hadpreviousexperiencewithbusinesssimulationsand6subjects(20%)wereacquainted
withconceptsfromeconomics.SubjectswerepaidanequivalentofUSD10forparticipation.Toprovideafurtherincentiveforgoodperformance,anextraamount,equivalenttoUSD5,wasawardedtoeach
winningparticipant.
http://jasss.soc.surrey.ac.uk/17/3/7.html
5
22/10/2015
Figure6:Interfaceofsimulationwithouttheinformationtoolsprovidedbytheagent-basedapproach.
5.5
Wepreparedtwoconditionsasindependentvariables:
Richinformation(RI)condition:heresubjectsplaythesimulationwiththecompletesetofinformationtoolsprovidedbytheagent-basedapproach.Accordingly,subjectsusethegraphicalinterfaceshownin
Fig.1.
Sparseinformation(SI)condition:subjectsplaythesimulationwithoutthesetofinformationtoolsprovidedbytheagent-basedapproach.SubjectsusethegraphicalinterfaceshowninFig.6.
5.6
Thedependentvariableofourstudywasthelearningoutcomeoftheparticipants.WeassessedthisvariableusingthereactionandlearninglevelsofKirckpatrick'sframeworkwiththefollowingmetrics:reaction
wasmeasuredbyanalyzingsubjects'answerstoapost-experimentquestionnaire;learningwasmeasuredbyusingtwoapproaches:
Pre-experimenttestandpost-experimenttest:Thesetestsaimedtomeasuretheknowledgeofparticipantsonarangeofeconomictopicsrelatedwiththebehavioroffirmsandmarkets.
Behaviorlogging:Thebehaviorofparticipantswasloggedduringthesimulationsessions.Toassesssubjects'performanceweusedthefollowingthreemetrics:(1)numberofroundstowingame,(2)profit
perround,and(3)cumulativeprofits.
5.7
Inthestudy,abetween-subjectdesignwasusedtocomparetheRIandSIconditions.Subjectswereassignedtotheconditionsrandomly.Duringtheexperiment,subjectswereseatedinfrontofalaptopcomputer
andusedamousetointeractwiththesimulation.
Materials
5.8
Inthissection,webrieflyreviewthematerialsusedintheexperiment.
Instructions
Theinstructionsprovidedtothesubjectsconsistedof(i)adescriptionofthepurposeoftheexperiment,(ii)instructionsonhowtoplaythegame,and(iii)theequationofmarketdemand.Participantsweretoldthat
theexperimentwasintendedtoinvestigatepeople'sdecisionprocessinabusinesssimulation.
Tests
Thepre-experimenttestandpost-experimenttestconsistedofthesamesetoffivemultiple-choicequestions.Eachquestionwasdevisedtoassessthesubject'sknowledgeaboutaparticulareconomicconcept.
Table1presentsthelistofquestionsused.
Questionnaire
Thepost-questionnaireisconcernedwiththereactionlevelofthesubjects.Weuseda7-pointLikertscale.Table2presentsthetwostatementsthataimedtoinvestigatesubjects'subjectiveperceptionoftheir
knowledgeacquisition.
Table1:Questionsofknowledgeacquisitiontests.
EconomicConcept
ID
Question
Lawofdemand
E-Q1
Ifafirmincreasesthepriceofaproduct(witheverythingelseconstant)whathappens?
E-Q2
ConsideramarketwithtwofirmsAandBcompetingtosellthesameproduct.IffirmAincreasesitsprice(witheverythingelseconstant)what
happens?
E-Q3
ConsideramarketwithtwofirmsAandBcompetingtosellthesameproduct.IffirmAdecreasesitsprice(witheverythingelseconstant)what
happens?
E-Q4
Consideramarketwithtwofirmscompetingtosellthesameproduct.Thegoalofeachfirmistoincreaseitsworkingcapital.Eachfirmcandecide
onthepriceandquantityforsaleofitsproduct.Inthisscenario,whatisthebeststrategyofafirm?
E-Q5
ConsideramarketwithtwofirmscompetingtoselltwoproductsAandB.Thegoalofeachfirmistoincreaseitsworkingcapital.Eachfirmcan
decideonthepriceandquantityforsaleofitsproduct.Assumeeachfirmhasalimitedwarehousecapacityandafixedcostperunit.Inthis
scenario,whatdowepredictthathappensaftersometime?
SubstituteProducts
Behavioroffirms
MarketBehavior
Table2:Statementsofreactionquestionnaire.
ID
Statement
R-Q1
Playingthesimulationincreasedmyknowledgeofthedynamicsofmarkets.
R-Q2
Playingthesimulationincreasedmyknowledgeofthestrategicbehavioroffirms.
Procedure
5.9
Thestudywasconductedbyanexperimenterandanon-technicalassistant,andwasdividedintofivemainparts:
1.
2.
3.
4.
5.
Welcome(5min):subjectsarewelcomedtothestudyandreadtheinstructions
Profilingandpre-experimenttest(10min):subjectsfillinaprofilingformandanswertheknowledgeacquisitiontestshowninTable1
Training(5min):subjectsplaythesimulationfortworoundsasatrainingsession
Simulation(40min):subjectsplaythebusinesssimulationfor15rounds
Reactionquestionnaireandpost-experimenttest(10min):subjectsanswerthequestionnaireshowninTable2andtheknowledgeacquisitiontestshowninTable1
http://jasss.soc.surrey.ac.uk/17/3/7.html
6
22/10/2015
5.10 Finally,thesubjectsreceivethereward.Eachexperimentlastedupto1hourand10minutes.
Results
ReactionHypothesis
5.11 InthissectionweanalyzetheresultsofthereactionquestionnaireinTable2tocomparethesubjectiveperceptionofsubjectsonhowmuchtheylearnedbetweentheRIconditionandtheSIcondition.Resultsfrom
thethequestionnairessuggestthatsubjectsintheRIconditionfeltmorestronglytheylearnedaboutmarkets(Mean=5.71,STD=0.73)andbehavioroffirms(Mean=5.93,STD=0.73)thanstudentsintheSI
condition(Mean=4.93,STD=1.53andMean=4.60,STD=1.30).1Inparticular,Studentt-tests(unequalvariances)showedthattheagreementlevelsofsubjectsonquestionsR-Q1andR-Q2werestatistically
differentbetweentheRIandSIconditions(p<0.05).Figure8summarizesthementionedfindings.Basedonthesefindings,weclaimthattheReactionHypothesisisconfirmed.
Figure8:AgreementlevelsonquestionsR-Q1andR-Q2forRIandSIconditions.1:"stronglydisagree";2:"disagree";3:"weaklydisagree";4:"neutral";5:"weaklyagree";6:"agree";7:
"stronglyagree".
EconomicConceptAcquisitionHypothesis
5.12 Inthissectionweanalyzetheresultsofthepre-experimentandpost-experimentknowledgeteststocomparetheknowledgeacquisitionofsubjectsbetweentheRIconditionandtheSIcondition.Here,knowledge
acquisitionorimprovementmeansthatsubjects,whogavewronganswersinthepre-experimenttest,wereabletogivethecorrectanswerinthepost-experimenttest.
5.13 Resultsfromourpre-experimentandpost-experimentknowledgetests(Table1)suggestthatahighpercentageofstudentsalreadyknewtheconceptofthelawofdemand(60%)andsubstitutegoods(70%)–
eventhoughonly20%ofthesubjectsreportedhavingpreviousknowledgeineconomics.OurresultssuggestthatthesubjectsunacquaintedwiththeseconceptswerebetterabletolearntheminconditionRIthan
inconditionSI.AsshowninFig.9,incomparisontoconditionSI,conditionRIhadahigherproportionofsubjectsshowingsignsofknowledgeimprovementforquestionsE-Q1(lawofdemand)andE-Q2
(substituteproducts).
Figure9:ProportionofsubjectswhoshowedsignsofknowlegeimprovementinconditionRIandSI.
5.14 Incontrastwiththelawofdemandandsubstituteproducts,theresultsofourpre-experimenttestsindicatethatonly27%ofstudentshadpriorknowledgeonstrategicbehavioroffirms(E-Q4)andonly30%of
studentshadpriorknowledgeontheexpectedoutcomeofthesimulatedmarket(E-Q5).Resultsofourpost-experimenttestsonthosequestionssuggestthatthesubjectsunacquaintedwiththetopicsoffirmand
marketbehaviorwerebetterabletolearntheminconditionRIratherthaninconditionSI.AsshowninFig.9,conditionRIhadahigherproportionofsubjectsshowingsignsofknowledgeimprovementinquestions
E-Q4andE-Q5,thanconditionSI.Differencesinlearningbetweenthetwoconditionswereparticularlypronouncedwhentestingtheknowledgeofparticipantsonthepredictedoutcomeofthesimulatedmarket
(differenceof26%onE-Q5).TheseresultsconfirmtheEconomicConceptAcquisitionHypothesis.
In-SimulationPerformanceHypothesis
5.15 Herewewanttocomparesubjects'actualin-simulationbehaviorbetweentheRIconditionandtheSIcondition.Asmetricsweused(1)thenumberofroundstodefeattheAIcompetitor,(2)theprofitsperround
(USD)and(3)cumulativeprofits(USD).
5.16 Amongthe19subjectswhowereabletodefeattheAIcompetitor,10wereundertheRIconditionand9undertheSIcondition.Interestingly,thesubjectsintheRIconditionwereabletowininlessrounds(Mean=
9.3,STD=1.16)thansubjectsintheSIcondition(Mean=13.1,STD=2.93).ThisdifferencewasstatisticallysignificantaccordingtoaStudentt-test(p<0.01).
5.17 WecomparedtheaverageprofitperroundofsubjectsunderRIconditionandunderSIcondition.Inlinewiththepreviousresults,wefoundthatsubjectshadhigherprofitsperroundinRIcondition(Mean=4,172,
STD=2,772)thaninSIcondition(Mean=2,045,STD=2,273)2.ThisdifferencewasstatisticallysignificantaccordingtoaStudentt-test(p<0.05).Figure10depictstheevolutionofaverageprofitperroundin
conditionRIandSI.Asshown,forthemajorityoftherounds(73.3%),subjectswereabletoachievehigherprofitsperroundundertheRIcondition.
http://jasss.soc.surrey.ac.uk/17/3/7.html
7
22/10/2015
Figure10:EvolutionofaverageprofitperroundinconditionRIandSI.
5.18 Finally,wealsocomparedRIandSIconditionsaccordingtocumulativeprofitsofsubjectsattheendoftheseventhround(theminimumsetofroundsplayedbysubjects).Ananalysisofourresultsshowedthat,as
expected,subjectshadhighercumulativeprofitsintheRIcondition(Mean=23,167,STD=16,628)thanintheSIcondition(Mean=10,508,STD=13,478)3.Thisdifferencewasstatisticallysignificantaccordingto
aStudentt-test(p<0.05).
DiscussionandConclusions
6.1
Wedescribedabusinesssimulationwherethemarketplaceismodeledasacomplexsystemcomposedofseveralinteractingconsumeragents.Theuniquefeatureofoursimulationisaninterfacethatanimates
theoperationoftheagents(consumers)andthusallowstheusertovisualizethedynamicsofthemarketplace.
6.2
Thecorequestionofourworkiswhetherusersactuallylearnbetterwhenprovidedwithinformationfacilitatedbyanagent-basedsimulationofaneconomicenvironment.Toanswerthisquestionwesetoutto
investigatelearningattwofundamentallevels:thereactionandlearninglevelsofKirckpatrick'sclassicalframework.
6.3
Thefirstlevel,reaction,consistedontheself-assessmentoflearningbysubjects.Ourresultssupportthehypothesisthatsubjectsperceivebetterlearningwiththeagent-basedinformation.Ofcourse,this
subjectivefindingisnotsufficienttosupporttheclaimofimprovedactuallearning.Ashasbeenshown(Sitzmannetal.2010),thecorrelationbetweenself-assessmentandactuallearningisonlymoderate.
However,thisisanimportantresultsinceitsuggeststhatparticipantsofbusinesssimulationsaremoreengagedintheirlearningprocesseswhenhavingaccesstoagent-basedinformation.
6.4
Thesecondlevel,learning,allowedustoobjectivelyassesslearningbasedonfactualevidence.Supportedbytheoutcomeofpre-experimentandpost-experimentknowledgeacquisitiontests,weconcludedthat
subjectsarebetterabletolearnconceptsfromeconomics,suchasthelawofdemand,substituteproducts,strategicbehavioranddynamicsofmarkets,whenagent-basedinformationispresent.Wealsoobserved
thatsubjectsdemonstratebetterin-simulationperformanceinthepresenceofagent-basedinformation.
6.5
Designingaconvincingreal-timeinterfaceforanagent-basedsystemisnotaneasytask.Thedifficultyofsuchendeavortendstoincreasewiththecomplexityoftheagent-basedmodel.Inourwork,wewereable
tocreateaninterfacethathadaprovenpositiveinfluenceonthelearningprocessesofusers,asevidencedbytheresultsofourstudy.Thisisasignificantachievementofourdesign.Nonetheless,noveltechniques
fromthefieldofintelligentuserinterfacescouldbeappliedtofurtherimprovethecurrentimplementation.Itseemstous,thatthistopicisquiteunexploredandcanbeofgreatinterestbothforthecommunityof
businesssimulationsandforthegeneralcommunityofartificialsocietiesandsocialsimulation.
6.6
Regardingthefutureimprovementofouragent-basedmodelwefacetwoofthetraditionalchallengesofdesignersofagent-basedmodels(Knoerietal.2011).Thefirstchallengeishowtoensuremodel
validation,andthesecondchallengeishowtocreateasimulationwithbehaviorallyrealisticagents.OurimplementationofrationalchoicetheoryisbasedontheBDIarchitecture,inlinewithwithclassical
microeconomics.PropervalidationoftheproposedmodelcanbeachievedusingoneormoreofthevalidationapproachesproposedbyMoss(2008).
6.7
Sun&Naveh(2004)raisedtheinterestingquestionwhetheragent-basedmodelsmightbenefitfromhavingagentswithmorecognitiverealism.Asstatedbefore,itisnotcleariflearning,theprimarypurposeof
businesssimulations,dependsonrealismandcomplexity.Accordingly,wemustcarefullystudythistopicinthefutureandtrytobalancetheneedforincreasedrealismwitheffectivelearningprocesses.
6.8
Tosummarize,wehaveoutlinedthethreemainresearchdirectionstofurtherourstudy:(1)intelligentagent-basedinterfaces,(2)extendingandvalidatingthepresentagent-basedmodel,and(3)increasingits
realismandcomplexity.
Notes
1
Priortostatisticalanalysis,anoutlierwasremovedfromthedataofthereactionquestionnaire.ThissubjectwasunderconditionRIandshowedstrongdisagreementwhenaskedifhehadlearnedaboutthe
behavioroffirms(R-Q1)andmarkets(R-Q2)withthesimulation.Whenaskedaboutthereasonsforhisstrongdisagreementthesubjectreportedthathefeltthatthesimulationdidnotportrayareal-lifemarket.
Surprisingly,thesubjectwasabletowinthesimulationinfewnumberofrounds(8comparedtothemeanof9.3ofconditionRI),answeredalltheknowledgeacquisitionquestionscorrectlyandshowedadeep
knowledgeofthesimulatedmarketwhenquestioned.Hisassessment(1:stronglydisagree)ofsentencesR-Q1andR-Q2largelydeviatedfromtheaverageclassification(Mean=5.71andMean=5.93)ofthe
remainingsubjectsunderthesamecondition.Insummary,ourexperimentwasnotdesignedforexpertsineconomics.
2
Priortothisstatisticalanalysis,anoutlierwasremovedfromthedata.ThissubjectwasunderRIconditionandexhibitedaverylowprofitperround(16.8)largelydeviatingfromtheaverage(Mean=4,172)ofthe
remainingdata.
3
Priortothisstatisticalanalysis,threeoutlierswereremovedfromthedata.OnewasinconditionRIreportingacumulativeprofitof298,comparablylowerthantheaverage(Mean=23,167)whiletheothertwowere
inconditionSI,reportingcumulativeprofitsof36,573and38,620,threetimeshigherthantheaverageoftheset(Mean=10,508).
References
ARAI,K.,DEGUCHI,H.&MATSUI, H.(2005).Agent-BasedModelingMeetsGamingSimulation.Springer.[doi:10.1007/4-431-29427-9]
BANDINI, S.,MANZONI,S.&VIZZARI, G.(2009).Agentbasedmodelingandsimulation:Aninformaticsperspective.JournalofArtificialSocietiesandSocialSimulation12(4),4.
http://jasss.soc.surrey.ac.uk/12/4/4.html.[doi:10.1007/978-0-387-30440-3_12]
BRATMAN,M.(1987).Intentions,Plans,andPracticalReason .HarvardUniversityPress.
CANNON, J.N.,C ANNON, H.M.&SCHWAIGER, M.(2012).Modelingthe'profitable-productdeathspiral':Accountingforstrategicproduct-mixinteractionsinmarketingsimulationgames. Simulation&Gaming43(6),
761-777.[doi:10.1177/1046878111434474]
CARVALHO,G.(1995).Modelingthelawofdemandinbusinesssimulators. Simulation&Gaming26(1),60-79.[doi:10.1177/1046878195261006]
COURNOT,A.-A.(1838).Recherchessurlesprincipesmathématiquesdelathéoriedesrichesses.Hachette.
http://jasss.soc.surrey.ac.uk/17/3/7.html
8
22/10/2015
DELRE, S.,JAGER, W.,BIJMOLT,T.&J ANSSEN,M.(2010).Willitspreadornot?Theeffectsofsocialinfluencesandnetworktopologyoninnovationdiffusion.JournalofProductInnovationManagement27(2),267282.[doi:10.1111/j.1540-5885.2010.00714.x]
F RANK,R.H.&GLASS, A.J.(1997).Microeconomicsandbehavior.McGraw-Hill.
G OLD, S.(1993).Modelinginteractiveeffectsinmathematicalfunctionsforbusinesssimulations:AcritiqueofGoosenandKusel'sinterpolationapproach.Simulation&Gaming24(1),90-94.
[doi:10.1177/1046878193241010]
G OLD, S.&PRAY, T.(1995).Theuseofthegammaprobabilitydistribution:AcritiqueofCarvalho'sdemandsimulator.Simulation&Gaming26(1),80-87.[doi:10.1177/1046878195261007]
G OLD, S.C.&PRAY, T.F.(1997).Modelingattributesindemandfunctionsofcomputerizedbusinesssimulations:AnextensionofTeach'sgravityflowalgorithm.DevelopmentsinBusinessSimulationand
ExperientialExercises24,132-141.
G OOSEN, K.,JENSEN,R.&WELLS, R.(2001).Purposeandlearningbenefitsofsimulations:Adesignanddevelopmentperspective.Simulation&Gaming32(1),21-39.[doi:10.1177/104687810103200104]
G OOSEN, K.&KUSEL, J.(1993).Aninterpolationapproachtodevelopingmathematicalfunctionsforbusinesssimulations.Simulation&Gaming24(1),76-89.[doi:10.1177/1046878193241009]
G OOSEN, K.R.(1981).Ageneralizedalgorithmfordesigninganddevelopingbusinesssimulations.DevelopmentsinBusinessSimulationandExperientialLearning8,41-47.
G RößLER, A.,MAIER,F.H.&MILLING, P.M.(2000).Enhancinglearningcapabilitiesbyprovidingtransparencyinbusinesssimulators.Simulation&Gaming31(2),257-278.[doi:10.1177/104687810003100209]
G RüNE-YANOFF, T.(2009).Theexplanatorypotentialofartificialsocieties.Synthese169(3),539-555.[doi:10.1007/s11229-008-9429-0]
KENWORTHY, J.&WONG, A.(2005).Developingmanagerialeffectiveness:Assessingandcomparingtheimpactofdevelopmentprogrammesusingamanagementsimulationoramanagementgame.Developments
inBusinessSimulationsandExperientialLearning 32,164-75.
KIRKPATRICK,D.(1959).Techniquesforevaluatingtrainingprograms(afour-partseriesbeginninginthenovemberissue).JournalofAmericanSocietyofTrainingDirectors13(11),3-9.
KNOERI, C.,BINDER,C.R.&ALTHAUS, H.-J.(2011).Anagentoperationalizationapproachforcontextspecificagent-basedmodeling. JournalofArtificialSocietiesandSocialSimulation14(2),4.
http://jasss.soc.surrey.ac.uk/14/2/4.html.
KOPAINSKY,B.,ALESSI, S.M.&PIRNAY-DUMMER,P.(2011).Providingstructuraltransparencywhenexploringamodel'sbehavior:Effectsonperformanceandknowledgeacquisition.In:29thInternational
ConferenceoftheSystemDynamicsSociety.
LEE, K.,KIM,S.,KIM,C.O.&PARK, T.(2013).Anagent-basedcompetitiveproductdiffusionmodelfortheestimationandsensitivityanalysisofsocialnetworkstructureandpurchasetimedistribution.Journalof
ArtificialSocietiesandSocialSimulation16(1),3.http://jasss.soc.surrey.ac.uk/16/1/3.html.
MACHUCA,J.A.(2000).Transparent-boxbusinesssimulators:Anaidtomanagethecomplexityoforganizations. Simulation&Gaming31(2),230-239.[doi:10.1177/104687810003100207]
MOSS, S.(2008).Alternativeapproachestotheempiricalvalidationofagent-basedmodels. JournalofArtificialSocietiesandSocialSimulation11(1),5.http://jasss.soc.surrey.ac.uk/11/1/5.html.
PARUNAK, H.V.D.,SAVIT,R.&RIOLO, R.L.(1998).Agent-basedmodelingvs.equation-basedmodeling:Acasestudyandusers'guide.In:Multi-AgentSystemsandAgent-BasedSimulation.Springer.
SAID,L.,BOURON, T.&DROGOUL,A.(2002).Agent-basedinteractionanalysisofconsumerbehavior.In:1stInternationalConferenceonAutonomousAgentsandMultiagentSystems.ACM.
SCHUMANN,P.L.,ANDERSON,P.H.,SCOTT, T.W.&LAWTON, L.(2001).Aframeworkforevaluatingsimulationsaseducationaltools.DevelopmentsinBusinessSimulationandExperientialLearning28,215-220.
SENGUPTA, A.&GLAVIN,S.E.(2013).Predictingvolatileconsumermarketsusingmulti-agentmethods:Theoryandvalidation.In:SimulationinComputationalFinanceandEconomics:ToolsandEmerging
Applications(ALEXANDROVA-KABADJOVA,B.,MARTINEZ-J ARAMILLO, S.,GARCIA-ALMANZA, A.L.&T SANG, E.,eds.).IGIGlobal,Hershey,pp.339-358.
SITZMANN,T.,ELY, K.,BROWN, K.G.&BAUER,K.N.(2010).Self-assessmentofknowledge:Acognitivelearningoraffectivemeasure?AcademyofManagementLearning&Education9(2),169-191.
SUMMERS,G.J.(2004).Today'sbusinesssimulationindustry. Simulation&Gaming35(2),208-241.[doi:10.1177/1046878104263546]
SUN,R.&NAVEH, I.(2004).Simulatingorganizationaldecision-makingusingacognitivelyrealisticagentmodel.JournalofArtificialSocietiesandSocialSimulation7(3)5.http://jasss.soc.surrey.ac.uk/7/3/5.html.
T EACH,R.D.(1990).Demandequationswhichincludeproductattributes.DevelopmentsinBusinessSimulationandExperientialLearning17,161-166.
T EACH,R.D.&MURFF,E.J.(2008).Arethebusinesssimulationsweplaytoocomplex? DevelopmentsinBusinessSimulation&ExperientialExercises35,205-211.
UMEDA,T.,ICHIKAWA,M.,KOYAMA, Y.&DEGUCHI,H.(2009).Evaluationofcollaborativefilteringbyagent-basedsimulationconsideringmarketenvironment.DevelopmentsinBusinessSimulationandExperiential
Learning36,214-222.
WALDHERR,A.&WIJERMANS, N.(2013).Communicatingsocialsimulationmodelstoscepticalminds.JournalofArtificialSocietiesandSocialSimulation16(4),13.http://jasss.soc.surrey.ac.uk/16/4/13.html.
YILMAZ, L.,ÖREN,T.&AGHAEE, N.-G.(2006).Intelligentagents,simulation,andgaming.Simulation&Gaming37(3),339-349.[doi:10.1177/1046878106289089]
http://jasss.soc.surrey.ac.uk/17/3/7.html
9
22/10/2015