naturalrealplayersp

HALId:inria-00349127https://hal.
inria.
fr/inria-00349127Submittedon23Dec2008HALisamulti-disciplinaryopenaccessarchiveforthedepositanddisseminationofsci-entificresearchdocuments,whethertheyarepub-lishedornot.
ThedocumentsmaycomefromteachingandresearchinstitutionsinFranceorabroad,orfrompublicorprivateresearchcenters.
L'archiveouvertepluridisciplinaireHAL,estdestinéeaudéptetàladiffusiondedocumentsscientifiquesdeniveaurecherche,publiésounon,émanantdesétablissementsd'enseignementetderecherchefranaisouétrangers,deslaboratoirespublicsouprivés.
Grimage:3DmodelingforremotecollaborationandtelepresenceBenjaminPetit,Jean-DenisLesage,Jean-SébastienFranco,EdmondBoyer,BrunoRainTocitethisversion:BenjaminPetit,Jean-DenisLesage,Jean-SébastienFranco,EdmondBoyer,BrunoRain.
Grimage:3Dmodelingforremotecollaborationandtelepresence.
VRST2008-ACMsymposiumonVirtualrealitysoftwareandtechnology,Oct2008,Bordeaux,France.
pp.
299-300,10.
1145/1450579.
1450662.
inria-00349127299Grimage:3DModelingforRemoteCollaborationandTelepresenceBenjaminPetitINRIAGrenobleUniversitiesJean-DenisLesageINRIAGrenobleUniversitiesJean-SebastienFrancoINRIAUniversiteBordeaux1EdmondBoyer§INRIAGrenobleUniversitiesBrunoRafnINRIAGrenobleUniversitiesAbstractReal-timemulti-camera3Dmodelingprovidesfull-bodygeometricandphotometricdataontheobjectspresentintheacquisitionspace.
Itcanbeusedasaninputdeviceforrenderingtextured3Dmodels,andforcomputinginteractionswithvirtualobjectsthroughaphys-icalsimulationengine.
Inthispaperwepresentaworkinprogresstobuildacollaborativeenvironmentwheretwodistantusers,eachone3Dmodeledinreal-time,interactinasharedvirtualworld.
CRCategories:I.
3.
2[ComputerGraphics]:GraphicsSystems—Distributed/networkgraphicsI.
4.
5[ImageProcessingandCom-puterVision]:ReconstructionKeywords:Telepresence;Collaborative3DInteractions;Marker-less3DModeling;Multi-cameras;PCCluster1IntroductionTelepresenceisofcentralimportanceinvirtualandaugmentedreal-ityapplications.
Itreferstotheabilitytoembedapersoninagivenlocationintovirtualenvironmentsatotherlocations.
Whileseveraltechnologiesalreadyallowvisualizationatdifferentlocations,thecreationofvirtualenvironmentswherepersonsinremotesitescaninteractwitheachother,aswellaswithvirtualobjects,isstillchal-lenging.
Oneofthedifcultyistobuildvirtualusermodelsthatcarryenoughinformationforsuchapplications.
Therearedifferentapplicationsofvirtualimmersiveandcollab-orativeenvironmentssuchasvideogames,teleconferencingwiththecapabilitytomanipulateandinteracttogetherwithaCADmodel,learnandtrainfromremotesitesormanageemergencycri-sis,amongothers.
Traditionaltechnologiesforsuchapplicationsaregenerallylimitedinseveralaspects.
Themainreasonsaretobefoundinthevirtualrepresentationsoftheuserwhichareoftencoarseandhencepreventpeoplefrominteractinginanaturalway.
Mostoftheexistingsystemsuses2Drepresentationsobtainedwithmono-camerasystems.
Whilegivingapartiallytruerepresentationoftheuser,theyarenotimmersiveenoughfor3Dvirtualworlds.
Othersystems,whicharetosomeextentmoresuitablefor3Dvir-tualworlds,useavatars,e.
gmulti-playergameslikeSecondLife.
e-mail:benjamin.
petit@inrialpes.
fre-mail:jean-denis.
lesage@imag.
fre-mail:jean-sebastien.
franco@labri.
fr§e-mail:edmond.
boyer@inrialpes.
fre-mail:bruno.
rafn@imag.
frButavatarsonlycarrypartialinformationaboutusers.
Thoughreal-timemotioncaptureenvironmentscanimprovesuchmodelsandallowforanimation,avatarsdonotyetprovidearealisticrepresen-tation.
Onewaytoimprovethesenseofpresenceandrealismistocom-putereal-time3Dmodelsbasedonbothphotometricandgeomet-ricdata.
Theyyieldmorefaithfullifelikerepresentationsthatcanincludeuserappearance,motionsandevenfacialexpressions.
Dif-ferentstrategiescanbefollowedtoobtainthe3Dmodelofaper-son.
Themostcommononeistousemulti-camerasystems.
Suchsystemsarealreadyusedfortelepresence[Grossetal.
2003].
Nev-ertheless,existing3Dpersonrepresentations,inreal-timesystems,oftenhavelimitationssuchasimperfectorcoarsegeometricmod-elsandlowresolutiontextures.
Thisasaresultofthemethodusedtoreconstructin3D,e.
g.
stereo-visionorvisualhull,andofthenumberofcamerasandthealgorithmcomputationtime.
Wepresentworksinprogressonusingmulti-camerareal-time3Dmodelingbetweenremotesites.
Suchworksarebasedonourpre-viousexperiencewiththeGrimagemulti-cameraplatformthaten-ablesoneorseveraluserspresentinthesameacquisitionspacetointeractwithvirtualobjectsanimatedbyaphysicalsimulation.
Weconsiderheretheextensiontomulti-sitefortelepresenceandcol-laborativeenvironments.
Theremainderofthepaperisasfollows.
Section2presentstheexistingGrimageplatformwhilesection3describeshowthisenvi-ronmenthasevolvedtosupportmulti-siteinteractions,remotecol-laborationandtelepresence.
2TheSingle-siteEnvironmentTheGrimageplatformconsistsinamulti-cameraenvironment,aPCclusterprovidingacquisition,computingandrenderingre-sources.
Wecompute3Dgeometricmodelsandextractmulti-viewtexturesfromasetofcalibratedcameraswithoutusingintrusivemarkersorassumptionsabouttheobjectsbeinglmed.
Wecomputethevisualhulloftheobjectsbyreconstructingtheirshapefromthesilhouettesextractedfromthevideostreams.
Thealgorithmwedevelopedanduse,calledtheExactPolyhedralVisualHullalgorithm[FrancoandBoyer2003],computesinreal-timethecompleteandexactvisualhullpolyhedronwithregardtosilhouetteinputs.
The3DmodelsareinjectedintotheSOFAphysicalsimulationsoft-ware[Sof].
SOFAanimatesvirtualobjectstakingintoaccountthepresenceofthereal-objectsthroughtheir3Dmodels.
Thesceneisrenderedaftertexturingthe3Dmodelsforanimprovedrealism.
Thissingle-siteplatformwaspresentedattheSiggraph2007EmergingTechnologiesshowbyAllard&al.
[2007].
Photosandvideosareavailableat[Gri].
Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprofitorcommercialadvantage,andthatcopiesbearthisnoticeandthefullcitationonthefirstpage.
Tocopyotherwise,torepublish,topostonserversortoredistributetolists,requirespriorspecificpermissionand/orafee.
VRST'08,October27-29,2008,Bordeaux,FranceCopyright2008ACMISBN978-1-59593-951-7/08/10.
.
.
$5.
003003RemoteCollaborationandTelepresenceWeextendGrimagetoadistributedcontext.
Weconnectdifferentclusterstogetherinsidethesametelepresenceapplication.
Eachclusterhostsasitewithanacquisitionareaandrendersthevir-tualworldindependently.
Acentralnodeoftheapplicationgath-ersthe3Dmodelsfromallsitesandrunsthephysicalsimulation(Figure1).
Thissimulationdetectscollisionsandcomputestheef-fectofeachuseronthevirtualworld.
Itisofcourseimpossibletochangethestateoftheinputmodelsthem-selvesastherearenoforce-feedbackdeviceontheplatform.
Physicalinteractionsbe-tweenparticipantsarealsoimpossibleforthesamereason.
Thesimulationsendsthe3Dscene,i.
e.
thevirtualobjects,toallrenderingnodesofeachsite.
Eachacquisitionplatformsendsits3Dtexturedmodelstotheothersites.
Therenderingnodesofeachclustermergethesedatasentbydifferentsourcesandprovideacon-sistentviewofthescene.
Renderingcanbedoneonvirtuallyanydisplaydevicewithdifferentlevelsofimmersion(e.
gstereoscopicdisplay,HMD,displaywall,Cave).
CouplingthedifferentsoftwarecomponentsinvolvedintothisprojectanddistributingthemonthenodesofaPCclusterforreachingreal-timeexecutionsisperformedusingFlowVR[Flo;LesageandRafn2008],ahierarchicalcomponentorientedmid-dlewarebasedonadata-owparadigm.
FlowVRenforcesamod-ularprogrammingthatleveragessoftwareengineeringissueswhileenablinghighperformanceexecutionsonmulti-clusterarchitec-tures.
Figure1:Applicationarchitecturefor2multi-cameraacquisitionspaces.
3.
1PracticalSetupThedemonstrationisconductedontwoplatformslocatedattheVRSTconferencesite.
Eachplatformhasanacquisitionspaceofonesquaredmeterattableheightwithverewirecameras.
Itissuitableforhands/armsinteractions.
EachplatformisequippedwithaPCclustercomposedofvePCsusedforcameraacquisition,onePCforcomputationandrenderingandalaptopforsupervision.
Thetwoclustersareconnectedbyagigabyteethernetnetworkus-ingonePCasgatewaybetweenthetwoplatforms.
ThisPCrunsthephysicalsimulation.
3.
2ResultsWesucceededinbroadcastingthe3Dmodelsbetweensitesatelevenframespersecondwhilethephysicalsimulationwasrun-ningat30iterationspersecond.
Performancesreachedduringex-perimentsenabledtwouserstointeractwithacommonphysicalsimulationwithoutsignicantlatency.
4ConclusionWepresentedanon-goingworkfortelepresenceandcollaborativeinteractionusingtwomulti-cameraandreal-time3Dmodelingplat-formsandaphysicalsimulationtohandlethevirtualinteractionspace.
Firstresultsareverypositive.
Fromtheuserpointofviewthesenseofpresenceisstrong.
Wealsonoticedthatusersdonotneedtolearntheinteractionparadigmsbasedontheonespeopleexperienceintherealworld.
Thisworkalsostressesanumberofweakpointsthatwewillad-dressinthefuture.
Texturing3Dmodelisakeycomponentoftelepresence,butimprovingtexturequalitytendstooverloadthecommunicationnetworkandleadstoextra-latencyintheapplica-tion.
Interactionsusinga3Dmodelandphysicalsimulationdonotallowsomeactionslikeobjectpreemption.
AcknowledgmentThisworkwaspartlyfundedbyAgenceNationaledelaRecherche,contractANR-06-MDCA-003.
ReferencesALLARD,J.
,MENIER,C.
,RAFFIN,B.
,BOYER,E.
ANDFAURE,F.
2007.
Grimage:Markerless3DInteractions.
InSIGGRAPH'07:ACMSIGGRAPH2007emergingtechnologies,ACM,NewYork,NY,USA,9.
FlowVR.
http://flowvr.
sf.
net.
FRANCO,J.
,ANDBOYER,E.
2003.
ExactPolyhedralVisualHulls.
InProceedingsofBMVC2003.
Grimage.
http://www.
inrialpes.
fr/grimage.
GROSS,M.
,W¨URMLIN,S.
,NAEF,M.
,LAMBORAY,E.
,SPAGNO,C.
,KUNZ,A.
,KOLLER-MEIER,E.
,SVOBODA,T.
,GOOL,L.
V.
,LANG,S.
,STREHLKE,K.
,MOERE,A.
V.
,ANDSTAADT,O.
2003.
Blue-C:aSpatiallyImmersiveDisplayand3DVideoPortalforTelepresence.
ACMTrans.
Graph.
22,3,819–827.
LESAGE,J.
-D.
,ANDRAFFIN,B.
2008.
AHierarchicalCompo-nentModelforLargeParallelInteractiveApplications.
JournalofSupercomputing(July).
Sofa.
http://www.
sofa-framework.
org.