intermediateav

MITSUBISHIELECTRICRESEARCHLABORATORIEShttp://www.
merl.
comOverviewoftheStereoandMultiviewVideoCodingExtensionsoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCStandardVetro,A.
;Wiegand,T.
;SullivanG.
J.
TR2011-022January2011AbstractSignicantimprovementsinvideocompressioncapabilityhCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeendemonstratedwiththein-troductionoftheH.
264/MPEG-4AdvancedVideoCoding(COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC)standard.
Sincedevelopingthisstandard,theJointVideoTeamoftheITU-TVideoCodingExpertsGroup(VCEG)andtheISO/IECMovingPictureExpertsGroup(MPEG)hasalsostandardizedanextensionofthattechnologythatisreferredtoasmultiviewvideocoding(MVC).
MVCprovidesacompactrep-resentationformultipleviewsofavideoscene,suchasmultiplesynchronizedvideocameras.
Stereo-pairedvideofor3DviewingisanimportantspecialcaseofMVC.
Thestandardenablesinter-viewpredictiontoimprovecompressioncapability,aswellassupportingordinarytempo-ralandspatialprediction.
ItalsosupportsbackwardcompatibilitywithexistinglegacysystemsbystructuringtheMVCbitstreamtoincludeacompatible"baseview".
Eachotherviewisencodedatthesamepictureresolutionasthebaseview.
Inrecognitionofitshighqualityen-codingcapabilityandsupportforbackwardcompatibility,theStereoHighproleoftheMVCextensionwasselectedbytheBlu-RayDiscAssociationasthecodingformatfor3Dvideowithhigh-denitionresolution.
ThispaperprovidesanoverviewofthealgorithmicdesignusedforextendingH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCtowardsMVC.
ThebasicapproachofMVCforenablinginter-viewpredictionandviewscalabilityinthecontextofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCisreviewed.
Relatedsupplementalenhancementinformation(SEI)metadataisalsodescribed.
Various"framecom-patible"approachesforsupportofstereo-viewvideoasanalternativetoMVCarealsodiscussed.
AsummaryofthecodingperformanceachievedbyMVCforbothstereoandmultiviewvideoisalsoprovided.
Futuredirectionsandchallengesrelatedto3Dvideoarealsobrieydiscussed.
ProceedingsoftheIEEEThisworkmaynotbecopiedorreproducedinwholeorinpartforanycommercialpurpose.
Permissiontocopyinwholeorinpartwithoutpaymentoffeeisgrantedfornonproteducationalandresearchpurposesprovidedthatallsuchwholeorpartialcopiesincludethefollowing:anoticethatsuchcopyingisbypermissionofMitsubishiElectricResearchLaboratories,Inc.
;anacknowledgmentoftheauthorsandindividualcontributionstothework;andallapplicableportionsofthecopyrightnotice.
Copying,reproduction,orrepublishingforanyotherpurposeshallrequirealicensewithpaymentoffeetoMitsubishiElectricResearchLaboratories,Inc.
Allrightsreserved.
CopyrightcMitsubishiElectricResearchLaboratories,Inc.
,2011201Broadway,Cambridge,Massachusetts02139MERLCoverPageSide2PROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN1Abstract—Significantimprovementsinvideocompressionca-pabilityhCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeendemonstratedwiththeintroductionoftheH.
264/MPEG-4AdvancedVideoCoding(COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC)standard.
Sincedevelopingthisstandard,theJointVideoTeamoftheITU-TVid-eoCodingExpertsGroup(VCEG)andtheISO/IECMovingPic-tureExpertsGroup(MPEG)hasalsostandardizedanextensionofthattechnologythatisreferredtoasmultiviewvideocoding(MVC).
MVCprovidesacompactrepresentationformultipleviewsofavideoscene,suchasmultiplesynchronizedvideocam-eras.
Stereo-pairedvideofor3DviewingisanimportantspecialcaseofMVC.
Thestandardenablesinter-viewpredictiontoim-provecompressioncapability,aswellassupportingordinarytemporalandspatialprediction.
Italsosupportsbackwardcom-patibilitywithexistinglegacysystemsbystructuringtheMVCbitstreamtoincludeacompatible"baseview".
Eachotherviewisencodedatthesamepictureresolutionasthebaseview.
Inrecog-nitionofitshighqualityencodingcapabilityandsupportforbackwardcompatibility,theStereoHighprofileoftheMVCex-tensionwasselectedbytheBlu-RayDiscAssociationasthecod-ingformatfor3Dvideowithhigh-definitionresolution.
Thispa-perprovidesanoverviewofthealgorithmicdesignusedforex-tendingH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCtowardsMVC.
ThebasicapproachofMVCforenablinginter-viewpredictionandviewscalabilityinthecontextofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCisreviewed.
Relatedsupple-mentalenhancementinformation(SEI)metadataisalsode-scribed.
Various"framecompatible"approachesforsupportofstereo-viewvideoasanalternativetoMVCarealsodiscussed.
AsummaryofthecodingperformanceachievedbyMVCforbothstereoandmultiviewvideoisalsoprovided.
Futuredirectionsandchallengesrelatedto3Dvideoarealsobrieflydiscussed.
IndexTerms—MVC,H.
264,MPEG-4,COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC,standards,stereovideo,multiviewvideocoding,inter-viewprediction,3Dvideo,Blu-rayDiscI.
INTRODUCTIONVIDEOiscurrentlybeingintroducedtothehomethroughvariouschannels,includingBlu-rayDisc,cableandsatellitetransmission,terrestrialbroadcast,andstreaminganddownloadthroughtheInternet.
Today's3DManuscriptreceivedApril1,2010.
RevisedversionsubmittedMMDD,2010.
A.
VetroiswithMitsubishiElectricResearchLabs,Cambridge,MA,02139USA(email:COLOR:#000000;BACKGROUND-COLOR:#ffff00">avetro@merl.
com).
T.
WiegandisjointlyaffiliatedwiththeBerlinInstituteoftechnologyandtheFraunhoferInstituteforTelecommunications–HeinrichHertzInstitute(HHI),Einsteinufer37,10587Berlin,Germany(email:wiegand@hhi.
de).
G.
J.
SullivaniswithMicrosoftCorporation,Redmond,WA,98052USA(email:garys@ieee.
org).
videooffersahigh-qualityandimmersivemultimediaexperi-ence,whichhasonlyrecentlybecomefeasibleonconsumerelectronicsplatformsthroughadvancesindisplaytechnology,signalprocessing,transmissiontechnology,andcircuitdesign.
Inadditiontoadvancesonthedisplayandreceiverside,therehasalsobeenanotableincreaseintheproductionof3Dcontent.
Thenumberof3Dfeaturefilmreleaseshasbeengrowingdramaticallyeachyear,andseveralmajorstudioshCOLOR:#000000;BACKGROUND-COLOR:#ffff00">aveannouncedthatalloftheirfuturereleaseswillbein3D.
Therearemajorinvestmentsbeingmadetoupgradedigitalcinematheaterswith3Dcapabilities,severalmajorfeaturefilmreleaseshCOLOR:#000000;BACKGROUND-COLOR:#ffff00">aveattractedamajorityoftheirtheaterrevenuein3Dshowings(includingCOLOR:#000000;BACKGROUND-COLOR:#ffff00">Avatar,thecurrenttopgrossingfeaturefilmofalltime1),andpremiumpricingfor3Dhasbe-comeasignificantfactorinthecinemarevenuemodel.
Thepushfromboththeproductionanddisplaysideshasplayedasignificantroleinfuellingaconsumerappetitefor3Dvideo.
Thereareanumberofchallengestoovercomeinmaking3Dvideoforconsumeruseinthehomebecomefullypracticalandshowsustainedmarketvalueforthelongterm.
Forone,theusabilityandconsumeracceptanceof3Dviewingtechnol-ogywillbecritical.
Inparticular,massconsumeracceptanceofthespecialeyewearneededtoview3Dinthehomewithcur-rentdisplaytechnologyisstillrelativelyunknown.
Ingeneral,contentcreators,serviceprovidersanddisplaymanufacturersneedtoensurethattheconsumerhasahighqualityexperienceandisnotburdenedwithhightransitioncostsorturnedoffbyviewingdiscomfortorfatigue.
TheCOLOR:#000000;BACKGROUND-COLOR:#ffff00">availabilityofpremium3Dcontentinthehomeisanothermajorfactortobeconsidered.
Thesearebroaderissuesthatwillsignificantlyinfluencetherateof3Dadoptionandmarketsize,butarebeyondthescopeofthispaper.
Withregardtothedeliveryof3Dvideo,itisessentialtode-termineanappropriatedataformat,takingintoconsiderationtheconstraintsimposedbyeachdeliverychannel–includingbitrateandcompatibilityrequirements.
Needlesstosay,inter-operabilitythroughthedeliverychainandamongvariousde-viceswillbeessential.
The3Drepresentation,compressionformats,andsignalingprotocolswilllargelydefinetheinter-operabilityofthesystem.
Forpurposesofthispaper,3Dvideoisconsideredtorefertoeitherageneraln-viewmultiviewvideorepresentationorits1Basedontotalrevenuewithoutinflationadjustments.
OverviewoftheStereoandMultiviewVideoCodingExtensionsoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCStandardANTHONYVETRO,FELLOW,IEEE,THOMASWIEGAND,FELLOW,IEEE,ANDGARYJ.
SULLIVAN,FELLOW,IEEE3DPROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN2importantstereo-viewspecialcase.
Efficientcompressionofsuchdataistheprimarysubjectofthispaper.
Thepaperalsodiscussesstereorepresentationformatsthatcouldbecodedusingexisting2Dvideocodingmethods–suchapproachesoftenbeingreferredtoasframe-compatibleencodingschemes.
Multiviewvideocoding(MVC)istheprocessbywhichste-reoandmultiviewvideosignalsareefficientlycoded.
ThebasicapproachofmostMVCschemesistoexploitnotonlytheredundanciesthatexisttemporallybetweentheframeswithinagivenview,butalsothesimilaritiesbetweenframesofneighboringviews.
Bydoingso,areductioninbitraterelativetoindependentcodingoftheviewscanbeachievedwithoutsacrificingthereconstructedvideoquality.
Inthispaper,thetermMVCisusedinterchangeablyforeitherthegeneralcon-ceptofcodingmultiviewvideoorfortheparticulardesignthathasbeenstandardizedasarecentextensionoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandard[1].
Thetopicofmultiviewvideocodinghasbeenanactivere-searchareaformorethan20years,withearlyworkondispar-ity-compensatedpredictionbyLukacsfirstappearingin1986[2],followedbyothercodingschemesinthelate1980'sandearly1990's[3][4].
In1996,theinternationalvideocodingstandardH.
262/MPEG-2Video[5]wasamendedtosupportthecodingofmultiviewvideobymeansofdesignfeaturesoriginallyintendedfortemporalscalability[6][7].
However,themultiviewextensionofH.
262/MPEG-2Videowasneverdeployedinactualproducts.
Itwasnottherighttimetointro-duce3Dvideointothemarketsincethemorefundamentaltransitionfromstandard-definitionanalogtohigh-definitiondigitalvideoserviceswasalargechallengeinitself.
Adequatedisplaytechnologyandhardwareprocessingcapabilitieswerealsolackingatthetime.
Inadditiontothis,theH.
262/MPEG-2Videosolutiondidnotofferaverycompellingcompressionimprovementduetolimitationsinthecodingtoolsenabledforinter-viewpredictioninthatdesign[8]-[10].
ThispaperfocusesontheMVCextensionoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandard.
Relevantsupplementalen-hancementinformation(SEI)metadataandalternativeap-proachestoenablingmultiviewservicesarealsodiscussed.
Thepaperisorganizedasfollows.
SectionIIexplainsthevari-ousmultiviewvideoapplicationsofMVCaswellastheirim-plicationsintermsofrequirements.
SectionIIIgivesthehis-toryofMVC,includingpriorstandardizationaction.
Sec-tionIVbrieflyreviewsbasicdesignconceptsofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC.
TheMVCdesignissummarizedinSectionV,includ-ingprofiledefinitionsandasummaryofcodingperformance.
AlternativestereorepresentationformatsandtheirsignalingintheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandardaredescribedinSectionVI.
ConcludingremarksaregiveninSectionVII.
Formorede-tailedinformationaboutMVCandstereosupportintheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandard,thereaderisreferredtothemostrecenteditionofthestandarditself[1],theamendmentcompletedinJuly2008thataddedtheMVCextensiontoit[11],andtheadditionalamendmentcompletedoneyearlaterthataddedtheStereoHighprofileandframepackingarrange-mentSEImessage[12].
II.
MULTIVIEWSCENARIOS,APPLICATIONS,ANDREQUIREMENTSThepredictionstructuresandcodingschemespresentedinthispaperhCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeendevelopedandinvestigatedinthecontextoftheMPEG,andlaterJVT,standardizationprojectforMVC.
Therefore,mostofthescenariosformultiviewcoding,appli-cationsandtheirrequirementsarespecifiedbytheMVCpro-ject[13]aspresentedinthenextsections.
A.
MultiviewScenariosandApplicationsTheprimaryusagescenarioformultiviewvideoistosup-port3Dvideoapplications,where3Ddepthperceptionofavisualsceneisprovidedbya3Ddisplaysystem.
Therearemanytypesof3Ddisplaysystems[14]includingclassicstereosystemsthatrequirespecial-purposeglassestomoresophisti-catedmultiviewauto-stereoscopicdisplaysthatdonotrequireglasses[15].
Thestereosystemsonlyrequiretwoviews,wherealeft-eyeviewispresentedtotheviewer'slefteye,andaright-eyeviewispresentedtotheviewer'srighteye.
The3Ddisplaytechnologyandglassesensurethattheappropriatesignalsareviewedbythecorrecteye.
Thisisaccomplishedwitheitherpassivepolarizationoractiveshuttertechniques.
Themul-tiviewdisplayshCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avemuchgreaterdatathroughputrequire-mentsrelativetoconventionalstereodisplaysinordertosup-portagivenpictureresolution,since3Disachievedbyessen-tiallyemittingmultiplecompletevideosamplearraysinordertoformview-dependentpictures.
Suchdisplayscanbeimple-mented,forexample,usingconventionalhigh-resolutiondis-playsandparallaxbarriers;othertechnologiesincludelenticu-laroverlaysheetsandholographicscreens.
Eachview-dependentvideosamplecanbethoughtofasemittingasmallnumberoflightraysinasetofdiscreteviewingdirections–typicallybetweeneightandafewdozenforanautostereo-scopicdisplay.
Oftenthesedirectionsaredistributedinahori-zontalplane,suchthatparallaxeffectsarelimitedtothehori-zontalmotionoftheobserver.
Amorecomprehensivereviewof3Ddisplaytechnologiesiscoveredbyotherarticlesinthisspecialissue.
Anothergoalofmultiviewvideoistoenablefree-viewpointvideo[16][17].
Inthisscenario,theviewpointandviewdirec-tioncanbeinteractivelychanged.
Eachoutputviewcaneitherbeoneoftheinputviewsoravirtualviewthatwasgeneratedfromasmallersetofmultiviewinputsandotherdatathatas-sistsintheviewgenerationprocess.
Withsuchasystem,view-erscanfreelynCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avigatethroughthedifferentviewpointsofthescene–withinarangecoveredbytheacquisitioncameras.
Suchanapplicationofmultiviewvideocouldbeimplementedwithconventional2Ddisplays.
However,moreadvancedver-sionsofthefree-viewpointsystemthatworkwith3Ddisplayscouldalsobeconsidered.
WehCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avealreadyseentheuseofthisfunctionalityinbroadcastproductionenvironments,e.
g.
,tochangetheviewpointofasportsscenetoshowabetterangleofaplay.
Suchfunctionalitymayalsobeofinterestinsurveil-lance,education,gaming,andsightseeingapplications.
Fi-nally,wemayalsoimagineprovidingthisinteractivecapabil-PROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN3itydirectlytothehomeviewer,e.
g.
,forspecialeventssuchasconcerts.
Anotherimportantapplicationofmultiviewvideoistosup-portimmersiveteleconferenceapplications.
Beyondtheadvan-tagesprovidedby3Ddisplays,ithasbeenreportedthatatele-conferencesystemscouldenableamorerealisticcommunica-tionexperiencewhenmotionparallaxissupported.
Motionparallaxiscausedbythechangeintheappearanceofascenewhentheviewershiftstheirviewingposition,e.
g.
,shiftingtheviewingpositiontorevealoccludedscenecontent.
Inaninter-activesystemdesign,itcanbepossibleforthetransmissionsystemtoadaptivelyshiftitsencodedviewingpositiontoachieveadynamicperspectivechange[18][19][20].
Perspec-tivechangescanbecontrolledexplicitlybyuserinterventionthroughauserinterfacecontrolcomponentorbyasystemthatsensestheobserver'sviewingpositionandadjuststhedis-playedsceneaccordingly.
OtherinterestingapplicationsofmultiviewvideohCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeendemonstratedbyWilburn,etal.
[21].
Inthiswork,ahighspa-tialsamplingofascenethroughalargemultiviewvideocam-eraarraywasusedforadvancedimaging.
Amongthecapabili-tiesshownwasaneffectiveincreaseofbitdepthandframerate,aswellassyntheticaperturephotographyeffects.
Sincethen,therehCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avealsobeenotherexcitingdevelopmentsintheareaofcomputationalimagingthatrelyontheacquisitionofmultiviewvideo[22].
Foralloftheaboveapplicationsandscenarios,thestorageandtransmissioncapacityrequirementsofthesystemaresig-nificantlyincreased.
Consequently,thereisastrongneedforefficientmultiviewvideocompressiontechniques.
Specificrequirementsarediscussedinthenextsubsection.
B.
StandardizationRequirementsThecentralrequirementformostvideocodingdesignsishighcompressionefficiency.
InthespecificcaseofMVCthismeansasignificantgaincomparedtoindependentcompres-sionofeachview.
Compressionefficiencymeasuresthetrade-offbetweencost(intermsofbitrate)andbenefit(intermsofvideoquality)–i.
e.
thequalityatacertainbitrateorthebitrateatacertainquality.
However,compressionefficiencyisnottheonlyfactorunderconsiderationforavideocodingstandard.
Somerequirementsmayevenbesomewhatconflict-ing,suchasdesiringbothgoodcompressionefficiencyandlowdelay.
Insuchcases,agoodtrade-offneedstobefound.
Generalrequirementsforvideocodingcapabilities,suchasminimumresourceconsumption(memory,processingpower),lowdelay,errorrobustness,andsupportofarangeofpictureresolutions,colorsamplingstructures,andbitdepthprecisions,tendtobeapplicabletonearlyanyvideocodingdesign.
SomerequirementsarespecifictoMVC–ashighlightedinthefollowing.
Temporalrandomaccessisarequirementforvirtuallyanyvideocodingdesign.
ForMVC,view-switchingrandomaccessalsobecomesimportant.
Bothtogetherensurethatanyimagecanbeaccessed,decoded,anddisplayedbystartingthedecoderatarandomaccesspointanddecodingarelativelysmallquantityofdataonwhichthatimagemayde-pend.
Randomaccesscanbeprovidedbyinsertionofpicturesthatareintra-picturecoded(i.
e.
,picturesthatarecodedwith-outanyuseofpredictionfromotherpictures).
Scalabilityisalsoadesirablefeatureforvideocodingdesigns.
Here,werefertotheabilityofadecodertoaccessonlyaportionofabitstreamwhilestillbeingabletogenerateeffectivevideoout-put–althoughreducedinqualitytoadegreecommensuratewiththequantityofdatainthesubsetusedforthedecodingprocess.
Thisreductioninqualitymayinvolvereducedtempo-ralorspatialresolution,orareducedqualityofrepresentationatthesametemporalandspatialresolution.
ForMVC,addi-tionally,viewscalabilityisdesirable.
Inthiscase,aportionofthebitstreamcanbeaccessedinordertooutputasubsetoftheencodedviews.
Also,backwardcompatibilitywasrequiredfortheMVCstandard.
ThismeansthatasubsetoftheMVCbit-streamcorrespondingtoone"baseview"needstobedecod-ablebyanordinary(non-MVC)H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCde-coder,andtheotherdatarepresentingotherviewsshouldbeencodedinwaythatwillnotaffectthatbaseviewdecodingcapability.
Achievingadesireddegreequalityconsistencyamongviewsisalsoaddressed–i.
e.
,itshouldbepossibletocontroltheencodingqualityofthevariousviews–forinstancetoprovideapproximatelyconstantqualityoverallviewsortoselectapreferentialqualityforencodingsomeviewsversusothers.
Theabilityofanencoderordecodertouseparallelprocessingwasrequiredtoenablepracticalimplementationandtomanageprocessingresourceseffectively.
Itshouldalsobepossibletoconveycameraparameters(extrinsicandintrin-sic)alongwiththebitstreaminordertosupportintermediateviewinterpolationatthedecoderandtoenableotherdecod-ing-sideenhancedcapabilitiessuchasmulti-viewfeaturede-tectionandclassification,e.
g.
,determiningtheposeofafacewithinascene,whichwouldtypicallyrequiresolvingacorre-spondenceproblembasedonthescenegeometry.
Moreover,foreaseofimplementation,itwashighlydesir-ablefortheMVCdesigntohCOLOR:#000000;BACKGROUND-COLOR:#ffff00">aveasmanydesignelementsincommonwithanordinaryH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCsystemaspossible.
Suchacommonalityofdesigncomponentscanen-ableanMVCsystemtobeconstructedrapidlyfromelementsofexistingH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCproductsandtobetestedmoreeasily.
III.
HISTORYOFMVCOneoftheearlieststudiesoncodingofmultiviewimageswasdonebyLukacs[2];inthiswork,theconceptofdisparity-compensatedinter-viewpredictionwasintroduced.
InlaterworkbyDinstein,etal.
[3],thepredictivecodingapproachwascomparedto3Dblocktransformcodingforstereoimagecompression.
In[4],Perkinspresentedatransform-domaintechniquefordisparity-compensatedprediction,aswellasamixed-resolutioncodingscheme.
Thefirstsupportformultiviewvideocodinginaninterna-tionalstandardwasina1996amendmenttotheH.
262/MPEG-2videocodingstandard[6].
Itsupportedthecodingoftwoviewsonly.
Inthatdesign,theleftviewwasreferredtoasthe"baseview"anditsencodingwascompatiblewiththatforordinarysingle-viewdecoders.
TherightviewPROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN4wasencodedasanenhancementviewthatusedthepicturesoftheleftviewasreferencepicturesforinter-viewprediction.
Thecodingtoolfeaturesthatwereusedforthisschemewereactuallythesameaswhathadpreviouslybeendesignedforprovidingtemporalscalability(i.
e.
,framerateenhancement)[7]-[10].
Fortheencodingoftheenhancementview,thesamebasiccodingtoolswereusedasinordinaryH.
262/MPEG-2videocoding,buttheselectionofthepicturesusedasrefer-enceswasaltered,sothatareferencepicturecouldeitherbeapicturefromwithintheenhancementvieworapicturefromthebaseview.
AnexampleofapredictionstructurethatcanbeusedintheH.
262/MPEG-2multiviewprofileisshowninFig.
1.
Arrowsinthefigureindicatetheuseofareferencepic-tureforthepredictiveencodingofanotherpicture.
Asignifi-cantbenefitofthisapproach,relativetosimulcastcodingofeachviewindependently,wastheabilitytouseinter-viewpre-dictionfortheencodingofthefirstenhancement-viewpictureineachrandom-accessibleencodedvideosegment.
However,theabilitytopredictinthereverse-temporaldirection,whichwasenabledforthebaseview,wasnotenabledfortheen-hancementview.
Thishelpedtominimizethememorystoragecapacityrequirementsforthescheme,butmayhCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avereducedthecompressioncapabilityofthedesign.
Fig.
1.
Illustrationofinter-viewpredictioninH.
262/MPEG-2.
Consideringrecentadvancementsinvideocompressiontechnologyandtheanticipatedneedsforstate-of-the-artcod-ingofmultiviewvideo,MPEGissuedaCallforProposals(CfP)forefficientmultiviewvideocodingtechnologyinOc-toberof2005.
Althoughnotanexplicitrequirementatthetime,allproposalresponseswerebasedonH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCandincludedsomeformofinter-viewprediction[23].
Asreportedin[24],significantgainsinvisualqualitywereob-servedfromtheformalsubjectiveteststhatwereconductedincomparisontoindependentsimulcastcodingofviewsbasedonH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC.
Specifically,whencomparingvisualqualityatthesamebitrate,MVCsolutionsachievedupto3MOSpoints(meanopinionscorepointsona0-10scale)bettervisualqualitythansimulcastH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCforlowandmediumbitratecoding,andabout1MOSpointbetterqualityforhighbitratecoding.
Whencomparingbitratesforseveralofthetestsequences,someproposedMVCsolutionsrequiredonlyabouthalfthebitratetoachieveequivalentorbettervis-ualqualitythantheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCcodedanchors2.
Theproposaldescribedin[25]wasfoundtoprovidethebestvisualqualityoverthewiderangeoftestsequencesandratepoints.
AkeyfeatureofthatproposalwasthatitdidnotintroduceanychangetothelowerlevelsofthesyntaxanddecodingprocessusedbyH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC,withoutanyapparentsacrificeofcompressioncapability.
Thisintentionaldesignfeatureal-lowsfortheimplementationofMVCdecoderstorequireonlyarathersimpleandstraightforwardchangetoexistingH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCdecodingchipsets.
Asaresultoftheseadvantages,thisproposalwasselectedasthestartingpointoftheMVCproject–formingwhatwascalledthejointmul-tiviewmodel(JMVM)version1.
0.
Inthesix-monthperiodthatfollowedtheresponsestoCfP,athoroughevaluationofthecodingschemedescribedin[25]wasmade.
Thisproposalmadeuseofhierarchicalpredictioninbothtimeandviewdimensionstoachievehighcompressionperformance.
However,viewswereencodedinaninterleCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avedmanneronagroup-of-picture(GOP)basis,whichresultedinasignificantdelayanddidnotallowforsimultaneousdecodingandoutputofviewsatagiventimeinstant.
Anumberofcon-tributionsweremadetoproposeadifferentapproachforrefer-encepicturemanagementandatime-firstcodingschemetoreduceencoding/decodingdelayandenableparallelinputandoutputofviews[26]-[29].
Theseproposalswereadoptedintothedesignatthestagereferredtoasjointmultiviewmodel(JMVM)version2.
0[30],whichwasanearlydraftofthestandardthatestablishedthebasicprinciplesoftheeventualMVCstandard.
DuringthedevelopmentofMVC,anumberofmacroblock-levelcodingtoolswerealsoexplored,includingthefollowing:Illuminationcompensation:Theobjectiveofthistoolistocompensateforilluminationdifferencesaspartoftheinter-viewpredictionprocess[31][32].
Adaptivereferencefiltering:ItwasobservedbyLai,etal.
[33][34]thatthereareothertypesofmismatchespresentinmultiviewvideoinadditiontoilluminationdifferences,whichledtothedevelopmentofanadap-tivereferencefilteringschemetocompensateforfocusmismatchesbetweendifferentviews.
Motionskipmode:Notingthecorrelationbetweenmo-tionvectorsindifferentviews,thismethodinfersmo-tionvectorsfrominter-viewreferencepictures[35][36].
Viewsynthesisprediction:Thiscodingtechniquepre-dictsapictureinthecurrentviewfromsynthesizedref-erencesgeneratedfromneighboringviews[37]-[39].
Itwasshownthatadditionalcodinggainscouldbeachievedbyusingtheseblock-levelcodingtools.
Inananalysisofthecodinggainsofferedbybothilluminationcompensationandmotionskipmodethatwasreportedin[40],anCOLOR:#000000;BACKGROUND-COLOR:#ffff00">averagebitrate2Inthatcomparison,theanchorbitstreamsusedforthesubjectiveevalua-tiontestingdidnotuseamulti-levelhierarchicalpredictionreferencingstruc-ture(asthistypeofreferencinghadnotyetbecomewellestablishedinindus-trypractice).
Ifsuchhierarchicalreferencinghadbeenusedintheanchors,theestimatedbitrategainswouldlikelyhCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeenmoremodest.
PROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN5reductionof10%(relativetoanMVCcodingdesignwithoutthesetools)wasreportedoverasignificantsetofsequences–withamaximumsequence-specificreductionofapproximately18%.
Whilethegainswerenotable,thesetoolswerenotadoptedintotheMVCstandardsincetheywouldrequiresyn-taxanddesignchangesaffectinglowlevelsoftheencodinganddecodingprocess(withinthemacroblocklevel).
Itwasbelievedthattheseimplementationconcernsoutweighedthecodinggainbenefitsatthetime.
Therewasalsosomeconcernthatthebenefitsoftheproposedtechniquesmightbereducedbyhigherqualityvideoacquisitionandpre-processingprac-tices.
However,asthe3Dmarketmatures,thebenefitsofblock-levelcodingtoolsmayberevisitedinthespecificationoffuture3Dvideoformats.
IV.
H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCBASICSMVCwasstandardizedasanextensionofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC.
Inordertokeepthepaperself-contained,thefollowingbriefdescriptionofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCislimitedtothosekeyfeaturesthatarerelevantforunderstandingtheconceptsofextendingH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCtowardsmultiviewvideocod-ing.
FormoredetailedinformationaboutH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC,thereaderisreferredtothestandarditself[1]andthevariousoverviewpapersthathCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avediscussedit(e.
g.
,[41]-[43]).
Conceptually,thedesignofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCcoversaVideoCodingLayer(VCL)andaNetworkAbstractionLayer(NAL).
WhiletheVCLcreatesacodedrepresentationofthesourcecontent,theNALformatsthesedataandprovidesheaderinformationinawaythatenablessimpleandeffectivecustomizationoftheuseofVCLdataforabroadvarietyofsystems.
A.
NetworkAbstractionLayer(NAL)AcodedH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCvideodatastreamisorgan-izedintoNALunits,whicharepacketsthateachcontainanintegernumberofbytes.
ANALunitstartswithaone-byteindicatorofthetypeofdataintheNALunit.
Theremainingbytesrepresentpayloaddata.
NALunitsareclassifiedintovideocodinglayer(VCL)NALunits,whichcontaincodeddataforareasofthepicturecontent(codedslicesorslicedatapartitions),andnon-VCLNALunits,whichcontainassociatedadditionalinformation.
Twokeytypesofnon-VCLNALunitsaretheparametersetsandthesupplementalenhancementin-formation(SEI)messages.
Thesequenceandpictureparame-tersetscontaininfrequentlychanginginformationforacodedvideosequence.
SEImessagesdonotaffectthecoredecodingprocessofthesamplesofacodedvideosequence.
However,theyprovideadditionalinformationtoassistthedecodingprocessoraffectsubsequentprocessingsuchasbitstreamma-nipulationordisplay.
ThesetofconsecutiveNALunitsasso-ciatedwithasinglecodedpictureisreferredtoasanaccessunit.
Asetofconsecutiveaccessunitswithcertainpropertiesisreferredtoasacodedvideosequence.
Acodedvideose-quence(togetherwiththeassociatedparametersets)representsanindependentlydecodablepartofavideobitstream.
Acodedvideosequencealwaysstartswithaninstantaneousdecodingrefresh(IDR)accessunit,whichsignalsthattheIDRaccessunitandallaccessunitsthatfollowitinthebitstreamcanbedecodedwithoutdecodinganyofthepicturesthatprecededit.
B.
VideoCodingLayer(VCL)TheVCLofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCfollowstheso-calledblock-basedhybridvideocodingapproach.
AlthoughitsbasicdesignisverysimilartothatofpriorvideocodingstandardssuchasH.
261,MPEG-1,H.
262/MPEG-2,H.
263,orMPEG-4Visual,H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCincludesnewfeaturesthaten-ableittoachieveasignificantimprovementincompressionefficiencyrelativetoanypriorvideocodingstandard[41]-[43].
ThemaindifferencerelativetopreviousstandardsisthegreatlyincreasedflexibilityandadaptabilityoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCdesign.
ThewaypicturesarepartitionedintosmallercodingunitsinH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC,however,followstherathertraditionalconceptofsubdivisionintosliceswhichinturnaresubdividedintomacroblocks.
Eachslicecanbeparsedindependentlyoftheotherslicesinthepicture.
Eachpictureispartitionedintomacroblocksthateachcoversarectangularpictureareaof16*16lumasamplesand,inthecaseofvideoin4:2:0chromasamplingformat,8*8sampleareasofeachofthetwochromacomponents.
Thesamplesofamacroblockareeitherspatiallyortemporallypredicted,andtheresultingpredictionresidualsignalisrepresentedusingtransformcoding.
Dependingonthedegreeoffreedomforgeneratingthepredictionsignal,H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCsupportsthreebasicslicecodingtypesthatspecifythetypesofcodingsupportedforthemacroblockswithintheslice:Islices,inwhicheachmacroblockusesintra-picturecodingusingspatialpredictionfromneighboringregions,Pslices,whichsupportbothintra-picturecodingandin-ter-picturepredictivecodingusingonepredictionsignalforeachpredictedregion,Bslices,whichsupportintra-picturecoding,inter-picturepredictivecoding,andalsointer-picturebi-predictivecodingusingtwopredictionsignalsthatarecombinedwithaweightedCOLOR:#000000;BACKGROUND-COLOR:#ffff00">averagetoformtheregionprediction.
ForIslices,H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCprovidesseveraldirec-tionalspatialintra-picturepredictionmodes,inwhichthepre-dictionsignalisgeneratedbyusingthedecodedsamplesofneighboringblocksthatprecedetheblocktobepredicted(incodinganddecodingorder).
Forthelumacomponent,thein-tra-picturepredictioncanbeappliedtoindividual4*4or8*8lumablockswithinthemacroblock,ortothefull16*16lumaarrayforthemacroblock;whereasforthechromacomponents,itisappliedonafull-macroblockregionbasis.
ForPandBslices,H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCadditionallyper-mitsvariableblocksizemotion-compensatedpredictionwithmultiplereferencepictures.
Themacroblocktypesignalsthepartitioningofamacroblockintoblocksof16*16,16*8,8*16,or8*8lumasamples.
Whenamacroblocktypespecifiespartitioningintofour8*8blocks,eachoftheseso-calledsub-PROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN6macroblockscanbefurthersplitinto8*4,4*8,or4*4blocks,asdeterminedbyasub-macroblocktypeindication.
ForPslic-es,onemotionvectoristransmittedforeachinter-picturepre-dictionblock.
Thereferencepicturetobeusedforinter-picturepredictioncanbeindependentlychosenforeach16*16,16*8,or8*16macroblockmotionpartitionor8*8sub-macroblock.
Theselectionofthereferencepictureissignaledbyarefer-enceindexparameter,whichisanindexintoalist(referredtoaslist0)ofpreviouslycodedreferencepicturesthatarestoredbythedecoderforsuchuseaftertheyhCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeendecoded.
InBslices,twodistinctreferencepicturelistsareused,andforeach16*16,16*8,or8*16macroblockpartitionor8*8sub-macroblock,thepredictionmethodcanbeselectedbe-tweenlist0,list1,orbi-prediction.
List0andlist1predictionrefertointer-picturepredictionusingthereferencepictureatthereferenceindexpositioninreferencepicturelist0and1,respectively,inamannersimilartothatsupportedinPslices.
However,inthebi-predictivemodethepredictionsignalisformedbyaweightedsumofthepredictionvaluesfrombothalist0andlist1predictionsignal.
Inaddition,specialmodesreferredtoasdirectmodesinBslicesandskipmodesinPandBslicesareprovided,whichoperatesimilarlytotheothermodes,butinwhichsuchdataasmotionvectorsandreferenceindicesarederivedfrompropertiesofneighboringpreviously-codedregionsratherthanbeingindicatedexplicitlybysyntaxforthedirectorskipmodemacroblock.
Fortransformcodingofthespatial-domainresidualdiffer-encesignalremainingafterthepredictionprocess,H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCspecifiesasetofintegertransformsofdifferentblocksizes.
Whileforintra-picturecodedmacro-blocksthetransformsizeisdirectlycoupledtothepredictionblocksize,thelumasignalofmotion-compensatedmacrob-locksthatdonotcontainblockssmallerthan8*8canbecodedbyusingeithera4*4or8*8transform.
Forthechromacom-ponents,atwo-stagetransformisemployed,consistingof4*4transformsandanadditionalHadamardtransformoftheresult-ingDCcoefficients.
Asimilarhierarchicaltransformisalsousedforthelumacomponentofmacroblockscodedinthe16*16intra-picturemacroblockcodingmode.
Allinversetransformsarespecifiedbyexactintegeroperations,sothatinverse-transformmismatchesareCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avoided.
H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCusesuniformreconstructionquantizers.
Thereconstruc-tionstepsizeforthequantizeriscontrolledforeachmacrob-lockbyaquantizationparameterQP.
For8-bit-per-samplevideo,52valuesofQPcanbeselected.
TheQPvalueismulti-pliedbyanentryinascalingmatrixtodetermineatransform-frequency-specificquantizationreconstructionstepsizeThescalingoperationsforthequantizationstepsizesarearrangedwithlogarithmicstepsizeincrements,suchthatanincrementoftheQPby6correspondstoadoublingofquantizationstepsize.
Forreducingblockingartifacts,whicharetypicallythemostdisturbingartifactsinblock-basedcoding,H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCspecifiesanadaptivedeblockingfilterthatoperateswithinthemotion-compensatedinter-picturepredictionloop.
H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCsupportstwomethodsofentropycod-ing,whichbothusecontext-basedadaptivitytoimproveper-formancerelativetopriorstandards.
WhileCCOLOR:#000000;BACKGROUND-COLOR:#ffff00">AVLC(context-basedadaptivevariable-lengthcoding)usesvariable-lengthcodesanditsadaptivityisrestrictedtothecodingoftransformcoefficientlevels,CABAC(context-basedadaptivebinaryarithmeticcoding)usesarithmeticcodingandamoresophisti-catedmechanismforemployingstatisticaldependencies,whichleadstotypicalbitratesCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avingsof10-15%relativetoCCOLOR:#000000;BACKGROUND-COLOR:#ffff00">AVLC.
Inadditiontoincreasedflexibilityatthemacroblocklevelandthelowerlevelswithinit,H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCalsoal-lowsmuchmoreflexibilityonapictureandsequencelevelcomparedtopriorvideocodingstandards.
Hereweprimarilyrefertoreferencepicturebufferingandtheassociatedbuffer-ingmemorycontrol.
InH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC,thecodinganddisplayorderofpicturesiscompletelydecoupled.
Further-more,anypicturecanbeusedasreferencepictureformotion-compensatedpredictionofsubsequentpictures,independentofitsslicecodingtypes.
ThebehCOLOR:#000000;BACKGROUND-COLOR:#ffff00">aviorofthedecodedpicturebuffer(DPB),whichcanholdupto16frames(dependingonthesupportedconformancepointandthedecodedpicturesize),canbeadaptivelycontrolledbymemorymanagementcontroloperation(MMCO)commands,andthereferencepic-tureliststhatareusedforcodingofPorBslicescanbearbi-trarilyconstructedfromthepicturesCOLOR:#000000;BACKGROUND-COLOR:#ffff00">availableintheDPBviareferencepicturelistmodification(RPLM)commands.
Forefficientsupportofthecodingofinterlaced-scanvideo,inamannersimilartopriorvideocodingstandards,acodedpicturemayeithercomprisethesetofslicesrepresentingacompletevideoframeorofjustoneofthetwofieldsofalter-natinglinesinsuchaframe.
Additionally,H.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCsupportsamacroblock-adaptiveswitchingbetweenframeandfieldcoding.
Inthisadaptiveoperation,each16*32regioninaframeistreatedasasinglecodingunitreferredtoasamacroblockpair,whichcanbeeithertransmittedastwomacroblocksrepresentingvertically-neighboring16*16rec-tangularareasintheframe,orasmacroblocksformedfromthede-interleCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avedlinesofthetopandbottomfieldsinthe16*32region.
Thisschemeisreferredtoasmacroblock-adaptiveframe-fieldcoding(MBAFF).
Togetherthesingle-fieldpicturecodingandMBAFFcodingfeaturesaresometimesreferredtoasinterlacecodingtools.
V.
EXTENDINGH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCFORMULTIVIEWThemostrecentmajorextensionoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandard[1]istheMultiviewVideoCoding(MVC)de-sign[11].
SeveralkeyfeaturesofMVCarereviewedbelow;someofwhichhCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avealsobeencoveredin[10]and[44].
Sev-eralotheraspectsoftheMVCdesignwerefurtherelaboratedonin[44],includingrandomaccessandviewswitching,ex-tractionofoperationpoints(setsofcodedviewsatparticularlevelsofanestedtemporalreferencingstructure)ofanMVCbitstreamforadaptationtonetworkanddeviceconstraints,parallelprocessing,andadescriptionofseveralnewlyadoptedPROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN7SEImessagesthatarerelevantformultiviewvideobitstreams.
AnanalysisofMVCdecodedpicturebufferrequirementswasalsoprovidedinthatwork.
A.
BitstreamStructureAkeyaspectoftheMVCdesignisthatitismandatoryforthecompressedmultiviewstreamtoincludeabaseviewbit-stream,whichiscodedindependentlyfromallotherviewsinamannercompatiblewithdecodersforsingle-viewprofileofthestandard,suchastheHighprofileortheConstrainedBaselineprofile.
Thisrequirementenablesavarietyofusescasesthatneeda2Dversionofthecontenttobeeasilyextractedanddecoded.
Forinstance,intelevisionbroadcast,thebaseviewcouldbeextractedanddecodedbylegacyreceivers,whilenewer3Dreceiverscoulddecodethecomplete3Dbitstreamincludingnon-baseviews.
AsdescribedinSectionIV.
A,codeddatainH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCisorganizedintoNALunits.
ThereexistvarioustypesofNALunits,someofwhicharedesignatedforcodedvideopic-tures,whileothersfornon-picturedatasuchasparametersetsandSEImessages.
MVCmakesuseoftheNALunittypestructuretoprovidebackwardcompatibilityformultiviewvid-eo.
SliceofIDRpictureNUT=5Sliceofnon-IDRpictureNUT=1Sliceofnon-IDRpictureNUT=1SPSNUT=7profile_idclevel_idcconstraint_setX_flagsBaseView:NALunitsthataredecodedbylegacyCOLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCdecodersSliceextensionNUT=20SliceextensionNUT=20Sliceextensionhassameslice-levelsyntaxasbaseviewNon-BaseView:NALunitsthataredecodedbyMVCdecoders,anddiscardedbylegacyCOLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCdecodersSubsetSPSNUT=15SubsetSPSincludesSPSsyntaxandSPSMVCextensionsyntaxViewidentificationViewdependenciesMVCprofile/levelSliceofIDRpictureNUT=5Sliceofnon-IDRpictureNUT=1Sliceofnon-IDRpictureNUT=1SPSNUT=7profile_idclevel_idcconstraint_setX_flagsBaseView:NALunitsthataredecodedbylegacyCOLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCdecodersSliceextensionNUT=20SliceextensionNUT=20Sliceextensionhassameslice-levelsyntaxasbaseviewNon-BaseView:NALunitsthataredecodedbyMVCdecoders,anddiscardedbylegacyCOLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCdecodersSubsetSPSNUT=15SubsetSPSincludesSPSsyntaxandSPSMVCextensionsyntaxViewidentificationViewdependenciesMVCprofile/levelFig.
2.
StructureofanMVCbitstreamincludingNALunitsthatareassoci-atedwithabaseviewandNALunitsthatareassociatedwithanon-baseview.
NALunittype(NUT)indicatorsareusedtodistinguishdifferenttypesofdatathatarecarriedinthebitstream.
Toachievethiscompatibility,thevideodataassociatedwithabaseviewisencapsulatedinNALunitsthathCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avepreviouslybeendefinedforthe2Dvideo,whilethevideodataassociatedwiththeadditionalviewsareencapsulatedinanextensionNALunittypethatisusedforbothscalablevideocoding(SVC)[45]andmultiviewvideo.
Aflagisspecifiedtodistin-guishwhethertheNALunitisassociatedwithanSVCorMVCbitstream.
ThebaseviewbitstreamconformstoexistingH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCprofilesforsingle-viewvideo,e.
g.
,Highprofile,anddecodersconformingtoanexistingsingleviewprofilewillignoreanddiscardtheNALunitsthatcontainthedataforthenon-baseviewssincetheywouldnotrecognizethoseNALunittypes.
DecodingtheadditionalviewswiththesenewNALunittypeswouldrequireadecoderthatrecog-nizestheextensionNALunittypeandconformstooneoftheMVCprofiles.
ThebasicstructureoftheMVCbitstreamin-cludingsomeNALunitsassociatedwithabaseviewandsomeNALunitsassociatedwithanon-baseviewisshowninFig.
2.
Furtherdiscussionofthehigh-levelsyntaxisgivenbelow.
MVCprofilesandlevelsarealsodiscussedlaterinthissec-tion.
B.
EnablingInter-viewPredictionThebasicconceptofinter-viewprediction,whichisem-ployedinallofthedescribeddesignsforefficientmultiviewvideocoding,istoexploitbothspatialandtemporalredun-dancyforcompression.
Sincethecameras(orrenderedview-pointperspectives)ofamultiviewscenariotypicallycapturethesamescenefromnearbyviewpoints,substantialinter-viewredundancyispresent.
AsamplepredictionstructureisshowninFig.
3.
Picturesarenotonlypredictedfromtemporalrefer-ences,butalsofrominter-viewreferences.
Thepredictionisadaptive,sothebestpredictoramongtemporalandinter-viewreferencescanbeselectedonablockbasisintermsofrate-distortioncost.
Fig.
3.
Illustrationofinter-viewpredictioninMVC.
Inter-viewpredictionisakeyfeatureoftheMVCdesign,anditisenabledinawaythatmakesuseoftheflexiblerefer-encepicturemanagementcapabilitiesthathadalreadybeendesignedintoH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC,bymakingthedecodedpicturesfromotherviewsCOLOR:#000000;BACKGROUND-COLOR:#ffff00">availableinthereferencepicturelistsforusebytheinter-picturepredictionprocessing.
Specifi-cally,thereferencepicturelistsaremaintainedforeachpicturetobedecodedinagivenview.
Eachsuchlistisinitializedasusualforsingle-viewvideo,whichwouldincludethetemporalreferencepicturesthatmaybeusedtopredictthecurrentpic-ture.
Additionally,inter-viewreferencepicturesareincludedinthelistandaretherebyalsomadeCOLOR:#000000;BACKGROUND-COLOR:#ffff00">availableforpredictionofthecurrentpicture.
AccordingtotheMVCspecification,inter-viewreferencepicturesmustbecontainedwithinthesameaccessunitasthecurrentpicture,whereanaccessunitcontainsalltheNALunitspertainingtoacertaincaptureordisplaytimeinstant.
TheMVCdesigndoesnotallowthepredictionofapictureinoneviewatagiventimeusingapicturefromanotherviewatadifferenttime.
Thiswouldinvolveinter-viewpredictionacrossdifferentaccessunits,whichwouldincuradditionalcomplex-ityforlimitedcodingbenefits.
Tokeepthemanagementofreferencepicturesconsistentwiththatforsingle-viewvideo,allthememorymanagementcontroloperationcommandsthatmaybesignaledthroughanH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCbitstreamapplytooneparticularviewinPROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN8whichtheassociatedsyntaxelementsappear.
Thesameistruefortheslidingwindowandadaptivememorycontrolprocessesthatcanbeusedtomarkpicturesasnotbeingusedforrefer-ence.
ThereferencepicturemarkingprocessofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCisappliedindependentlyforeachview,sothattheen-codercanusetheCOLOR:#000000;BACKGROUND-COLOR:#ffff00">availabledecodermemorycapacityinaflexiblemanner.
Moreover,justasitispossibleforanencodertore-orderthepositionsofthereferencepicturesinareferencepicturelistthatincludestemporalreferencepictures,itcanalsoplacetheinter-viewreferencepicturesatanydesiredposi-tionsinthelists.
Anextendedsetofre-orderingcommandsareprovidedintheMVCspecificationforthispurpose.
Itisimportanttoemphasizethatthecoremacroblock-levelandlower-leveldecodingmodulesofanMVCdecoderarethesame,regardlessofwhetherareferencepictureisatemporalreferenceoraninter-viewreference.
Thisdistinctionisman-agedatahigherlevelofthedecodingprocess.
Intermsofsyntax,supportingMVConlyinvolvessmallchangestohigh-levelsyntax,e.
g.
,anindicationofthepredic-tiondependencyasdiscussedinthenextsubsection.
Amajorconsequenceofnotrequiringchangestolowerlevelsofthesyntax(atthemacroblocklevelandbelowit)isthatMVCiscompatiblewithexistinghardwarefordecodingsingle-viewvideowithH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC.
Inotherwords,supportingMVCaspartofanexistingH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCdecodershouldnotrequiresubstantialdesignchanges.
SinceMVCintroducesdependenciesbetweenviews,ran-domaccessmustalsobeconsideredintheviewdimension.
Specifically,inadditiontotheviewstobeaccessed(calledthetargetviews),anyviewsonwhichtheydependforpurposesofinter-viewreferencingalsoneedtobeaccessedanddecoded,whichtypicallyrequiressomeadditionaldecodingtimeordelay.
Forapplicationsinwhichrandomaccessorviewswitchingisimportant,thepredictionstructurecanbede-signedtominimizeaccessdelay,andtheMVCdesignpro-videsawayforanencodertodescribethepredictionstructureforthispurpose.
Toachieveaccesstoaparticularpictureinagivenview,thedecodershouldfirstdetermineanappropriateaccesspoint.
InH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC,eachinstantaneousdecodingrefresh(IDR)pictureprovidesacleanrandomaccesspoint,sincethesepicturescanbeindependentlydecodedandallthecodedpicturesthatfollowtheminbitstreamordercanalsobede-codedwithouttemporalpredictionfromanypicturedecodedpriortotheIDRpicture.
InthecontextofMVC,anIDRpic-tureinagivenviewprohibitstheuseoftemporalpredictionforanyoftheviewsonwhichaparticularviewdependsatthatparticularinstantoftime;however,inter-viewpredictionmaybeusedforencodingthenon-baseviewsofanIDRpicture.
Thisabilitytouseinter-viewpredictionforencodinganIDRpicturereducesthebitrateneededtoencodethenon-baseviews,whilestillenablingrandomaccessatthattemporallo-cationinthebitstream.
Additionally,MVCalsointroducesanadditionalpicturetype,referredtoasananchorpictureforaview.
AnchorpicturesaresimilartoIDRpicturesinthattheydonotusetemporalpredictionfortheencodingofanyviewonwhichagivenviewdepends,althoughtheydoallowinter-viewpredictionfromotherviewswithinthesameaccessunit.
Moreover,itisprohibitedforanypicturethatfollowsthean-chorpictureinbothbitstreamorderanddisplayordertouseanypicturethatprecedestheanchorpictureinbitstreamorderasareferenceforinter-pictureprediction,andforanypicturethatprecedestheanchorpictureindecodingordertofollowitindisplayorder.
Thisprovidesacleanrandomaccesspointforaccesstoagivenview.
ThedifferencebetweenanchorpicturesandIDRpicturesissimilartothedifferencebetweenthe"openGOP"and"closedGOP"conceptsthatpreviouslyappliedintheH.
262/MPEG-2context3,withclosedGOPsbeingassoci-atedwithIDRpicturesandopenGOPsbeingassociatedwithanchorpictures[44].
Withananchorpicture,itispermissibletousepicturesthatprecedetheanchorpictureinbitstreamorderasreferencepicturesforinter-picturepredictionofpic-turesthatfollowaftertheanchorpictureinbitstreamorder,butonlyifthepicturesthatusethistypeofreferencingprecedetheanchorpictureindisplayorder.
InMVC,bothIDRandanchorpicturesareefficientlycoded,andtheyenablerandomaccessinthetimeandviewdimensions.
C.
High-levelSyntaxThedecodingprocessofMVCrequiresseveraladditionstothehigh-levelsyntax,whichareprimarilysignaledthroughamultiviewextensionofthesequenceparameterset(SPS)de-finedbyH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC.
Threeimportantpiecesofin-formationarecarriedintheSPSextension:ViewidentificationViewdependencyinformationLevelindexforoperationpointsTheviewidentificationpartincludesanindicationoftheto-talnumberofviews,aswellasalistingofviewidentifiers.
Theviewidentifiersareimportantforassociatingaparticularviewtoaspecificindex,whiletheorderoftheviewidentifierssignalsthevieworderindex.
Thevieworderindexiscriticaltothedecodingprocessasitdefinestheorderinwhichviewsaredecoded.
Theviewdependencyinformationiscomposedofasetofsignalsthatindicatethenumberofinter-viewreferencepic-turesforeachofthetworeferencepictureliststhatareusedinthepredictionprocess,aswellastheviewsthatmaybeusedforpredictingaparticularview.
Separateviewdependencyinformationisprovidedforanchorandnon-anchorpicturestoprovidesomeflexibilityinthepredictionwhilenotover-burdeningdecoderswithdependencyinformationthatcouldchangeforeachunitoftime.
Fornon-anchorpictures,theviewdependencyonlyindicatesthatagivensetofviewsmaybeusedforinter-viewprediction.
ThereisadditionalsignalingintheNALunitheaderindicatingwhetheraparticularviewatagiventimemaybeusedforinter-viewreferenceforanyother3ForthosefamiliarwiththemoremodernversionofthisconceptasfoundinH.
264/MPEG-4,anMVCanchorpictureisalsoanalogoustotheuseoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCrecoverypointSEImessagewitharecoveryframecountequalto0.
PROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN9pictureinthesameaccessunit.
Theviewdependencyinforma-tionintheSPSisusedtogetherwiththissyntaxelementintheNALunitheadertocreatereferencepictureliststhatincludeinter-viewreferences,asdescribedintheprevioussubsection.
ThefinalportionoftheSPSextensionisthesignalingoflevelinformationandinformationabouttheoperatingpointstowhichitcorrespond.
Thelevelindexisanindicatoroftheresourcerequirementsforadecoderthatconformstoaparticu-larlevel;itismainlyusedtoestablishaboundonthecomplex-ityofadecoderandisdiscussedfurtherbelow.
InthecontextofMVC,anoperatingpointcorrespondstoaspecifictemporalsubsetandasetofviewsincludingthoseintendedforoutputandtheviewsthattheydependon.
Forexample,anMVCbit-streamwith8viewsmayprovideinformationforseveraloper-atingpoints,e.
g.
,onecorrespondingtoall8viewstogether,anothercorrespondingtoastereopair,andanothercorre-spondingtoasetofthreeparticularviews.
AccordingtotheMVCstandard,multiplelevelvaluescouldbesignaledaspartoftheSPSextension,witheachlevelbeingassociatedwithaparticularoperatingpoint.
Thesyntaxindicatesthenumberofviewsthataretargetedforoutputaswellasthenumberofviewsthatwouldberequiredfordecodingparticularoperatingpoints.
ConstrainedBaselineIandPsliceMCpredictionIn-loopdeblockingIntrapredictionCCOLOR:#000000;BACKGROUND-COLOR:#ffff00">AVLCBsliceFieldcodingMBAFFWeightedpredictionCABACHigh8x8spatialprediction8x8transformMonochromeformatScalingmatricesInter-viewprediction(2views)StereoStereoHighHighMultiviewMultiviewHighHighInter-viewprediction(morethan2views)CompatiblewithbothMultiviewHighandStereoHighprofilesInter-viewPredictionwithFieldandMBAFF(2views)Fig.
4.
IllustrationofMVCprofiles,consistingoftheMultiviewHighandStereoHighprofiles,togetherwithillustrationofthefeaturescompatiblewithbothprofilesandprofilesthatcanbeusedfortheencodingofthebaseview.
D.
ProfilesandLevelsAswithpriorvideocodingstandards,profilesdeterminethesubsetofcodingtoolsthatmustbesupportedbyconformingdecoders.
TherearetwoprofilescurrentlydefinedbyMVCwithsupportformorethanoneview:theMultiviewHighpro-fileandtheStereoHighprofile.
BotharebasedontheHighprofileofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC,withafewdifferences.
TheMultiviewHighprofilesupportsmultipleviewsanddoesnotsupportinterlacecodingtools.
TheStereoHighprofileislimitedtotwoviews,butdoessupportinterlacecodingtools.
Foreitheroftheseprofiles,thebaseviewcanbeencodedusingeithertheHighprofileofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC,oramoreconstrainedprofileknownastheConstrainedBaselineprofilewhichwasaddedtothestandardmorerecently[12].
WhentheHighprofileisusedforthebaseviewfortheMul-tiviewHighprofile,theinterlacecodingtools(fieldpicturecodingandMBAFF),whichareordinarilysupportedintheHighprofile,cannotbeusedinthebaselayersincetheyarenotsupportedintheMultiviewHighprofile.
(TheConstrainedBaselineprofiledoesnotsupportinterlacecodingtools.
)AnillustrationoftheseprofilespecificationsrelativetotheHighandConstrainedBaselineprofilesofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCisprovidedinFig.
4.
ItispossibletohCOLOR:#000000;BACKGROUND-COLOR:#ffff00">aveabitstreamthatconformstoboththeStereoHighprofileandMultiviewHighprofile,whenthereareonlytwoviewsthatarecodedandtheinterlacecodingtoolsarenotused.
Inthiscase,aflagsig-nalingtheircompatibilityisset.
LevelsimposeconstraintsonthebitstreamsproducedbyMVCencoders,toestablishboundsonthenecessarydecoderresourcesandcomplexity.
Thelevellimitsincludelimitsontheamountofframememoryrequiredforthedecodingofabitstream,themaximumthroughputintermsofmacroblockspersecond,maximumpicturesize,overallbitrate,etc.
ThegeneralapproachtodefininglevellimitsinMVCwastoenabletherepurposingofthedecodingresourcesofsingle-viewdecodersforthecreationofmultiviewdecoders.
Inthisway,somelevellimitsareunchanged,suchastheoverallbitrate;inthisway,aninputbitstreamcanbeprocessedbyade-coderregardlessofwhetheritencodesasingleviewormulti-pleviews.
However,otherlevellimitsareincreased,suchasforthemaximumdecodedpicturebuffercapacityandthroughput;afixedscalefactoroftwowasappliedtothesedecoderresourcerequirements.
Assumingafixedresolution,thisscalefactorenablesthedecodingofstereovideousingthesamelevelasisspecifiedforsingle-viewvideoatthesameresolution.
Forinstance,thesameLevel4.
0designationisusedforsingle-viewvideoat1920*1080pat24HzusingtheHighprofileandforstereo-viewvideoat1920*1080pat24HzforeachofthetwoviewsusingtheStereoHighprofile.
Todecodeahighernumberofviews,onewouldeitheruseahigherleveland/orreducethespatialortemporalresolutionofthemultiviewvideo.
E.
CodingPerformanceIthasbeenshownthatcodingmultiviewvideowithinter-viewpredictiondoesgivesignificantlybetterresultscomparedtoindependentcoding[47].
Forsomecases,gainsashighas3dB,roughlycorrespondingtoa50%sCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avingsinbitrate,hCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeenreported.
Acomprehensivesetofresultsformultiviewvideocodingoverabroadrangeoftestmaterialwaspresentedin[40]accordingtoasetofcommontestconditionsandtestmaterialspecifiedin[48].
Formultiviewvideowithupto8views,anCOLOR:#000000;BACKGROUND-COLOR:#ffff00">averageof20%reductioninbitratewasreported,relativetothetotalsimulcastbitrate,basedonBjntegaarddeltameasures[49].
Inotherstudies[50],anCOLOR:#000000;BACKGROUND-COLOR:#ffff00">averagereductionof20-30%ofthebitrateforthesecond(dependent)viewoftypicalstereomoviecontentwasreported,withapeakreduc-tionforanindividualtestsequenceof43%ofthebitrateofthedependentview.
Fig.
5showssamplerate-distortioncurvescomparingtheperformanceofsimulcastcodingwiththeper-formanceofMVCreferencesoftwarethatemployshierarchi-PROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN10calpredictionsinboththetemporalandviewdimensions.
Ballroom31323334353637383940020040060080010001200140016001800Bitrate(Kb/s)PSNR(db)SimulcastMVCRace1323334353637383940414202004006008001000120014001600Bitrate(Kb/s)PSNR(db)SimulcastMVCRena3536373839404142434445460100200300400500600700800900Bitrate(Kb/s)PSNR(db)SimulcastMVCFig.
5.
SamplecodingresultsforseveralMVCtestsequences,includingBallroom,Race1,andRenasequences,accordingtocommontestcondi-tions[48].
Therearemanypossiblevariationsonthepredictionstruc-tureconsideringbothtemporalandinter-viewdependencies.
Thestructurenotonlyaffectscodingperformance,buthasnotableimpactondelay,memoryrequirementsandrandomaccess.
Ithasbeenconfirmedthatthemajorityofgainsareobtainedusinginter-viewpredictionatanchorpositions.
AnCOLOR:#000000;BACKGROUND-COLOR:#ffff00">averagedecreaseinbitrateofapproximately5-15%atequiva-lentqualitycouldbeexpectediftheinter-viewpredictionsatnon-anchorpositionsarenotused[51].
Theupsideisthatde-layandrequiredmemorywouldalsobereduced.
Priorstudiesonasymmetricalcodingofstereovideo,inwhichoneoftheviewsisencodedwithlowerqualitythantheother,suggestthatafurthersubstantialsCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avingsinbitrateforthenon-baseviewcouldbeachievedusingthattechnique.
Inthisscheme,oneoftheviewsissignificantlyblurredormorecoarselyquantizedthantheother[52],oriscodedwithare-ducedspatialresolution[53][54],withanimpactonthestereoqualitythatmaybeimperceptible.
Withmixedresolutioncod-ing,ithasbeenreportedthatanadditionalviewcouldbesup-portedwithminimalrateoverhead,e.
g.
,ontheorderof25-30%additionalrateaddedtoabaseviewencodingforcodingtheotherviewatquarterresolution.
Furtherstudyisneededtounderstandhowthisphenomenonextendstomultiviewvideowithmorethantwoviews.
Thecurrently-standardizedMVCdesignprovidestheencoderwithagreatdealoffreedomtoselecttheencodedfidelityforeachviewandtoperformpre-processingsuchasblurringifdesired;however,itusesthesamesamplearrayresolutionfortheencodingofallviews.
F.
SEIMessagesforMultiviewVideoSeveralnewSEImessagesformultiviewvideoapplicationshCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avealsobeenspecifiedaspartoftheMVCextensionofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC.
However,itshouldbenotedthat,ingeneral,SEImessagesonlysupplysupplementalinformationthatisnotusedwithinthestandardizedprocessforthedecod-ingofthesamplevaluesofthecodedpictures,andtheuseofanygivenSEImessagemaynotbenecessaryorappropriateinsomeparticularMVCapplicationenvironment.
Abriefsum-maryofthesemessagesandtheirprimaryintendedusesareincludedbelow.
ParalleldecodinginformationSEImessage:indicatesthattheviewsofanaccessunitareencodedwithcertainconstraintsthatenableparalleldecoding.
Specifically,itsignalsalimita-tionthathasbeenimposedbytheMVCencoderwherebyamacroblockinacertainviewisonlyallowedtodependonreconstructionvaluesofasubsetofmacroblocksinotherviews.
Byconstrainingthereferencearea,itispossibletoen-ablebetterparallelizationinthedecodingprocess[44].
MVCscalablenestingSEImessage:enablesthereuseofex-istingSEImessagesinthemultiviewvideocontextbyindicat-ingtheviewsortemporallevelstowhichthemessagesapply.
ViewscalabilityinformationSEImessage:containsviewandscalabilityinformationforparticularoperationpoints(setsofcodedviewsatparticularlevelsofanestedtemporalrefer-encingstructure)inthecodedvideosequence.
Informationsuchasbitrateandframerate,amongothers,aresignaledaspartofthemessageforthesubsetoftheoperationpoints.
Thisinformationcanbeusefultoguideabitstreamextractionprocess[44].
MultiviewsceneinformationSEImessage:indicatesthemaximumdisparityamongmultipleviewcomponentsinanaccessunit.
Thismessagecanbeusedforprocessingthede-codedviewcomponentspriortorenderingona3Ddisplay.
Itmayalsobeusefulintheplacementofgraphicoverlays,subti-PROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN11tles,andcaptionsina3Dscene.
MultiviewacquisitioninformationSEImessage:thisSEImessagespecifiesvariousparametersoftheacquisitionenvi-ronment,andspecifically,theintrinsicandextrinsiccameraparameters.
Theseparametersareusefulforviewwarpingandinterpolation,aswellassolvingothercorrespondenceprob-lemsmentionedaboveinsectionII.
B.
Non-requiredviewcomponentSEImessage:indicatesthataparticularviewcomponentisnotneededfordecoding.
ThismayoccurifaparticularsetofviewshCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeenidentifiedforoutputandthereareotherviewsinthebitstreamthatthesetargetoutputviewsdonotdependon.
ViewdependencychangeSEImessage:withthisSEImes-sage,itispossibletosignalchangesintheviewdependencystructure.
OperationpointnotpresentSEImessage:indicatesopera-tionpointsthatarenotpresentinthebitstream.
Thismaybeusefulinstreamingandnetworkingscenariosthatareconsider-ingCOLOR:#000000;BACKGROUND-COLOR:#ffff00">availableoperationpointsinthecurrentbitstreamthatcouldsatisfynetworkordeviceconstraints.
BaseviewtemporalHRDSEImessage:whenpresent,thisSEImessageisassociatedwithanIDRaccessunitandsignalsinformationrelevanttothehypotheticalreferencedecoder(HRD)parametersassociatedwiththebaseview.
VI.
FRAME-COMPATIBLESTEREOENCODINGFORMATSFramecompatibleformatsrefertoaclassofstereovideofor-matsinwhichthetwostereoviewsareessentiallymultiplexedintoasinglecodedframeorsequenceofframes.
Somecom-monsuchformatsareshowninFig.
6.
OthercommonnamesincludestereointerleCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avingorspatial/temporalmultiplexingformats.
Inthefollowing,ageneraloverviewoftheseformatsalongwiththekeybenefitsanddrawbacksarediscussed.
ThesignalingfortheseformatsthathasbeenstandardizedaspartoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandardisalsodescribed.
A.
BasicPrinciplesWithaframe-compatibleformat,theleftandrightviewsarepackedtogetherinthesamplesofasinglevideoframe.
Insuchaformat,halfofthecodedsamplesrepresenttheleftviewandtheotherhalfrepresenttherightview.
Thus,eachcodedviewhashalftheresolutionofthefullcodedframe.
Thereisavari-etyofoptionsCOLOR:#000000;BACKGROUND-COLOR:#ffff00">availableforhowthepackingcanbeperformed.
Forexample,eachviewmayhCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avehalfhorizontalresolutionorhalfverticalresolution.
Thetwosuchhalf-resolutionviewscanbeinterleCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avedinalternatingsamplesofeachcolumnorrow,respectively,orcanbeplacednexttoeachotherinarrange-mentsknownastheside-by-sideandtop-bottompackings(seeFig.
6).
Thetop-bottompackingisalsosometimesreferredtoasover-underpacking[55].
Alternatively,a"checkerboard"(quincunx)samplingmaybeappliedtoeachview,withthetwoviewsinterleCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avedinalternatingsamplesinboththehori-zontalandverticaldimensions(asalsoshowninFig.
6).
Temporalmultiplexingisalsopossible.
Inthisapproach,theleftandrightviewswouldbeinterleCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avedasalternatingframesorfieldsofacodedvideosequence.
Theseformatsarereferredtoasframesequentialandfieldsequential.
Theframerateofeachviewmaybereducedsothattheamountofdataisequivalenttothatofasingleview.
Fig.
6.
Commonframe-compatibleformatswhere'x'representsthesamplesfromoneviewand'o'representsthesamplesfromtheotherview.
Frame-compatibleformatshCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avereceivedconsiderableatten-tionfromthebroadcastindustrysincetheyfacilitatetheintro-ductionofstereoscopicservicesthroughexistinginfrastructureandequipment.
Thecodedvideocanbeprocessedbyencodersanddecodersthatwerenotspecificallydesignedtohandlestereovideo–suchthatonlythedisplaysubsystemthatfol-lowsthedecodingprocessneedstobealteredtosupport3D.
Representingthestereovideoinawaythatismaximallycom-patiblewithexistingencoding,decodinganddeliveryinfra-structureisthemajoradvantageofthisformat.
Thevideocanbecompressedwithexistingencoders,transmittedthroughexistingchannels,anddecodedbyexistingreceivers.
Onlythefinaldisplaystagerequiressomecustomizationforrecogniz-ingandproperlyrenderingthevideotoenablea3Dviewingexperience.
Althoughcompressionperformancemayvaryde-pendingonthecontent,theacquisitionandpre-processingtechnology,andtheframepackingarrangementthatareused,thebitratesforsupportingstereovideointhismannermaynotneedtobesubstantiallyhigherthanforacompressedsingleviewatanequivalentspatialresolution(althoughasomewhathigherbitratemaybedesirable,sincetheframe-compatiblestereovideowouldtendtohCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avehigherspatialfrequencycon-tentcharacteristics).
Thisformatessentiallytunnelsthestereovideothroughexistinghardwareanddeliverychannels.
Duetotheseminimalchanges,stereovideoservicecanbequicklydeployedto3Dcapabledisplays(whicharealreadyCOLOR:#000000;BACKGROUND-COLOR:#ffff00">availableinthemarket–e.
g.
,usingtheHDMI1.
4aspecification[56]).
ThedrawbackofrepresentingthestereosignalinthiswayisthatspatialortemporalresolutionwouldbeonlyhalfofthatPROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN12usedfor2Dvideowiththesame(total)encodedresolution.
Thekeyadditionalissuewithframe-compatibleformatsisdistinguishingtheleftandrightviews.
Toperformthede-interleCOLOR:#000000;BACKGROUND-COLOR:#ffff00">aving,itisnecessaryforreceiverstobeabletoparseandinterpretsomesignalthatindicatesthattheframepackingisbeingused.
Sincethissignalingmaynotbeunderstoodbylegacyreceivers,itmaynotevenbepossibleforsuchdevicestoextract,decodeanddisplaya2Dversionofthe3Dprogram.
However,thismaynotnecessarilybeconsideredsoproblem-atic,asitisnotalwaysconsidereddesirabletoenable2Dvideoextractionfroma3Dstream.
Thecontentproductionpracticesfor2Dand3Dprogramsmaybedifferent,and2Dand3Dversionsofaprogrammaybeediteddifferently(e.
g.
,usingmorefrequentscenecutsandmoreglobalmotionfor2Dprogrammingthanfor3D).
Moreover,thefirmwareonsomedevices,suchascableset-topboxes,couldbeupgradedtounderstandthenewsignalingthatdescribesthevideoformat(althoughthesameisnotnecessarilytrueforbroadcastreceiv-ersandalltypesofequipment).
B.
SignalingThesignalingforacompletesetofframe-compatiblefor-matshasbeenstandardizedwithintheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandardassupplementalenhancementinformation(SEI)mes-sages.
AdecoderthatunderstandstheSEImessagecaninter-prettheformatofthedecodedvideoanddisplaythestereocontentappropriately.
Anearliereditionofthestandardthatwascompletedin2004specifiedastereovideoinformation(SVI)SEImessagethatcouldidentifytwotypesofframe-compatibleencodingforleftandrightviews.
Morespecifically,itwasabletoindicateeitherarow-basedinterleCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avingofviewsthatwouldberepre-sentedasindividualfieldsofavideoframeoratemporalmul-tiplexingofviewswheretheleftandrightviewswouldbeinatemporallyalternatingsequenceofframes.
TheSVISEImes-sagealsohadthecapabilityofindicatingwhethertheencodingofaparticularviewisself-contained,i.
e.
,whethertheframesorfieldscorrespondingtotheleftviewareonlypredictedfromotherframesorfieldsoftheleftview.
Inter-viewpredictionforstereoispossiblewhentheself-containedflagisdisabled.
AlthoughthespecificationoftheSVISEImessageisstillincludedinthecurrentversionofthestandard[1],thefunc-tionalityofthisSEImessagehasrecentlybeenincorporated,alongwithadditionalsignalingcapabilitiesandsupportofvariousotherspatiallymultiplexedformats(asdescribedabove),intoanewSEImessage.
ThustheneweditionofthestandardexpressesapreferencefortheuseofthenewSEImessageratherthantheSVISEImessage.
ThenewSEImes-sageisreferredtoastheframepackingarrangement(FPA)SEImessage.
ItwasspecifiedinanamendmentoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandard[12]andwasincorporatedintothelatestedition[1].
ThisnewSEImessageisthecurrentsug-gestedwaytosignalframe-compatiblestereovideoinforma-tion,anditisabletosignalallofthevariousframepackingarrangementsshowninFig.
6.
Withtheside-by-sideandtop-bottomarrangements,itisalsopossibletosignalwhetheroneoftheviewshasbeenflippedsoastocreateamirrorimageinthehorizontalorverticaldirection,respectively.
Independentoftheframepackingarrangement,theSEImessagealsoindi-cateswhethertheleftandrightviewshCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeensubjecttoaquincunx(checkerboard)sampling.
Forinstance,itispossibletoapplyaquincunxfilterandsub-samplingprocess,butthenrearrangethevideosamplesintoaside-by-sideformat.
SuchschemesarealsosupportedintheFPASEImessage.
Finally,theSEImessageindicateswhethertheupper-leftsampleofapackedframeisfortheleftorrightviewanditalsosupportsadditionalsyntaxtoindicatethepreciserelativegridalignmentpositionsofthesamplesoftheleftandrightviews,usingaprecisionofonesixteenthofthesamplegridspacingbetweentherowsandcolumnsofthedecodedvideoarray.
C.
DiscussionIndustryisnowpreparingfortheintroductionofnew3Dservices.
WiththeexceptionofBlu-rayDiscs,whichwillofferastereoformatwithHDresolutionforeachviewbasedontheStereoHighprofileoftheMVCextensions,themajorityofserviceswillstartbasedonframe-compatibleformatsthatwillhCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avealowerresolutionforeachcodedviewthanthefullreso-lutionofthecodedframe[57].
Somebenefitsanddrawbacksofthevariousformatsarediscussedbelow;furtherdiscussioncanalsobefoundin[57].
Intheproductionanddistributiondomains,theside-by-sideandtop-bottomformatscurrentlyappeartobethemostfa-vored(e.
g.
,in[55]and[58]).
Relativetoroworcolumninter-leCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avingandthecheckerboardformat,thequalityoftherecon-structedstereosignalaftercompressioncanbebettermain-tained.
TheinterleCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avedformatsintroducesignificanthighfre-quencycontentintotheframe-compatiblesignal–therebyrequiringahigherbitrateforencodingwithadequatequality.
Also,theinterleCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avingandcompressionprocesscancreatecross-talkartifactsandcolorbleedingacrossviews.
Fromthepuresamplingperspective,therehCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeensomestudiesthatadvocatedbenefitsofquincunxsampling.
Inpar-ticular,quincunxsamplingpreservesmoreoftheoriginalsig-nalanditsfrequency-domainrepresentationissimilartothatofthehumanvisualsystem.
Theresolutionlossisequallydis-tributedintheverticalandhorizontaldirections.
So,whileitmaynotbeadistribution-friendlyformat,quincunxsamplingfollowedbyarearrangementtoside-by-sideortop-bottomformatcouldpotentiallyleadtohigherqualitycomparedtodirecthorizontalorverticaldecimationoftheleftandrightviewsbyafactoroftwo.
Ontheotherhand,quincunxsam-plingmayintroducehighfrequenciesintothevideosignalthataredifficulttoencode,sinceitcreatesfrequencycontentthatisneitherpurelyverticalnorpurelyhorizontal.
Thismayresultinasignalthatrequiresahigherbitratetoencodewithadequatequality[55].
Anotherissuetoconsiderregardingframe-compatiblefor-matsiswhetherthesourcematerialisinterlaced.
Sincethetop-bottomformatincursaresolutionlossintheverticaldimen-sionandaninterlacedfieldisalreadyhalftheresolutionofthePROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN13decodedframe,theside-by-sideformatisgenerallypreferredoverthetop-bottomformatforinterlacedcontent[55][58].
Sincetherearedisplaysinthemarketthatsupportinter-leCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avedformatsastheirnativedisplayformat,suchaschecker-boardforDLPtelevisionsandroworcolumninterleCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avingforsomeLCD-baseddisplays,itislikelythatthedistributionfor-matswillbeconvertedtothesedisplayformatspriortoreach-ingthedisplay.
ThenewestHigh-DefinitionMultimediaInter-facespecificationbetweenset-topboxesanddisplays,HDMI1.
4a[56],addssupportforthefollowing3Dvideoformatstructures:framepacking(forprogressiveandinterlacedscanformats),side-by-side(halforfullhorizontalresolution),top-bottom(halfverticalresolutiononly),fieldalternating(forinterlacedformats),andlinealternating(forprogressivefor-mats).
4Therefore,thesignalingoftheseformatsoverthedis-playinterfacewouldbenecessaryalongwiththesignalingofthevariousdistributionformats.
TheSEImessagethathasbeenspecifiedinthelatestversionoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandardsupportsabroadsetofpossibleframe-compatibleformats.
Itisexpectedtobeusedthroughoutthedeliverychainfromproductiontodistribution,throughthereceivingdevices,andpossiblyallthewaytothedisplayinsomecases.
Anaturalquestionthatarisesinregardtothedeploymentofframe-compatiblestereovideoishowtothenmigratetoaser-vicethatprovideshigherresolutionforeachview.
VariousapproachestothisquestionarecurrentlyunderstudyintheMPEGstandardizationworkinggroup–enhancingtheresolu-tionofeachviewwithacodedresolutionenhancementbit-streaminalayeredscalablefashion[59].
ThebestapproachforthismayinvolvesomecombinationofMVCwithanothersetofrecentextensionsofH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC–namelythescalablevideocoding(SVC)extension[45]–perhapsalongwithadditionalnewtechnology.
VII.
CONCLUSIONSANDFURTHERWORK3Dvideohasdrawnsignificantattentionrecentlyamongin-dustry,standardizationforums,andacademicresearchers.
Theefficientrepresentationandcompressionofstereoandmul-tiviewvideoisacentralcomponentofany3Dormultiviewsystemsinceitdefinestheformattobeproduced,stored,transmittedanddisplayed.
ThisarticlereviewedtherecentextensionstothewidelydeployedH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstan-dardthatsupport3Dstereoandmultiviewvideo.
TheMVCstandardincludessupportforimprovedcompressionofstereoandmultiviewvideobyenablinginter-viewpredictionaswellastemporalinter-pictureprediction.
Anotherimportantdevel-opmenthasbeentheefficientrepresentation,codingandsig-nalingofframe-compatiblestereovideoformats.
AssociatedstandardsforthetransportandstorageofstereoandmultiviewvideousingH.
222.
0/MPEG-2Systems,RTPandtheISObasemediafileformathCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avealsobeenspecified,andaredescribed4InadditiontotheHDMIformatsrelevanttothispaper,alsotheformatsleftplusdepth(forprogressive-scanformatsonly),leftplusdepth,andgraph-icsplusgraphics-depth(forprogressive-scanformatsonly)arespecified.
in[60].
Wearenowwitnessingtheroll-outofnew3Dservicesandequipmentbasedonthesetechnologiesandstandards.
Asthemarketevolvesandnewtypesofdisplaysandservicesareof-fered,additionalnewtechnologiesandstandardswillneedtobeintroduced.
Forexample,itisanticipatedthatanew3Dvideoformattosupportthegenerationofthelargenumberofviewsrequiredbyauto-stereoscopicdisplayswouldbeneeded.
Solutionsthatconsidertheinclusionofdepthmapinformationforthispurposeareasignificantareaoffocusforfuturede-signs,asdiscussedin[61].
ACKNOWLEDGMENTTheauthorsthanktheexpertsoftheJointVideoTeam(JVT)ofISO/IECMPEGandITU-TVCEGfortheircontribu-tionsandfruitfuldiscussions.
WeespeciallythankthosethathCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avecontributedtothetextediting,softwaredevelopmentandconformancetesting,namelyPhilippMerkle,KarstenMüller,Ye-KuiWang,YingChen,PurvinPandit,AljoschaSmolic,SehoonYea,ShinyaShimizu,HideakiKimata,Chong-SoonLim,DongTian,andTeruhikoSuzuki.
REFERENCES(COLOR:#000000;BACKGROUND-COLOR:#ffff00">Availabilitynote:JointVideoTeam(JVT)documentscitedbelowareCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avail-ableathttp://ftp3.
itu.
int/COLOR:#000000;BACKGROUND-COLOR:#ffff00">av-arch/jvt-site.
)[1]ITU-TandISO/IECJTC1,"Advancedvideocodingforgenericaudio-visualservices",ITU-TRecommendationH.
264andISO/IEC14496-10(MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC),2010.
[2]M.
E.
Lukacs,"Predictivecodingofmulti-viewpointimagesets",Proc.
IEEEInternationalConf.
onAcoustics,SpeechandSignalProcessing,vol.
1,pp.
521-524,Tokyo,Japan,1986.
[3]I.
Dinstein,G.
Guy,J.
Rabany,J.
Tzelgov,andA.
Henik,"Onthecom-pressionofstereoimages:Preliminaryresults",SignalProcessing:Im-ageCommunications,vol.
17,no.
4,pp.
373-382,Aug.
1989.
[4]M.
G.
Perkins,"Datacompressionofstereopairs",IEEETrans.
Com-munications,vol.
40,no.
4,pp.
684-696,April1992.
[5]ITU-TandISO/IECJTC1,"Genericcodingofmovingpicturesandassociatedaudioinformation–Part2:Video",ITU-TRecommendationH.
262andISO/IEC13818-2(MPEG-2Video),1994.
[6]ITU-TandISO/IECJTC1,"FinalDraftAmendment3",Amendment3toITU-TRecommendationH.
262andISO/IEC13818-2(MPEG-2Vid-eo),ISO/IECJTC1/SC29/WG11(MPEG)Doc.
N1366,Sept.
1996.
[7]A.
Puri,R.
V.
Kollarits,andB.
G.
Haskell.
"Stereoscopicvideocom-pressionusingtemporalscalability",Proc.
SPIEConf.
VisualCommu-nicationsandImageProcessing,vol.
2501,pp.
745–756,1995.
[8]X.
ChenandA.
Luthra,"MPEG-2multi-viewprofileanditsapplicationin3DTV",Proc.
SPIEIS&TMultimediaHardwareArchitectures,SanDiego,USA,Vol.
3021,pp.
212-223,February1997.
[9]J.
-R.
Ohm,"Stereo/MultiviewVideoEncodingUsingtheMPEGFamilyofStandards",Proc.
SPIEConf.
StereoscopicDisplaysandVirtualRe-alitySystemsVI,SanJose,CA,Jan.
1999.
[10]G.
J.
Sullivan,"Standards-basedapproachesto3Dandmultiviewvideocoding",Proc.
SPIEConf.
ApplicationsofDigitalImageProcessingXXXII,SanDiego,CA,Aug.
2009.
[11]A.
Vetro,P.
Pandit,H.
Kimata,A.
Smolic,andY.
-K.
Wang,eds.
,"JointDraft8ofMultiviewVideoCoding",JointVideoTeam(JVT)Doc.
JVT-AB204,Hannover,Germany,July2008.
[12]G.
J.
Sullivan,A.
M.
Tourapis,T.
Yamakage,C.
S.
Lim,eds.
,"DraftCOLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCamendmenttexttospecifyConstrainedBaselineprofile,StereoHighprofile,andframepackingSEImessage",JointVideoTeam(JVT)Doc.
JVT-AE204,London,UnitedKingdom,July2009.
[13]MPEGrequirementssub-group,"RequirementsonMulti-viewVideoCodingv.
7",ISO/IECJTC1/SC29/WG11(MPEG)Doc.
N8218,Kla-genfurt,Austria,July2006.
PROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN14[14]J.
KonradandM.
Halle,"3-DDisplaysandSignalProcessing–AnAnswerto3-DIlls",IEEESignalProcessingMagazine,Vol.
24,No.
6,Nov.
2007.
[15]N.
A.
Dodgson,"Autostereoscopic3DDisplays",IEEEComputer,vol.
38,no.
8,pp.
31-36,Aug.
2005.
[16]A.
SmolicandP.
Kauff,"Interactive3-Dvideorepresentationandcod-ingtechnologies",ProceedingsoftheIEEE,vol.
93,no.
1,pp98-110,Jan.
2005.
[17]T.
Fujii,T.
Kimono,andM.
Tanimoto,"Free-viewpointTVsystembasedonray-spacerepresentation",Proc.
SPIEITCom,vol.
4864-22,pp.
175-189,2002.
[18]P.
Kauff,O.
Schreer,andR.
Tanger,"VirtualTeamUserEnvironments-AMixedRealityApproachforImmersiveTele-Collaboration",Int.
WorkshoponImmersiveTelepresence(ITP2002),pp.
1-4,January2002.
[19]I.
Feldmann,O.
Schreer,P.
Kauff,R.
Schfer,Z.
Fei,H.
J.
W.
Belt,.
DivorraEscoda,"ImmersiveMulti-User3DVideoCommunication",Proc.
ofInternationalBroadcastConference(IBC2009),Amsterdam,Netherlands,September2009.
[20]D.
FlorencioandC.
Zhang,"MultiviewvideoCompressionandStream-ingBasedonPredictedViewerPosition",Proc.
IEEEInternationalConf.
onAcoustics,SpeechandSignalProcessing,Taipei,Taiwan,Apr.
2009.
[21]B.
Wilburn,etal.
,"HighPerformanceImagingUsingLargeCameraArrays",ACMTransactionsonGraphics,vol.
24,no.
3,pp.
765-776,July2005.
[22]R.
RaskarandJ.
Tumblin,ComputationalPhotography:MasteringNewTechniquesforLenses,Lighting,andSensors,AKPeters,Ltd.
,ISBN978-1-56881-313-4,2010.
[23]MPEGvideosub-groupchair(J.
-R.
Ohm),"SubmissionsreceivedinCfPonMultiviewVideoCoding",ISO/IECJTC1/SC29/WG11(MPEG)Doc.
M12969,Bangkok,Thailand,Jan.
2006.
[24]MPEGvideoandtestsub-groups,"SubjectivetestresultsfortheCfPonMulti-viewVideoCoding",ISO/IECJTC1/SC29/WG11(MPEG)Doc.
N7799,Bangkok,Thailand,Jan.
2006.
[25]K.
Müller,P.
Merkle,A.
Smolic,andT.
Wiegand,"MultiviewCodingusingCOLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC",ISO/IECJTC1/SC29/WG11(MPEG)Doc.
M12945,Bangkok,Thailand,Jan.
2006.
[26]E.
Martinian,S.
Yea,andA.
Vetro,"ResultsofCoreExperiment1BonMultiviewCoding",ISO/IECJTC1/SC29/WG11(MPEG)Doc.
M13122,Montreux,Switzerland,Apr.
2006.
[27]E.
Martinian,A.
Behrens,J.
Xin,A.
Vetro,andH.
Sun,"ExtensionsofH.
264/COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCforMultiviewVideoCompression",Proc.
IEEEInterna-tionalConf.
onImageProcessing,Atlanta,USA,Oct.
2006[28]MPEGvideosub-group,"TechnologiesunderStudyforReferencePic-tureManagementandHigh-LevelSyntaxforMultiviewVideoCoding",ISO/IECJTC1/SC29/WG11(MPEG)Doc.
N8018,Montreux,Swit-zerland,Apr.
2006.
[29]Y.
-K.
Wang,Y.
Chen,andM.
M.
Hannuksela,"Time-firstcodingformulti-viewvideocoding",JointVideoTeam(JVT)Doc.
JVT-U104,Hangzhou,China,Oct.
2006.
[30]A.
Vetro,Y.
Su,H.
Kimata,andA.
Smolic,eds.
,"JointMultiviewVid-eoModel2.
0",JointVideoTeam(JVT)Doc.
JVT-U207,Hangzhou,China,Oct.
2006.
[31]Y.
L.
Lee,J.
H.
Hur,Y.
K.
Lee,K.
H.
Han,S.
H.
Cho,N.
H.
Hur,J.
W.
Kim,J.
H.
Kim,P.
Lai,A.
Ortega,Y.
Su,P.
Yin,andC.
Gomila,"CE11:Illuminationcompensation",JointVideoTeam(JVT)Doc.
JVT-U052,Hangzhou,China,2006.
[32]J.
H.
Hur,S.
Cho,andY.
L.
Lee,"AdaptivelocalilluminationchangecompensationmethodforH.
264/COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVC-basedmultiviewvideocoding",IEEETransactionsonCircuitsandSystemsforVideoTechnology,vol.
17,no.
11,pp.
1496-1505,Nov.
2007.
[33]P.
Lai,A.
Ortega,P.
Pandit,P.
Yin,andC.
Gomila,"AdaptivereferencefilteringforMVC",JointVideoTeam(JVT)Doc.
JVT-W065,SanJose,CA,April2007.
[34]P.
Lai,A.
Ortega,P.
Pandit,P.
Yin,andC.
Gomila,"Focusmismatchesinmultiviewsystemsandefficientadaptivereferencefilteringformul-tiviewvideocoding",Proc.
SPIEConferenceonVisualCommunica-tionsandImageProcessing,SanJose,CA,Jan.
2008.
[35]H.
S.
Koo,Y.
J.
Jeon,andB.
M.
Jeon,"MVCmotionskipmode",JointVideoTeam(JVT)Doc.
JVT-W081,SanJose,CA,April2007.
[36]H.
S.
Koo,Y.
J.
Jeon,andB.
M.
Jeon,"Motioninformationinferringschemeformulti-viewvideocoding.
IEICETransactionsonCommuni-cations,E91-B(4),pp.
1247-1250,2008.
[37]E.
Martinian,A.
Behrens,J.
Xin,A.
Vetro,"Viewsynthesisformul-tiviewvideocompression",Proc.
PictureCodingSymposium,Beijing,China,2006.
[38]S.
YeaandA.
Vetro,"Viewsynthesispredictionformultiviewvideocoding",ImageCommunication,vol.
24,no.
1-2,pp.
89-100,Jan.
2009.
[39]M.
Kitahara,H.
Kimata,S.
Shimizu,K.
Kamikura,Y.
Yashima,K.
Yamamoto,T.
Yendo,T.
Fujii,andM.
Tanimoto,"Multi-viewvideocodingusingviewinterpolationandreferencepictureselection",Proc.
IEEEInternationalConferenceonMultimedia&Expo,Toronto,Can-ada,pp.
97-100,July2006.
[40]D.
Tian,P.
Pandit,P.
Yin,andC.
Gomila,"StudyofMVCcodingtools",JointVideoTeam(JVT)Doc.
JVT-Y044,Shenzhen,China,Oct.
2007.
[41]T.
Wiegand,G.
J.
Sullivan,G.
Bjntegaard,andA.
Luthra,"OverviewoftheH.
264/COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCvideocodingstandard",IEEETransactionsonCir-cuitsandSystemsforVideoTechnology,vol.
13,no.
7,pp.
560-576,Ju-ly2003.
[42]G.
J.
SullivanandT.
Wiegand,"Videocompression–fromconceptstotheH.
264/COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandard",ProceedingsofIEEE,vol.
93,no.
1,pp.
18-31,Jan.
2005.
[43]D.
Marpe,T.
Wiegand,andG.
J.
Sullivan,"TheH.
264/MPEG4Ad-vancedVideoCodingstandardanditsapplications",IEEECommunica-tionsMagazine,vol.
44,no.
8,pp.
134-144,Aug.
2006.
[44]Y.
Chen,Y.
-K.
Wang,K.
Ugur,M.
M.
Hannuksela,J.
Lainema,andM.
Gabbouj,"3DvideoserviceswiththeemergingMVCstandard",EURASIPJournalonAdvancesinSignalProcessing,2009.
[45]H.
Schwarz,D.
Marpe,andT.
Wiegand:"OverviewoftheScalableVideoCodingExtensionoftheH.
264/COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCStandard",IEEETransac-tionsonCircuitsandSystemsforVideoTechnology,SpecialIssueonScalableVideoCoding,vol.
17,no.
9,pp.
1103-1120,September2007.
[46]Y.
Chen,P.
Pandit,S.
Yea,andC.
S.
Lim,eds.
,"Draftreferencesoft-wareforMVC(JMVC6.
0)",JointVideoTeam(JVT)Doc.
JVT-AE207,London,UnitedKingdom,July2009.
[47]P.
Merkle,A.
Smolic,K.
Mueller,andT.
Wiegand,"EfficientPredictionStructuresforMultiviewVideoCoding",IEEETransactionsonCir-cuitsandSystemsforVideoTechnology,vol.
17,no.
11,Nov.
2007.
[48]Y.
Su,A.
Vetro,andA.
Smolic,"CommonTestConditionsforMul-tiviewVideoCoding",JointVideoTeam(JVT)Doc.
JVT-U211,Hang-zhou,China,October2006.
[49]G.
Bjntegaard,"CalculationofCOLOR:#000000;BACKGROUND-COLOR:#ffff00">averagePSNRdifferencesbetweenRD-curves",ITU-TSG16/Q.
6,Doc.
VCEG-M033,Austin,TX,April2001.
[50]T.
Chen,Y.
Kashiwagi,C.
S.
Lim,andT.
Nishi,"CodingperformanceofStereoHighProfileformoviesequences",JointVideoTeam(JVT)Doc.
JVT-AE022,London,UnitedKingdom,July2009.
[51]M.
DroeseandC.
Clemens,"ResultsofCE1-Donmultiviewvideocoding",ISO/IECJTC1/SC29/WG11(MPEG)Doc.
M13247,Montreux,Switzerland,Apr.
2006.
[52]L.
Stelmach,W.
J.
Tam,D.
Meegan,andA.
Vincent,"Stereoimagequality:effectsofmixedspatio-temporalresolution",IEEETransac-tionsonCircuitsandSystemsforVideoTechnology,Vol.
10,No.
2,pp.
188-193,Mar.
2000.
[53]C.
Fehn,P.
Kauff,S.
Cho,H.
Kwon,N.
Hur,andJ.
Kim,"AsymmetriccodingofstereoscopicvideofortransmissionoverT-DMB",Proc.
3DTV-CON2007,Kos,Greece,May2007.
[54]H.
Brust,A.
Smolic,K.
Müller,G.
Tech,andT.
Wiegand,"MixedReso-lutionCodingofStereoscopicVideoforMobileDevices",Proc.
3DTV-CON2009,Potsdam,Germany,May2009.
[55]DolbyLaboratories,"DolbyOpenSpecificationforFrame-Compatible3DSystems",Issue1,COLOR:#000000;BACKGROUND-COLOR:#ffff00">availableathttp://www.
dolby.
com/uploadedFiles/Assets/US/Doc/Professional/3DFrameCompatibleOpenStandard.
pdf,April2010.
[56]HDMIFounders,"HDMISpecification",version1.
4a,COLOR:#000000;BACKGROUND-COLOR:#ffff00">availableathttp://www.
hdmi.
org/manufacturer/specification.
aspx,March2010.
[57]D.
K.
Broberg,"InfrastructuresforHomeDelivery,Interfacing,Caption-ing,andViewingof3DContent",thisspecialissue.
[58]CableTelevisionLaboratories,"ContentEncodingProfiles3.
0Specifi-cationOC-SP-CEP3.
0-I01-100827",versionI01,COLOR:#000000;BACKGROUND-COLOR:#ffff00">availableathttp://www.
cablelabs.
com/specifications/OC-SP-CEP3.
0-I01-100827.
pdf,August2010.
PROCEEDINGSOFTHEIEEE(2011):VETRO,WIEGAND,SULLIVAN15[59]G.
J.
Sullivan,W.
Husak,A.
LuthraforMPEGRequirementsSub-Group,"Problemstatementforscalableresolutionenhancementofframe-compatiblestereoscopic3Dvideo",ISO/IECJTC1/SC29/WG11(MPEG)Doc.
N11526,Geneva,Switzerland,July2010.
[60]T.
SchierlandS.
Narasimhan,"Transportandstoragesystemsfor3DvideousingMPEG-2systems,RTP,andISOfileformats",thisspecialissue.
[61]K.
Müller,P.
Merkle,andT.
Wiegand,"3Dvideorepresentationusingdepthmaps",thisspecialissue.
AnthonyVetro(S'92–M'96–SM'04–F'11)receivedtheB.
S.
,M.
S.
,andPh.
D.
degreesinelectricalengi-neeringfromPolytechnicUniversity,Brooklyn,NY.
HejoinedMitsubishiElectricResearchLabs,Cam-bridge,MA,in1996,whereheiscurrentlyaGroupManagerresponsibleforresearchandstandardiza-tiononvideocoding,aswellasworkondisplayprocessing,informationsecurity,speechprocessing,andradarimaging.
Hehaspublishedmorethan150papersintheseareas.
HehasalsobeenanactivememberoftheISO/IECandITU-Tstandardizationcommitteesonvideocodingformanyyears,wherehehasservedasanad-hocgroupchairandeditorforseveralprojectsandspecifications.
Mostrecently,hewasakeycontributortotheMultiviewVideoCodingextensionoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandard.
HealsoservesasVice-ChairoftheU.
S.
delegationtoMPEG.
Dr.
VetroisalsoactiveinvariousIEEEconferences,technicalcommittees,andeditorialboards.
HecurrentlyservesontheEditorialBoardsofIEEESignalProcessingMagazineandIEEEMultiMedia,andasanAssociateEdi-torforIEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGYandIEEETRANSACTIONSONIMAGEPROCESSING.
HeservedasChairoftheTechnicalCommitteeonMultimediaSignalProcessingoftheIEEESignalProcessingSocietyandonthesteeringcommitteesforICMEandtheIEEETRANSACTIONSONMULTIMEDIA.
HeservedasanAssociateEditorforIEEESignalProcessingMagazine(2006–2007),asConferenceChairforICCE2006,TutorialsChairforICME2006,andasamemberofthePublica-tionsCommitteeoftheIEEETRANSACTIONSONCONSUMERELECTRONICS(2002–2008).
HeisamemberoftheTechnicalCommitteesonVisualSignalProcessing&Communications,andMultimediaSystems&ApplicationsoftheIEEECircuitsandSystemsSociety.
Hehasalsoreceivedseveralawardsforhisworkontranscoding,includingthe2003IEEECircuitsandSystemsCSVTTransactionsBestPaperAward.
ThomasWiegand(M'05–SM'08–F'11)isaprofessoratthedepartmentofElectricalEngineeringandComputerScienceattheBerlinInstituteofTech-nology,chairingtheImageCommunicationLaboratory,andisjointlyheadingtheImageProcessingdepartmentoftheFraunhoferInstituteforTelecommunications-HeinrichHertzInstitute,Berlin,Germany.
HereceivedtheDipl.
-Ing.
degreeinElectricalEngineeringfromtheTechnicalUniversityofHamburg-Harburg,Germany,in1995andtheDr.
-Ing.
degreefromtheUniversityofErlangen-Nuremberg,Germany,in2000.
HejoinedtheHeinrichHertzInstitutein2000astheheadoftheImageCommunicationgroupintheImageProcessingdepartment.
Hisresearchinterestsincludevideoprocessingandcoding,multimediatransmission,aswellascomputervisionandgraphicsFrom1993to1994,hewasaVisitingResearcheratKobeUniversity,Ja-pan.
In1995,hewasaVisitingScholarattheUniversityofCaliforniaatSantaBarbara,USA.
From1997to1998,hewasaVisitingResearcheratStanfordUniversity,USAandservedasaconsultantto8x8,Inc.
,SantaClara,CA,USA.
From2006-2008,hewasaconsultanttoStreamProcessors,Inc.
,Sunnyvale,CA,USA.
From2007-2009,hewasaconsultanttoSkyfire,Inc.
,MountainView,CA,USA.
Since2006,hehasbeenamemberofthetechni-caladvisoryboardofVidyo,Inc.
,Hackensack,NJ,USA.
Since1995,hehasbeenanactiveparticipantinstandardizationformul-timediawithsuccessfulsubmissionstoITU-TVCEG,ISO/IECMPEG,3GPP,DVB,andIETF.
InOctober2000,hewasappointedastheAssociatedRapporteurofITU-TVCEG.
InDecember2001,hewasappointedastheAssociatedRapporteur/Co-ChairoftheJVT.
InFebruary2002,hewasap-pointedastheEditoroftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCvideocodingstandardanditsextensions(FRExtandSVC).
From2005-2009,hewasCo-ChairofMPEGVideo.
In1998,hereceivedtheSPIEVCIPBestStudentPaperAward.
In2004,hereceivedtheFraunhoferAwardandtheITGAwardoftheGermanSocietyforInformationTechnology.
Theprojectsthatheco-chairedfordevelopmentoftheH.
264/COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCstandardhCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeenrecognizedbythe2008ATASPrime-timeEmmyEngineeringAwardandapairofNATASTechnology&Engi-neeringEmmyAwards.
In2009,hereceivedtheInnovationsAwardoftheVodafoneFoundation,theEURASIPGroupTechnicalAchievementAward,andtheBestPaperAwardofIEEETransactionsonCircuitsandSystemsforVideoTechnology.
In2010,hereceivedtheEduardRheinTechnologyAward.
ProfessorWiegandwaselectedFellowoftheIEEEin2011'forhiscontributionstovideocodinganditsstandardization.
'HewasaGuestEditorfortheIEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGYforitsSpecialIssueontheH.
264/COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCVideoCodingStandardinJuly2003,itsSpecialIssueonScalableVideoCoding-StandardizationandBeyondinSeptember2007,anditsSpecialSec-tionontheJointCallforProposalsonHighEfficiencyVideoCoding(HEVC)Standardization.
SinceJanuary2006,hehasbeenanAssociateEdi-torofIEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY.
GaryJ.
Sullivan(S'83–M'91–SM'01–F'06)receivedtheB.
S.
andM.
Eng.
degreesinelectricalengineeringfromtheUniversityofLouisvilleJ.
B.
SpeedSchoolofEngineering,Louisville,KY,in1982and1983,respec-tively,andthePh.
D.
andEngineerdegreesinelectricalengineeringfromtheUniversityofCalifornia,LosAngeles,in1991.
Hehasheldleadershippositionsinanumberofvideoandimagecodingstandardizationorganizationssince1996,includingchairmanshiporco-chairmanshipoftheITU-TVideoCodingExpertsGroup(VCEG),thevideosubgroupoftheISO/IECMovingPictureExpertsGroup(MPEG),theITU-T/ISO/IECJointVideoTeam(JVT),theITU-T/ISO/IECJointCollabora-tiveTeamonVideoCoding(JCT-VC),andtheJPEGXRsubgroupoftheITU-T/ISO/IECJointPhotographicExpertsGroup(JPEG).
Heisavideo/imagetechnologyarchitectintheWindowsEcosystemEngagementteamofMicrosoftCorporation.
AtMicrosofthedesignedandremainsleadengineerfortheDirectXVideoAcceleration(DXVA)videodecodingfeatureoftheMicrosoftWindowsoperatingsystem.
PriortojoiningMicrosoftin1999,hewasthemanagerofCommunicationsCoreResearchatPictureTelCorporation.
HewaspreviouslyaHowardHughesFellowandMemberoftheTechnicalStaffintheAdvancedSystemsDivisionofHughesAircraftCorpo-rationandaTerrain-FollowingRadar(TFR)SystemSoftwareEngineerforTexasInstruments.
Hisresearchinterestsandareasofpublicationincludeimageandvideocompressionandrate-distortionoptimization,videomotionestimationandcompensation,scalarandvectorquantization,andscalable,multiviewandloss-resilientvideocoding.
Dr.
SullivanhasreceivedtheIEEEConsumerElectronicsEngineeringEx-cellenceAward,theINCITSTechnicalExcellenceAward,theIMTCLeader-shipAward,theJ.
B.
SpeedProfessionalAwardinEngineering,theMicrosoftTechnicalAchievementinStandardizationAward,andtheMicrosoftBusi-nessAchievementinStandardizationAward.
ThestandardizationprojectsthatheledfordevelopmentoftheH.
264/MPEG-4COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCvideocodingstan-dardhCOLOR:#000000;BACKGROUND-COLOR:#ffff00">avebeenrecognizedbyanATASPrimetimeEmmyEngineeringAwardandapairofNATASTechnology&EngineeringEmmyAwards.
HeisaFellowoftheIEEEandSPIE.
HewasaGuestEditorfortheIEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGYforitsSpecialIssueontheH.
264/COLOR:#000000;BACKGROUND-COLOR:#ffff00">AVCVideoCodingStandardinJuly2003anditsSpecialIssueonScalableVideoCoding—StandardizationandBeyondinSeptember2007.