cwise_ops_commonyc8

FAQandTroubleshootingBitfusionGuideWHITEPAPER–OCTOBER2019WHITEPAPER|2Bitfusion:FAQandTroubleshootingTableofContentsCanIuseFlexDirectonmyownhardware3Whatismyperformancegoingtobelike3"YourkernelmaynothavebeenbuiltwithNUMAsupport"3Runningoutofmemoryerrors3Errorestablishingconnection:Cannotallocatememory3WorkingwithHTTP_PROXYsettings4CUDA9.
0"memoryoperationsarenotsupportedonthisdevice"4CUDA_ERROR_PEER_ACCESS_UNSUPPORTED5Utility,nvidia-smi,notrunning5ErrorMessage:couldnotfind=char5ErrorMessage:allCUDA-capabledevicesarebusyorunavailable5WHITEPAPER|3CanIuseFlexDirectonmyownhardwareYes,itcanbeusedbothon-premiseinyourdatacenteraswellinpubliccloudslikeAWS,Azure,etc.
WhatismyperformancegoingtobelikeGreatquestion,itreallydependsonthemodelandinstancesyouchoose.
Wedorecommendatleast10GbEnetworkingformostuse-cases.
High-speedfabricssuchasInfinibandandthosewithRDMAsupportwillbenecessaryformulti-serverscenarios.
Thebestthingtodoistotestitoutyourselfandcontactusifyouwantustodivedeeperwithyou.
"YourkernelmaynothavebeenbuiltwithNUMAsupport"WhenrunningwithFlexDirectyoumayseethewarningmessage,"YourkernelmaynothavebeenbuiltwithNUMAsupport.
".
ThesemessageshavenoimpactonperformanceoraccuracyofTensorFlowresults.
TheyarecausedbyTensorFlowlookingforhardwarepropertiesthoughsysfs,and,ofcourse,suchinformationwillnotbeavailableonaCPUnodebecauseitisusingnetwork-attachedGPUs.
TheFlexDirectruntimeperformancebenefitsfromNUMAoptimizationswhenappropriate,soyoucansafelyignorethesewarnings.
RunningoutofmemoryerrorsWhenrunninglargemodelsorbatchsizes,frameworkssuchasTensorFlowcanreportoutofmemoryerrors:TextTextWtensorflow/core/common_runtime/gpu/gpu_bfc_allocator.
cc:211]Ranoutofmemorytryingtoallocate877.
38MiB.
SeelogsformemorystateWtensorflow/core/kernels/cwise_ops_common.
cc:56]Resourceexhausted:OOMwhenallocatingtensorwithshape[10000,23000]$ulimit-n4096#or$ulimit-nunlimitedThesearelegitimateerrors.
TheapplicationrequiresmorememorythanyouhaveassignedorisavailablefromtheGPUs.
Avoidingtheseissuescanbeacombinationofoneormorestrategies:ReducebatchsizeUsealargerGPUsizeIncreasemodelparallelismbysplittingyourmodelintosmallerchunksErrorestablishingconnection:CannotallocatememoryThiserrorcanoccurifthesystemhasaresourcelimitthatistoorestrictive.
Toavoidthisissueincreasethenumberofopenfilesallowedwiththeulimitcommand.
WHITEPAPER|4WorkingwithHTTP_PROXYsettingsBydefault,thehttp_proxyandhttps_proxyenvironmentvariablesarenothonoredbyFlexDirectforcommunicationsbetweentheclientandserver(s).
Thisisbydesign,asin-clusternetworkingperformancecanpotentiallybereducedbywebproxies.
ToforceFlexDirecttousethesystem'sproxysettings,usetheBF_USE_PROXYenvironmentvariableeitherinyourstartupscriptsorpriortolaunchinganyserverorclient:TextTextText$exportBF_USE_PROXY=1$sudormmodnvidianvidia_uvmnvidia_drmnvidia_modeset$sudomodprobenvidiaNVreg_EnableStreamMemOPs=1$psauxf#Examineprocessand,forexample,notethat"lightdm"isrunning,whichusestheGPU$sudokill#Or$sudosystemctlstop//e.
g.
lightdmCUDA9.
0"memoryoperationsarenotsupportedonthisdevice"CUDA9.
0,asofJanuary24,2018,disablesbatchmemoryoperationsbydefaultasanerrata.
TheseoperationsaremainlyusedforGPUDirect-enabledapplications.
Thus,itisrecommendedtoenablethissettingforbestresults.
Tore-enable,removeallNVIDIAmodulesandre-installwiththeNVreg_EnableStreamMemOPsparameterenabled:Sometimes,amodulecannotberemovedbecauseanotherapplicationisusingit.
Itcanbedifficulttodeterminewhatthespecificapplicationis.
Youmayneedtomanuallyexaminethelistofrunningprocessesandkilllikelycandidates.
TheremaydesktoporgraphicalservicesrunningaknownserviceoftenfoundinVMwareenvironmentsislightdm.
Dosomeexplorationtofindwhichapplicationisresponsible.
Desktoporothergraphicalservicesandapplicationsaregoodcandidates.
Youcanseeeverythingthatisrunningwith:Thentryagaintouninstall-reinstallthenvidiamodule.
WHITEPAPER|5CUDA_ERROR_PEER_ACCESS_UNSUPPORTEDTensorFlowmayemitanerror,CUDA_ERROR_PEER_ACCESS_UNSUPPORTED,whenitfindsGPUpairsnotconnectedbythePCIeandsystemtopology.
Youmayignoretheseerrors.
ThejobofFlexDirectvirtualizationistohandlethenecessarycommunicationviathenetwork(e.
g.
,ethernetofInfiniBand).
Anexampleoftheerrormessageishere:2018-09-0520:42:10.
049855:Wtensorflow/core/common_runtime/gpu/gpu_device.
cc:1331]Unabletoenablepeeraccessbetweendeviceordinals0and6,status:Internal:failedtoenablepeeraccessfrom0x55ef97c9fef0to0x55ef97cb2520:CUDA_ERROR_PEER_ACCESS_UNSUPPORTEDUtility,nvidia-smi,notrunningtheNvidiautility,nvidia-smi,isreleasedwiththeNvidiadriver.
Theutilityisoftenupdatedaswellasthedriver.
Anoldernvidia-smimaynotworkwithalaterdriver.
Forexample,theversionofnvidia-smithatcomeswiththe410driverversion,doesnotworkwithdriverversion418.
Errormessage:couldnotfind=charThiserrormessageissometimesseennearthebeginningoftheFlexDirectoutput.
Itmaybeignored.
Itmayberepeatedseveraltimes:couldnotfind=charcouldnotfind=charcouldnotfind=charcouldnotfind=charUltimatelyitcomesfromathird-partylibrary,ibverbs.
ThebestwaytopreventunnecessaryoccurancesistoconfigureFlexDirecttoexploreanduseonlythenetworkinterfacesandtransportmechanismsyouwantittouse.
ThiscanbeconfiguredisdocumentedunderAdvancedNetworkingConfiguration.
ErrorMessage:allCUDA-capabledevicesarebusyorunavailableIfyourattempttorunmultipleapplicationsonaGPUfails(orallbutoneoftheapplicationsfail)withanerrormessagesuchas,Cudafailurep2pBandwidthLatencyTest.
cu:68:'allCUDA-capabledevicesarebusyorunavailable',thenchangetheNVIDIAGPUcomputemodesettingfrom"Exclusive"to"Default.
"sudonvidia-smi-c0ComputeMode:DefaultThe"Default"modeallowsGPUsharing.
Youcanseethecurrentcomputemodewithnvidia-smi-a(alongwithalotofotherinformation),e.
g.
,VMware,Inc.
3401HillviewAvenuePaloAltoCA94304USATel877-486-9273Fax650-427-5001vmware.
comCopyright2019VMware,Inc.
Allrightsreserved.
ThisproductisprotectedbyU.
S.
andinternationalcopyrightandintellectualpropertylaws.
VMwareproductsarecoveredbyoneormorepatentslistedatvmware.
com/go/patents.
VMwareisaregisteredtrademarkortrademarkofVMware,Inc.
anditssubsidiariesintheUnitedStatesandotherjurisdictions.
Allothermarksandnamesmentionedhereinmaybetrademarksoftheirrespectivecompanies.
ItemNo:VMW-0518-1843_VMW_CPBUTechnicalWhitePapers_BitfusionDocs_10FAQandTroubleshooting_1.
2_YC8/19