sequentiallyyc8
WWW YC8 COM 时间:2021-03-02 阅读:()
IntegrationandAdvancedUsageBitfusionGuideWHITEPAPER–OCTOBER2019WHITEPAPER|2Bitfusion:IntegrationandAdvancedUsageTableofcontentsIntroduction3StartingFlexDirectDaemonsasServersviaCLI3RunClientApplicationswithFlexDirectviaflexdirectclient4ConfiguringIPAddressesasPartofClientConfiguration.
5Advanced:ClusterCommunications7Advanced:FlexibleDynamicGPUConfigurations7ExposingOneGPUoutoftheFourGPUsforApps8ExposingTwoGPUsoutoftheFourGPUsforApps8PartialGPUConfigurations9WHITEPAPER|3IntroductionForengineersintegratingBitfusiontechnologyintotheirownresourceschedulerorperhapsforadvancedusersneedingmorecontroloverGPUsareresourcing,thisguideshowshowtostartandinvokebothserverandclientprocesseswithlow-levelaccess.
YouwillstartaserverdaemonforaparticularGPUconfiguration(e.
g.
,partialmemory)andwriteaclient-sideconfigurationfile'adaptor.
conf'asshownintheexamplesbelow.
Wehavedoneintegrationsforseveraljobschedulersandresourcemanagers,socontactusifyou'relookingforhelp.
StartingFlexDirectDaemonsasServersviaCLIThedrawingbelowshowsthefourprocessesthatarerunningonaclient(orCPU)nodeandonaserver(orGPU)nodewhenyouareinteractingwiththeFlexDirectServer(Dispatcher).
Itshouldhelpyouunderstandtheconcepts,commandsandusagethatthismanualdiscusses.
Onlytwoprocessesaredirectlylaunchedbytheuser.
Thesearetheonesshowninafixedfontasyouwouldtypetheminacommandshell.
ThedrawingalsoshowstheTCPportsusedbytheGPUserverprocesses.
USERGPUGPUGPUGPUCOMPUTESERVERGPUSERVERALLOCATEINUSEflexdirectserver(Dispatcher)setupclients.
confforGPUserver:55001flexdirectclient--listeningonport55001(default)listeningonports45201+fordatapathmessageslinkedtoVMwareBitfusionCUDAlibCUDAServerCoolAppYoumuststartFlexDirectasaserver(whichiscalledDispatcher)onalltheinstancesthathaveGPUswhichyou'dliketomakeavailabletoyourclientnodesandapplications.
Shellflexdirectserver[-pport]WHITEPAPER|4YoucanalsostartaFlexDirectserver(Dispatcherprocess)fromtheclientmachinewiththerequest_gpuscommand.
However,thisrequiresthattheGPUserverisalreadyrunningtheresourcescheduler.
Advantagesinclude:PreventsmultipleusersfromtryingtoservethesameGPUsCreatesadaptors.
conffileforyouDoesnotautomaticallydeallocatetheGPUsafteraclientapplicationhasfinishedsoyoucanrunseveralapplicationssequentiallyHowever,thisdocumentcoversmanuallaunchesoftheFlexDirectserver.
RunClientApplicationswithFlexDirectviaflexdirectclientOncetheFlexDirectServersarerunning,runapplicationsusingflexdirectclient.
Passthe-lparameterasalistoftheIPaddressesofthenodesonwhichyouhaveFlexDirectServerrunning.
Usesemicolonstoseparatetheaddresses.
Replacewiththeapplicationyouwouldliketorun.
Useadoubledash--beforetheapplicationifitrequiresitsownarguments.
Shellflexdirectclient-l"172.
31.
51.
20;172.
31.
51.
26"[--]CPUServerCommandLineflexdirectclient-l172.
31.
51.
20:55002nvidia-smiGPUServerCommandLinenvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11439MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Youmayspecifyaportnumberwiththestandardcolonnotation:WHITEPAPER|5USERGPUGPUGPUGPUGPUGPUGPUGPUCOMPUTESERVERGPUSERVERGPUSERVERflexdirectclientflexdirectserverflexdirectserverConfiguringIPAddressesasPartofClientConfigurationIfyouwanttosimplifytheflexdirectclientcommand,youcanputyourBitfusionserverIPaddressesintothe/etc/bitfusionio/adaptor.
conffile.
Overridethedefaultportbyadding:.
CPUServerCommandLineCPUServerCommandLinecat/etc/bitfusionio/adaptor.
conf172.
31.
51.
20172.
31.
51.
26:57001flexdirectclientnvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Afterwritingadaptor.
conf,simplyrunflexdirectclientwithaGPUapplication.
Forexample,ifyourunflexdirectclientwithnvidia-smiitwilllisttheGPUsconfigured.
Typeflexdirecthelporflexdirecthelp[command]foradditionalhelpfulcommandsandinformation.
WHITEPAPER|6SampleOutput$flexdirecthelpNAME:flexdirect-RunapplicationwithBitfusionFlexDirectUSAGE:flexdirect"application"flexdirect--[application]flexdirecthelp[command]Formoreinformation,systemrequirements,andadvancedusagepleasevisithttps://www-review.
vmware.
com/solutions/business-critical-apps/hardwareaccelerators-virtualization.
htmlCOMMANDS:init,iInitializeconfiguration.
Requiresrootpriviledges.
version,vDisplayfullFlexDirectversion.
localhealth,LHRunhealthcheckoncurrentnodeonly.
upgrade,UUpgradeversion.
Requiresrootpriviledges.
uninstallUninstallFlexDirect.
Requiresrootpriviledges.
deallocDeallocatelicensecertificate.
Requiresrootpriviledges.
crashreportSendcrashreporttoBitfusion.
licenseChecklicensestatus.
list_gpusListtheavailableGPUsinasharedpool.
help,hShowsalistofcommandsorhelpforonecommand.
ClientCommands:client,cRunapplication.
health,HRunhealthcheckonallspecifiedserversandcurrentnode.
request_gpusRequestGPUsfromasharedpool.
release_gpusReleaseGPUsbackintoasharedpool.
Optionsmustmatchapreviousrequest_gpuscommand.
runRequestGPUsfromasharedpool,runaclientcommand,thenreleasetheGPUs.
statsGatherstatsfromallservers.
smiDisplaysmi-likeinfoforallservers.
localRunaCUDAapplicationlocally.
net_perfGathernetworkperformancedatafromallSRSservers.
ServerCommands:server,sRunserver.
resource_schedulerRunFlexDirectresourcescheduler(SRS)onGPUserverEXAMPLES:$sudoflexdirectinit-l$flexdirectresource_scheduler--srs_port50001$flexdirectrun-n4--Herearesomeflexdirectexampleswithexplanatorycomments.
WHITEPAPER|7TextInitializeflexdirectlicensebeforethefirstrunofserveronasystem$sudoflexdirectinit-lRunaflexdirectserverwithdefaultport55001$flexdirectserverRunaflexdirectserverwithadifferentport$flexdirectserver-p55010Runanapplicationwithaserverrunninglocalwithdefaultport55001$flexdirectclient-l"localhost"Runanapplicationwithmultipleservers,localorremote$flexdirectclient-l"192.
168.
0.
2:55010;192.
168.
0.
6:51234"Runanapplicationwithserversspecifiedinoneofthedefaultconfigfiles(~/.
bitfusionio/adaptor.
confand/etc/bitfusionio/adaptor.
confinpriorityorder)$flexdirectclientRunanapplicationwithserversspecifiedinaconfigfile$flexdirectclient-fRunaserverwitharesourcescheduleronacustomport$flexdirectresource_scheduler--srs_port50001--port55010Runanapplicationwith4sharedGPUs$flexdirectrun-n4Runanapplicationwith2sharedGPUs,usinghalftheavailablememory,andacustomservers.
conf$flexdirectrun-n2-p0.
5-sservers.
confRunanapplicationwith4sharedGPUswithInfiniBand$flexdirectrun-n4Runanapplicationlocally,restrictedtoonlyhalfthephysicalGPUmemory$flexdirectlocal-p0.
5Request8remoteGPUs$flexdirectrequest_gpus-sservers.
conf-fadaptor_8gpu.
conf-n8Runanapplicationwiththegeneratedconfigfile$flexdirectclient-fadaptor_8gpu.
confReleasethe8remoteGPUsaftertheapplicationhasfinished$flexdirectrelease_gpus-fadaptor_8gpu.
confGethelponaspecificcommand(theclientcommandinthisexample)$flexdirecthelpclientAdvanced:ClusterCommunicationsIfyouareunabletoopenupthedefault45201-46225portrangeforin-clustercommunication,youcanoverridethisrangebyexportingtheseenvironmentvariablesonyourGPUserversbeforerunningtheFlexDirectServer(alsocalledDispatcher):GPUServerCommandLine$exportBF_SERVER_PORT_MIN=$exportBF_SERVER_PORT_MAX=Advanced:FlexibleDynamicGPUConfigurationsTheexamplesbelowassumethatyouhaveafour-GPUserveratIPaddress123.
45.
67.
890.
WewillusethisoneGPUnodeforthreedifferentclientapplicationswithslightlydifferentresourceconfigurations,allsharingthesameGPUnode.
WHITEPAPER|8NOTENotehowasweprogressthroughtheexamples,weusedifferentportssothateachserverprocessisutilizinguniqueportsforcommunication.
BF_VISIBLE_DEVICESreferstotheIDnumberofeachGPUdevice,whichstartsat0.
Ifyouhavea4GPUinstance,theIDswouldbe0,1,2,and3respectively.
YoucanseethedevicesandtheirspecificIDsbyrunningnvidia-smi.
ExposingOneGPUoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefirstGPUdevice(outofthefourweareassumingfortheseexamples)withthefollowingcommand:NowruntheFlexDirectClientonyourCPUnode.
Inthisexample,we'lldoitwithapplication"nvidia-smi",butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
ExposingTwoGPUsoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefour-GPUnodewiththefollowingcommand:GPUServerCommandLineGPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001flexdirectclient-l123.
45.
67.
89:55001nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|BF_VISIBLE_DEVICES=0,1flexdirectserver-p55002WHITEPAPER|9RuntheFlexDirectClient.
Inthisexample,we'lldoitwithapplication"nvidia-smi:,butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
GPUServerCommandLineflexdirectclient-l123.
45.
67.
89:55002nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|PartialGPUConfigurations1/2-GPUavailableonport55001ThisisdonebysettingenvironmentalvariableBF_GPU_DEVICE_MEMORY_LIMITtohalfoftheGPUsmemory.
NVIDIAGPUSETTINGTOALLOWSHARINGWhenyoupartitionaGPU,presumablyyouwanttobeabletousebothpartitionssimultaneously.
NVIDIAGPUshaveacomputemodethatshouldbesetto"Default"(not"Exclusive")sothatmultipleapplicationscanshareaccess.
Usethenvidia-smi-acommandtoseethecurrentcomputemodesetting.
Andsetthemodeto"Default"withthecommandsudonvidia-smi-c0.
Server-sidecommandsshown,seeaboveonhowtoinvoketheclient.
GPUServerCommandLineBF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001WHITEPAPER|10TwoGPUsAvailableonPort5500121/2-GPUsAvailableonPort55001Fortwohalf-sizedGPUs:161/2GPUsAssignedtoTwoDifferentClients(Acrosstwofour-GPUnodes).
EachclientseeseightpartialGPUs.
Usetwodifferentportnumbers,oneforeachclient.
Commentsareinterlacedwithcommands:GPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001BF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001GPUServerCommandLines#server1:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002#server2:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002SampleOutput#server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002#server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001ConfigurationFiles#client1adaptor.
conf:5500155001#client2adaptor.
conf:5500255002WHITEPAPER|11161/2GPUs(acrosstwofour-GPUnodes)available.
TwodifferentclientseachallocateonepartialGPU.
GPUServerCommandLines#Server1:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&#Server2:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&SampleOutput#Server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008#Server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008SampleOutput#Client1adaptor.
conf(firstpartialGPUofserver1):55001#Client2adaptor.
conf(secondpartialGPUofserver1):55002VMware,Inc.
3401HillviewAvenuePaloAltoCA94304USATel877-486-9273Fax650-427-5001vmware.
comCopyright2019VMware,Inc.
Allrightsreserved.
ThisproductisprotectedbyU.
S.
andinternationalcopyrightandintellectualpropertylaws.
VMwareproductsarecoveredbyoneormorepatentslistedatvmware.
com/go/patents.
VMwareisaregisteredtrademarkortrademarkofVMware,Inc.
anditssubsidiariesintheUnitedStatesandotherjurisdictions.
Allothermarksandnamesmentionedhereinmaybetrademarksoftheirrespectivecompanies.
ItemNo:VMW-0518-1843_VMW_CPBUTechnicalWhitePapers_BitfusionDocs_08IntegrationandAdvancedUsage_1.
5_YC8/19
5Advanced:ClusterCommunications7Advanced:FlexibleDynamicGPUConfigurations7ExposingOneGPUoutoftheFourGPUsforApps8ExposingTwoGPUsoutoftheFourGPUsforApps8PartialGPUConfigurations9WHITEPAPER|3IntroductionForengineersintegratingBitfusiontechnologyintotheirownresourceschedulerorperhapsforadvancedusersneedingmorecontroloverGPUsareresourcing,thisguideshowshowtostartandinvokebothserverandclientprocesseswithlow-levelaccess.
YouwillstartaserverdaemonforaparticularGPUconfiguration(e.
g.
,partialmemory)andwriteaclient-sideconfigurationfile'adaptor.
conf'asshownintheexamplesbelow.
Wehavedoneintegrationsforseveraljobschedulersandresourcemanagers,socontactusifyou'relookingforhelp.
StartingFlexDirectDaemonsasServersviaCLIThedrawingbelowshowsthefourprocessesthatarerunningonaclient(orCPU)nodeandonaserver(orGPU)nodewhenyouareinteractingwiththeFlexDirectServer(Dispatcher).
Itshouldhelpyouunderstandtheconcepts,commandsandusagethatthismanualdiscusses.
Onlytwoprocessesaredirectlylaunchedbytheuser.
Thesearetheonesshowninafixedfontasyouwouldtypetheminacommandshell.
ThedrawingalsoshowstheTCPportsusedbytheGPUserverprocesses.
USERGPUGPUGPUGPUCOMPUTESERVERGPUSERVERALLOCATEINUSEflexdirectserver(Dispatcher)setupclients.
confforGPUserver:55001flexdirectclient--listeningonport55001(default)listeningonports45201+fordatapathmessageslinkedtoVMwareBitfusionCUDAlibCUDAServerCoolAppYoumuststartFlexDirectasaserver(whichiscalledDispatcher)onalltheinstancesthathaveGPUswhichyou'dliketomakeavailabletoyourclientnodesandapplications.
Shellflexdirectserver[-pport]WHITEPAPER|4YoucanalsostartaFlexDirectserver(Dispatcherprocess)fromtheclientmachinewiththerequest_gpuscommand.
However,thisrequiresthattheGPUserverisalreadyrunningtheresourcescheduler.
Advantagesinclude:PreventsmultipleusersfromtryingtoservethesameGPUsCreatesadaptors.
conffileforyouDoesnotautomaticallydeallocatetheGPUsafteraclientapplicationhasfinishedsoyoucanrunseveralapplicationssequentiallyHowever,thisdocumentcoversmanuallaunchesoftheFlexDirectserver.
RunClientApplicationswithFlexDirectviaflexdirectclientOncetheFlexDirectServersarerunning,runapplicationsusingflexdirectclient.
Passthe-lparameterasalistoftheIPaddressesofthenodesonwhichyouhaveFlexDirectServerrunning.
Usesemicolonstoseparatetheaddresses.
Replacewiththeapplicationyouwouldliketorun.
Useadoubledash--beforetheapplicationifitrequiresitsownarguments.
Shellflexdirectclient-l"172.
31.
51.
20;172.
31.
51.
26"[--]CPUServerCommandLineflexdirectclient-l172.
31.
51.
20:55002nvidia-smiGPUServerCommandLinenvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11439MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Youmayspecifyaportnumberwiththestandardcolonnotation:WHITEPAPER|5USERGPUGPUGPUGPUGPUGPUGPUGPUCOMPUTESERVERGPUSERVERGPUSERVERflexdirectclientflexdirectserverflexdirectserverConfiguringIPAddressesasPartofClientConfigurationIfyouwanttosimplifytheflexdirectclientcommand,youcanputyourBitfusionserverIPaddressesintothe/etc/bitfusionio/adaptor.
conffile.
Overridethedefaultportbyadding:.
CPUServerCommandLineCPUServerCommandLinecat/etc/bitfusionio/adaptor.
conf172.
31.
51.
20172.
31.
51.
26:57001flexdirectclientnvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Afterwritingadaptor.
conf,simplyrunflexdirectclientwithaGPUapplication.
Forexample,ifyourunflexdirectclientwithnvidia-smiitwilllisttheGPUsconfigured.
Typeflexdirecthelporflexdirecthelp[command]foradditionalhelpfulcommandsandinformation.
WHITEPAPER|6SampleOutput$flexdirecthelpNAME:flexdirect-RunapplicationwithBitfusionFlexDirectUSAGE:flexdirect"application"flexdirect--[application]flexdirecthelp[command]Formoreinformation,systemrequirements,andadvancedusagepleasevisithttps://www-review.
vmware.
com/solutions/business-critical-apps/hardwareaccelerators-virtualization.
htmlCOMMANDS:init,iInitializeconfiguration.
Requiresrootpriviledges.
version,vDisplayfullFlexDirectversion.
localhealth,LHRunhealthcheckoncurrentnodeonly.
upgrade,UUpgradeversion.
Requiresrootpriviledges.
uninstallUninstallFlexDirect.
Requiresrootpriviledges.
deallocDeallocatelicensecertificate.
Requiresrootpriviledges.
crashreportSendcrashreporttoBitfusion.
licenseChecklicensestatus.
list_gpusListtheavailableGPUsinasharedpool.
help,hShowsalistofcommandsorhelpforonecommand.
ClientCommands:client,cRunapplication.
health,HRunhealthcheckonallspecifiedserversandcurrentnode.
request_gpusRequestGPUsfromasharedpool.
release_gpusReleaseGPUsbackintoasharedpool.
Optionsmustmatchapreviousrequest_gpuscommand.
runRequestGPUsfromasharedpool,runaclientcommand,thenreleasetheGPUs.
statsGatherstatsfromallservers.
smiDisplaysmi-likeinfoforallservers.
localRunaCUDAapplicationlocally.
net_perfGathernetworkperformancedatafromallSRSservers.
ServerCommands:server,sRunserver.
resource_schedulerRunFlexDirectresourcescheduler(SRS)onGPUserverEXAMPLES:$sudoflexdirectinit-l$flexdirectresource_scheduler--srs_port50001$flexdirectrun-n4--Herearesomeflexdirectexampleswithexplanatorycomments.
WHITEPAPER|7TextInitializeflexdirectlicensebeforethefirstrunofserveronasystem$sudoflexdirectinit-lRunaflexdirectserverwithdefaultport55001$flexdirectserverRunaflexdirectserverwithadifferentport$flexdirectserver-p55010Runanapplicationwithaserverrunninglocalwithdefaultport55001$flexdirectclient-l"localhost"Runanapplicationwithmultipleservers,localorremote$flexdirectclient-l"192.
168.
0.
2:55010;192.
168.
0.
6:51234"Runanapplicationwithserversspecifiedinoneofthedefaultconfigfiles(~/.
bitfusionio/adaptor.
confand/etc/bitfusionio/adaptor.
confinpriorityorder)$flexdirectclientRunanapplicationwithserversspecifiedinaconfigfile$flexdirectclient-fRunaserverwitharesourcescheduleronacustomport$flexdirectresource_scheduler--srs_port50001--port55010Runanapplicationwith4sharedGPUs$flexdirectrun-n4Runanapplicationwith2sharedGPUs,usinghalftheavailablememory,andacustomservers.
conf$flexdirectrun-n2-p0.
5-sservers.
confRunanapplicationwith4sharedGPUswithInfiniBand$flexdirectrun-n4Runanapplicationlocally,restrictedtoonlyhalfthephysicalGPUmemory$flexdirectlocal-p0.
5Request8remoteGPUs$flexdirectrequest_gpus-sservers.
conf-fadaptor_8gpu.
conf-n8Runanapplicationwiththegeneratedconfigfile$flexdirectclient-fadaptor_8gpu.
confReleasethe8remoteGPUsaftertheapplicationhasfinished$flexdirectrelease_gpus-fadaptor_8gpu.
confGethelponaspecificcommand(theclientcommandinthisexample)$flexdirecthelpclientAdvanced:ClusterCommunicationsIfyouareunabletoopenupthedefault45201-46225portrangeforin-clustercommunication,youcanoverridethisrangebyexportingtheseenvironmentvariablesonyourGPUserversbeforerunningtheFlexDirectServer(alsocalledDispatcher):GPUServerCommandLine$exportBF_SERVER_PORT_MIN=$exportBF_SERVER_PORT_MAX=Advanced:FlexibleDynamicGPUConfigurationsTheexamplesbelowassumethatyouhaveafour-GPUserveratIPaddress123.
45.
67.
890.
WewillusethisoneGPUnodeforthreedifferentclientapplicationswithslightlydifferentresourceconfigurations,allsharingthesameGPUnode.
WHITEPAPER|8NOTENotehowasweprogressthroughtheexamples,weusedifferentportssothateachserverprocessisutilizinguniqueportsforcommunication.
BF_VISIBLE_DEVICESreferstotheIDnumberofeachGPUdevice,whichstartsat0.
Ifyouhavea4GPUinstance,theIDswouldbe0,1,2,and3respectively.
YoucanseethedevicesandtheirspecificIDsbyrunningnvidia-smi.
ExposingOneGPUoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefirstGPUdevice(outofthefourweareassumingfortheseexamples)withthefollowingcommand:NowruntheFlexDirectClientonyourCPUnode.
Inthisexample,we'lldoitwithapplication"nvidia-smi",butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
ExposingTwoGPUsoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefour-GPUnodewiththefollowingcommand:GPUServerCommandLineGPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001flexdirectclient-l123.
45.
67.
89:55001nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|BF_VISIBLE_DEVICES=0,1flexdirectserver-p55002WHITEPAPER|9RuntheFlexDirectClient.
Inthisexample,we'lldoitwithapplication"nvidia-smi:,butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
GPUServerCommandLineflexdirectclient-l123.
45.
67.
89:55002nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|PartialGPUConfigurations1/2-GPUavailableonport55001ThisisdonebysettingenvironmentalvariableBF_GPU_DEVICE_MEMORY_LIMITtohalfoftheGPUsmemory.
NVIDIAGPUSETTINGTOALLOWSHARINGWhenyoupartitionaGPU,presumablyyouwanttobeabletousebothpartitionssimultaneously.
NVIDIAGPUshaveacomputemodethatshouldbesetto"Default"(not"Exclusive")sothatmultipleapplicationscanshareaccess.
Usethenvidia-smi-acommandtoseethecurrentcomputemodesetting.
Andsetthemodeto"Default"withthecommandsudonvidia-smi-c0.
Server-sidecommandsshown,seeaboveonhowtoinvoketheclient.
GPUServerCommandLineBF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001WHITEPAPER|10TwoGPUsAvailableonPort5500121/2-GPUsAvailableonPort55001Fortwohalf-sizedGPUs:161/2GPUsAssignedtoTwoDifferentClients(Acrosstwofour-GPUnodes).
EachclientseeseightpartialGPUs.
Usetwodifferentportnumbers,oneforeachclient.
Commentsareinterlacedwithcommands:GPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001BF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001GPUServerCommandLines#server1:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002#server2:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002SampleOutput#server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002#server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001ConfigurationFiles#client1adaptor.
conf:5500155001#client2adaptor.
conf:5500255002WHITEPAPER|11161/2GPUs(acrosstwofour-GPUnodes)available.
TwodifferentclientseachallocateonepartialGPU.
GPUServerCommandLines#Server1:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&#Server2:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&SampleOutput#Server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008#Server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008SampleOutput#Client1adaptor.
conf(firstpartialGPUofserver1):55001#Client2adaptor.
conf(secondpartialGPUofserver1):55002VMware,Inc.
3401HillviewAvenuePaloAltoCA94304USATel877-486-9273Fax650-427-5001vmware.
comCopyright2019VMware,Inc.
Allrightsreserved.
ThisproductisprotectedbyU.
S.
andinternationalcopyrightandintellectualpropertylaws.
VMwareproductsarecoveredbyoneormorepatentslistedatvmware.
com/go/patents.
VMwareisaregisteredtrademarkortrademarkofVMware,Inc.
anditssubsidiariesintheUnitedStatesandotherjurisdictions.
Allothermarksandnamesmentionedhereinmaybetrademarksoftheirrespectivecompanies.
ItemNo:VMW-0518-1843_VMW_CPBUTechnicalWhitePapers_BitfusionDocs_08IntegrationandAdvancedUsage_1.
5_YC8/19
- sequentiallyyc8相关文档
- Statisticsyc8
- stackyc8
- 系统一级医院系统集成专业方案Com
- 调度使用COM—Express Nano工控板实现IP调度设备
- 接口基于COM技术的开放式数控软件系统的开发与研究
- 共产主义com开头词汇
RAKsmart秒杀服务器$30/月,洛杉矶/圣何塞/香港/日本站群特价
RAKsmart发布了9月份优惠促销活动，从9月1日~9月30日期间，爆款美国服务器每日限量抢购最低$30.62-$46/月起，洛杉矶/圣何塞/香港/日本站群大量补货特价销售，美国1-10Gbps大带宽不限流量服务器低价热卖等。RAKsmart是一家华人运营的国外主机商，提供的产品包括独立服务器租用和VPS等，可选数据中心包括美国加州圣何塞、洛杉矶、中国香港、韩国、日本、荷兰等国家和地区数据中心(...
rfchost：洛杉矶vps/双向CN2 GIA,1核/1G/10G SSD/500G流量/100Mbps/季付$23.9
rfchost怎么样?rfchost是一家开办了近六年的国人主机商，一般能挺过三年的国人商家，还是值得入手的，商家主要销售VPS，机房有美国洛杉矶/堪萨斯、中国香港，三年前本站分享过他家堪萨斯机房的套餐。目前rfchost商家的洛杉矶机房还是非常不错的，采用CN2优化线路，电信双程CN2 GIA，联通去程CN2 GIA，回程AS4837，移动走自己的直连线路，目前季付套餐还是比较划算的，有需要的可...
老周互联24小时无理由退款，香港原生IP，28元起
老周互联怎么样？老周互联隶属于老周网络科技部旗下，创立于2019年12月份，是一家具有代表性的国人商家。目前主营的产品有云服务器，裸金属服务器。创办一年多以来，我们一直坚持以口碑至上，服务宗旨为理念，为用户提供7*24小时的轮班服务，目前已有上千多家中小型站长选择我们!服务宗旨：老周互联提供7*24小时轮流值班客服，用户24小时内咨询问题可提交工单，我们会在30分钟内为您快速解答!另免费部署服务器...
WWW YC8 COM为你推荐
-
iphone5解锁iphone5怎么解锁spgnux怎么安装思普操作系统怎么在qq空间里添加背景音乐怎么在QQ空间里免费添加背景音乐？？？？网站联盟网站联盟的运作流程ps抠图技巧ps中怎么抠图？唱吧电脑版官方下载唱吧有电脑版的么？ 在哪里下载啊？彩信中心移动的彩信中心是？主页是？收不到彩信，怎么设置？蘑菇街美丽说蘑菇街、美丽说这类网站前期是怎么推广的？保护气球抖音里面看的，这是什么游戏奇虎论坛奇虎是中国的吗？