platformsblaze
blaze 时间:2021-01-12 阅读:()
ExportMemblazePBlaze4NVMeSSDbyNVMeOverFabricOrNVMeOverFabricinSPDKEnvWHITEPAPERExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2ContentsExecutiveSummary.
3NVMeOVERFABRICBriefIntroduction.
3SPDKBriefIntroduction.
4AbouttheTest4Testenvironment.
4ConfigurationofNVMeOverFabricTargetofLinuxKernel4ConfigurationofNVMeOverFabricClient8ConfigurationforNVMeOverFabricTargetofSPDK11ConfigurationforNVMeOverFabricClientofSPDK13Configurationofperformancetestingtool(fio)15Configurationofperformancetestingtool(perf)16TestResults.
16Instructionsoftestitems164KBRandomReadLatencyTestResults164KBRandomWriteLatencyTestResults164KBRandomReadTestResults.
174KBRandomWriteTestResults.
1764KBSequentialReadTestResults.
2064KBSequentialWriteTestResults.
21Conclusions.
22Reference22ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
3ExecutiveSummaryAsSSDsbecomemorecommon,you'llalsohearmoreaboutNon-VolatileMemoryExpress,a.
k.
a.
NVMExpress,ormorecommonly-NVMe.
NVMeSSDoffersbothhighthroughputandlowlatencyfeatures.
ThecapacityofNVMeSSDbecomeslargerthanlargerupto3.
2or6.
4TBandoneservercanalsosupportupto24or48U.
2NVMeSSDdisknow.
SothecapacityoftheserverfullyfilledwithU.
2NVMeSSDcanreachabortthehundredsofTBcapacity.
Thereareveryfewsinglenodeapplicationcanusesuchalargecapacity.
Sotheproblemhowtoexportthelargestoragecapacityforotherapplicationnodesisneededtosolve.
PBlazeSeriesisMembalzeenterpriselevelPCIeSSDwhichsupportstheindustry-standardNVMe,providingextremeIOPSandhelpremovetheI/Ostoragebottleneckthatplaguesthemodern-daydatacenters.
Inthiswhitepaper,comparestheperformanceandIOlatencyconsistencyforthefollowingtwoscenes:ItalsounveilsoutstandingperformanceofPBlazePCIeSSDandsuperiorQoS.
NVMeOVERFABRICBriefIntroductionMuchhasalreadybeensaidaboutNVMeoverFabrics.
Itwasfirstpublicallydemonstratedin2014andthe1.
0specificationisfinallycomplete.
Inessence,NVMeoverFabrics(NVMf)istheNVMExpress*(NVMe)blockprotocoltunneledthroughanRDMAfabric,andithasenormouspotentialtoenablethenextgenerationofdatacenterstoragedevelopment.
TosupportbroadadoptionofNVMf,theStoragePerformanceDevelopmentKit(SPDK)hascreatedareferenceuser-spaceNVMftargetimplementationforLinux,releasedforcommunityinvolvementviaGitHubunderBSDlicense.
Inparallel,community-basedLinux*kerneleffortshavecreatedbothahostandatargetunderGPLlicense.
Allthreeoftheseimplementationsarenowreleasedwiththefinal1.
0specificationasofJune8,2016.
[1]TopologyofNVMeOverFabric1.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabric(usingNVMeoverfabrictargetprogramofLinuxkernel);2.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabricinSPDKenvironment(usingNVMeoverfabrictargetprogramofSPDK).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
4SPDKBriefIntroductionTheStoragePerformanceDevelopmentKit(SPDK)providesasetoftoolsandlibrariesforwritinghighperformance,scalable,user-modestorageapplications.
Itachieveshighperformancebymovingallofthenecessarydriversintouserspaceandoperatinginapolledmodeinsteadofrelyingoninterrupts,whichavoidskernelcontextswitchesandeliminatesinterrupthandlingoverhead.
ThebedrockofSPDKisauserspace,polled-mode,asynchronous,locklessNVMedriver.
Thisprovideszero-copy,highlyparallelaccessdirectlytoanSSDfromauserspaceapplication.
ThedriveriswrittenasaClibrarywithasinglepublicheader.
Similarly,SPDKprovidesauserspacedriverfortheI/OATDMAenginepresentonmanyIntelXeon-basedplatformswithallofthesamepropertiesastheNVMedriver.
SPDKalsoprovidesNVMe-oFandiSCSIserversbuiltontopoftheseuserspacedriversthatarecapableofservingdisksoverthenetwork.
ThestandardLinuxkerneliSCSIandNVMe-oFinitiatorcanbeused(ortheWindowsiSCSIinitiatoreven)toconnectclientstotheservers.
TheseserverscanbeuptoanorderofmagnitudemoreCPUefficientthanotherimplementations.
[1]AbouttheTestTestenvironmentConfigurationofNVMeOverFabricTargetofLinuxKernelLoadnvmetandnvmet-rdmakernelmodules.
Server:2*PowerEdgeR730xd(Intel(R)Xeon(R)CPUE5-2630v3@2.
40GHz,8core,128GBDRAM)Storage:Memblaze3.
2TPBlaze4U.
2Disk(20WPowerLimit)OS:CentOS4.
7.
0-rc2+SPDK:spdk-masterDPDK:dpdk-stable-16.
07.
1NVMeOverFabricTargetofLinuxKernel:nvmet.
ko(DEFDD1F71C669E1679530B9)NVMeOverFabricTargetDriverofLinuxKernel:nvmet-rdma(865E9D5BA7A00E6EC149124)NVMeOverFabricClient:nvme-cli(nvmeversion0.
9.
77.
gc5e4)NVMeOverFabricClientHostDriver:nvme-rdma.
ko(1E2A340FDC19F7E5BF87F8C)NVMeOverFabricTargetofSPDK:nvmf_tgtTestTool:fio2.
2.
9&SPDKperf(.
/examples/nvme/perf/perf)SDPKTopologyExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
5FirmwareversionandporttypeofMellanoxHCA.
RoCEModesRoCEmodesareincludingRoCEV1andRoCEV2asfollowingfigure.
InthiscaseRoCEModesisRoCEv1asthefollowingscreenshot.
TheLinux-IOTargetusesconfigFSforallfabricmoduleconfiguration(accordingtohttps://linux-iscsi.
org/wiki/ConfigFS).
1.
Createnvmet-rdmaSubsystem.
mkdir/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
6cd/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameThenvmetisafabricmodule.
2.
Allowanyhoststobeconnectedtothistarget.
echo1>attr_allow_any_host3.
Createnamespaceinsidethesubsystemmkdirnamespaces/10cdnamespaces/104.
SetthepathtotheNVMedevice(/dev/nvme0n1)andenablethenamespace.
echo-n/dev/nvme0n1>device_pathecho1>enable5.
CreatethefollowingdirectorywithanNVMeport.
mkdir/sys/kernel/config/nvmet/ports/1cd/sys/kernel/config/nvmet/ports/16.
SettheIPaddressoftherelevantportecho10.
10.
10.
10>addr_traddr7.
SetRDMAastransporttype,andsetthetransportRDMAport.
echordma>addr_trtypeecho1023>addr_trsvcid8.
SetAddressFamilytoport.
echoipv4>addr_adrfam9.
Createasoftlink.
ln-s/sys/kernel/config/nvmet/subsystems/nvme-subsystem-namesubsystems/nvme-subsystem-name10.
Checkdmesgtomakesurethatthenvmetargetislisteningontheport.
Oct2810:51:15localhostkernel:nvmet:addingnsid10tosubsystemnvme-subsystem-nameOct2810:53:54localhostkernel:enablingport1(10.
10.
10.
10:1023)11.
Afterfinishconfiguration.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
7BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAfromcore2tocore18.
Targetnvmessdirqbind:PBlaze4NVMeSSDisalsoatnumanode1.
StopirqbalanceandbindirqsofthePBlaze4NVMeSSDfromcore0tocore31.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
8ConfigurationofNVMeOverFabricClientLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
9RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetConnecttothediscoveredsubsystems#nvmeconnect-trdma-nnvme-subsystem-name-a10.
10.
10.
10-s1023ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
10BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
11ConfigurationforNVMeOverFabricTargetofSPDKSettheconfigurationoftheNVMeoverfabrictargetofSPDK.
PBlazeIVNVMeSSDandMellanoxHCAareallatthenumanode1.
TheconfigurationoftheNVMeoverfabrictargetofSPDK(configurenvmf.
conf).
[Global]ReactorMask0xaaaaaaaa[Nvmf]MaxQueuesPerSession256MaxQueueDepth512InCapsuleDataSize4096MaxIOSize131072AcceptorCore31AcceptorPollRate0[Subsystem1]NQNnqn.
2016-06.
io.
spdk:cnode1Core3ModeDirectListenRDMA10.
10.
10.
10:1023NVMe0000:86:00.
0ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
12Configurationofthehugepages(32*1GB).
DPDKneedtouse1GBsizeLinuxHugepages.
StarttheNVMeoverfabrictargetofSPDK.
NoneedtobindirqswhendoingSPDKNVMeoverfabrictest.
Accordingtothefollowingpicture,youcansettherearenointerruptsofNVMeSSDandMellanoxHCAtobedealtwithwhendoingSPDKNVMeoverfabrictest.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
13ConfigurationforNVMeOverFabricClientofSPDKLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
14RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetandconnecttothediscoveredsubsystems.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
15BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
Configurationofperformancetestingtool(fio)FIOisanopensourcetoolfortestingthestorageIOPS.
ItsupportsavarietyofdifferentI/Oengine,includingsync,mmap,libaio,posixaio,SGv3,splice,null,network,syslet,guasiandsolarisaio.
Inthistest,weuselibaioengineinordertoincreasetheSSDconcurrentaccess.
FIOcommontestsettingisontherightsideofthelist.
ioengine=libaio,random_generator=tausworthe64,direct=1,thread=1,norandommap,randrepeat=0,Theparametersincludingbs/rw/numjobs/iodepthareareconfiguredaccordingtothetestrequirements.
ThenumberofSSDconcurrentaccessis256(numjobs*iodepth).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
16Configurationofperformancetestingtool(perf)TestResultsInstructionsoftestitemsTestItemsTestDescriptionLinuxKernelLocalTestWeuseLinuxkernelNVMedrivertoaccessNVMeSSDanddoIOtestbyusingfiotool.
NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofLinuxKernelanddoIOtestbyusingfiotoolinclientserver.
SPDKLocalTestWeuseSPDKusermodeNVMedrivertoaccessNVMeSSDanddoIOtestbyusingSPDKperftool.
SPDK+NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofSPDKanddoIOtestbyusingfiotoolinclientserver.
4KBRandomReadLatencyTestResultsFollowingtestresultsshowthatNVMeOFonlyaddabout10~15uslatencywhendoing4KBrandomreadtest,etherinornotinSPDKenvironment.
4KBRandomRead(1OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)Avg.
Latency(us)98.
56109.
50113.
9399.
00%IOLatencyUnder(us)1211311294KBRandomWriteLatencyTestResultsFollowingresultsshowthatNVMeOFonlyaddabout9~10uslatencywhendoing4KBrandomwritetest,etherinornotinSPDKenvironment.
4KBRandomWrite(1OutstandingIO)LinuxKernelNVMeOFTestSPDK+NVMeOFExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
17LocalTest(NVMeDriver)(TargetofLinuxKernel)Test(TargetofSPDK)Avg.
Latency(us)14.
6025.
9623.
5399.
00%IOLatencyUnder(us)1833374KBRandomReadTestResultsFollowingtestresultsshowthatNVMeOFhasfewperformanceimpactwhendoingtheultimate4KBrandomreadperformancetest,etherinornotinSPDKenvironment.
4KBRandomRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDKLocalTestSPDK+NVMeOFTest(TargetofSPDK)IOPS714,643701,184666,615(1lcore)689,950(2lcore)638,296Avg.
Latency(us)354.
18359.
75191.
99395.
5999.
00%IOLatencyUnder(us)1,176884N/A844Figure1showsthat4KBrandomreadperformanceofLinuxKerneltargetisalittlebetterthanSPDKtarget.
Figure2showsthat4KBrandomreadlatencyconsistencyofSPDKtargetisalittlebetterthanLinuxKerneltarget.
Figure1,4KRRIOPSFigure2,4KRRLatencyConsistency(lowerisbetter)4KBRandomWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate4KBrandomwriteperformancetest,etherinornotinSPDKenvironment.
4KBRandomWrite(256OutstandingIO)LinuxKernelLocalTestNVMeOFTest(TargetofSPDK+NVMeOF1stTestSPDK+NVMeOF2ndTestExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
18(NVMeDriver)LinuxKernel)(TargetofSPDK)(TargetofSPDK)IOPS163,112IOPS163,120IOPS163,063IOPS164,682IOPSAvg.
Latency1.
56ms1.
56ms1.
56ms1.
55ms99.
00%IOLatencyUnder10ms10ms5.
73ms5.
66ms99.
99%IOLatencyUnder27.
26ms27.
01ms12.
48ms11.
97msAndfromtheabovetable,wecanseethewritelatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxkerneltarget.
Thenwedothefollowingteststofindtheinflectionpointofthewritelatencyconsistency.
4KBRandomWriteItemsNVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)1*1OutstandingIOIOPS33,60435,174Avg.
Latency(us)25.
9624.
6299.
00%IOLatencyUnder(us)333999.
99%IOLatencyUnder(us)1071,1442*2OutstandingIOIOPS125,676134,161Avg.
Latency(us)28.
1626.
1499.
00%IOLatencyUnder(us)714199.
99%IOLatencyUnder(us)1221,1444*4OutstandingIOIOPS155,293155,459Avg.
Latency(us)98.
9298.
8299.
00%IOLatencyUnder(us)1,4001,49699.
99%IOLatencyUnder(us)4,1924,2568*8OutstandingIOIOPS155,199158,744Avg.
Latency(us)407.
97398.
8099.
00%IOLatencyUnder(us)2,8642,89699.
99%IOLatencyUnder(us)5,7925,792ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
194*16OutstandingIOIOPS154,099155,441Avg.
Latency(us)411.
22407.
6199.
00%IOLatencyUnder(us)2,8962,89699.
99%IOLatencyUnder(us)5,8565,9844*32OutstandingIOIOPS155,912156,171Avg.
Latency(us)816.
81815.
4699.
00%IOLatencyUnder(us)4,5764,44899.
99%IOLatencyUnder(us)9,4089,1524*64OutstandingIOIOPS155,192157,084Avg.
Latency(us)1,634.
51,625.
2899.
00%IOLatencyUnder(us)9,5365,47299.
99%IOLatencyUnder(us)20,86410,0484*128OutstandingIOIOPS154,697157,202Avg.
Latency(us)3,283.
913,231.
4299.
00%IOLatencyUnder(us)19,3287,45699.
99%IOLatencyUnder(us)46,33612,480Wesummarythedatainabovetableanddrawthefollowingpicture.
FromtheFigure3,youcanseethewritelatencyconsistencyofSPDKtargetisalmostthesameasLinuxkerneltargetwhenoutstandingIOofthetestisunder4*32.
ButiftheoutstandingIOofthetestisbiggerthan4*32,thewritelatencyconsistencyofLinuxkerneltargetwillbecomepoorsoon.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
20Figure3,4KRWIOPSandlatencyconsistencyinornotinSPDKenv.
Figure4showsthat4KBrandomwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure5showsthat4KBrandomreadlatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxKerneltarget.
Figure4,4KRWIOPSFigure5,4KRWLatencyConsistency(lowerisbetter)64KBSequentialReadTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialreadperformancetest,etherinornotinSPDKenvironment.
64KBSeqRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS2770.
8MB/s2778.
2MB/s2669.
1MB/sAvg.
Latency2.
88ms2.
87ms2.
99msExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2199.
00%IOLatencyUnder7.
52ms6.
88ms7.
01msFigure6showsthat64KBsequentialreadperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure7showsthat64KBsequentialreadlatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure6,64SRIOPSFigure7,64KSRLatencyConsistency(lowerisbetter)64KBSequentialWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialwriteperformancetest,etherinornotinSPDKenvironment.
64KBSeqWrite(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS1728.
1MB/s1731.
6MB/s1732.
6MB/sAvg.
Latency1.
15ms1.
14ms1.
14ms99.
00%IOLatencyUnder3.
57ms3.
34ms3.
53msFigure8showsthat64KBsequentialwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure9showsthat64KBsequentialwritelatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure8,64KSWIOPSFigure9,64KSWLatencyConsistency(lowerisbetter)ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
22ConclusionsThistestreportclearlydemonstratestheutilityofMemblazePBlazeSSDonNVMeoverfabricalmosthasnoperformanceimpactwhendoingultimateperformancetest,nomatterinSPDKenvironmentornot.
Thelatencywillonlyinduceextra9~15uswhenusingNVMeoverfabric.
ThewritelatencyconsistencyofSPDKtargetkeepsstableevenunderhighoutstandingIOpressure,whilethewritelatencyconsistencyofLinuxkerneltargetisgettingpoorerandpoorerwithoutstandingIOpressureincreased.
SoitisrecommendedtouseSPDKtargetunderheavybusinesswritepressuretofullydeliverMemblazePBlazeSeriesSSDperformance.
Reference[1]http://www.
spdk.
io/.
DISCLAIMERInformationinthisdocumentisprovidedinconnectionwithMemblazeproducts.
Memblazeprovidesthisdocument"asis",withoutwarrantyofanykind,neitherexpressednorimplied,including,butnotlimitedto,theparticularpurpose.
Memblazemaymakeimprovementsand/orchangesinthisdocumentorintheproductdescribedinthisdocumentatanytimewithoutnotice.
Theproductsdescribedinthisdocumentmaycontaindesigndefectsorerrorsknownasanomaliesorerratawhichmaycausetheproductsfunctionstodeviatefrompublishedspecifications.
COPYRIGHT2016MemblazeCorp.
Allrightsreserved.
Nopartofthisdocumentmaybereproduced,transmitted,transcribed,storedinaretrievalsystem,ortranslatedintoanylanguageinanyformorbyanymeanswithoutthewrittenpermissionofMemblazeCorp.
TRADEMARKSMemblazeisatrademarkofMemblazeCorporation.
Othernamesmentionedinthisdocumentaretrademarks/registeredtrademarksoftheirrespectiveowners.
USINGTHISDOCUMENTThoughMemblazehasreviewedthisdocumentandveryefforthasbeenmadetoensurethatthisdocumentiscurrentandaccurate,moreinformationmayhavebecomeavailablesubsequenttotheproductionofthisguide.
Inthatevent,pleasecontactyourlocalMemblazesalesofficeoryourdistributorforlatestspecificationsbeforeplacingyourproductorder.
,Ltd.
2ContentsExecutiveSummary.
3NVMeOVERFABRICBriefIntroduction.
3SPDKBriefIntroduction.
4AbouttheTest4Testenvironment.
4ConfigurationofNVMeOverFabricTargetofLinuxKernel4ConfigurationofNVMeOverFabricClient8ConfigurationforNVMeOverFabricTargetofSPDK11ConfigurationforNVMeOverFabricClientofSPDK13Configurationofperformancetestingtool(fio)15Configurationofperformancetestingtool(perf)16TestResults.
16Instructionsoftestitems164KBRandomReadLatencyTestResults164KBRandomWriteLatencyTestResults164KBRandomReadTestResults.
174KBRandomWriteTestResults.
1764KBSequentialReadTestResults.
2064KBSequentialWriteTestResults.
21Conclusions.
22Reference22ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
3ExecutiveSummaryAsSSDsbecomemorecommon,you'llalsohearmoreaboutNon-VolatileMemoryExpress,a.
k.
a.
NVMExpress,ormorecommonly-NVMe.
NVMeSSDoffersbothhighthroughputandlowlatencyfeatures.
ThecapacityofNVMeSSDbecomeslargerthanlargerupto3.
2or6.
4TBandoneservercanalsosupportupto24or48U.
2NVMeSSDdisknow.
SothecapacityoftheserverfullyfilledwithU.
2NVMeSSDcanreachabortthehundredsofTBcapacity.
Thereareveryfewsinglenodeapplicationcanusesuchalargecapacity.
Sotheproblemhowtoexportthelargestoragecapacityforotherapplicationnodesisneededtosolve.
PBlazeSeriesisMembalzeenterpriselevelPCIeSSDwhichsupportstheindustry-standardNVMe,providingextremeIOPSandhelpremovetheI/Ostoragebottleneckthatplaguesthemodern-daydatacenters.
Inthiswhitepaper,comparestheperformanceandIOlatencyconsistencyforthefollowingtwoscenes:ItalsounveilsoutstandingperformanceofPBlazePCIeSSDandsuperiorQoS.
NVMeOVERFABRICBriefIntroductionMuchhasalreadybeensaidaboutNVMeoverFabrics.
Itwasfirstpublicallydemonstratedin2014andthe1.
0specificationisfinallycomplete.
Inessence,NVMeoverFabrics(NVMf)istheNVMExpress*(NVMe)blockprotocoltunneledthroughanRDMAfabric,andithasenormouspotentialtoenablethenextgenerationofdatacenterstoragedevelopment.
TosupportbroadadoptionofNVMf,theStoragePerformanceDevelopmentKit(SPDK)hascreatedareferenceuser-spaceNVMftargetimplementationforLinux,releasedforcommunityinvolvementviaGitHubunderBSDlicense.
Inparallel,community-basedLinux*kerneleffortshavecreatedbothahostandatargetunderGPLlicense.
Allthreeoftheseimplementationsarenowreleasedwiththefinal1.
0specificationasofJune8,2016.
[1]TopologyofNVMeOverFabric1.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabric(usingNVMeoverfabrictargetprogramofLinuxkernel);2.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabricinSPDKenvironment(usingNVMeoverfabrictargetprogramofSPDK).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
4SPDKBriefIntroductionTheStoragePerformanceDevelopmentKit(SPDK)providesasetoftoolsandlibrariesforwritinghighperformance,scalable,user-modestorageapplications.
Itachieveshighperformancebymovingallofthenecessarydriversintouserspaceandoperatinginapolledmodeinsteadofrelyingoninterrupts,whichavoidskernelcontextswitchesandeliminatesinterrupthandlingoverhead.
ThebedrockofSPDKisauserspace,polled-mode,asynchronous,locklessNVMedriver.
Thisprovideszero-copy,highlyparallelaccessdirectlytoanSSDfromauserspaceapplication.
ThedriveriswrittenasaClibrarywithasinglepublicheader.
Similarly,SPDKprovidesauserspacedriverfortheI/OATDMAenginepresentonmanyIntelXeon-basedplatformswithallofthesamepropertiesastheNVMedriver.
SPDKalsoprovidesNVMe-oFandiSCSIserversbuiltontopoftheseuserspacedriversthatarecapableofservingdisksoverthenetwork.
ThestandardLinuxkerneliSCSIandNVMe-oFinitiatorcanbeused(ortheWindowsiSCSIinitiatoreven)toconnectclientstotheservers.
TheseserverscanbeuptoanorderofmagnitudemoreCPUefficientthanotherimplementations.
[1]AbouttheTestTestenvironmentConfigurationofNVMeOverFabricTargetofLinuxKernelLoadnvmetandnvmet-rdmakernelmodules.
Server:2*PowerEdgeR730xd(Intel(R)Xeon(R)CPUE5-2630v3@2.
40GHz,8core,128GBDRAM)Storage:Memblaze3.
2TPBlaze4U.
2Disk(20WPowerLimit)OS:CentOS4.
7.
0-rc2+SPDK:spdk-masterDPDK:dpdk-stable-16.
07.
1NVMeOverFabricTargetofLinuxKernel:nvmet.
ko(DEFDD1F71C669E1679530B9)NVMeOverFabricTargetDriverofLinuxKernel:nvmet-rdma(865E9D5BA7A00E6EC149124)NVMeOverFabricClient:nvme-cli(nvmeversion0.
9.
77.
gc5e4)NVMeOverFabricClientHostDriver:nvme-rdma.
ko(1E2A340FDC19F7E5BF87F8C)NVMeOverFabricTargetofSPDK:nvmf_tgtTestTool:fio2.
2.
9&SPDKperf(.
/examples/nvme/perf/perf)SDPKTopologyExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
5FirmwareversionandporttypeofMellanoxHCA.
RoCEModesRoCEmodesareincludingRoCEV1andRoCEV2asfollowingfigure.
InthiscaseRoCEModesisRoCEv1asthefollowingscreenshot.
TheLinux-IOTargetusesconfigFSforallfabricmoduleconfiguration(accordingtohttps://linux-iscsi.
org/wiki/ConfigFS).
1.
Createnvmet-rdmaSubsystem.
mkdir/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
6cd/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameThenvmetisafabricmodule.
2.
Allowanyhoststobeconnectedtothistarget.
echo1>attr_allow_any_host3.
Createnamespaceinsidethesubsystemmkdirnamespaces/10cdnamespaces/104.
SetthepathtotheNVMedevice(/dev/nvme0n1)andenablethenamespace.
echo-n/dev/nvme0n1>device_pathecho1>enable5.
CreatethefollowingdirectorywithanNVMeport.
mkdir/sys/kernel/config/nvmet/ports/1cd/sys/kernel/config/nvmet/ports/16.
SettheIPaddressoftherelevantportecho10.
10.
10.
10>addr_traddr7.
SetRDMAastransporttype,andsetthetransportRDMAport.
echordma>addr_trtypeecho1023>addr_trsvcid8.
SetAddressFamilytoport.
echoipv4>addr_adrfam9.
Createasoftlink.
ln-s/sys/kernel/config/nvmet/subsystems/nvme-subsystem-namesubsystems/nvme-subsystem-name10.
Checkdmesgtomakesurethatthenvmetargetislisteningontheport.
Oct2810:51:15localhostkernel:nvmet:addingnsid10tosubsystemnvme-subsystem-nameOct2810:53:54localhostkernel:enablingport1(10.
10.
10.
10:1023)11.
Afterfinishconfiguration.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
7BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAfromcore2tocore18.
Targetnvmessdirqbind:PBlaze4NVMeSSDisalsoatnumanode1.
StopirqbalanceandbindirqsofthePBlaze4NVMeSSDfromcore0tocore31.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
8ConfigurationofNVMeOverFabricClientLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
9RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetConnecttothediscoveredsubsystems#nvmeconnect-trdma-nnvme-subsystem-name-a10.
10.
10.
10-s1023ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
10BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
11ConfigurationforNVMeOverFabricTargetofSPDKSettheconfigurationoftheNVMeoverfabrictargetofSPDK.
PBlazeIVNVMeSSDandMellanoxHCAareallatthenumanode1.
TheconfigurationoftheNVMeoverfabrictargetofSPDK(configurenvmf.
conf).
[Global]ReactorMask0xaaaaaaaa[Nvmf]MaxQueuesPerSession256MaxQueueDepth512InCapsuleDataSize4096MaxIOSize131072AcceptorCore31AcceptorPollRate0[Subsystem1]NQNnqn.
2016-06.
io.
spdk:cnode1Core3ModeDirectListenRDMA10.
10.
10.
10:1023NVMe0000:86:00.
0ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
12Configurationofthehugepages(32*1GB).
DPDKneedtouse1GBsizeLinuxHugepages.
StarttheNVMeoverfabrictargetofSPDK.
NoneedtobindirqswhendoingSPDKNVMeoverfabrictest.
Accordingtothefollowingpicture,youcansettherearenointerruptsofNVMeSSDandMellanoxHCAtobedealtwithwhendoingSPDKNVMeoverfabrictest.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
13ConfigurationforNVMeOverFabricClientofSPDKLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
14RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetandconnecttothediscoveredsubsystems.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
15BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
Configurationofperformancetestingtool(fio)FIOisanopensourcetoolfortestingthestorageIOPS.
ItsupportsavarietyofdifferentI/Oengine,includingsync,mmap,libaio,posixaio,SGv3,splice,null,network,syslet,guasiandsolarisaio.
Inthistest,weuselibaioengineinordertoincreasetheSSDconcurrentaccess.
FIOcommontestsettingisontherightsideofthelist.
ioengine=libaio,random_generator=tausworthe64,direct=1,thread=1,norandommap,randrepeat=0,Theparametersincludingbs/rw/numjobs/iodepthareareconfiguredaccordingtothetestrequirements.
ThenumberofSSDconcurrentaccessis256(numjobs*iodepth).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
16Configurationofperformancetestingtool(perf)TestResultsInstructionsoftestitemsTestItemsTestDescriptionLinuxKernelLocalTestWeuseLinuxkernelNVMedrivertoaccessNVMeSSDanddoIOtestbyusingfiotool.
NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofLinuxKernelanddoIOtestbyusingfiotoolinclientserver.
SPDKLocalTestWeuseSPDKusermodeNVMedrivertoaccessNVMeSSDanddoIOtestbyusingSPDKperftool.
SPDK+NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofSPDKanddoIOtestbyusingfiotoolinclientserver.
4KBRandomReadLatencyTestResultsFollowingtestresultsshowthatNVMeOFonlyaddabout10~15uslatencywhendoing4KBrandomreadtest,etherinornotinSPDKenvironment.
4KBRandomRead(1OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)Avg.
Latency(us)98.
56109.
50113.
9399.
00%IOLatencyUnder(us)1211311294KBRandomWriteLatencyTestResultsFollowingresultsshowthatNVMeOFonlyaddabout9~10uslatencywhendoing4KBrandomwritetest,etherinornotinSPDKenvironment.
4KBRandomWrite(1OutstandingIO)LinuxKernelNVMeOFTestSPDK+NVMeOFExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
17LocalTest(NVMeDriver)(TargetofLinuxKernel)Test(TargetofSPDK)Avg.
Latency(us)14.
6025.
9623.
5399.
00%IOLatencyUnder(us)1833374KBRandomReadTestResultsFollowingtestresultsshowthatNVMeOFhasfewperformanceimpactwhendoingtheultimate4KBrandomreadperformancetest,etherinornotinSPDKenvironment.
4KBRandomRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDKLocalTestSPDK+NVMeOFTest(TargetofSPDK)IOPS714,643701,184666,615(1lcore)689,950(2lcore)638,296Avg.
Latency(us)354.
18359.
75191.
99395.
5999.
00%IOLatencyUnder(us)1,176884N/A844Figure1showsthat4KBrandomreadperformanceofLinuxKerneltargetisalittlebetterthanSPDKtarget.
Figure2showsthat4KBrandomreadlatencyconsistencyofSPDKtargetisalittlebetterthanLinuxKerneltarget.
Figure1,4KRRIOPSFigure2,4KRRLatencyConsistency(lowerisbetter)4KBRandomWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate4KBrandomwriteperformancetest,etherinornotinSPDKenvironment.
4KBRandomWrite(256OutstandingIO)LinuxKernelLocalTestNVMeOFTest(TargetofSPDK+NVMeOF1stTestSPDK+NVMeOF2ndTestExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
18(NVMeDriver)LinuxKernel)(TargetofSPDK)(TargetofSPDK)IOPS163,112IOPS163,120IOPS163,063IOPS164,682IOPSAvg.
Latency1.
56ms1.
56ms1.
56ms1.
55ms99.
00%IOLatencyUnder10ms10ms5.
73ms5.
66ms99.
99%IOLatencyUnder27.
26ms27.
01ms12.
48ms11.
97msAndfromtheabovetable,wecanseethewritelatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxkerneltarget.
Thenwedothefollowingteststofindtheinflectionpointofthewritelatencyconsistency.
4KBRandomWriteItemsNVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)1*1OutstandingIOIOPS33,60435,174Avg.
Latency(us)25.
9624.
6299.
00%IOLatencyUnder(us)333999.
99%IOLatencyUnder(us)1071,1442*2OutstandingIOIOPS125,676134,161Avg.
Latency(us)28.
1626.
1499.
00%IOLatencyUnder(us)714199.
99%IOLatencyUnder(us)1221,1444*4OutstandingIOIOPS155,293155,459Avg.
Latency(us)98.
9298.
8299.
00%IOLatencyUnder(us)1,4001,49699.
99%IOLatencyUnder(us)4,1924,2568*8OutstandingIOIOPS155,199158,744Avg.
Latency(us)407.
97398.
8099.
00%IOLatencyUnder(us)2,8642,89699.
99%IOLatencyUnder(us)5,7925,792ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
194*16OutstandingIOIOPS154,099155,441Avg.
Latency(us)411.
22407.
6199.
00%IOLatencyUnder(us)2,8962,89699.
99%IOLatencyUnder(us)5,8565,9844*32OutstandingIOIOPS155,912156,171Avg.
Latency(us)816.
81815.
4699.
00%IOLatencyUnder(us)4,5764,44899.
99%IOLatencyUnder(us)9,4089,1524*64OutstandingIOIOPS155,192157,084Avg.
Latency(us)1,634.
51,625.
2899.
00%IOLatencyUnder(us)9,5365,47299.
99%IOLatencyUnder(us)20,86410,0484*128OutstandingIOIOPS154,697157,202Avg.
Latency(us)3,283.
913,231.
4299.
00%IOLatencyUnder(us)19,3287,45699.
99%IOLatencyUnder(us)46,33612,480Wesummarythedatainabovetableanddrawthefollowingpicture.
FromtheFigure3,youcanseethewritelatencyconsistencyofSPDKtargetisalmostthesameasLinuxkerneltargetwhenoutstandingIOofthetestisunder4*32.
ButiftheoutstandingIOofthetestisbiggerthan4*32,thewritelatencyconsistencyofLinuxkerneltargetwillbecomepoorsoon.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
20Figure3,4KRWIOPSandlatencyconsistencyinornotinSPDKenv.
Figure4showsthat4KBrandomwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure5showsthat4KBrandomreadlatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxKerneltarget.
Figure4,4KRWIOPSFigure5,4KRWLatencyConsistency(lowerisbetter)64KBSequentialReadTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialreadperformancetest,etherinornotinSPDKenvironment.
64KBSeqRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS2770.
8MB/s2778.
2MB/s2669.
1MB/sAvg.
Latency2.
88ms2.
87ms2.
99msExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2199.
00%IOLatencyUnder7.
52ms6.
88ms7.
01msFigure6showsthat64KBsequentialreadperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure7showsthat64KBsequentialreadlatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure6,64SRIOPSFigure7,64KSRLatencyConsistency(lowerisbetter)64KBSequentialWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialwriteperformancetest,etherinornotinSPDKenvironment.
64KBSeqWrite(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS1728.
1MB/s1731.
6MB/s1732.
6MB/sAvg.
Latency1.
15ms1.
14ms1.
14ms99.
00%IOLatencyUnder3.
57ms3.
34ms3.
53msFigure8showsthat64KBsequentialwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure9showsthat64KBsequentialwritelatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure8,64KSWIOPSFigure9,64KSWLatencyConsistency(lowerisbetter)ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
22ConclusionsThistestreportclearlydemonstratestheutilityofMemblazePBlazeSSDonNVMeoverfabricalmosthasnoperformanceimpactwhendoingultimateperformancetest,nomatterinSPDKenvironmentornot.
Thelatencywillonlyinduceextra9~15uswhenusingNVMeoverfabric.
ThewritelatencyconsistencyofSPDKtargetkeepsstableevenunderhighoutstandingIOpressure,whilethewritelatencyconsistencyofLinuxkerneltargetisgettingpoorerandpoorerwithoutstandingIOpressureincreased.
SoitisrecommendedtouseSPDKtargetunderheavybusinesswritepressuretofullydeliverMemblazePBlazeSeriesSSDperformance.
Reference[1]http://www.
spdk.
io/.
DISCLAIMERInformationinthisdocumentisprovidedinconnectionwithMemblazeproducts.
Memblazeprovidesthisdocument"asis",withoutwarrantyofanykind,neitherexpressednorimplied,including,butnotlimitedto,theparticularpurpose.
Memblazemaymakeimprovementsand/orchangesinthisdocumentorintheproductdescribedinthisdocumentatanytimewithoutnotice.
Theproductsdescribedinthisdocumentmaycontaindesigndefectsorerrorsknownasanomaliesorerratawhichmaycausetheproductsfunctionstodeviatefrompublishedspecifications.
COPYRIGHT2016MemblazeCorp.
Allrightsreserved.
Nopartofthisdocumentmaybereproduced,transmitted,transcribed,storedinaretrievalsystem,ortranslatedintoanylanguageinanyformorbyanymeanswithoutthewrittenpermissionofMemblazeCorp.
TRADEMARKSMemblazeisatrademarkofMemblazeCorporation.
Othernamesmentionedinthisdocumentaretrademarks/registeredtrademarksoftheirrespectiveowners.
USINGTHISDOCUMENTThoughMemblazehasreviewedthisdocumentandveryefforthasbeenmadetoensurethatthisdocumentiscurrentandaccurate,moreinformationmayhavebecomeavailablesubsequenttotheproductionofthisguide.
Inthatevent,pleasecontactyourlocalMemblazesalesofficeoryourdistributorforlatestspecificationsbeforeplacingyourproductorder.
- platformsblaze相关文档
- Assembliesblaze
- 请浏览网站获取更多信息:www.memblaze.com
- hitblaze
- averageblaze
- Ellefsenblaze
- 系列blaze
老薛主机入门建站月付34/月,年付345元，半价香港VPS主机
老薛主机怎么样?老薛主机这个商家有存在有一些年头。如果没有记错的话，早年老薛主机是做虚拟主机业务的，还算不错在异常激烈的市场中生存到现在，应该算是在众多商家中早期积累到一定的用户群的，主打小众个人网站业务所以能持续到现在。这不，站长看到商家有在进行夏季促销，比如我们很多网友可能有需要的香港vps主机季度及以上可以半价优惠，如果有在选择不同主机商的香港机房的可以看看老薛主机商家的香港vps。点击进入...
AlphaVPS（€3.99/月）VPS年付15欧，AMD EYPC+NVMe系列起
AlphaVPS是一家保加利亚本土主机商(DA International Group Ltd)，提供VPS主机及独立服务器租用等，数据中心包括美国(洛杉矶/纽约)、德国、英国和保加利亚等，公司办公地点跟他们提供的保加利亚数据中心在一栋楼内，自有硬件，提供IPv4+IPv6，支持PayPal或者信用卡等方式付款。商家提供的大硬盘VPS主机，提供128GB-2TB磁盘，最低年付15欧元起，也可以选择...
[6.18]DogYun：充100送10元,态云7折,经典云8折,独立服务器月省100元,幸运大转盘最高5折
DogYun是一家2019年成立的国人主机商，提供VPS和独立服务器租用等，数据中心包括中国香港、美国洛杉矶、日本、韩国、德国、荷兰等，其中VPS包括常规VPS(经典云)和按小时计费VPS(动态云)，使用自行开发的面板和管理系统，支持自定义配置，动态云各个硬件独立按小时计费，带宽按照用户使用量计费(不使用不计费)或者购买流量包，线路也可以自行切换。目前商家发布了6.18促销方案，新购动态云7折，经...
blaze为你推荐
-
企业虚拟主机购买虚拟主机要注意些什么？企业网站有什么好的虚拟主机推荐吗？域名服务域名系统主要是什么？免费网站域名申请哪有里可以申请免费域名的网站？网站空间域名网站空间,域名,操作海外域名外贸网站如何选择合适的海外域名？域名申请申请域名需要哪些流程具体点 谢谢虚拟空间免费试用哪有免费试用的虚拟主机？北京网站空间自己弄一个简单的网站，大概需要办理什么，大概需要多少钱？网站空间免备案免备案网站空间哪个好虚拟主机是什么什么是虚拟主机