platformsblaze
blaze 时间:2021-01-12 阅读:()
ExportMemblazePBlaze4NVMeSSDbyNVMeOverFabricOrNVMeOverFabricinSPDKEnvWHITEPAPERExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2ContentsExecutiveSummary.
3NVMeOVERFABRICBriefIntroduction.
3SPDKBriefIntroduction.
4AbouttheTest4Testenvironment.
4ConfigurationofNVMeOverFabricTargetofLinuxKernel4ConfigurationofNVMeOverFabricClient8ConfigurationforNVMeOverFabricTargetofSPDK11ConfigurationforNVMeOverFabricClientofSPDK13Configurationofperformancetestingtool(fio)15Configurationofperformancetestingtool(perf)16TestResults.
16Instructionsoftestitems164KBRandomReadLatencyTestResults164KBRandomWriteLatencyTestResults164KBRandomReadTestResults.
174KBRandomWriteTestResults.
1764KBSequentialReadTestResults.
2064KBSequentialWriteTestResults.
21Conclusions.
22Reference22ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
3ExecutiveSummaryAsSSDsbecomemorecommon,you'llalsohearmoreaboutNon-VolatileMemoryExpress,a.
k.
a.
NVMExpress,ormorecommonly-NVMe.
NVMeSSDoffersbothhighthroughputandlowlatencyfeatures.
ThecapacityofNVMeSSDbecomeslargerthanlargerupto3.
2or6.
4TBandoneservercanalsosupportupto24or48U.
2NVMeSSDdisknow.
SothecapacityoftheserverfullyfilledwithU.
2NVMeSSDcanreachabortthehundredsofTBcapacity.
Thereareveryfewsinglenodeapplicationcanusesuchalargecapacity.
Sotheproblemhowtoexportthelargestoragecapacityforotherapplicationnodesisneededtosolve.
PBlazeSeriesisMembalzeenterpriselevelPCIeSSDwhichsupportstheindustry-standardNVMe,providingextremeIOPSandhelpremovetheI/Ostoragebottleneckthatplaguesthemodern-daydatacenters.
Inthiswhitepaper,comparestheperformanceandIOlatencyconsistencyforthefollowingtwoscenes:ItalsounveilsoutstandingperformanceofPBlazePCIeSSDandsuperiorQoS.
NVMeOVERFABRICBriefIntroductionMuchhasalreadybeensaidaboutNVMeoverFabrics.
Itwasfirstpublicallydemonstratedin2014andthe1.
0specificationisfinallycomplete.
Inessence,NVMeoverFabrics(NVMf)istheNVMExpress*(NVMe)blockprotocoltunneledthroughanRDMAfabric,andithasenormouspotentialtoenablethenextgenerationofdatacenterstoragedevelopment.
TosupportbroadadoptionofNVMf,theStoragePerformanceDevelopmentKit(SPDK)hascreatedareferenceuser-spaceNVMftargetimplementationforLinux,releasedforcommunityinvolvementviaGitHubunderBSDlicense.
Inparallel,community-basedLinux*kerneleffortshavecreatedbothahostandatargetunderGPLlicense.
Allthreeoftheseimplementationsarenowreleasedwiththefinal1.
0specificationasofJune8,2016.
[1]TopologyofNVMeOverFabric1.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabric(usingNVMeoverfabrictargetprogramofLinuxkernel);2.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabricinSPDKenvironment(usingNVMeoverfabrictargetprogramofSPDK).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
4SPDKBriefIntroductionTheStoragePerformanceDevelopmentKit(SPDK)providesasetoftoolsandlibrariesforwritinghighperformance,scalable,user-modestorageapplications.
Itachieveshighperformancebymovingallofthenecessarydriversintouserspaceandoperatinginapolledmodeinsteadofrelyingoninterrupts,whichavoidskernelcontextswitchesandeliminatesinterrupthandlingoverhead.
ThebedrockofSPDKisauserspace,polled-mode,asynchronous,locklessNVMedriver.
Thisprovideszero-copy,highlyparallelaccessdirectlytoanSSDfromauserspaceapplication.
ThedriveriswrittenasaClibrarywithasinglepublicheader.
Similarly,SPDKprovidesauserspacedriverfortheI/OATDMAenginepresentonmanyIntelXeon-basedplatformswithallofthesamepropertiesastheNVMedriver.
SPDKalsoprovidesNVMe-oFandiSCSIserversbuiltontopoftheseuserspacedriversthatarecapableofservingdisksoverthenetwork.
ThestandardLinuxkerneliSCSIandNVMe-oFinitiatorcanbeused(ortheWindowsiSCSIinitiatoreven)toconnectclientstotheservers.
TheseserverscanbeuptoanorderofmagnitudemoreCPUefficientthanotherimplementations.
[1]AbouttheTestTestenvironmentConfigurationofNVMeOverFabricTargetofLinuxKernelLoadnvmetandnvmet-rdmakernelmodules.
Server:2*PowerEdgeR730xd(Intel(R)Xeon(R)CPUE5-2630v3@2.
40GHz,8core,128GBDRAM)Storage:Memblaze3.
2TPBlaze4U.
2Disk(20WPowerLimit)OS:CentOS4.
7.
0-rc2+SPDK:spdk-masterDPDK:dpdk-stable-16.
07.
1NVMeOverFabricTargetofLinuxKernel:nvmet.
ko(DEFDD1F71C669E1679530B9)NVMeOverFabricTargetDriverofLinuxKernel:nvmet-rdma(865E9D5BA7A00E6EC149124)NVMeOverFabricClient:nvme-cli(nvmeversion0.
9.
77.
gc5e4)NVMeOverFabricClientHostDriver:nvme-rdma.
ko(1E2A340FDC19F7E5BF87F8C)NVMeOverFabricTargetofSPDK:nvmf_tgtTestTool:fio2.
2.
9&SPDKperf(.
/examples/nvme/perf/perf)SDPKTopologyExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
5FirmwareversionandporttypeofMellanoxHCA.
RoCEModesRoCEmodesareincludingRoCEV1andRoCEV2asfollowingfigure.
InthiscaseRoCEModesisRoCEv1asthefollowingscreenshot.
TheLinux-IOTargetusesconfigFSforallfabricmoduleconfiguration(accordingtohttps://linux-iscsi.
org/wiki/ConfigFS).
1.
Createnvmet-rdmaSubsystem.
mkdir/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
6cd/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameThenvmetisafabricmodule.
2.
Allowanyhoststobeconnectedtothistarget.
echo1>attr_allow_any_host3.
Createnamespaceinsidethesubsystemmkdirnamespaces/10cdnamespaces/104.
SetthepathtotheNVMedevice(/dev/nvme0n1)andenablethenamespace.
echo-n/dev/nvme0n1>device_pathecho1>enable5.
CreatethefollowingdirectorywithanNVMeport.
mkdir/sys/kernel/config/nvmet/ports/1cd/sys/kernel/config/nvmet/ports/16.
SettheIPaddressoftherelevantportecho10.
10.
10.
10>addr_traddr7.
SetRDMAastransporttype,andsetthetransportRDMAport.
echordma>addr_trtypeecho1023>addr_trsvcid8.
SetAddressFamilytoport.
echoipv4>addr_adrfam9.
Createasoftlink.
ln-s/sys/kernel/config/nvmet/subsystems/nvme-subsystem-namesubsystems/nvme-subsystem-name10.
Checkdmesgtomakesurethatthenvmetargetislisteningontheport.
Oct2810:51:15localhostkernel:nvmet:addingnsid10tosubsystemnvme-subsystem-nameOct2810:53:54localhostkernel:enablingport1(10.
10.
10.
10:1023)11.
Afterfinishconfiguration.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
7BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAfromcore2tocore18.
Targetnvmessdirqbind:PBlaze4NVMeSSDisalsoatnumanode1.
StopirqbalanceandbindirqsofthePBlaze4NVMeSSDfromcore0tocore31.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
8ConfigurationofNVMeOverFabricClientLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
9RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetConnecttothediscoveredsubsystems#nvmeconnect-trdma-nnvme-subsystem-name-a10.
10.
10.
10-s1023ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
10BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
11ConfigurationforNVMeOverFabricTargetofSPDKSettheconfigurationoftheNVMeoverfabrictargetofSPDK.
PBlazeIVNVMeSSDandMellanoxHCAareallatthenumanode1.
TheconfigurationoftheNVMeoverfabrictargetofSPDK(configurenvmf.
conf).
[Global]ReactorMask0xaaaaaaaa[Nvmf]MaxQueuesPerSession256MaxQueueDepth512InCapsuleDataSize4096MaxIOSize131072AcceptorCore31AcceptorPollRate0[Subsystem1]NQNnqn.
2016-06.
io.
spdk:cnode1Core3ModeDirectListenRDMA10.
10.
10.
10:1023NVMe0000:86:00.
0ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
12Configurationofthehugepages(32*1GB).
DPDKneedtouse1GBsizeLinuxHugepages.
StarttheNVMeoverfabrictargetofSPDK.
NoneedtobindirqswhendoingSPDKNVMeoverfabrictest.
Accordingtothefollowingpicture,youcansettherearenointerruptsofNVMeSSDandMellanoxHCAtobedealtwithwhendoingSPDKNVMeoverfabrictest.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
13ConfigurationforNVMeOverFabricClientofSPDKLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
14RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetandconnecttothediscoveredsubsystems.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
15BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
Configurationofperformancetestingtool(fio)FIOisanopensourcetoolfortestingthestorageIOPS.
ItsupportsavarietyofdifferentI/Oengine,includingsync,mmap,libaio,posixaio,SGv3,splice,null,network,syslet,guasiandsolarisaio.
Inthistest,weuselibaioengineinordertoincreasetheSSDconcurrentaccess.
FIOcommontestsettingisontherightsideofthelist.
ioengine=libaio,random_generator=tausworthe64,direct=1,thread=1,norandommap,randrepeat=0,Theparametersincludingbs/rw/numjobs/iodepthareareconfiguredaccordingtothetestrequirements.
ThenumberofSSDconcurrentaccessis256(numjobs*iodepth).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
16Configurationofperformancetestingtool(perf)TestResultsInstructionsoftestitemsTestItemsTestDescriptionLinuxKernelLocalTestWeuseLinuxkernelNVMedrivertoaccessNVMeSSDanddoIOtestbyusingfiotool.
NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofLinuxKernelanddoIOtestbyusingfiotoolinclientserver.
SPDKLocalTestWeuseSPDKusermodeNVMedrivertoaccessNVMeSSDanddoIOtestbyusingSPDKperftool.
SPDK+NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofSPDKanddoIOtestbyusingfiotoolinclientserver.
4KBRandomReadLatencyTestResultsFollowingtestresultsshowthatNVMeOFonlyaddabout10~15uslatencywhendoing4KBrandomreadtest,etherinornotinSPDKenvironment.
4KBRandomRead(1OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)Avg.
Latency(us)98.
56109.
50113.
9399.
00%IOLatencyUnder(us)1211311294KBRandomWriteLatencyTestResultsFollowingresultsshowthatNVMeOFonlyaddabout9~10uslatencywhendoing4KBrandomwritetest,etherinornotinSPDKenvironment.
4KBRandomWrite(1OutstandingIO)LinuxKernelNVMeOFTestSPDK+NVMeOFExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
17LocalTest(NVMeDriver)(TargetofLinuxKernel)Test(TargetofSPDK)Avg.
Latency(us)14.
6025.
9623.
5399.
00%IOLatencyUnder(us)1833374KBRandomReadTestResultsFollowingtestresultsshowthatNVMeOFhasfewperformanceimpactwhendoingtheultimate4KBrandomreadperformancetest,etherinornotinSPDKenvironment.
4KBRandomRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDKLocalTestSPDK+NVMeOFTest(TargetofSPDK)IOPS714,643701,184666,615(1lcore)689,950(2lcore)638,296Avg.
Latency(us)354.
18359.
75191.
99395.
5999.
00%IOLatencyUnder(us)1,176884N/A844Figure1showsthat4KBrandomreadperformanceofLinuxKerneltargetisalittlebetterthanSPDKtarget.
Figure2showsthat4KBrandomreadlatencyconsistencyofSPDKtargetisalittlebetterthanLinuxKerneltarget.
Figure1,4KRRIOPSFigure2,4KRRLatencyConsistency(lowerisbetter)4KBRandomWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate4KBrandomwriteperformancetest,etherinornotinSPDKenvironment.
4KBRandomWrite(256OutstandingIO)LinuxKernelLocalTestNVMeOFTest(TargetofSPDK+NVMeOF1stTestSPDK+NVMeOF2ndTestExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
18(NVMeDriver)LinuxKernel)(TargetofSPDK)(TargetofSPDK)IOPS163,112IOPS163,120IOPS163,063IOPS164,682IOPSAvg.
Latency1.
56ms1.
56ms1.
56ms1.
55ms99.
00%IOLatencyUnder10ms10ms5.
73ms5.
66ms99.
99%IOLatencyUnder27.
26ms27.
01ms12.
48ms11.
97msAndfromtheabovetable,wecanseethewritelatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxkerneltarget.
Thenwedothefollowingteststofindtheinflectionpointofthewritelatencyconsistency.
4KBRandomWriteItemsNVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)1*1OutstandingIOIOPS33,60435,174Avg.
Latency(us)25.
9624.
6299.
00%IOLatencyUnder(us)333999.
99%IOLatencyUnder(us)1071,1442*2OutstandingIOIOPS125,676134,161Avg.
Latency(us)28.
1626.
1499.
00%IOLatencyUnder(us)714199.
99%IOLatencyUnder(us)1221,1444*4OutstandingIOIOPS155,293155,459Avg.
Latency(us)98.
9298.
8299.
00%IOLatencyUnder(us)1,4001,49699.
99%IOLatencyUnder(us)4,1924,2568*8OutstandingIOIOPS155,199158,744Avg.
Latency(us)407.
97398.
8099.
00%IOLatencyUnder(us)2,8642,89699.
99%IOLatencyUnder(us)5,7925,792ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
194*16OutstandingIOIOPS154,099155,441Avg.
Latency(us)411.
22407.
6199.
00%IOLatencyUnder(us)2,8962,89699.
99%IOLatencyUnder(us)5,8565,9844*32OutstandingIOIOPS155,912156,171Avg.
Latency(us)816.
81815.
4699.
00%IOLatencyUnder(us)4,5764,44899.
99%IOLatencyUnder(us)9,4089,1524*64OutstandingIOIOPS155,192157,084Avg.
Latency(us)1,634.
51,625.
2899.
00%IOLatencyUnder(us)9,5365,47299.
99%IOLatencyUnder(us)20,86410,0484*128OutstandingIOIOPS154,697157,202Avg.
Latency(us)3,283.
913,231.
4299.
00%IOLatencyUnder(us)19,3287,45699.
99%IOLatencyUnder(us)46,33612,480Wesummarythedatainabovetableanddrawthefollowingpicture.
FromtheFigure3,youcanseethewritelatencyconsistencyofSPDKtargetisalmostthesameasLinuxkerneltargetwhenoutstandingIOofthetestisunder4*32.
ButiftheoutstandingIOofthetestisbiggerthan4*32,thewritelatencyconsistencyofLinuxkerneltargetwillbecomepoorsoon.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
20Figure3,4KRWIOPSandlatencyconsistencyinornotinSPDKenv.
Figure4showsthat4KBrandomwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure5showsthat4KBrandomreadlatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxKerneltarget.
Figure4,4KRWIOPSFigure5,4KRWLatencyConsistency(lowerisbetter)64KBSequentialReadTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialreadperformancetest,etherinornotinSPDKenvironment.
64KBSeqRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS2770.
8MB/s2778.
2MB/s2669.
1MB/sAvg.
Latency2.
88ms2.
87ms2.
99msExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2199.
00%IOLatencyUnder7.
52ms6.
88ms7.
01msFigure6showsthat64KBsequentialreadperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure7showsthat64KBsequentialreadlatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure6,64SRIOPSFigure7,64KSRLatencyConsistency(lowerisbetter)64KBSequentialWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialwriteperformancetest,etherinornotinSPDKenvironment.
64KBSeqWrite(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS1728.
1MB/s1731.
6MB/s1732.
6MB/sAvg.
Latency1.
15ms1.
14ms1.
14ms99.
00%IOLatencyUnder3.
57ms3.
34ms3.
53msFigure8showsthat64KBsequentialwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure9showsthat64KBsequentialwritelatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure8,64KSWIOPSFigure9,64KSWLatencyConsistency(lowerisbetter)ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
22ConclusionsThistestreportclearlydemonstratestheutilityofMemblazePBlazeSSDonNVMeoverfabricalmosthasnoperformanceimpactwhendoingultimateperformancetest,nomatterinSPDKenvironmentornot.
Thelatencywillonlyinduceextra9~15uswhenusingNVMeoverfabric.
ThewritelatencyconsistencyofSPDKtargetkeepsstableevenunderhighoutstandingIOpressure,whilethewritelatencyconsistencyofLinuxkerneltargetisgettingpoorerandpoorerwithoutstandingIOpressureincreased.
SoitisrecommendedtouseSPDKtargetunderheavybusinesswritepressuretofullydeliverMemblazePBlazeSeriesSSDperformance.
Reference[1]http://www.
spdk.
io/.
DISCLAIMERInformationinthisdocumentisprovidedinconnectionwithMemblazeproducts.
Memblazeprovidesthisdocument"asis",withoutwarrantyofanykind,neitherexpressednorimplied,including,butnotlimitedto,theparticularpurpose.
Memblazemaymakeimprovementsand/orchangesinthisdocumentorintheproductdescribedinthisdocumentatanytimewithoutnotice.
Theproductsdescribedinthisdocumentmaycontaindesigndefectsorerrorsknownasanomaliesorerratawhichmaycausetheproductsfunctionstodeviatefrompublishedspecifications.
COPYRIGHT2016MemblazeCorp.
Allrightsreserved.
Nopartofthisdocumentmaybereproduced,transmitted,transcribed,storedinaretrievalsystem,ortranslatedintoanylanguageinanyformorbyanymeanswithoutthewrittenpermissionofMemblazeCorp.
TRADEMARKSMemblazeisatrademarkofMemblazeCorporation.
Othernamesmentionedinthisdocumentaretrademarks/registeredtrademarksoftheirrespectiveowners.
USINGTHISDOCUMENTThoughMemblazehasreviewedthisdocumentandveryefforthasbeenmadetoensurethatthisdocumentiscurrentandaccurate,moreinformationmayhavebecomeavailablesubsequenttotheproductionofthisguide.
Inthatevent,pleasecontactyourlocalMemblazesalesofficeoryourdistributorforlatestspecificationsbeforeplacingyourproductorder.
,Ltd.
2ContentsExecutiveSummary.
3NVMeOVERFABRICBriefIntroduction.
3SPDKBriefIntroduction.
4AbouttheTest4Testenvironment.
4ConfigurationofNVMeOverFabricTargetofLinuxKernel4ConfigurationofNVMeOverFabricClient8ConfigurationforNVMeOverFabricTargetofSPDK11ConfigurationforNVMeOverFabricClientofSPDK13Configurationofperformancetestingtool(fio)15Configurationofperformancetestingtool(perf)16TestResults.
16Instructionsoftestitems164KBRandomReadLatencyTestResults164KBRandomWriteLatencyTestResults164KBRandomReadTestResults.
174KBRandomWriteTestResults.
1764KBSequentialReadTestResults.
2064KBSequentialWriteTestResults.
21Conclusions.
22Reference22ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
3ExecutiveSummaryAsSSDsbecomemorecommon,you'llalsohearmoreaboutNon-VolatileMemoryExpress,a.
k.
a.
NVMExpress,ormorecommonly-NVMe.
NVMeSSDoffersbothhighthroughputandlowlatencyfeatures.
ThecapacityofNVMeSSDbecomeslargerthanlargerupto3.
2or6.
4TBandoneservercanalsosupportupto24or48U.
2NVMeSSDdisknow.
SothecapacityoftheserverfullyfilledwithU.
2NVMeSSDcanreachabortthehundredsofTBcapacity.
Thereareveryfewsinglenodeapplicationcanusesuchalargecapacity.
Sotheproblemhowtoexportthelargestoragecapacityforotherapplicationnodesisneededtosolve.
PBlazeSeriesisMembalzeenterpriselevelPCIeSSDwhichsupportstheindustry-standardNVMe,providingextremeIOPSandhelpremovetheI/Ostoragebottleneckthatplaguesthemodern-daydatacenters.
Inthiswhitepaper,comparestheperformanceandIOlatencyconsistencyforthefollowingtwoscenes:ItalsounveilsoutstandingperformanceofPBlazePCIeSSDandsuperiorQoS.
NVMeOVERFABRICBriefIntroductionMuchhasalreadybeensaidaboutNVMeoverFabrics.
Itwasfirstpublicallydemonstratedin2014andthe1.
0specificationisfinallycomplete.
Inessence,NVMeoverFabrics(NVMf)istheNVMExpress*(NVMe)blockprotocoltunneledthroughanRDMAfabric,andithasenormouspotentialtoenablethenextgenerationofdatacenterstoragedevelopment.
TosupportbroadadoptionofNVMf,theStoragePerformanceDevelopmentKit(SPDK)hascreatedareferenceuser-spaceNVMftargetimplementationforLinux,releasedforcommunityinvolvementviaGitHubunderBSDlicense.
Inparallel,community-basedLinux*kerneleffortshavecreatedbothahostandatargetunderGPLlicense.
Allthreeoftheseimplementationsarenowreleasedwiththefinal1.
0specificationasofJune8,2016.
[1]TopologyofNVMeOverFabric1.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabric(usingNVMeoverfabrictargetprogramofLinuxkernel);2.
PBlaze43.
2TBU.
2NVMeSSDdiskexportedbyNVMeoverfabricinSPDKenvironment(usingNVMeoverfabrictargetprogramofSPDK).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
4SPDKBriefIntroductionTheStoragePerformanceDevelopmentKit(SPDK)providesasetoftoolsandlibrariesforwritinghighperformance,scalable,user-modestorageapplications.
Itachieveshighperformancebymovingallofthenecessarydriversintouserspaceandoperatinginapolledmodeinsteadofrelyingoninterrupts,whichavoidskernelcontextswitchesandeliminatesinterrupthandlingoverhead.
ThebedrockofSPDKisauserspace,polled-mode,asynchronous,locklessNVMedriver.
Thisprovideszero-copy,highlyparallelaccessdirectlytoanSSDfromauserspaceapplication.
ThedriveriswrittenasaClibrarywithasinglepublicheader.
Similarly,SPDKprovidesauserspacedriverfortheI/OATDMAenginepresentonmanyIntelXeon-basedplatformswithallofthesamepropertiesastheNVMedriver.
SPDKalsoprovidesNVMe-oFandiSCSIserversbuiltontopoftheseuserspacedriversthatarecapableofservingdisksoverthenetwork.
ThestandardLinuxkerneliSCSIandNVMe-oFinitiatorcanbeused(ortheWindowsiSCSIinitiatoreven)toconnectclientstotheservers.
TheseserverscanbeuptoanorderofmagnitudemoreCPUefficientthanotherimplementations.
[1]AbouttheTestTestenvironmentConfigurationofNVMeOverFabricTargetofLinuxKernelLoadnvmetandnvmet-rdmakernelmodules.
Server:2*PowerEdgeR730xd(Intel(R)Xeon(R)CPUE5-2630v3@2.
40GHz,8core,128GBDRAM)Storage:Memblaze3.
2TPBlaze4U.
2Disk(20WPowerLimit)OS:CentOS4.
7.
0-rc2+SPDK:spdk-masterDPDK:dpdk-stable-16.
07.
1NVMeOverFabricTargetofLinuxKernel:nvmet.
ko(DEFDD1F71C669E1679530B9)NVMeOverFabricTargetDriverofLinuxKernel:nvmet-rdma(865E9D5BA7A00E6EC149124)NVMeOverFabricClient:nvme-cli(nvmeversion0.
9.
77.
gc5e4)NVMeOverFabricClientHostDriver:nvme-rdma.
ko(1E2A340FDC19F7E5BF87F8C)NVMeOverFabricTargetofSPDK:nvmf_tgtTestTool:fio2.
2.
9&SPDKperf(.
/examples/nvme/perf/perf)SDPKTopologyExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
5FirmwareversionandporttypeofMellanoxHCA.
RoCEModesRoCEmodesareincludingRoCEV1andRoCEV2asfollowingfigure.
InthiscaseRoCEModesisRoCEv1asthefollowingscreenshot.
TheLinux-IOTargetusesconfigFSforallfabricmoduleconfiguration(accordingtohttps://linux-iscsi.
org/wiki/ConfigFS).
1.
Createnvmet-rdmaSubsystem.
mkdir/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
6cd/sys/kernel/config/nvmet/subsystems/nvme-subsystem-nameThenvmetisafabricmodule.
2.
Allowanyhoststobeconnectedtothistarget.
echo1>attr_allow_any_host3.
Createnamespaceinsidethesubsystemmkdirnamespaces/10cdnamespaces/104.
SetthepathtotheNVMedevice(/dev/nvme0n1)andenablethenamespace.
echo-n/dev/nvme0n1>device_pathecho1>enable5.
CreatethefollowingdirectorywithanNVMeport.
mkdir/sys/kernel/config/nvmet/ports/1cd/sys/kernel/config/nvmet/ports/16.
SettheIPaddressoftherelevantportecho10.
10.
10.
10>addr_traddr7.
SetRDMAastransporttype,andsetthetransportRDMAport.
echordma>addr_trtypeecho1023>addr_trsvcid8.
SetAddressFamilytoport.
echoipv4>addr_adrfam9.
Createasoftlink.
ln-s/sys/kernel/config/nvmet/subsystems/nvme-subsystem-namesubsystems/nvme-subsystem-name10.
Checkdmesgtomakesurethatthenvmetargetislisteningontheport.
Oct2810:51:15localhostkernel:nvmet:addingnsid10tosubsystemnvme-subsystem-nameOct2810:53:54localhostkernel:enablingport1(10.
10.
10.
10:1023)11.
Afterfinishconfiguration.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
7BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAfromcore2tocore18.
Targetnvmessdirqbind:PBlaze4NVMeSSDisalsoatnumanode1.
StopirqbalanceandbindirqsofthePBlaze4NVMeSSDfromcore0tocore31.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
8ConfigurationofNVMeOverFabricClientLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
9RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetConnecttothediscoveredsubsystems#nvmeconnect-trdma-nnvme-subsystem-name-a10.
10.
10.
10-s1023ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
10BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
11ConfigurationforNVMeOverFabricTargetofSPDKSettheconfigurationoftheNVMeoverfabrictargetofSPDK.
PBlazeIVNVMeSSDandMellanoxHCAareallatthenumanode1.
TheconfigurationoftheNVMeoverfabrictargetofSPDK(configurenvmf.
conf).
[Global]ReactorMask0xaaaaaaaa[Nvmf]MaxQueuesPerSession256MaxQueueDepth512InCapsuleDataSize4096MaxIOSize131072AcceptorCore31AcceptorPollRate0[Subsystem1]NQNnqn.
2016-06.
io.
spdk:cnode1Core3ModeDirectListenRDMA10.
10.
10.
10:1023NVMe0000:86:00.
0ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
12Configurationofthehugepages(32*1GB).
DPDKneedtouse1GBsizeLinuxHugepages.
StarttheNVMeoverfabrictargetofSPDK.
NoneedtobindirqswhendoingSPDKNVMeoverfabrictest.
Accordingtothefollowingpicture,youcansettherearenointerruptsofNVMeSSDandMellanoxHCAtobedealtwithwhendoingSPDKNVMeoverfabrictest.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
13ConfigurationforNVMeOverFabricClientofSPDKLoadnvme-rdmakernelmodules.
FirmwareversionandporttypeofMellanoxHCA.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
14RoCEModes(RoCEv1)Installandcompilenvmecli.
#nvmenvme-0.
9.
77.
gc5e4DiscoveravailablesubsystemsonNVMFtargetandconnecttothediscoveredsubsystems.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
15BindirqsofMellanoxConnectX-3ProHCA.
MellanoxHCAisatnumanode1.
StopirqbalanceandbindirqsoftheMellanoxHCAtocore1,core3……core31ofthenumanode1.
Configurationofperformancetestingtool(fio)FIOisanopensourcetoolfortestingthestorageIOPS.
ItsupportsavarietyofdifferentI/Oengine,includingsync,mmap,libaio,posixaio,SGv3,splice,null,network,syslet,guasiandsolarisaio.
Inthistest,weuselibaioengineinordertoincreasetheSSDconcurrentaccess.
FIOcommontestsettingisontherightsideofthelist.
ioengine=libaio,random_generator=tausworthe64,direct=1,thread=1,norandommap,randrepeat=0,Theparametersincludingbs/rw/numjobs/iodepthareareconfiguredaccordingtothetestrequirements.
ThenumberofSSDconcurrentaccessis256(numjobs*iodepth).
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
16Configurationofperformancetestingtool(perf)TestResultsInstructionsoftestitemsTestItemsTestDescriptionLinuxKernelLocalTestWeuseLinuxkernelNVMedrivertoaccessNVMeSSDanddoIOtestbyusingfiotool.
NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofLinuxKernelanddoIOtestbyusingfiotoolinclientserver.
SPDKLocalTestWeuseSPDKusermodeNVMedrivertoaccessNVMeSSDanddoIOtestbyusingSPDKperftool.
SPDK+NVMeOFTestWeusenvme-cliconnecttotheNVMeoverfabrictargetofSPDKanddoIOtestbyusingfiotoolinclientserver.
4KBRandomReadLatencyTestResultsFollowingtestresultsshowthatNVMeOFonlyaddabout10~15uslatencywhendoing4KBrandomreadtest,etherinornotinSPDKenvironment.
4KBRandomRead(1OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)Avg.
Latency(us)98.
56109.
50113.
9399.
00%IOLatencyUnder(us)1211311294KBRandomWriteLatencyTestResultsFollowingresultsshowthatNVMeOFonlyaddabout9~10uslatencywhendoing4KBrandomwritetest,etherinornotinSPDKenvironment.
4KBRandomWrite(1OutstandingIO)LinuxKernelNVMeOFTestSPDK+NVMeOFExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
17LocalTest(NVMeDriver)(TargetofLinuxKernel)Test(TargetofSPDK)Avg.
Latency(us)14.
6025.
9623.
5399.
00%IOLatencyUnder(us)1833374KBRandomReadTestResultsFollowingtestresultsshowthatNVMeOFhasfewperformanceimpactwhendoingtheultimate4KBrandomreadperformancetest,etherinornotinSPDKenvironment.
4KBRandomRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDKLocalTestSPDK+NVMeOFTest(TargetofSPDK)IOPS714,643701,184666,615(1lcore)689,950(2lcore)638,296Avg.
Latency(us)354.
18359.
75191.
99395.
5999.
00%IOLatencyUnder(us)1,176884N/A844Figure1showsthat4KBrandomreadperformanceofLinuxKerneltargetisalittlebetterthanSPDKtarget.
Figure2showsthat4KBrandomreadlatencyconsistencyofSPDKtargetisalittlebetterthanLinuxKerneltarget.
Figure1,4KRRIOPSFigure2,4KRRLatencyConsistency(lowerisbetter)4KBRandomWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate4KBrandomwriteperformancetest,etherinornotinSPDKenvironment.
4KBRandomWrite(256OutstandingIO)LinuxKernelLocalTestNVMeOFTest(TargetofSPDK+NVMeOF1stTestSPDK+NVMeOF2ndTestExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
18(NVMeDriver)LinuxKernel)(TargetofSPDK)(TargetofSPDK)IOPS163,112IOPS163,120IOPS163,063IOPS164,682IOPSAvg.
Latency1.
56ms1.
56ms1.
56ms1.
55ms99.
00%IOLatencyUnder10ms10ms5.
73ms5.
66ms99.
99%IOLatencyUnder27.
26ms27.
01ms12.
48ms11.
97msAndfromtheabovetable,wecanseethewritelatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxkerneltarget.
Thenwedothefollowingteststofindtheinflectionpointofthewritelatencyconsistency.
4KBRandomWriteItemsNVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)1*1OutstandingIOIOPS33,60435,174Avg.
Latency(us)25.
9624.
6299.
00%IOLatencyUnder(us)333999.
99%IOLatencyUnder(us)1071,1442*2OutstandingIOIOPS125,676134,161Avg.
Latency(us)28.
1626.
1499.
00%IOLatencyUnder(us)714199.
99%IOLatencyUnder(us)1221,1444*4OutstandingIOIOPS155,293155,459Avg.
Latency(us)98.
9298.
8299.
00%IOLatencyUnder(us)1,4001,49699.
99%IOLatencyUnder(us)4,1924,2568*8OutstandingIOIOPS155,199158,744Avg.
Latency(us)407.
97398.
8099.
00%IOLatencyUnder(us)2,8642,89699.
99%IOLatencyUnder(us)5,7925,792ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
194*16OutstandingIOIOPS154,099155,441Avg.
Latency(us)411.
22407.
6199.
00%IOLatencyUnder(us)2,8962,89699.
99%IOLatencyUnder(us)5,8565,9844*32OutstandingIOIOPS155,912156,171Avg.
Latency(us)816.
81815.
4699.
00%IOLatencyUnder(us)4,5764,44899.
99%IOLatencyUnder(us)9,4089,1524*64OutstandingIOIOPS155,192157,084Avg.
Latency(us)1,634.
51,625.
2899.
00%IOLatencyUnder(us)9,5365,47299.
99%IOLatencyUnder(us)20,86410,0484*128OutstandingIOIOPS154,697157,202Avg.
Latency(us)3,283.
913,231.
4299.
00%IOLatencyUnder(us)19,3287,45699.
99%IOLatencyUnder(us)46,33612,480Wesummarythedatainabovetableanddrawthefollowingpicture.
FromtheFigure3,youcanseethewritelatencyconsistencyofSPDKtargetisalmostthesameasLinuxkerneltargetwhenoutstandingIOofthetestisunder4*32.
ButiftheoutstandingIOofthetestisbiggerthan4*32,thewritelatencyconsistencyofLinuxkerneltargetwillbecomepoorsoon.
ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
20Figure3,4KRWIOPSandlatencyconsistencyinornotinSPDKenv.
Figure4showsthat4KBrandomwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure5showsthat4KBrandomreadlatencyconsistencyofSPDKtargetistwotimesbetterthanLinuxKerneltarget.
Figure4,4KRWIOPSFigure5,4KRWLatencyConsistency(lowerisbetter)64KBSequentialReadTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialreadperformancetest,etherinornotinSPDKenvironment.
64KBSeqRead(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS2770.
8MB/s2778.
2MB/s2669.
1MB/sAvg.
Latency2.
88ms2.
87ms2.
99msExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
2199.
00%IOLatencyUnder7.
52ms6.
88ms7.
01msFigure6showsthat64KBsequentialreadperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure7showsthat64KBsequentialreadlatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure6,64SRIOPSFigure7,64KSRLatencyConsistency(lowerisbetter)64KBSequentialWriteTestResultsFollowingtestresultsshowthatNVMeOFhasnoperformanceimpactwhendoingtheultimate64KBsequentialwriteperformancetest,etherinornotinSPDKenvironment.
64KBSeqWrite(256OutstandingIO)LinuxKernelLocalTest(NVMeDriver)NVMeOFTest(TargetofLinuxKernel)SPDK+NVMeOFTest(TargetofSPDK)MBPS1728.
1MB/s1731.
6MB/s1732.
6MB/sAvg.
Latency1.
15ms1.
14ms1.
14ms99.
00%IOLatencyUnder3.
57ms3.
34ms3.
53msFigure8showsthat64KBsequentialwriteperformanceofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure9showsthat64KBsequentialwritelatencyconsistencyofLinuxKerneltargetisalmostthesameasSPDKtarget.
Figure8,64KSWIOPSFigure9,64KSWLatencyConsistency(lowerisbetter)ExportMemblazePBlazeNVMeSSDByNVMeOFOrNVMeOFInSPDKEnvWHITEPAPERBeijingMemblazeTechnologyCo.
,Ltd.
22ConclusionsThistestreportclearlydemonstratestheutilityofMemblazePBlazeSSDonNVMeoverfabricalmosthasnoperformanceimpactwhendoingultimateperformancetest,nomatterinSPDKenvironmentornot.
Thelatencywillonlyinduceextra9~15uswhenusingNVMeoverfabric.
ThewritelatencyconsistencyofSPDKtargetkeepsstableevenunderhighoutstandingIOpressure,whilethewritelatencyconsistencyofLinuxkerneltargetisgettingpoorerandpoorerwithoutstandingIOpressureincreased.
SoitisrecommendedtouseSPDKtargetunderheavybusinesswritepressuretofullydeliverMemblazePBlazeSeriesSSDperformance.
Reference[1]http://www.
spdk.
io/.
DISCLAIMERInformationinthisdocumentisprovidedinconnectionwithMemblazeproducts.
Memblazeprovidesthisdocument"asis",withoutwarrantyofanykind,neitherexpressednorimplied,including,butnotlimitedto,theparticularpurpose.
Memblazemaymakeimprovementsand/orchangesinthisdocumentorintheproductdescribedinthisdocumentatanytimewithoutnotice.
Theproductsdescribedinthisdocumentmaycontaindesigndefectsorerrorsknownasanomaliesorerratawhichmaycausetheproductsfunctionstodeviatefrompublishedspecifications.
COPYRIGHT2016MemblazeCorp.
Allrightsreserved.
Nopartofthisdocumentmaybereproduced,transmitted,transcribed,storedinaretrievalsystem,ortranslatedintoanylanguageinanyformorbyanymeanswithoutthewrittenpermissionofMemblazeCorp.
TRADEMARKSMemblazeisatrademarkofMemblazeCorporation.
Othernamesmentionedinthisdocumentaretrademarks/registeredtrademarksoftheirrespectiveowners.
USINGTHISDOCUMENTThoughMemblazehasreviewedthisdocumentandveryefforthasbeenmadetoensurethatthisdocumentiscurrentandaccurate,moreinformationmayhavebecomeavailablesubsequenttotheproductionofthisguide.
Inthatevent,pleasecontactyourlocalMemblazesalesofficeoryourdistributorforlatestspecificationsbeforeplacingyourproductorder.
- platformsblaze相关文档
- Assembliesblaze
- 请浏览网站获取更多信息:www.memblaze.com
- hitblaze
- averageblaze
- Ellefsenblaze
- 系列blaze
RAKsmart含站群服务器/10G带宽不限流量首月半价
RAKsmart 商家估摸着前段时间服务器囤货较多，这两个月的促销活动好像有点针对独立服务器。前面才整理到七月份的服务器活动在有一些配置上比上个月折扣力度是大很多，而且今天看到再来部分的服务器首月半价，一般这样的促销有可能是商家库存充裕。比如近期有一些服务商挖矿服务器销售不好，也都会采用这些策略，就好比电脑硬件最近也有下降。不管如何，我们选择服务器或者VPS主机要本着符合自己需求，如果业务不需要，...
atcloud：480G超高防御VPS低至$4/月，美国/新加坡等6机房，512m内存/1核/500g硬盘/不限流量
atcloud主要提供常规cloud（VPS）和storage（大硬盘存储）系列VPS，其数据中心分布在美国(俄勒冈、弗吉尼亚)、加拿大、英国、法国、德国、新加坡，所有VPS默认提供480Gbps的超高DDoS防御+不限流量，杜绝DDoS攻击骚扰，比较适合海外建站等相关业务。ATCLOUD.NET是一家成立于2020年的海外主机商，主要提供KVM架构的VPS产品、LXC容器化产品、权威DNS智能解...
老薛主机入门建站月付34/月,年付345元，半价香港VPS主机
老薛主机怎么样?老薛主机这个商家有存在有一些年头。如果没有记错的话，早年老薛主机是做虚拟主机业务的，还算不错在异常激烈的市场中生存到现在，应该算是在众多商家中早期积累到一定的用户群的，主打小众个人网站业务所以能持续到现在。这不，站长看到商家有在进行夏季促销，比如我们很多网友可能有需要的香港vps主机季度及以上可以半价优惠，如果有在选择不同主机商的香港机房的可以看看老薛主机商家的香港vps。点击进入...
blaze为你推荐
-
免备案虚拟空间教你怎么看免备案虚拟主机空间便宜虚拟主机哪里有国内便宜虚拟主机虚拟主机评测浅谈建站新手如何挑选虚拟主机北京虚拟主机虚拟主机 那个好用又实惠论坛虚拟主机我要做个论坛，是用虚拟主机呢？还是用空间？除论坛外还有好及个单页，还带数据库。jsp虚拟主机java虚拟主机空间怎么选择，国内jsp虚拟主机比较稳定，现在java项目做好后需要推荐一下吧双线虚拟主机G型双线虚拟主机是什么意思域名交易域名可以过户吗？解析域名域名解析是什么意思域名批量查询有好米域名批量查询好用吗？