hitpagedefrag

ServerVMCapacityandtheNeedforMoreMemoryRunningalargenumberofVMsonasingleserver(byeithersingleormulti-tenant)isanestablishedbestpracticeforincreasingtheutilizationofservers.
TheabilitytomaximizethenumberofVMsperserverhasadirectinfluenceontheeconomicsofaninfrastructure.
WiththeincreaseincorecountandCPUcapabilities,capacityplanninginVMdeploymentsisnowmorefocusedonmemoryrequirements,asthesystemcouldrunoutofthememoryresourcewithoutmaximizingthecomputingpoweroftheCPUs.
ThisstoragecapacitylimitationhasdriventheuseoflesspowerfulCPUsandcurbthememoryofworkloadsrunningVMs.
ThisuseofinsufficientamountofphysicalmemoryresultsinthehostusingLinux*Swap(evenwithhigh-endSSDs),typicallyproducingpoorperformance.
Inmostcases,customersconsiderthisoptionasalastresorttopreventout-of-memorycrash,andcarefullyplancapacitytoavoidusingLinuxSwap.
Memory-inducedScalabilityChallengeforVMsDeploymentAserver'sVMscapacityistypicallylimitedbytheamountofmemoryavailableonthatsingleserver.
DuetothehighcostofDRAM,organizationsseekthe"sweet-spot"of$/GB;itiscost-prohibitivetopurchasehigh-densityDIMMS.
Tocompoundthesedifficulties,themaximumamountofmemoryperserveriscappedymodernserverarchitecture:generally12DIMMSperprocessorsocket,however,inhigh-densityservers,thecapisaslowas6-8DIMMSpersocket.
Thesefinanciallimitationsaresignificant.
TheperGBcostofDRAMincreasesexponentiallywithsizeforhigh-densityDIMMs.
Currently,inmid-2018,DIMMslargerthan64GBareeffectivelycost-prohibitive,leadingtoapracticalsystemmemorylimitof768GBto1.
5TBforastandarddualsocketserver.
Asaresult,thenumberofservernodesrequiredforalarge-scalevirtualizedinfrastructure,forpublicorprivateclouds,istypicallydeterminedbytheamountofmemoryineachnode,ratherthanbythecomputecapacityavailablewiththelatestprocessormodels.
ThisisevidentintypicaldeploymentscenarioswhereCPUutilizationisfarfrommaxingout,whichresultsinthecustomer'schoosinglowergradeparts.
TosupportalargenumberofVMs,organizationsareforcedtouseexpensivehigh-densityDIMMs,ortoincreasethenumberofnodesforalarge-scaleinfrastructure,resultinginmuchhighercostsfortheinfrastructure,aswellasalargerdatacenterfootprintandsubsequentcost.
AuthorFrankOberEnterpriseArchitectIntelOptaneDCSSDswithIntelMemoryDriveTechnology:IncreasesServer'sVMCapacityCloudVirtualizationEnablingmoreaffordableorbiggermemorysolutionsfordatacenters.
2TechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnologyIntelMemoryDriveTechnologyBenefitsandCapabilitiesIncreasingthetotalmemorypernode,orreducingthecostperGBofmemory,cansignificantlyimprovethecostefficiencyofaVMinfrastructure.
IntelMemoryDriveTechnologyisarevolutionarysoftware-definedmemory(SDM),whichtransparentlyintegratestheIntelOptaneSSDintothememorysubsystemandmakesitappearsimilartoDRAMtotheOSandapplications.
UnlikeLinuxSwap,IntelMemoryDriveTechnologyincreasesmemorycapacitybeyondDRAMlimitationsinasignificantlygreater,morecost-effectiveway,whiledeliveringDRAM-likeperformanceinacompletelytransparentmannertotheoperatingsystemandapplications.
Itisaplug-inaddition/upgradetoanexistingserver,andnochangesarerequiredtotheOS,VMsorapplications.
Linux*KVM+RedisPerformanceBenchmark–BasedonCustomerUse-CaseBasedonarealuse-case,acustomerwasexperiencinglowCPUutilizationduetothelimitednumberofvirtualmachinesabletorunonasingleserver.
WetestedtheoptionofexpandingmemorywithIntelOptaneDCSSDP4800XwithIntelMemoryDriveTechnologytoincreaseaserver'scapacitytorunmoreVMswithoutasignificantimpactonperformance,ascomparedtotheperformancedrawbackofLinuxSwap.
WeusedasetofVMsbasedontheLinux*KernelVirtualMachine(KVM),eachVMrunningaredisserverandaredisclient,performingamixtureofsetandgetoperations.
Thebenchmarkwasfirstrunonabare-metal,DRAM-onlysystemwith192GBDDR4DRAM,tovalidatethatitcanfitno-morethan31VMsduetotheamountofmemoryrequired(~5.
7GB/VM),andthenre-testedwithtwoconfigurationsproviding4xmorememory-768GBeach:1.
Asystemwith768GBofvirtualmemoryandLinuxSwap,using192GBofDDR4plusthebestavailableIntelOptaneSSDsastheLinuxSwapdevice2.
Asystemwith768GBofmemoryusing192GBofDDR4augmentedwithIntelMemoryDriveTechnologyandIntelOptaneSSDs.
WorkloadCharacteristicsofEachVM(Allclaimsbasedonthefollowingconfiguration.
2)MaximumthroughputperVMwasthrottledto:5,000requests/sec(operations/sec)tosimulaterealuse-caseworkload,toreachanaveragelatencytargetof50s,anda99%latencyof300s(whichisconsideredagoodpracticeforsuchscenarios)EachRedis*serverinstance(oneperVM)wasloadedwith3.
3Mrecordsof1KBeach,resultinginamemoryfootprintof5.
7GBforeachVMTheoperationsweretestedwithdifferentset/getratiosaswellasdifferentrandomGaussiandistributions(bellcurve)withdifferentinfluencefactors1(Gaussianwidth)o100%Random,flatdistribution–influence=0:100%GET,70%GEToRandom,Gaussiandistribution–influence=10:100%GET,70%GET4iterationswereperformedforeachtest:o1stiterationwasusedforwarmupo3measurediterationsfromwhichdatawasderivedforresultsSoftwarestack:oCentOSLinuxrelease7.
4.
1708(Core)okernel:4.
15.
12-1.
el7.
elrepo.
x86_64(el7.
x86_64)oVMs:Ubuntu*16.
04.
2LTS(GNU/Linux4.
4.
0-62-genericx86_64)oIntelMemoryDriveTechnology8.
5.
2401.
0Hardwaresetup:oDual-socketserver:2xIntelXeonGold6154CPU@3.
00GHzoIntelMemoryDriveTechnology:192GBDDR4DRAM+IntelMemoryDriveTechnology:2x750GB(cappedto768GB)oLinuxSwap:192GBDDR4DRAM+LinuxSwap:2x750GBIntelOptaneSSDsIntelbestknownmethods(BKMs)forLinuxSwapapplied(seeappendix)3TechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnologyIntelOptaneSSDDCP4800XwithIntelMemoryDriveTechnologyMeasuredPerformanceThefollowingchartsshowtheperformanceofasystemrunning108VMsconcurrently,andcomparesthetwoconfigurationspreviouslydiscussed:1.
LinuxSwap:Asystemusing192GBDDR4DRAM+SWAPonIntelOptaneSSDs2.
IntelMemoryDriveTechnology:Asystemwith192GBDDR4DRAM+IntelOptaneSSDDCP4800XwithIntelMemoryDriveTechnologytoachievea768GBofmemoryinacost-effectivesetupTargetaveragelatencylevels(50saverageanda99%latencyof300saremarkedinred).
AverageLatencywithIntelMemoryDriveTechnologyisbelow50sLatencyfor99%ofoperationsisbelow300sAverageLatencywithIntelMemoryDriveTechnologyis2x-4xbetterthanLinuxSwapLatencyfor99%is7xto10xbetterthanLinuxSwapOverallQualityofServiceWhenevaluatingthequalityofservice,checkthemaximumlatencythatisachievedfromeachoption,indicatedontheworst-caseperformance.
Asshownonthecharttotheright,themaximumlatencyachievedwithSSDDCP4800XwithIntelMemoryDriveTechnologyiswithinacceptablerange,whilewithLinuxSwapweseethatpeaklatencyis30xhigherandreacheslevelswhichareunacceptableforsuchaworkloadenvironment.
4TechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnologyCostEfficiencyandOverallSavingsIntelOptaneDCSSDwithIntelMemoryDriveTechnologyenablesthecost-effectiveexpansionofaserver'smemory,enabling3.
5xthenumberofconcurrentVMs(108vs.
31)withintherequired99%latencyacceptablerange.
Thecostchartontheright,comparesthecostperVMandnumberofconcurrentVMsof:3DRAM-onlyserverwith192GBofmemoryAserverconfigurationwith192GBRAM+SSDDCP4800XwithIntelMemoryDriveTechnologytoreachthesameoverallamountofmemory(768GB)Additionalcostsavingsareachievedasthesolutionrequiresasmallerdatacenterfootprint,energysavings,andreducedmaintenancecosts.
Upgradinganexisting192GBserverwithIntelOptaneDCSSDwithIntelMemoryDriveTechnologyprovidesadditionalcapacityof350%,andreducesthecostperVMbyupto60%!
HigherHitRatioAllowsforGreaterThroughputInadditiontothescenariosabove,wehavetestedwithaGaussiandistributioninfluencefactorof25.
Thissimulatesanarrowerbellcurve(andasaresultahigherhitratio),whichrequireslessmemorybandwidthtotheIntelOptaneDCSSDswithNVMe*.
Thisallowstheabilitytorunupto126VMs,whilemeetingthelatencytargets.
IntelOptaneDCSSDwithIntelMemoryDriveTechnologyareatruememoryreplacementtechnologySummaryLeveragingIntelOptaneDCSSDswithIntelMemoryDriveTechnologyprovidesacost-effectiveway,aswellasanin-placetransparentupgrade,tosupportservernodeswithupto8x4morememorythanserver'sspecifications'limits(orcurrentmemoryconfiguration),enablingacost-effectiveinfrastructureforVMs,withminimalimpactonperformance.
UnlikeLinuxSwap,IntelOptaneDCSSDswithIntelMemoryDriveTechnologyenablesallthiswhilemaintainingthecommonlyacceptablelatencyrequirements.
5TechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnologyAppendixA:SystemandBenchmarkInstallationandConfigurationBenchmarkScriptsandCodeRunscript#!
/bin/bashif["_$1"thenecho"Usage:$0"exit1fiBASE="$PWD"OUTDIR="$PWD/$1"if[-e"$OUTDIR"]thenecho"Outputdirectoryalreadyexists.
"exit1fimkdir"$OUTDIR"exec&>>(tee-a$OUTDIR/log.
txt).
/tune.
shdmesg>$OUTDIR/dmesgsleep15cdansible#######VMssetupvms=96echo"CloneVMs:$vms"|tee-a$OUTDIR/free.
txtfree-m>>$OUTDIR/free.
txt.
.
/vms-clone.
sh$vmsecho"SpawnVMs:$vms"|tee-a$OUTDIR/free.
txtfree-m>>$OUTDIR/free.
txt.
.
/vms-spawn.
sh$vmsecho"RunBenchmark"ansible-iinventoryall-mpingsleep15#######fillexportREDIS_OP=fillexportREDIS_VALSIZE=1024exportREDIS_CUSTOMERS=3300000json=$OUTDIR/fill_${REDIS_CUSTOMERS}_${REDIS_VALSIZE}.
`date+%y%m%d-%H%M%S`.
jsonecho"FillRedis-$REDIS_CUSTOMERSx$REDIS_VALSIZEBytes-before-`date`"|tee-a$OUTDIR/free.
txtfree-m>>$OUTDIR/free.
txtansible-playbook-iinventoryprovisioning/benchmark-random-bias-mixed-redis.
yml>$jsonecho"FillRedis-$REDIS_CUSTOMERSx$REDIS_VALSIZEBytes-after-`date`"|tee-a$OUTDIR/free.
txtfree-m>>$OUTDIR/free.
txt#Makesuretheresultsarerepresentitive,repeateachstepsthisnumbreoftimesITERS=3exportREDIS_OP=testexportREDIS_RANDOM_BENCHMARK_RATE=5000#######test#wemustrun0first,toneutralizethesequentialdatafillforinfin01025;doexportREDIS_RANDOM_BENCHMARK_INFLUENCE=${inf}formixin10070;doexportREDIS_RANDOM_BENCHMARK_MIX=${mix}for((iter=0;iter>$OUTDIR/free.
txtansible-playbook-iinventoryprovisioning/benchmark-random-bias-mixed-redis.
yml>$json.
/stats$json|grepHosts-A5|pastedonedonedoneecho"Done"echo6vms-clone.
sh#!
/bin/bashvms=$1vm_user="intel"vm_master="ubuntu1604-good-backup"virshdestroy$vm_mastervirshundefine$vm_masterforvm_namein`virshlist--name--all`;doecho$vm_namevirshdestroy$vm_namevirshundefine$vm_name--remove-all-storagedonevms=$((vms-1))virshcreate/home/swapstream/$vm_master.
xmlvirshsuspend$vm_masterforiin`seq0$vms`;doecho$ivm_name="ubuntu1604-$i"virt-clone-o$vm_master-n$vm_name--auto-clonedonevirshdestroy$vm_mastervirshundefine$vm_mastervms-spawn.
sh#!
/bin/bashvms=$1vm_user="intel"vms=$((vms-1))foriin`seq0$vms`;doecho$ivm_name="ubuntu1604-$i"virshstart$vm_namesleep1doneecho"Sleeping"sleep15.
.
/get_ips.
sh$((vms+1))echo"Sleeping"sleep15get_ips.
sh#!
/bin/bashvms=$1vm_user="intel"vms=$((vms-1))while[1];doecho"GettingIPs"rminventoryforiin`seq0$vms`;dovm_name="ubuntu1604-$i"ip=$(virshdomifaddr"$vm_name"|tail-2|awk'{print$4}'|cut-d/-f1)echo"$ipansible_user=$vm_user">>inventorydoneif[`grep-c192.
168inventory`-gt$vms];thenbreak;fiecho"WaitingforallVMIPaddresses.
.
.
"sleep10donecatinventoryTechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnology7tune.
sh#!
/bin/bashecho"Setscalinggovernor"forCPUFREQin/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor;do[-f$CPUFREQ]||continue;echo-nperformance>$CPUFREQ;doneecho"Showscalinggovernor"cat/sys/devices/system/cpu/cpu0/cpufreq/scaling_governorechoforfnamein`echo-n/usr/local/bin/version|sed's/usr\/local\/bin\/vsmpversion\1\3\2/'`;dostrings$fname2>&1|grep-iq"vsmpversion"if[$==0];thenUTILPX=`basename$fname|awk'{printsubstr($0,0,4)}'`;break;fidoneecho"Runningenvironment"NATIVE=0${UTILPX}version--long[$-ne0]&&NATIVE=1echoecho"Removeallswapdevices"swapoff-aechoif[$NATIVE-eq1];thenif[`free-g|grepMem:|awk'{print$2}'`-le500];thenSWAP-SPECIFICSETTINGS######SpecifytheNVMEname-spacetouse.
Donotuse/devand#partitionswillbecreatedautomaticallypername-space.
#WARNING:SETTHISCAREFULLYASITCANDESTROYTHEOSswapdevlist="nvme0n1nvme1n1"#ForallvaluesbelowUSE-1FORDEFAULTpartitions_per_device=-1cluster=-1watermark_scale_factor=400max_sectors_kb=-1io_queue_depth=-1nomerges=-1echo"SettingOptaneasswapon$swapdevlistwith$partitions_per_deviceperdevice"if[!
-x/usr/sbin/partprobe-o!
-x/usr/sbin/parted];thenecho"pleaseinstallpartprobeandparted"exitfifordevin$swapdevlist;doecho$dev;ddif=/dev/zeroof=/dev/${dev}bs=1Mcount=10oflag=direct;done2>&1/usr/sbin/partprobe;sleep1;/usr/sbin/partprobe;sleep1if[$partitions_per_device-gt1];thenpchunk=$((100/partitions_per_device))fordevin$swapdevlist;do/usr/sbin/parted/dev/${dev}mklabelmsdos/usr/sbin/parted-anone/dev/${dev}mkpartextended0100%forpin`seq1$partitions_per_device`;doecho/dev/${dev}$(((p-1)*pchunk))%$((p*pchunk))%/usr/sbin/parted-anone/dev/${dev}mkpartlogical$(((p-1)*pchunk))%$((p*pchunk))%donedone2>&1/usr/sbin/partprobe;sleep1;/usr/sbin/partprobe;sleep1fordevin$swapdevlist;doforpartin/dev/${dev}p*;doif[[$part=~p1$]];thenecho"====skippingextpartition$part";continue;fi/usr/sbin/mkswap-f$part;/usr/sbin/swapon-p0$partdone;done2>&1elsefordevin$swapdevlist;do/usr/sbin/mkswap-f$dev;/usr/sbin/swapon-p0$dev;done2>&1fiswapon-sechoecho"Swaptunning"grep-H^/proc/sys/vm/page-clusterif[$cluster-ge0];thenecho$cluster>/proc/sys/vm/page-clustergrep-H^/proc/sys/vm/page-clusterfigrep-H^/proc/sys/vm/watermark_scale_factorif[$watermark_scale_factor-ge0];thenecho$watermark_scale_factor>/proc/sys/vm/watermark_scale_factorgrep-H^/proc/sys/vm/watermark_scale_factorfigrep-H^/sys/block/nvme*/queue/max_sectors_kbif[$max_sectors_kb-ge0];thenfordevin$swapdevlist;doecho$max_sectors_kb>/sys/block/${dev}/queue/max_sectors_kb;donegrep-H^/sys/block/nvme*/queue/max_sectors_kbfigrep-H^/sys/module/nvme/parameters/io_queue_depthif[$io_queue_depth-ge0];thenecho$io_queue_depth>/sys/module/nvme/parameters/io_queue_depthgrep-H^/sys/module/nvme/parameters/io_queue_depthfiTechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnology8grep-H^/sys/block/nvme*/queue/nomergesif[$nomerges-ge0];thenfordevin$swapdevlist;doecho$nomerges>/sys/block/${dev}/queue/nomerges;donegrep-H^/sys/block/nvme*/queue/nomergesfiechofifiecho"Disablehugepages"echo'never'>/sys/kernel/mm/transparent_hugepage/enabledecho'never'>/sys/kernel/mm/transparent_hugepage/defragechoecho"ShowNUMATopology"numactl-Hechoecho"ShowMemoryusage"free-gechoecho"Show/proc/meminfo"cat/proc/meminfoecho#Lastly,stopksmtuned-asitisnoteffectivewithrandomdata.
echo"Stopksmtunedandksm"systemctlstopksmtuned.
serviceecho0>/sys/kernel/mm/ksm/run#IfKSMistobeused,wehavetwooptions:#1)Startksmmanually.
ThisisdonetomakesureKSM#isrunninginbothSWAPandIMDT,unrelatedtomemorysize.
#2)Change/etc/ksmtuned.
conffromKSM_THRES_COEF=20to#KSM_THRES_COEF=100-(DRAM/SYSTEM_MEMORY)*(100-20)#echo"Startksm"#echo1>/sys/kernel/mm/ksm/runechoecho"Showkernelmmtuning"grep-r^/sys/kernel/mm/ansible/provisioning/benchmark-random-bias-mixed-redis.
yml-hosts:allbecome:truetasks:-name:Makesurewecanconnectping:-name:Removeswappartitioncommand:swapoff-a-name:copyefficienttoVMcopy:src:/root/newswapstream/solotestdest:/root/intel-benchmode:0755-name:BenchmarkRediscommand:.
/solotest$REDIS_OP6379$REDIS_VALSIZE$REDIS_CUSTOMERS$REDIS_RANDOM_BENCHMARK_INFLUENCE$REDIS_RANDOM_BENCHMARK_RATE$REDIS_RANDOM_BENCHMARK_MIXregister:benchmarkargs:chdir:/root/intel-benchenvironment:REDIS_OP:"{{lookup('env','REDIS_OP')}}"REDIS_VALSIZE:"{{lookup('env','REDIS_VALSIZE')}}"REDIS_CUSTOMERS:"{{lookup('env','REDIS_CUSTOMERS')}}"REDIS_RANDOM_BENCHMARK_INFLUENCE:"{{lookup('env','REDIS_RANDOM_BENCHMARK_INFLUENCE')}}"REDIS_RANDOM_BENCHMARK_RATE:"{{lookup('env','REDIS_RANDOM_BENCHMARK_RATE')}}"REDIS_RANDOM_BENCHMARK_MIX:"{{lookup('env','REDIS_RANDOM_BENCHMARK_MIX')}}"-name:Outputbenchmarkresultsdebug:msg="{{benchmark.
stdout}}"TechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnology9ansible/stats#!
/usr/bin/envnodevarfs=require('fs')varfile=process.
argv[2];varjson=fs.
readFileSync(file).
toString("utf-8");if(json[0]varlines=json.
split("\n");lines.
shift()json=lines.
join("\n");}varcontents=JSON.
parse(json)contents.
plays.
forEach(function(play){vartasks=play.
tasksvarbenchmarkResults=tasks.
filter(function(task){returntask.
task.
name=="BenchmarkRedis"})varhosts=benchmarkResults[0].
hostsvartotalHosts=0;vartotalAvg=0,totalP95=0,totalP99=0,totalMin=99999999,totalMax=0;for(varhostinhosts){varresults=hosts[host]if(totalHosts==0){console.
log()}if(results.
stdout==undefined)continue;++totalHosts;console.
log(results.
stdout)varavg=parseFloat(results.
stdout.
split("\n")[1])*1000varmin=parseFloat(results.
stdout.
split("\n")[2])*1000varmax=parseFloat(results.
stdout.
split("\n")[3])*1000varP99=parseFloat(results.
stdout.
split("\n")[4])*1000varP95=parseFloat(results.
stdout.
split("\n")[5])*1000if(isNaN(avg)){--totalHosts}else{totalAvg+=avgtotalP95+=P95totalP99+=P99if(mintotalMax){totalMax=max}}}console.
log()varavgAvg=totalAvg/totalHostsvaravgP95=totalP95/totalHostsvaravgP99=totalP99/totalHostsconsole.
log("Hosts:",totalHosts)console.
log("Avg:",avgAvg.
toFixed(3))console.
log("Min:",totalMin.
toFixed(3))console.
log("P95:",avgP95.
toFixed(3))console.
log("P99:",avgP99.
toFixed(3))console.
log("Max:",totalMax.
toFixed(3))})Redisclientcode:Tocompile:gcc-O2-osolotestsolotest.
c-Iredis-4.
0.
2/deps/hiredis-Lredis-4.
0.
2/deps/hiredis-lhiredis-lmTechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnology10solotest.
c#define_GNU_SOURCE#include#include#include#include#include#include"hiredis.
h"#include#include#include#include//DEFAULTS:#defineDEF_PORT6379#defineDEF_SIZE1000#defineDEF_COUNT20000000#defineDEF_BIAS(-1)#defineDEF_FREQ1000000#defineDEF_READ100#defineMAX_HISTOGRAM1000000//1secchar*value;charhostname[256]={"127.
0.
0.
1"};intport=DEF_PORT;intval_size=DEF_SIZE;longkey_count=DEF_COUNT;longbias=DEF_BIAS;longfrequency=DEF_FREQ;longreadmix=DEF_READ;unsignedintseed;long*histogram;intfill=0;inttest=0;inlinedoublegauss(doublex,doubleD){doublea=1;doubleb=50;doublec=D;returna*exp(-(x-b)*(x-b)/(2*c*c));}voidselect_mode(char*modestr){fill=strcasestr(modestr,"fill")==NULL0:1;test=strcasestr(modestr,"test")==NULL0:1;}longtimestamp(void){staticlongstart_time=-1;structtimevaltv;gettimeofday(&tv,NULL);if(start_time==-1)start_time=tv.
tv_sec;return((long)(tv.
tv_sec-start_time))*1000000L+tv.
tv_usec;}char*datestr(void){time_tnow;time(&now);returnstrtok(ctime(&now),"\n");}intget_seed(){structsysinfos_info;interror=sysinfo(&s_info);if(error!
=0)printf("codeerror=%d\n",error);returns_info.
uptime*timestamp();}char*prep_value(char*value,intval_size){inti;for(i=0;i1)select_mode(argv[1]);if(argc>2)port=atoi(argv[2]);if(argc>3)val_size=atoi(argv[3]);if(argc>4)key_count=atol(argv[4]);if(argc>5)bias=atol(argv[5]);if(argc>6)frequency=atol(argv[6]);TechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnology11if(argc>7)readmix=atol(argv[7]);fprintf(stderr,"%s:fill%dtest%dport%dvalue_len%dkey_count%ldbias%ldfrequency%ldreadmix%ld\n",datestr(),fill,test,port,val_size,key_count,bias,frequency,readmix);sleep(5);//holdonforawhile,soIcanstartonallVMs.
seed=get_seed();redisContext*con;redisReply*reply;structtimevaltimeout={1,500000};//1.
5secondslongi;char*value=(char*)malloc(((int)((val_size+3)/4))*4+1);long*histogram=calloc(MAX_HISTOGRAM+1,sizeof(long));if(NULL==value){fprintf(stderr,"OutOfMemory\n");exit(1);}fprintf(stderr,"%s:Value(s)allocated,connecting\n",datestr());con=redisConnectWithTimeout(hostname,port,timeout);if(NULL==con){fprintf(stderr,"port%s:%dconnectionerror:can'tallocaterediscontext\n",hostname,port);exit(1);}if(con->err!
=0){fprintf(stderr,"port%s:%dconnectionerror:%s\n",hostname,port,con->errstr);redisFree(con);exit(1);}fprintf(stderr,"port%s:%dconnected\n",hostname,port);longstart,total_elapse;if(fill){fprintf(stderr,"%s:port%s:%dflushall-starting\n",datestr(),hostname,port);reply=redisCommand(con,"flushall");fprintf(stderr,"%s:port%s:%dflushall-%s\n",datestr(),hostname,port,reply->str);freeReplyObject(reply);fprintf(stderr,"%s:port%s:%dset-starting\n",datestr(),hostname,port);start=timestamp();for(i=0;istr,"OK")!
=0)fprintf(stderr,"port%s:%dSET%ld%sreplyerror!
reply:%s\n",hostname,port,i,"",reply->str);freeReplyObject(reply);}total_elapse=timestamp()-start;fprintf(stderr,"%s:port%s:%dset-%dkeys,elapse%ldusecs\n",datestr(),hostname,port,key_count,total_elapse);}intset=0;longmax=0,min=99999999,total=0,singlestart,elapsed,now;if(test){fprintf(stderr,"%s:port%s:%dtest-starting\n",datestr(),hostname,port);start=timestamp();for(i=0;ikey_count/2)(k+(int)(gauss(influence,bias)*(key_count/2-k))):(k-(int)(gauss(influence,bias)*(k-key_count/2)));}if(readmixreadmix)1:0;}if(set){prep_value(value,val_size);singlestart=timestamp();reply=redisCommand(con,"SET%ld%s",k,value);elapsed=timestamp()-singlestart;if(strcmp(reply->str,"OK")!
=0)fprintf(stderr,"port%s:%dSET%ld=%ld%sreplyerror!
reply:%s\n",hostname,port,i,k,"",reply->str);}else{singlestart=timestamp();reply=redisCommand(con,"GET%ld",k);elapsed=timestamp()-singlestart;if(NULL==reply)fprintf(stderr,"port%s:%dGET%ld=%ldreplyisNULL!
\n",hostname,port,i,k);}freeReplyObject(reply);total+=elapsed;histogram[elapsed>MAX_HISTOGRAMMAX_HISTOGRAM:elapsed]++;TechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnology12if(elapsed>max){max=elapsed;fprintf(stderr,"port%s:%dtest%skey%8ld(%d)-maxtime%6ldusecs\n",hostname,port,set"SET":"GET",i,k,max);}if(elapsed0)usleep(elapsed);}total_elapse=timestamp()-start;longpct=0,pct95=0,pct99=0;for(i=0;i=((key_count*95)/100)))pct95=i;if(!
pct99&&(pct>=((key_count*99)/100)))pct99=i;}fprintf(stderr,"%s:port%s:%dtest-%dkeys,readmix%d%%,elapse%ldusecs,min%ld,max%ld,P95%ld,P99%ld\n",datestr(),hostname,port,key_count,readmix,total_elapse,min,max,pct95,pct99);printf("%ld\n%e\n%e\n%e\n%e\n%e\n",key_count,(double)total/(double)key_count/1000000,(double)min/1000000,(double)max/1000000,(double)pct99/1000000,(double)pct95/1000000);}redisFree(con);free(value);fprintf(stderr,"%s:Complete.
\n",datestr());TechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnology13TechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnologyAppendixB:TuningVMs:oMinimizelatencybyadding"nohz=offhighres=offlapic=notscdeadline"tokernel(guest)commandlineHost:oScalinggovernor:performanceoDisablehugepagesoDisableKSM(randombenchmark)LinuxSwap:oEqualprioritydevicesoSetwatermark_scale_factorto4%oIncreasingnumberofpartitionsdoesnotimproveperformanceAppendixC:IntelBestPracticesforuseofLinux*Swap1.
ItishighlyrecommendedtohaveatleastoneIntelOptaneSSDDCP4800XforeachCPUsocketinthesystem,withallSSDsdividedequallyacrossallsockets.
ConsultyoursystemmanualforPCIetosocketmapping.
Forexample,inadualsocketsystem,two375GBDCP4800XdrivesattachedtoeachCPUsocketwouldprovidebetterperformancethanasingle750GBDCP4800Xdriveattachedtooneofthesockets.
AvoidconnectinganIntelOptaneSSDDCP4800XtoaPCIeslotassociatedwithaPCH.
2.
Determinethedevicename(s)ofyourSSD(s).
lsblk--outputNAME,MODEL,SIZE,REV,VENDORThedevicenameofanNVMeSSDliketheIntelOptaneSSDDCP4800Xshouldbesimilartonvme0n1.
Usethisnamewhereverthetext[devicename]appears.
Ensurethattheblockdevice(s)representsyourIntelOptaneSSDDCP4800X(s)andnot,forexample,adatadriveoryoursystembootdrive.
3.
ForeachIntelOptaneSSDDCP4800X:Note:Thesesettingsarenotretainedonreboot.
Turnoffallexistingswappartitions.
swapoff-aWipeallfilesystemsontheSSD.
WARNING:ThiswilldeletealldataontheSSD.
wipefs-a/dev/[devicename]MakeaswapareaontheSSD.
mkswap/dev/[devicename]Enabletheswapareawithpriority10.
swapon-p10/dev/[devicename]ItishighlyrecommendedthatallDCP4800Xswapdevicesusethesameswapprioritylevel,sothatswapaccessesareequallydistributedacrossbothdevices.
WhenextendingmainmemoryusingtheIntelOptaneSSDDCP4800XandLinuxSwap,itishighlyrecommendedtosetallCPUsinthesystemasNo-Callbacks(No-CBs)CPUs.
ANo-CBsCPUisaCPUwhoseRCUcallbacksareallowedtorunonCPUsotherthantheCPUmakingthecallback,whichimprovesthepageaccessqualityofservice(QoS)oftheworkloadCPUbutmayloweraveragepageaccessthroughput.
Read-Copy-Update(RCU)isaLinuxKernellockingmechanismforsharedmemory.
Whenasharedobjectinmemoryisupdated,anewobjectiscreatedinmemoryandallpointerstotheoldobjectaresettopointtothenewobject.
Oldobjectsthenneedtobeperiodicallycleanedout(garbagecollected).
TheRCUcallbackisanimplementationofgarbagecollection.
OnadefaultLinuxkernel,RCUcallbacksareruninsoftirqcontextonthesameCPUthatperformedthecallback.
ThiscancausejitterandpoorpageaccessQoSwhenRCUcallbacksarerunonworkloadCPUs,especiallywhentheworkloadmakesheavyuseofswapspace.
CPUswhicharesettobe"NoCallbacks14TechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnologyCPUs,"or"No-CBsCPUs,"areallowedtooffloadtheirRCUcallbackstothreadswhichcanthenrunonother(non-workload)CPUs.
ThisallowsforbetterpageaccessQoSontheworkloadCPU.
1.
ConfirmthatyourexistingLinuxkernelconfigurationsupportsNo-CBsCPUs.
No-CBsCPUsupportwasintroducedinLinuxKernel3.
10,butnotallLinuxdistributionkernelsenableNo-CBsCPUsupport.
CentOS7.
3supportsNo-CBsCPUs–skiptoStep4.
Ubuntu16.
04.
1LTSdoesnotsupportNo-CBsCPUs–seeStep3.
Forallotherdistributions,checkfortheCONFIG_RCU_NOCB_CPUconfigoptioninyourkernel'sconfigfile.
grep"CONFIG_RCU_NOCB_CPU"/boot/config-`uname-r`IfthecommandreturnsCONFIG_RCU_NOCB_CPU=y,yourkernelsupportsNo-CBsCPUs.
SkiptoStep4.
IfCONFIG_RCU_NOCB_CPU_ALL=y,yourkernelisalreadysettohaveallCPUsbeNo-CBsCPUs,andyoudonotneedtofollowthisguide.
Otherwise,ifthecommandreturns#CONFIG_RCU_NOCB_CPUisnotsetordoesnotreturnanytext,yourkerneldoesnotsupportNo-CBsCPUs.
SeeStep3.
2.
IfyourkerneldoesnotsupportNo-CBsCPUs,youwillneedtorebuildyourkernelwithenabledNo-CBssupport.
–Obtainthekernelsourceofyourdistribution,runmakeoldconfig,theneditthe.
configfileandaddthefollowinglines:CONFIG_RCU_NOCB_CPU=yCONFIG_RCU_NOCB_CPU_NONE=y–Buildandinstallthekernel.
–Setthercu_nocbskernelbootparameterintheGRUBconfig.
rcu_nocbsspecifieswhichCPUsaretobeNo-CBsCPUsatboottime.
OnaRedHat/CentOSmachine:oUsegrubbytoaddrcu_nocbs=0-X(whereXisthehighestCPUcorenumberinthesystem)tothekernelbootargumentslist.
oForexample,ona32-coremachine:grubby--update-kernel=ALL--args=rcu_nocbs=0-31OnanUbuntumachine:oEdit/etc/default/grubandappendrcu_nocbs=0-X(whereXisthehighestCPUcorenumberinthesystem)toGRUB_CMDLINE_LINUX_DEFAULT.
oForexample,ona32-coremachine:GRUB_CMDLINE_LINUX_DEFAULT="rcu_nocbs=0-31"oRunupdate-grub,thenrebootyourmachine.
rcu_nocbsisatunablebootparameterwhichcanalsospecifyonlyspecificCPUstobeNo-CBsCPUs,insteadofallCPUs.
Additionally,theRCUcallbackthreadsmaybeaffinitizedtospecificCPUs.
Suchtweaksarebeyondthescopeofthisguide.
Thefollowingconfigurationchangesmayfurtherimprovepagingperformance,butarenotappropriateforallsystemsorworkloads.
Ifbenchmarking,followthestepsin"SystemTuningforOptimumPerformance.
"DisablepageclusteringBydefault,whenreadingapagefromswap,multipleconsecutivepagesarereadintoDRAMatonce.
OnIntelOptaneTechnology,pageclusteringmayincuralatencypenaltyoverheadwhenreadingmultiplerandomindividualpages.
echo'0'>/proc/sys/vm/page-clusterDisabletransparenthugepages151Seeappendixforsolotest.
cofGaussianinfluenceimplementation.
2Thebenchmarkresultsmayneedtoberevisedasadditionaltestingisconducted.
Theresultsdependonthespecificplatformconfigurationsandworkloadsutlizedinthetestingandmaynotbeapplicabletoanyparticularuser'scomponents,computersystemorworkloads.
Theresultsarenonecessarilyrepresentativeofotherbenchmarksandotherbenchmarkresultsmayshowgreaterorlesserimpactfrommitigations.
Implementationdetails:SystemBIOS:00.
01.
0013(Link);Kernel4.
15.
12;Mitigationwasvalidatedforvariants1through3usingacheckerscript(Link–accessedJune21,2018).
3Source–Intel:MSRPforIntelMemoryDriveTechnologylicensewithIntelOptaneSSD(April2018).
Trendforce*-ServerDRAMPriceReportSep2017(MarketPrice)4Source-IntelMemoryDriveTechnologySetUpandConfigurationGuide,page43.
https://nsgresources.
intel.
com/asset-library/intel-memory-drive-technology-set-up-and-configuration-guide/Inteltechnologies'featuresandbenefitsdependonsystemconfigurationandmayrequireenabledhardware,softwareorserviceactivation.
Performancevariesdependingonsystemconfiguration.
Nocomputersystemcanbeabsolutelysecure.
Checkwithyoursystemmanufacturerorretailerorlearnmoreatintel.
com.
Nocomputersystemcanbeabsolutelysecure.
Softwareandworkloadsusedinperformancetestsmayhavebeenoptimizedforperformanceonlyonintelmicroprocessors.
Performancetests,suchasSYSmarkandMobileMark,aremeasuredusingspecificcomputersystems,components,software,operationsandfunctions.
Anychangetoanyofthosefactorsmaycausetheresultstovary.
Youshouldconsultotherinformationandperformanceteststoassistyouinfullyevaluatingyourcontemplatedpurchases,includingtheperformanceofthatproductwhencombinedwithotherproducts.
Formorecompleteinformationvisitwww.
intel.
com/benchmarks.
Nolicense(expressorimplied,byestoppelorotherwise)toanyintellectualpropertyrightsisgrantedbythisdocument.
Theproductsdescribedmaycontaindesigndefectsorerrorsknownaserratawhichmaycausetheproducttodeviatefrompublishedspecifications.
Currentcharacterizederrataareavailableonrequest.
Inteldisclaimsallexpressandimpliedwarranties,includingwithoutlimitation,theimpliedwarrantiesofmerchantability,fitnessforaparticularpurpose,andnon-infringement,aswellasanywarrantyarisingfromcourseofperformance,courseofdealing,orusageintrade.
Intel,theIntellogo,IntelOptanearetrademarksofIntelCorporationoritssubsidiariesintheU.
S.
and/orothercountries.
*OthernamesandbrandsmaybeclaimedaspropertyofothersIntelCorporationPrintedinUSA0419/FO/HCPleaseRecycle337919-003USTechnologyBrief|IntelOptaneDCSSDswithIntelMemoryDriveTechnologyThiswillremovetheoverheadofcoalescingpagesintohugepagesandthenbreakingthemupinswap.
echo'never'>/sys/kernel/mm/transparent_hugepage/enabledecho'never'>/sys/kernel/mm/transparent_hugepage/defragDisableNUMAbalancing.
BothswapandNUMAbalancingfrequentlyaccessLRUlists,whichmaintainthehot/coldpagesinthesystem.
DisablingNUMAbalancingpreventsitfromcontendingwithswapovertheLRUlists,improvingperformance.
echo'0'>/proc/sys/kernel/numa_balancing(Kernel4.
6andlater)Increasethewatermark_scale_factorWatermark_scale_factorisatunablevariablewhichaffectsthememorythresholdsatwhichtheswapdaemonwakesuporsleeps.
Wewantkswapdtowakeupearlier,sowemakeitwakeupwhenthereis4%ofavailablememoryleft,insteadofthedefault0.
1%ofavailablememory.
echo'400'>/proc/sys/vm/watermark_scale_factor(Kernel4.
11orlater)Recompilethekernelwithmulti-queuedeadlineI/Oschedulersupport.
ThisschedulerpreventswriteI/Orequestmergesfromblockingreadrequestsforalongtime.
Inthekernel.
config,setCONFIG_MQ_IOSCHED_DEADLINE,thenrecompilethekernel.
Useashortqueuedepthof64andsmallmax_sectors_kbto32preventwritefromblockingreadinI/Oforswapforbetterswapinlatency.