EOI_INDUCEDivybridge
ivybridge 时间:2021-03-28 阅读:()
MessagePassingWorkloadsinKVMDavidMatlack,dmatlack@google.
com1MessagePassingWorkloadsLoopbackTCP_RRIPIandHLTDISCLAIMER:x86andIntelVT-xHaltPollingInterruptsandquestionsarewelcome!
Overview2Usually,anythingthatfrequentlyswitchesbetweenrunningandidle.
Event-drivenworkloadsMemcacheLAMPserversRedisMultithreadedworkloadsusinglowlatencywait/signalprimitivesforcoordination.
WindowsEventObjectspthread_cond_wait/pthread_cond_signalInter-processcommunicationTCP_RR(benchmark)MessagePassingWorkloads3Intuition:Workloadswhichdon'tinvolveIOvirtualizationshouldrunatnearnativeperformance.
Reality:MessagePassingWorkloadsmaynotinvolveanyIObutwillstillperformnXworsethannative.
(loopback)Memcache:2xhigherlatency.
WindowsEventObjects:3-4xhigherlatency.
MessagePassingWorkloads4MessagePassingWorkloads2.
Receive1bytefromclient.
Send1byteback.
1.
Send1bytetoserver.
3.
Receive1bytefromserver.
Microbenchmark:LoopbackTCP_RRClientandServerping-pong1-byteofdataoveranestablishedTCPconnection.
Loopback:Nonetworkingdevices(realorvirtual)involved.
Performance:Latencyofeachtransaction.
Onetransaction:(idle)(idle)(idle)ClientServer5LoopbackTCP_RRPerformance6Host:IvyBridge3.
11KernelGuest:DebianWheezyBackports(3.
16Kernel)3xhigherlatency25usslowerMessagePassingon1CPUContextSwitchMessagePassingon>1CPUInterprocessor-InterruptsWhat'sgoingonunderthehoodVMEXITsareagoodplacetostartlooking.
KVMhasbuilt-inVMEXITcountersandtimers.
perf-kvm(1)VirtualOverheadsofTCP_RR7VirtualOverheadsofTCP_RRTotalNumberofVMEXITsVMEXITs/Transaction1VCPU2VCPU1VCPU2VCPUEXTERNAL_INTERRUPT16705123710.
020.
07MSR_WRITE259917043340.
009.
58IO_INSTRUCTION17867620.
000.
00EOI_INDUCED613250.
000.
00EXCEPTION_NMI289310.
000.
00CPUID2521120.
000.
00CR_ACCESS1712720.
000.
00HLT343543930.
001.
99EPT_VIOLATION200.
000.
00PAUSE_INSTRUCTION020140.
000.
012HLTperTransaction10MSR_WRITEperTransaction8HLTsofTCP_RR2HLTCPUinstruction.
StopexecutinginstructionsonthisCPUuntilaninterruptarrives.
VCPUwishestostopexecutinginstructions.
GuestOShasdecidedthatthereisnothingtodo.
Nothingtodo==idle.
Messagepassingworkloadsswitchbetweenrunningandidle.
.
.
910MSR_WRITE"WritetoModelSpecificRegister"instructionexecutedintheguest.
8APICTimer"InitialCount"Register(MSR838)Writtentostartaper-CPUtimer.
"Startcountingdownandfireaninterruptwhenyougettozero.
"ArtifactofNOHZguestkernel.
2APICInterruptCommandRegister(MSR830)Usedtosendinterprocessor-interrupts(IPI).
Usedtodeliver"messages"betweenclient/serverprocessesrunningonseparateCPUs.
MSR_WRITEsofTCP_RR10VMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU111HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERVMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU112HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERCriticalPath8pertransaction4onthecriticalpathNOHZ(ticklessguestkernel)"Disable"scheduler-tickuponenteringidle.
"Enable"scheduler-tickuponleavingidle.
scheduler-tick==APICTimer(couldalsobeTSCDeadlineTimer)Why2writespertransitioninto/outofidlehrtimer_cancelhrtimer_startAdds3-5ustoround-triplatency.
APICTimer"InitialCount"Register13HLT:x86Instruction.
CPUstopsexecutinginstructionsuntilaninterruptarrives.
ThispartofHLTisnotonthecriticalpath!
HowitworksinKVMPlaceVCPUthreadonawaitqueue.
YieldtheCPUtoanotherthread.
HLTkvm_vcpu_block->schedule()VMEXITHLTcontextswitchtoanotherusertask,kernelthread,oridleVCPU(guest)PCPU(KVM)14kvm_sched_outSendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
IPI+HLTWRMSR:APICInterruptCommandRegisterkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)guesthostkvm_sched_in*VMEXITandVMRESUMEimplementedinHardware.
time15SendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
Sameoperationonbaremetalisentirelyimplementedinhardware.
HowmuchoverheadfromvirtualizationUnlikeAPIC_TMICT,can'tjusttimeVMEXITs.
Wecancomparewiththesameoperationonphysicalhardware.
IPI+HLT16KVMversusHardwareRing0Microbenchmark(kvm-unit-tests)1.
VCPU0:HLT.
2.
~100usdelay3.
VCPU1:A=RDTSC4.
VCPU1:SendIPIto[V]CPU0.
5.
VCPU0:B=RDTSC(firstinstructionofIPIISR).
6.
Latency=B-A7.
Repeat.
RuninKVMguestandonbare-metal.
Compare!
17VMRESUMEWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVCPU1VCPU0(HLT-ed)guesthostkvm_sched_intimeKVMversusHardwareA=RDTSCB=RDTSC18Median:KVMis12xslowerPathologicalcase(witnessed):KVMis400xslowerBestcase(witnessed):KVMis11xslowerKVM:5.
7us;Hardware:0.
5usKVMversusHardwareCyclesKVMHardwareMin137001200Average15800120050%ile14900120090%ile16000130099%ile249001300Max5210001400Host:SandyBridge@2.
6GHz3.
11KernelKVMperformanceissimilaronIvyBridge(5.
6us)andHaswell(4.
9us).
19Notesaboutthisbenchmark:NoguestFPUtosave/restore.
Hostotherwiseidle(VCPUcontextswitchestoidleonHLT).
Hostpowermanagementnottheculprit.
KVMversusHardware20KVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRvmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()kvm_vcpu_kick21VMRESUMEvmx_vcpu_runkvm_vcpu_kickKVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRIPIISRVMEXITVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()22RDTSCRDTSCRDTSCRDTSCRDTSCKVMHLTInternalsWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEMin(cycles):400600730032001300VCPU1VCPU0guesthostVT-xKVMSchedulerkvm_sched_in:4924001200850034001400Median(cycles):23Unsurprisingly,theschedulertakessometimetoruntheVCPUSlowevenintheuncontended,cache-hot,case.
ImagineiftheVCPUiscontendingforCPUtimewithotherthreads.
Experiment:Don'tscheduleonHLT.
JustpollfortheIPIinkvm_vcpu_block.
KVMHLTInternals24Whathappenswhenyoudon'tscheduleonHLTKVM(Alwaysschedule)5.
7usKVM(Neverschedule)1.
7usHardware(SandyBridge)0.
5usNeverschedule!
CyclesKVM(Alwaysschedule)KVM(Neverschedule)HardwareMin1380040001200Average158004400120050%ile149004300120090%ile160004500130099%ile2490069001300Max52100050000140025SimilarimprovementsonIvyBridge(5.
6us->1.
6us)Haswell(4.
9us->1.
5us).
Neverschedule!
WRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEAlwaysschedule:4001200850034001400VCPU1VCPU0guesthostVT-xKVMSchedulerNeverschedule:300130011004001200(mediancycles)26Neverschedule!
WeeliminatealmostallofthelatencyoverheadbynotschedulingonHLT.
Schedulingisoftentherightthingtodo.
LetotherthreadsrunorsavehostCPUpower.
Mostofthetimeimprovesguestperformance(lettheIOthreadsrun!
).
Canhurtperformance.
Seemicrobenchmark.
SeeTCP_RR.
27Halt-PollingStep1:PollForuptoXnanoseconds:IfataskiswaitingtorunonourCPU,gotoStep2.
Checkifaguestinterruptarrived.
Ifso,wearedone.
Repeat.
Step2:schedule()Scheduleoutuntilit'stimetocomeoutofHLT.
Pros:WorksonshortHLTs(Cons:IncreasesCPUusage(~1%foridleVCPUsifX=200,000ns)Doesnotappeartonegativelyaffectturboofactivecores.
28Halt-PollingMemcache:1.
5xlatencyimprovementWindowsEventObjects:2xlatencyimprovementReducemessagepassinglatencyby10-15us(includingnetworklatency).
29Halt-PollingMergedintothe4.
0kernel[PATCH]kvm:addhalt_poll_nsmoduleparameterThankstoPaoloBonziniUsetheKVMmoduleparameterhalt_poll_nstocontrolhowlongtopolloneachHLT.
Futureimprovements:Automaticpolltoggling(removeidleCPUoverheadbyturningpollingoff).
Automatichalt_poll_nsKVMwillset(andvary)halt_poll_nsdynamically.
Howtodothisisanopenquestion.
.
.
ideasLazyContextSwitchingEquivalentfeature,butavailableforanykernelcomponenttouse.
30ConclusionMessagePassingEvenloopbackmessagepassingrequiresvirtualization.
Beingidle(asaLinuxguest)requiresvirtualization.
Cross-CPUcommunicationrequiresvirtualization.
Halt-Pollingsaves10-15usonmessagepassinground-triplatency.
Remaininground-triplatency:4MSRwritestotheAPICtimer(3-5us)IPIsend(~2us)HLTwakeup(evenwithhalt-polling,stilladds~3us!
)31
com1MessagePassingWorkloadsLoopbackTCP_RRIPIandHLTDISCLAIMER:x86andIntelVT-xHaltPollingInterruptsandquestionsarewelcome!
Overview2Usually,anythingthatfrequentlyswitchesbetweenrunningandidle.
Event-drivenworkloadsMemcacheLAMPserversRedisMultithreadedworkloadsusinglowlatencywait/signalprimitivesforcoordination.
WindowsEventObjectspthread_cond_wait/pthread_cond_signalInter-processcommunicationTCP_RR(benchmark)MessagePassingWorkloads3Intuition:Workloadswhichdon'tinvolveIOvirtualizationshouldrunatnearnativeperformance.
Reality:MessagePassingWorkloadsmaynotinvolveanyIObutwillstillperformnXworsethannative.
(loopback)Memcache:2xhigherlatency.
WindowsEventObjects:3-4xhigherlatency.
MessagePassingWorkloads4MessagePassingWorkloads2.
Receive1bytefromclient.
Send1byteback.
1.
Send1bytetoserver.
3.
Receive1bytefromserver.
Microbenchmark:LoopbackTCP_RRClientandServerping-pong1-byteofdataoveranestablishedTCPconnection.
Loopback:Nonetworkingdevices(realorvirtual)involved.
Performance:Latencyofeachtransaction.
Onetransaction:(idle)(idle)(idle)ClientServer5LoopbackTCP_RRPerformance6Host:IvyBridge3.
11KernelGuest:DebianWheezyBackports(3.
16Kernel)3xhigherlatency25usslowerMessagePassingon1CPUContextSwitchMessagePassingon>1CPUInterprocessor-InterruptsWhat'sgoingonunderthehoodVMEXITsareagoodplacetostartlooking.
KVMhasbuilt-inVMEXITcountersandtimers.
perf-kvm(1)VirtualOverheadsofTCP_RR7VirtualOverheadsofTCP_RRTotalNumberofVMEXITsVMEXITs/Transaction1VCPU2VCPU1VCPU2VCPUEXTERNAL_INTERRUPT16705123710.
020.
07MSR_WRITE259917043340.
009.
58IO_INSTRUCTION17867620.
000.
00EOI_INDUCED613250.
000.
00EXCEPTION_NMI289310.
000.
00CPUID2521120.
000.
00CR_ACCESS1712720.
000.
00HLT343543930.
001.
99EPT_VIOLATION200.
000.
00PAUSE_INSTRUCTION020140.
000.
012HLTperTransaction10MSR_WRITEperTransaction8HLTsofTCP_RR2HLTCPUinstruction.
StopexecutinginstructionsonthisCPUuntilaninterruptarrives.
VCPUwishestostopexecutinginstructions.
GuestOShasdecidedthatthereisnothingtodo.
Nothingtodo==idle.
Messagepassingworkloadsswitchbetweenrunningandidle.
.
.
910MSR_WRITE"WritetoModelSpecificRegister"instructionexecutedintheguest.
8APICTimer"InitialCount"Register(MSR838)Writtentostartaper-CPUtimer.
"Startcountingdownandfireaninterruptwhenyougettozero.
"ArtifactofNOHZguestkernel.
2APICInterruptCommandRegister(MSR830)Usedtosendinterprocessor-interrupts(IPI).
Usedtodeliver"messages"betweenclient/serverprocessesrunningonseparateCPUs.
MSR_WRITEsofTCP_RR10VMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU111HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERVMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU112HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERCriticalPath8pertransaction4onthecriticalpathNOHZ(ticklessguestkernel)"Disable"scheduler-tickuponenteringidle.
"Enable"scheduler-tickuponleavingidle.
scheduler-tick==APICTimer(couldalsobeTSCDeadlineTimer)Why2writespertransitioninto/outofidlehrtimer_cancelhrtimer_startAdds3-5ustoround-triplatency.
APICTimer"InitialCount"Register13HLT:x86Instruction.
CPUstopsexecutinginstructionsuntilaninterruptarrives.
ThispartofHLTisnotonthecriticalpath!
HowitworksinKVMPlaceVCPUthreadonawaitqueue.
YieldtheCPUtoanotherthread.
HLTkvm_vcpu_block->schedule()VMEXITHLTcontextswitchtoanotherusertask,kernelthread,oridleVCPU(guest)PCPU(KVM)14kvm_sched_outSendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
IPI+HLTWRMSR:APICInterruptCommandRegisterkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)guesthostkvm_sched_in*VMEXITandVMRESUMEimplementedinHardware.
time15SendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
Sameoperationonbaremetalisentirelyimplementedinhardware.
HowmuchoverheadfromvirtualizationUnlikeAPIC_TMICT,can'tjusttimeVMEXITs.
Wecancomparewiththesameoperationonphysicalhardware.
IPI+HLT16KVMversusHardwareRing0Microbenchmark(kvm-unit-tests)1.
VCPU0:HLT.
2.
~100usdelay3.
VCPU1:A=RDTSC4.
VCPU1:SendIPIto[V]CPU0.
5.
VCPU0:B=RDTSC(firstinstructionofIPIISR).
6.
Latency=B-A7.
Repeat.
RuninKVMguestandonbare-metal.
Compare!
17VMRESUMEWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVCPU1VCPU0(HLT-ed)guesthostkvm_sched_intimeKVMversusHardwareA=RDTSCB=RDTSC18Median:KVMis12xslowerPathologicalcase(witnessed):KVMis400xslowerBestcase(witnessed):KVMis11xslowerKVM:5.
7us;Hardware:0.
5usKVMversusHardwareCyclesKVMHardwareMin137001200Average15800120050%ile14900120090%ile16000130099%ile249001300Max5210001400Host:SandyBridge@2.
6GHz3.
11KernelKVMperformanceissimilaronIvyBridge(5.
6us)andHaswell(4.
9us).
19Notesaboutthisbenchmark:NoguestFPUtosave/restore.
Hostotherwiseidle(VCPUcontextswitchestoidleonHLT).
Hostpowermanagementnottheculprit.
KVMversusHardware20KVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRvmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()kvm_vcpu_kick21VMRESUMEvmx_vcpu_runkvm_vcpu_kickKVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRIPIISRVMEXITVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()22RDTSCRDTSCRDTSCRDTSCRDTSCKVMHLTInternalsWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEMin(cycles):400600730032001300VCPU1VCPU0guesthostVT-xKVMSchedulerkvm_sched_in:4924001200850034001400Median(cycles):23Unsurprisingly,theschedulertakessometimetoruntheVCPUSlowevenintheuncontended,cache-hot,case.
ImagineiftheVCPUiscontendingforCPUtimewithotherthreads.
Experiment:Don'tscheduleonHLT.
JustpollfortheIPIinkvm_vcpu_block.
KVMHLTInternals24Whathappenswhenyoudon'tscheduleonHLTKVM(Alwaysschedule)5.
7usKVM(Neverschedule)1.
7usHardware(SandyBridge)0.
5usNeverschedule!
CyclesKVM(Alwaysschedule)KVM(Neverschedule)HardwareMin1380040001200Average158004400120050%ile149004300120090%ile160004500130099%ile2490069001300Max52100050000140025SimilarimprovementsonIvyBridge(5.
6us->1.
6us)Haswell(4.
9us->1.
5us).
Neverschedule!
WRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEAlwaysschedule:4001200850034001400VCPU1VCPU0guesthostVT-xKVMSchedulerNeverschedule:300130011004001200(mediancycles)26Neverschedule!
WeeliminatealmostallofthelatencyoverheadbynotschedulingonHLT.
Schedulingisoftentherightthingtodo.
LetotherthreadsrunorsavehostCPUpower.
Mostofthetimeimprovesguestperformance(lettheIOthreadsrun!
).
Canhurtperformance.
Seemicrobenchmark.
SeeTCP_RR.
27Halt-PollingStep1:PollForuptoXnanoseconds:IfataskiswaitingtorunonourCPU,gotoStep2.
Checkifaguestinterruptarrived.
Ifso,wearedone.
Repeat.
Step2:schedule()Scheduleoutuntilit'stimetocomeoutofHLT.
Pros:WorksonshortHLTs(
28Halt-PollingMemcache:1.
5xlatencyimprovementWindowsEventObjects:2xlatencyimprovementReducemessagepassinglatencyby10-15us(includingnetworklatency).
29Halt-PollingMergedintothe4.
0kernel[PATCH]kvm:addhalt_poll_nsmoduleparameterThankstoPaoloBonziniUsetheKVMmoduleparameterhalt_poll_nstocontrolhowlongtopolloneachHLT.
Futureimprovements:Automaticpolltoggling(removeidleCPUoverheadbyturningpollingoff).
Automatichalt_poll_nsKVMwillset(andvary)halt_poll_nsdynamically.
Howtodothisisanopenquestion.
.
.
ideasLazyContextSwitchingEquivalentfeature,butavailableforanykernelcomponenttouse.
30ConclusionMessagePassingEvenloopbackmessagepassingrequiresvirtualization.
Beingidle(asaLinuxguest)requiresvirtualization.
Cross-CPUcommunicationrequiresvirtualization.
Halt-Pollingsaves10-15usonmessagepassinground-triplatency.
Remaininground-triplatency:4MSRwritestotheAPICtimer(3-5us)IPIsend(~2us)HLTwakeup(evenwithhalt-polling,stilladds~3us!
)31
- EOI_INDUCEDivybridge相关文档
- slightivybridge
- Revivybridge
- 1.0ivybridge
- 采购ivybridge
- Extreme3ivybridge
- subgroupivybridge
快快云：香港沙田CN2/美国Cera大宽带/日本CN2,三网直连CN2 GIA云服务器和独立服务器
快快云怎么样?快快云是一家成立于2021年的主机服务商，致力于为用户提供高性价比稳定快速的主机托管服务，快快云目前提供有香港云服务器、美国云服务器、日本云服务器、香港独立服务器、美国独立服务器，日本独立服务器。快快云专注为个人开发者用户,中小型,大型企业用户提供一站式核心网络云端服务部署,促使用户云端部署化简为零,轻松快捷运用云计算!多年云计算领域服务经验,遍布亚太地区的海量节点为业务推进提供强大...
Contabo美国独立日促销,独立服7月€3.99/月
Contabo自4月份在新加坡增设数据中心以后，这才短短的过去不到3个月，现在同时新增了美国纽约和西雅图数据中心。可见Contabo加速了全球布局，目前可选的数据中心包括：德国本土、美国东部（纽约）、美国西部（西雅图）、美国中部（圣路易斯）和亚洲的新加坡数据中心。为了庆祝美国独立日和新增数据中心，自7月4日开始，购买美国地区的VPS、VDS和独立服务器均免设置费。Contabo是德国的老牌服务商，...
PacificRack：洛杉矶KVM月付1.5美元起,1G内存套餐年付12美元起
PacificRack在本月发布了几款特价产品，其中最低款支持月付仅1.5美元，基于KVM架构，洛杉矶机房，PR-M系列。PacificRack简称PR，QN机房旗下站点，主要提供低价VPS主机产品，基于KVM架构，数据中心为自营洛杉矶机房，现在只有PR-M一个系列，分为了2个类别：常规(Elastic Compute Service)和多IP产品(Multi IP Server)。下面列出几款秒...
ivybridge为你推荐
-
美国互联网瘫痪如果全球网络瘫痪3分钟会造成多少损失老虎数码1200万相素的数码相机都有哪些款？大概价钱是多少？www.765.com有没好的学习网站www.gegeshe.comSHE个人资料www.33xj.compro/engineer 在哪里下载，为什么找不到下载网站？avtt4.comCOM1/COM3/COM4是什么意思？？/baqizi.cc汉字的故事100字www.mfav.org手机登录WWW.brcbc.org 能注册么官人放题求日本放题系列电影，要全集越多越好，求给力ename.com要怎么在Ename.cn上注册个人域名？