EOI_INDUCEDivybridge
ivybridge 时间:2021-03-28 阅读:()
MessagePassingWorkloadsinKVMDavidMatlack,dmatlack@google.
com1MessagePassingWorkloadsLoopbackTCP_RRIPIandHLTDISCLAIMER:x86andIntelVT-xHaltPollingInterruptsandquestionsarewelcome!
Overview2Usually,anythingthatfrequentlyswitchesbetweenrunningandidle.
Event-drivenworkloadsMemcacheLAMPserversRedisMultithreadedworkloadsusinglowlatencywait/signalprimitivesforcoordination.
WindowsEventObjectspthread_cond_wait/pthread_cond_signalInter-processcommunicationTCP_RR(benchmark)MessagePassingWorkloads3Intuition:Workloadswhichdon'tinvolveIOvirtualizationshouldrunatnearnativeperformance.
Reality:MessagePassingWorkloadsmaynotinvolveanyIObutwillstillperformnXworsethannative.
(loopback)Memcache:2xhigherlatency.
WindowsEventObjects:3-4xhigherlatency.
MessagePassingWorkloads4MessagePassingWorkloads2.
Receive1bytefromclient.
Send1byteback.
1.
Send1bytetoserver.
3.
Receive1bytefromserver.
Microbenchmark:LoopbackTCP_RRClientandServerping-pong1-byteofdataoveranestablishedTCPconnection.
Loopback:Nonetworkingdevices(realorvirtual)involved.
Performance:Latencyofeachtransaction.
Onetransaction:(idle)(idle)(idle)ClientServer5LoopbackTCP_RRPerformance6Host:IvyBridge3.
11KernelGuest:DebianWheezyBackports(3.
16Kernel)3xhigherlatency25usslowerMessagePassingon1CPUContextSwitchMessagePassingon>1CPUInterprocessor-InterruptsWhat'sgoingonunderthehoodVMEXITsareagoodplacetostartlooking.
KVMhasbuilt-inVMEXITcountersandtimers.
perf-kvm(1)VirtualOverheadsofTCP_RR7VirtualOverheadsofTCP_RRTotalNumberofVMEXITsVMEXITs/Transaction1VCPU2VCPU1VCPU2VCPUEXTERNAL_INTERRUPT16705123710.
020.
07MSR_WRITE259917043340.
009.
58IO_INSTRUCTION17867620.
000.
00EOI_INDUCED613250.
000.
00EXCEPTION_NMI289310.
000.
00CPUID2521120.
000.
00CR_ACCESS1712720.
000.
00HLT343543930.
001.
99EPT_VIOLATION200.
000.
00PAUSE_INSTRUCTION020140.
000.
012HLTperTransaction10MSR_WRITEperTransaction8HLTsofTCP_RR2HLTCPUinstruction.
StopexecutinginstructionsonthisCPUuntilaninterruptarrives.
VCPUwishestostopexecutinginstructions.
GuestOShasdecidedthatthereisnothingtodo.
Nothingtodo==idle.
Messagepassingworkloadsswitchbetweenrunningandidle.
.
.
910MSR_WRITE"WritetoModelSpecificRegister"instructionexecutedintheguest.
8APICTimer"InitialCount"Register(MSR838)Writtentostartaper-CPUtimer.
"Startcountingdownandfireaninterruptwhenyougettozero.
"ArtifactofNOHZguestkernel.
2APICInterruptCommandRegister(MSR830)Usedtosendinterprocessor-interrupts(IPI).
Usedtodeliver"messages"betweenclient/serverprocessesrunningonseparateCPUs.
MSR_WRITEsofTCP_RR10VMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU111HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERVMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU112HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERCriticalPath8pertransaction4onthecriticalpathNOHZ(ticklessguestkernel)"Disable"scheduler-tickuponenteringidle.
"Enable"scheduler-tickuponleavingidle.
scheduler-tick==APICTimer(couldalsobeTSCDeadlineTimer)Why2writespertransitioninto/outofidlehrtimer_cancelhrtimer_startAdds3-5ustoround-triplatency.
APICTimer"InitialCount"Register13HLT:x86Instruction.
CPUstopsexecutinginstructionsuntilaninterruptarrives.
ThispartofHLTisnotonthecriticalpath!
HowitworksinKVMPlaceVCPUthreadonawaitqueue.
YieldtheCPUtoanotherthread.
HLTkvm_vcpu_block->schedule()VMEXITHLTcontextswitchtoanotherusertask,kernelthread,oridleVCPU(guest)PCPU(KVM)14kvm_sched_outSendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
IPI+HLTWRMSR:APICInterruptCommandRegisterkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)guesthostkvm_sched_in*VMEXITandVMRESUMEimplementedinHardware.
time15SendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
Sameoperationonbaremetalisentirelyimplementedinhardware.
HowmuchoverheadfromvirtualizationUnlikeAPIC_TMICT,can'tjusttimeVMEXITs.
Wecancomparewiththesameoperationonphysicalhardware.
IPI+HLT16KVMversusHardwareRing0Microbenchmark(kvm-unit-tests)1.
VCPU0:HLT.
2.
~100usdelay3.
VCPU1:A=RDTSC4.
VCPU1:SendIPIto[V]CPU0.
5.
VCPU0:B=RDTSC(firstinstructionofIPIISR).
6.
Latency=B-A7.
Repeat.
RuninKVMguestandonbare-metal.
Compare!
17VMRESUMEWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVCPU1VCPU0(HLT-ed)guesthostkvm_sched_intimeKVMversusHardwareA=RDTSCB=RDTSC18Median:KVMis12xslowerPathologicalcase(witnessed):KVMis400xslowerBestcase(witnessed):KVMis11xslowerKVM:5.
7us;Hardware:0.
5usKVMversusHardwareCyclesKVMHardwareMin137001200Average15800120050%ile14900120090%ile16000130099%ile249001300Max5210001400Host:SandyBridge@2.
6GHz3.
11KernelKVMperformanceissimilaronIvyBridge(5.
6us)andHaswell(4.
9us).
19Notesaboutthisbenchmark:NoguestFPUtosave/restore.
Hostotherwiseidle(VCPUcontextswitchestoidleonHLT).
Hostpowermanagementnottheculprit.
KVMversusHardware20KVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRvmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()kvm_vcpu_kick21VMRESUMEvmx_vcpu_runkvm_vcpu_kickKVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRIPIISRVMEXITVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()22RDTSCRDTSCRDTSCRDTSCRDTSCKVMHLTInternalsWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEMin(cycles):400600730032001300VCPU1VCPU0guesthostVT-xKVMSchedulerkvm_sched_in:4924001200850034001400Median(cycles):23Unsurprisingly,theschedulertakessometimetoruntheVCPUSlowevenintheuncontended,cache-hot,case.
ImagineiftheVCPUiscontendingforCPUtimewithotherthreads.
Experiment:Don'tscheduleonHLT.
JustpollfortheIPIinkvm_vcpu_block.
KVMHLTInternals24Whathappenswhenyoudon'tscheduleonHLTKVM(Alwaysschedule)5.
7usKVM(Neverschedule)1.
7usHardware(SandyBridge)0.
5usNeverschedule!
CyclesKVM(Alwaysschedule)KVM(Neverschedule)HardwareMin1380040001200Average158004400120050%ile149004300120090%ile160004500130099%ile2490069001300Max52100050000140025SimilarimprovementsonIvyBridge(5.
6us->1.
6us)Haswell(4.
9us->1.
5us).
Neverschedule!
WRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEAlwaysschedule:4001200850034001400VCPU1VCPU0guesthostVT-xKVMSchedulerNeverschedule:300130011004001200(mediancycles)26Neverschedule!
WeeliminatealmostallofthelatencyoverheadbynotschedulingonHLT.
Schedulingisoftentherightthingtodo.
LetotherthreadsrunorsavehostCPUpower.
Mostofthetimeimprovesguestperformance(lettheIOthreadsrun!
).
Canhurtperformance.
Seemicrobenchmark.
SeeTCP_RR.
27Halt-PollingStep1:PollForuptoXnanoseconds:IfataskiswaitingtorunonourCPU,gotoStep2.
Checkifaguestinterruptarrived.
Ifso,wearedone.
Repeat.
Step2:schedule()Scheduleoutuntilit'stimetocomeoutofHLT.
Pros:WorksonshortHLTs(Cons:IncreasesCPUusage(~1%foridleVCPUsifX=200,000ns)Doesnotappeartonegativelyaffectturboofactivecores.
28Halt-PollingMemcache:1.
5xlatencyimprovementWindowsEventObjects:2xlatencyimprovementReducemessagepassinglatencyby10-15us(includingnetworklatency).
29Halt-PollingMergedintothe4.
0kernel[PATCH]kvm:addhalt_poll_nsmoduleparameterThankstoPaoloBonziniUsetheKVMmoduleparameterhalt_poll_nstocontrolhowlongtopolloneachHLT.
Futureimprovements:Automaticpolltoggling(removeidleCPUoverheadbyturningpollingoff).
Automatichalt_poll_nsKVMwillset(andvary)halt_poll_nsdynamically.
Howtodothisisanopenquestion.
.
.
ideasLazyContextSwitchingEquivalentfeature,butavailableforanykernelcomponenttouse.
30ConclusionMessagePassingEvenloopbackmessagepassingrequiresvirtualization.
Beingidle(asaLinuxguest)requiresvirtualization.
Cross-CPUcommunicationrequiresvirtualization.
Halt-Pollingsaves10-15usonmessagepassinground-triplatency.
Remaininground-triplatency:4MSRwritestotheAPICtimer(3-5us)IPIsend(~2us)HLTwakeup(evenwithhalt-polling,stilladds~3us!
)31
com1MessagePassingWorkloadsLoopbackTCP_RRIPIandHLTDISCLAIMER:x86andIntelVT-xHaltPollingInterruptsandquestionsarewelcome!
Overview2Usually,anythingthatfrequentlyswitchesbetweenrunningandidle.
Event-drivenworkloadsMemcacheLAMPserversRedisMultithreadedworkloadsusinglowlatencywait/signalprimitivesforcoordination.
WindowsEventObjectspthread_cond_wait/pthread_cond_signalInter-processcommunicationTCP_RR(benchmark)MessagePassingWorkloads3Intuition:Workloadswhichdon'tinvolveIOvirtualizationshouldrunatnearnativeperformance.
Reality:MessagePassingWorkloadsmaynotinvolveanyIObutwillstillperformnXworsethannative.
(loopback)Memcache:2xhigherlatency.
WindowsEventObjects:3-4xhigherlatency.
MessagePassingWorkloads4MessagePassingWorkloads2.
Receive1bytefromclient.
Send1byteback.
1.
Send1bytetoserver.
3.
Receive1bytefromserver.
Microbenchmark:LoopbackTCP_RRClientandServerping-pong1-byteofdataoveranestablishedTCPconnection.
Loopback:Nonetworkingdevices(realorvirtual)involved.
Performance:Latencyofeachtransaction.
Onetransaction:(idle)(idle)(idle)ClientServer5LoopbackTCP_RRPerformance6Host:IvyBridge3.
11KernelGuest:DebianWheezyBackports(3.
16Kernel)3xhigherlatency25usslowerMessagePassingon1CPUContextSwitchMessagePassingon>1CPUInterprocessor-InterruptsWhat'sgoingonunderthehoodVMEXITsareagoodplacetostartlooking.
KVMhasbuilt-inVMEXITcountersandtimers.
perf-kvm(1)VirtualOverheadsofTCP_RR7VirtualOverheadsofTCP_RRTotalNumberofVMEXITsVMEXITs/Transaction1VCPU2VCPU1VCPU2VCPUEXTERNAL_INTERRUPT16705123710.
020.
07MSR_WRITE259917043340.
009.
58IO_INSTRUCTION17867620.
000.
00EOI_INDUCED613250.
000.
00EXCEPTION_NMI289310.
000.
00CPUID2521120.
000.
00CR_ACCESS1712720.
000.
00HLT343543930.
001.
99EPT_VIOLATION200.
000.
00PAUSE_INSTRUCTION020140.
000.
012HLTperTransaction10MSR_WRITEperTransaction8HLTsofTCP_RR2HLTCPUinstruction.
StopexecutinginstructionsonthisCPUuntilaninterruptarrives.
VCPUwishestostopexecutinginstructions.
GuestOShasdecidedthatthereisnothingtodo.
Nothingtodo==idle.
Messagepassingworkloadsswitchbetweenrunningandidle.
.
.
910MSR_WRITE"WritetoModelSpecificRegister"instructionexecutedintheguest.
8APICTimer"InitialCount"Register(MSR838)Writtentostartaper-CPUtimer.
"Startcountingdownandfireaninterruptwhenyougettozero.
"ArtifactofNOHZguestkernel.
2APICInterruptCommandRegister(MSR830)Usedtosendinterprocessor-interrupts(IPI).
Usedtodeliver"messages"betweenclient/serverprocessesrunningonseparateCPUs.
MSR_WRITEsofTCP_RR10VMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU111HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERVMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU112HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERCriticalPath8pertransaction4onthecriticalpathNOHZ(ticklessguestkernel)"Disable"scheduler-tickuponenteringidle.
"Enable"scheduler-tickuponleavingidle.
scheduler-tick==APICTimer(couldalsobeTSCDeadlineTimer)Why2writespertransitioninto/outofidlehrtimer_cancelhrtimer_startAdds3-5ustoround-triplatency.
APICTimer"InitialCount"Register13HLT:x86Instruction.
CPUstopsexecutinginstructionsuntilaninterruptarrives.
ThispartofHLTisnotonthecriticalpath!
HowitworksinKVMPlaceVCPUthreadonawaitqueue.
YieldtheCPUtoanotherthread.
HLTkvm_vcpu_block->schedule()VMEXITHLTcontextswitchtoanotherusertask,kernelthread,oridleVCPU(guest)PCPU(KVM)14kvm_sched_outSendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
IPI+HLTWRMSR:APICInterruptCommandRegisterkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)guesthostkvm_sched_in*VMEXITandVMRESUMEimplementedinHardware.
time15SendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
Sameoperationonbaremetalisentirelyimplementedinhardware.
HowmuchoverheadfromvirtualizationUnlikeAPIC_TMICT,can'tjusttimeVMEXITs.
Wecancomparewiththesameoperationonphysicalhardware.
IPI+HLT16KVMversusHardwareRing0Microbenchmark(kvm-unit-tests)1.
VCPU0:HLT.
2.
~100usdelay3.
VCPU1:A=RDTSC4.
VCPU1:SendIPIto[V]CPU0.
5.
VCPU0:B=RDTSC(firstinstructionofIPIISR).
6.
Latency=B-A7.
Repeat.
RuninKVMguestandonbare-metal.
Compare!
17VMRESUMEWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVCPU1VCPU0(HLT-ed)guesthostkvm_sched_intimeKVMversusHardwareA=RDTSCB=RDTSC18Median:KVMis12xslowerPathologicalcase(witnessed):KVMis400xslowerBestcase(witnessed):KVMis11xslowerKVM:5.
7us;Hardware:0.
5usKVMversusHardwareCyclesKVMHardwareMin137001200Average15800120050%ile14900120090%ile16000130099%ile249001300Max5210001400Host:SandyBridge@2.
6GHz3.
11KernelKVMperformanceissimilaronIvyBridge(5.
6us)andHaswell(4.
9us).
19Notesaboutthisbenchmark:NoguestFPUtosave/restore.
Hostotherwiseidle(VCPUcontextswitchestoidleonHLT).
Hostpowermanagementnottheculprit.
KVMversusHardware20KVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRvmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()kvm_vcpu_kick21VMRESUMEvmx_vcpu_runkvm_vcpu_kickKVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRIPIISRVMEXITVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()22RDTSCRDTSCRDTSCRDTSCRDTSCKVMHLTInternalsWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEMin(cycles):400600730032001300VCPU1VCPU0guesthostVT-xKVMSchedulerkvm_sched_in:4924001200850034001400Median(cycles):23Unsurprisingly,theschedulertakessometimetoruntheVCPUSlowevenintheuncontended,cache-hot,case.
ImagineiftheVCPUiscontendingforCPUtimewithotherthreads.
Experiment:Don'tscheduleonHLT.
JustpollfortheIPIinkvm_vcpu_block.
KVMHLTInternals24Whathappenswhenyoudon'tscheduleonHLTKVM(Alwaysschedule)5.
7usKVM(Neverschedule)1.
7usHardware(SandyBridge)0.
5usNeverschedule!
CyclesKVM(Alwaysschedule)KVM(Neverschedule)HardwareMin1380040001200Average158004400120050%ile149004300120090%ile160004500130099%ile2490069001300Max52100050000140025SimilarimprovementsonIvyBridge(5.
6us->1.
6us)Haswell(4.
9us->1.
5us).
Neverschedule!
WRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEAlwaysschedule:4001200850034001400VCPU1VCPU0guesthostVT-xKVMSchedulerNeverschedule:300130011004001200(mediancycles)26Neverschedule!
WeeliminatealmostallofthelatencyoverheadbynotschedulingonHLT.
Schedulingisoftentherightthingtodo.
LetotherthreadsrunorsavehostCPUpower.
Mostofthetimeimprovesguestperformance(lettheIOthreadsrun!
).
Canhurtperformance.
Seemicrobenchmark.
SeeTCP_RR.
27Halt-PollingStep1:PollForuptoXnanoseconds:IfataskiswaitingtorunonourCPU,gotoStep2.
Checkifaguestinterruptarrived.
Ifso,wearedone.
Repeat.
Step2:schedule()Scheduleoutuntilit'stimetocomeoutofHLT.
Pros:WorksonshortHLTs(
28Halt-PollingMemcache:1.
5xlatencyimprovementWindowsEventObjects:2xlatencyimprovementReducemessagepassinglatencyby10-15us(includingnetworklatency).
29Halt-PollingMergedintothe4.
0kernel[PATCH]kvm:addhalt_poll_nsmoduleparameterThankstoPaoloBonziniUsetheKVMmoduleparameterhalt_poll_nstocontrolhowlongtopolloneachHLT.
Futureimprovements:Automaticpolltoggling(removeidleCPUoverheadbyturningpollingoff).
Automatichalt_poll_nsKVMwillset(andvary)halt_poll_nsdynamically.
Howtodothisisanopenquestion.
.
.
ideasLazyContextSwitchingEquivalentfeature,butavailableforanykernelcomponenttouse.
30ConclusionMessagePassingEvenloopbackmessagepassingrequiresvirtualization.
Beingidle(asaLinuxguest)requiresvirtualization.
Cross-CPUcommunicationrequiresvirtualization.
Halt-Pollingsaves10-15usonmessagepassinground-triplatency.
Remaininground-triplatency:4MSRwritestotheAPICtimer(3-5us)IPIsend(~2us)HLTwakeup(evenwithhalt-polling,stilladds~3us!
)31
- EOI_INDUCEDivybridge相关文档
- slightivybridge
- Revivybridge
- 1.0ivybridge
- 采购ivybridge
- Extreme3ivybridge
- subgroupivybridge
趣米云月付460元，香港CN2云服务器VPS月付低至18元
趣米云早期为做技术起家，为3家IDC提供技术服务2年多，目前商家在售的服务有香港vps、香港独立服务器、香港站群服务器等，线路方面都是目前最优质的CN2，直连大陆，延时非常低，适合做站，目前商家正在做七月优惠活动，VPS低至18元，价格算是比较便宜的了。趣米云vps优惠套餐：KVM虚拟架构，香港沙田机房，线路采用三网(电信，联通，移动)回程电信cn2、cn2 gia优质网络，延迟低，速度快。自行封...
提速啦(900元/月),杭州BGP E5-2665/89*2 32核 48G 100G防御
提速啦的来历提速啦是 网站 本着“良心 便宜 稳定”的初衷 为小白用户避免被坑提速啦的市场定位提速啦主要代理市场稳定速度的云服务器产品，避免新手购买云服务器的时候众多商家不知道如何选择，妮妮云就帮你选择好了产品，无需承担购买风险，不用担心出现被跑路 被诈骗的情况。提速啦的售后保证提速啦退款 通过于合作商的友好协商，云服务器提供3天内全额退款，超过3天不退款 物理机部分支持当天全额退款提速啦提现 充...
PacificRack：洛杉矶KVM月付1.5美元起,1G内存套餐年付12美元起
PacificRack在本月发布了几款特价产品，其中最低款支持月付仅1.5美元，基于KVM架构，洛杉矶机房，PR-M系列。PacificRack简称PR，QN机房旗下站点，主要提供低价VPS主机产品，基于KVM架构，数据中心为自营洛杉矶机房，现在只有PR-M一个系列，分为了2个类别：常规(Elastic Compute Service)和多IP产品(Multi IP Server)。下面列出几款秒...
ivybridge为你推荐
-
阿丽克丝·布莱肯瑞吉阿丽克斯布莱肯瑞吉演的美国恐怖故事哪两集access数据库什么是ACCESS数据库同ip网站查询服务器禁PING 是不是就可以解决同IP网站查询问题7788k.com以前有个网站是7788MP3.com后来改成KK130现在又改网站域名了。有知道现在是什么域名么？地陷裂口天上顿时露出一个大窟窿地上也裂开了,一到黑幽幽的深沟可以用什么四字词语来？www.119mm.comwww.993mm+com精品集！336.com求那个网站 你懂得 1552517773@qqwww.baitu.com韩国片爱人.欲望的观看地址dadi.tvapple TV 功能介绍www.15job.com南方人才市场有官方网站是什么？