EOI_INDUCEDivybridge
ivybridge 时间:2021-03-28 阅读:()
MessagePassingWorkloadsinKVMDavidMatlack,dmatlack@google.
com1MessagePassingWorkloadsLoopbackTCP_RRIPIandHLTDISCLAIMER:x86andIntelVT-xHaltPollingInterruptsandquestionsarewelcome!
Overview2Usually,anythingthatfrequentlyswitchesbetweenrunningandidle.
Event-drivenworkloadsMemcacheLAMPserversRedisMultithreadedworkloadsusinglowlatencywait/signalprimitivesforcoordination.
WindowsEventObjectspthread_cond_wait/pthread_cond_signalInter-processcommunicationTCP_RR(benchmark)MessagePassingWorkloads3Intuition:Workloadswhichdon'tinvolveIOvirtualizationshouldrunatnearnativeperformance.
Reality:MessagePassingWorkloadsmaynotinvolveanyIObutwillstillperformnXworsethannative.
(loopback)Memcache:2xhigherlatency.
WindowsEventObjects:3-4xhigherlatency.
MessagePassingWorkloads4MessagePassingWorkloads2.
Receive1bytefromclient.
Send1byteback.
1.
Send1bytetoserver.
3.
Receive1bytefromserver.
Microbenchmark:LoopbackTCP_RRClientandServerping-pong1-byteofdataoveranestablishedTCPconnection.
Loopback:Nonetworkingdevices(realorvirtual)involved.
Performance:Latencyofeachtransaction.
Onetransaction:(idle)(idle)(idle)ClientServer5LoopbackTCP_RRPerformance6Host:IvyBridge3.
11KernelGuest:DebianWheezyBackports(3.
16Kernel)3xhigherlatency25usslowerMessagePassingon1CPUContextSwitchMessagePassingon>1CPUInterprocessor-InterruptsWhat'sgoingonunderthehoodVMEXITsareagoodplacetostartlooking.
KVMhasbuilt-inVMEXITcountersandtimers.
perf-kvm(1)VirtualOverheadsofTCP_RR7VirtualOverheadsofTCP_RRTotalNumberofVMEXITsVMEXITs/Transaction1VCPU2VCPU1VCPU2VCPUEXTERNAL_INTERRUPT16705123710.
020.
07MSR_WRITE259917043340.
009.
58IO_INSTRUCTION17867620.
000.
00EOI_INDUCED613250.
000.
00EXCEPTION_NMI289310.
000.
00CPUID2521120.
000.
00CR_ACCESS1712720.
000.
00HLT343543930.
001.
99EPT_VIOLATION200.
000.
00PAUSE_INSTRUCTION020140.
000.
012HLTperTransaction10MSR_WRITEperTransaction8HLTsofTCP_RR2HLTCPUinstruction.
StopexecutinginstructionsonthisCPUuntilaninterruptarrives.
VCPUwishestostopexecutinginstructions.
GuestOShasdecidedthatthereisnothingtodo.
Nothingtodo==idle.
Messagepassingworkloadsswitchbetweenrunningandidle.
.
.
910MSR_WRITE"WritetoModelSpecificRegister"instructionexecutedintheguest.
8APICTimer"InitialCount"Register(MSR838)Writtentostartaper-CPUtimer.
"Startcountingdownandfireaninterruptwhenyougettozero.
"ArtifactofNOHZguestkernel.
2APICInterruptCommandRegister(MSR830)Usedtosendinterprocessor-interrupts(IPI).
Usedtodeliver"messages"betweenclient/serverprocessesrunningonseparateCPUs.
MSR_WRITEsofTCP_RR10VMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU111HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERVMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU112HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERCriticalPath8pertransaction4onthecriticalpathNOHZ(ticklessguestkernel)"Disable"scheduler-tickuponenteringidle.
"Enable"scheduler-tickuponleavingidle.
scheduler-tick==APICTimer(couldalsobeTSCDeadlineTimer)Why2writespertransitioninto/outofidlehrtimer_cancelhrtimer_startAdds3-5ustoround-triplatency.
APICTimer"InitialCount"Register13HLT:x86Instruction.
CPUstopsexecutinginstructionsuntilaninterruptarrives.
ThispartofHLTisnotonthecriticalpath!
HowitworksinKVMPlaceVCPUthreadonawaitqueue.
YieldtheCPUtoanotherthread.
HLTkvm_vcpu_block->schedule()VMEXITHLTcontextswitchtoanotherusertask,kernelthread,oridleVCPU(guest)PCPU(KVM)14kvm_sched_outSendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
IPI+HLTWRMSR:APICInterruptCommandRegisterkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)guesthostkvm_sched_in*VMEXITandVMRESUMEimplementedinHardware.
time15SendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
Sameoperationonbaremetalisentirelyimplementedinhardware.
HowmuchoverheadfromvirtualizationUnlikeAPIC_TMICT,can'tjusttimeVMEXITs.
Wecancomparewiththesameoperationonphysicalhardware.
IPI+HLT16KVMversusHardwareRing0Microbenchmark(kvm-unit-tests)1.
VCPU0:HLT.
2.
~100usdelay3.
VCPU1:A=RDTSC4.
VCPU1:SendIPIto[V]CPU0.
5.
VCPU0:B=RDTSC(firstinstructionofIPIISR).
6.
Latency=B-A7.
Repeat.
RuninKVMguestandonbare-metal.
Compare!
17VMRESUMEWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVCPU1VCPU0(HLT-ed)guesthostkvm_sched_intimeKVMversusHardwareA=RDTSCB=RDTSC18Median:KVMis12xslowerPathologicalcase(witnessed):KVMis400xslowerBestcase(witnessed):KVMis11xslowerKVM:5.
7us;Hardware:0.
5usKVMversusHardwareCyclesKVMHardwareMin137001200Average15800120050%ile14900120090%ile16000130099%ile249001300Max5210001400Host:SandyBridge@2.
6GHz3.
11KernelKVMperformanceissimilaronIvyBridge(5.
6us)andHaswell(4.
9us).
19Notesaboutthisbenchmark:NoguestFPUtosave/restore.
Hostotherwiseidle(VCPUcontextswitchestoidleonHLT).
Hostpowermanagementnottheculprit.
KVMversusHardware20KVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRvmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()kvm_vcpu_kick21VMRESUMEvmx_vcpu_runkvm_vcpu_kickKVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRIPIISRVMEXITVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()22RDTSCRDTSCRDTSCRDTSCRDTSCKVMHLTInternalsWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEMin(cycles):400600730032001300VCPU1VCPU0guesthostVT-xKVMSchedulerkvm_sched_in:4924001200850034001400Median(cycles):23Unsurprisingly,theschedulertakessometimetoruntheVCPUSlowevenintheuncontended,cache-hot,case.
ImagineiftheVCPUiscontendingforCPUtimewithotherthreads.
Experiment:Don'tscheduleonHLT.
JustpollfortheIPIinkvm_vcpu_block.
KVMHLTInternals24Whathappenswhenyoudon'tscheduleonHLTKVM(Alwaysschedule)5.
7usKVM(Neverschedule)1.
7usHardware(SandyBridge)0.
5usNeverschedule!
CyclesKVM(Alwaysschedule)KVM(Neverschedule)HardwareMin1380040001200Average158004400120050%ile149004300120090%ile160004500130099%ile2490069001300Max52100050000140025SimilarimprovementsonIvyBridge(5.
6us->1.
6us)Haswell(4.
9us->1.
5us).
Neverschedule!
WRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEAlwaysschedule:4001200850034001400VCPU1VCPU0guesthostVT-xKVMSchedulerNeverschedule:300130011004001200(mediancycles)26Neverschedule!
WeeliminatealmostallofthelatencyoverheadbynotschedulingonHLT.
Schedulingisoftentherightthingtodo.
LetotherthreadsrunorsavehostCPUpower.
Mostofthetimeimprovesguestperformance(lettheIOthreadsrun!
).
Canhurtperformance.
Seemicrobenchmark.
SeeTCP_RR.
27Halt-PollingStep1:PollForuptoXnanoseconds:IfataskiswaitingtorunonourCPU,gotoStep2.
Checkifaguestinterruptarrived.
Ifso,wearedone.
Repeat.
Step2:schedule()Scheduleoutuntilit'stimetocomeoutofHLT.
Pros:WorksonshortHLTs(Cons:IncreasesCPUusage(~1%foridleVCPUsifX=200,000ns)Doesnotappeartonegativelyaffectturboofactivecores.
28Halt-PollingMemcache:1.
5xlatencyimprovementWindowsEventObjects:2xlatencyimprovementReducemessagepassinglatencyby10-15us(includingnetworklatency).
29Halt-PollingMergedintothe4.
0kernel[PATCH]kvm:addhalt_poll_nsmoduleparameterThankstoPaoloBonziniUsetheKVMmoduleparameterhalt_poll_nstocontrolhowlongtopolloneachHLT.
Futureimprovements:Automaticpolltoggling(removeidleCPUoverheadbyturningpollingoff).
Automatichalt_poll_nsKVMwillset(andvary)halt_poll_nsdynamically.
Howtodothisisanopenquestion.
.
.
ideasLazyContextSwitchingEquivalentfeature,butavailableforanykernelcomponenttouse.
30ConclusionMessagePassingEvenloopbackmessagepassingrequiresvirtualization.
Beingidle(asaLinuxguest)requiresvirtualization.
Cross-CPUcommunicationrequiresvirtualization.
Halt-Pollingsaves10-15usonmessagepassinground-triplatency.
Remaininground-triplatency:4MSRwritestotheAPICtimer(3-5us)IPIsend(~2us)HLTwakeup(evenwithhalt-polling,stilladds~3us!
)31
com1MessagePassingWorkloadsLoopbackTCP_RRIPIandHLTDISCLAIMER:x86andIntelVT-xHaltPollingInterruptsandquestionsarewelcome!
Overview2Usually,anythingthatfrequentlyswitchesbetweenrunningandidle.
Event-drivenworkloadsMemcacheLAMPserversRedisMultithreadedworkloadsusinglowlatencywait/signalprimitivesforcoordination.
WindowsEventObjectspthread_cond_wait/pthread_cond_signalInter-processcommunicationTCP_RR(benchmark)MessagePassingWorkloads3Intuition:Workloadswhichdon'tinvolveIOvirtualizationshouldrunatnearnativeperformance.
Reality:MessagePassingWorkloadsmaynotinvolveanyIObutwillstillperformnXworsethannative.
(loopback)Memcache:2xhigherlatency.
WindowsEventObjects:3-4xhigherlatency.
MessagePassingWorkloads4MessagePassingWorkloads2.
Receive1bytefromclient.
Send1byteback.
1.
Send1bytetoserver.
3.
Receive1bytefromserver.
Microbenchmark:LoopbackTCP_RRClientandServerping-pong1-byteofdataoveranestablishedTCPconnection.
Loopback:Nonetworkingdevices(realorvirtual)involved.
Performance:Latencyofeachtransaction.
Onetransaction:(idle)(idle)(idle)ClientServer5LoopbackTCP_RRPerformance6Host:IvyBridge3.
11KernelGuest:DebianWheezyBackports(3.
16Kernel)3xhigherlatency25usslowerMessagePassingon1CPUContextSwitchMessagePassingon>1CPUInterprocessor-InterruptsWhat'sgoingonunderthehoodVMEXITsareagoodplacetostartlooking.
KVMhasbuilt-inVMEXITcountersandtimers.
perf-kvm(1)VirtualOverheadsofTCP_RR7VirtualOverheadsofTCP_RRTotalNumberofVMEXITsVMEXITs/Transaction1VCPU2VCPU1VCPU2VCPUEXTERNAL_INTERRUPT16705123710.
020.
07MSR_WRITE259917043340.
009.
58IO_INSTRUCTION17867620.
000.
00EOI_INDUCED613250.
000.
00EXCEPTION_NMI289310.
000.
00CPUID2521120.
000.
00CR_ACCESS1712720.
000.
00HLT343543930.
001.
99EPT_VIOLATION200.
000.
00PAUSE_INSTRUCTION020140.
000.
012HLTperTransaction10MSR_WRITEperTransaction8HLTsofTCP_RR2HLTCPUinstruction.
StopexecutinginstructionsonthisCPUuntilaninterruptarrives.
VCPUwishestostopexecutinginstructions.
GuestOShasdecidedthatthereisnothingtodo.
Nothingtodo==idle.
Messagepassingworkloadsswitchbetweenrunningandidle.
.
.
910MSR_WRITE"WritetoModelSpecificRegister"instructionexecutedintheguest.
8APICTimer"InitialCount"Register(MSR838)Writtentostartaper-CPUtimer.
"Startcountingdownandfireaninterruptwhenyougettozero.
"ArtifactofNOHZguestkernel.
2APICInterruptCommandRegister(MSR830)Usedtosendinterprocessor-interrupts(IPI).
Usedtodeliver"messages"betweenclient/serverprocessesrunningonseparateCPUs.
MSR_WRITEsofTCP_RR10VMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU111HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERVMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU112HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERCriticalPath8pertransaction4onthecriticalpathNOHZ(ticklessguestkernel)"Disable"scheduler-tickuponenteringidle.
"Enable"scheduler-tickuponleavingidle.
scheduler-tick==APICTimer(couldalsobeTSCDeadlineTimer)Why2writespertransitioninto/outofidlehrtimer_cancelhrtimer_startAdds3-5ustoround-triplatency.
APICTimer"InitialCount"Register13HLT:x86Instruction.
CPUstopsexecutinginstructionsuntilaninterruptarrives.
ThispartofHLTisnotonthecriticalpath!
HowitworksinKVMPlaceVCPUthreadonawaitqueue.
YieldtheCPUtoanotherthread.
HLTkvm_vcpu_block->schedule()VMEXITHLTcontextswitchtoanotherusertask,kernelthread,oridleVCPU(guest)PCPU(KVM)14kvm_sched_outSendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
IPI+HLTWRMSR:APICInterruptCommandRegisterkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)guesthostkvm_sched_in*VMEXITandVMRESUMEimplementedinHardware.
time15SendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
Sameoperationonbaremetalisentirelyimplementedinhardware.
HowmuchoverheadfromvirtualizationUnlikeAPIC_TMICT,can'tjusttimeVMEXITs.
Wecancomparewiththesameoperationonphysicalhardware.
IPI+HLT16KVMversusHardwareRing0Microbenchmark(kvm-unit-tests)1.
VCPU0:HLT.
2.
~100usdelay3.
VCPU1:A=RDTSC4.
VCPU1:SendIPIto[V]CPU0.
5.
VCPU0:B=RDTSC(firstinstructionofIPIISR).
6.
Latency=B-A7.
Repeat.
RuninKVMguestandonbare-metal.
Compare!
17VMRESUMEWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVCPU1VCPU0(HLT-ed)guesthostkvm_sched_intimeKVMversusHardwareA=RDTSCB=RDTSC18Median:KVMis12xslowerPathologicalcase(witnessed):KVMis400xslowerBestcase(witnessed):KVMis11xslowerKVM:5.
7us;Hardware:0.
5usKVMversusHardwareCyclesKVMHardwareMin137001200Average15800120050%ile14900120090%ile16000130099%ile249001300Max5210001400Host:SandyBridge@2.
6GHz3.
11KernelKVMperformanceissimilaronIvyBridge(5.
6us)andHaswell(4.
9us).
19Notesaboutthisbenchmark:NoguestFPUtosave/restore.
Hostotherwiseidle(VCPUcontextswitchestoidleonHLT).
Hostpowermanagementnottheculprit.
KVMversusHardware20KVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRvmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()kvm_vcpu_kick21VMRESUMEvmx_vcpu_runkvm_vcpu_kickKVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRIPIISRVMEXITVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()22RDTSCRDTSCRDTSCRDTSCRDTSCKVMHLTInternalsWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEMin(cycles):400600730032001300VCPU1VCPU0guesthostVT-xKVMSchedulerkvm_sched_in:4924001200850034001400Median(cycles):23Unsurprisingly,theschedulertakessometimetoruntheVCPUSlowevenintheuncontended,cache-hot,case.
ImagineiftheVCPUiscontendingforCPUtimewithotherthreads.
Experiment:Don'tscheduleonHLT.
JustpollfortheIPIinkvm_vcpu_block.
KVMHLTInternals24Whathappenswhenyoudon'tscheduleonHLTKVM(Alwaysschedule)5.
7usKVM(Neverschedule)1.
7usHardware(SandyBridge)0.
5usNeverschedule!
CyclesKVM(Alwaysschedule)KVM(Neverschedule)HardwareMin1380040001200Average158004400120050%ile149004300120090%ile160004500130099%ile2490069001300Max52100050000140025SimilarimprovementsonIvyBridge(5.
6us->1.
6us)Haswell(4.
9us->1.
5us).
Neverschedule!
WRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEAlwaysschedule:4001200850034001400VCPU1VCPU0guesthostVT-xKVMSchedulerNeverschedule:300130011004001200(mediancycles)26Neverschedule!
WeeliminatealmostallofthelatencyoverheadbynotschedulingonHLT.
Schedulingisoftentherightthingtodo.
LetotherthreadsrunorsavehostCPUpower.
Mostofthetimeimprovesguestperformance(lettheIOthreadsrun!
).
Canhurtperformance.
Seemicrobenchmark.
SeeTCP_RR.
27Halt-PollingStep1:PollForuptoXnanoseconds:IfataskiswaitingtorunonourCPU,gotoStep2.
Checkifaguestinterruptarrived.
Ifso,wearedone.
Repeat.
Step2:schedule()Scheduleoutuntilit'stimetocomeoutofHLT.
Pros:WorksonshortHLTs(
28Halt-PollingMemcache:1.
5xlatencyimprovementWindowsEventObjects:2xlatencyimprovementReducemessagepassinglatencyby10-15us(includingnetworklatency).
29Halt-PollingMergedintothe4.
0kernel[PATCH]kvm:addhalt_poll_nsmoduleparameterThankstoPaoloBonziniUsetheKVMmoduleparameterhalt_poll_nstocontrolhowlongtopolloneachHLT.
Futureimprovements:Automaticpolltoggling(removeidleCPUoverheadbyturningpollingoff).
Automatichalt_poll_nsKVMwillset(andvary)halt_poll_nsdynamically.
Howtodothisisanopenquestion.
.
.
ideasLazyContextSwitchingEquivalentfeature,butavailableforanykernelcomponenttouse.
30ConclusionMessagePassingEvenloopbackmessagepassingrequiresvirtualization.
Beingidle(asaLinuxguest)requiresvirtualization.
Cross-CPUcommunicationrequiresvirtualization.
Halt-Pollingsaves10-15usonmessagepassinground-triplatency.
Remaininground-triplatency:4MSRwritestotheAPICtimer(3-5us)IPIsend(~2us)HLTwakeup(evenwithhalt-polling,stilladds~3us!
)31
- EOI_INDUCEDivybridge相关文档
- slightivybridge
- Revivybridge
- 1.0ivybridge
- 采购ivybridge
- Extreme3ivybridge
- subgroupivybridge
VinaHost，越南vps,国内延时100MS;不限流量100Mbps
vinahost怎么样?vinahost是一家越南的主机商家，至今已经成13年了，企业运营，老牌商家，销售VPS、虚拟主机、域名、邮箱、独立服务器等，机房全部在越南，有Viettle和VNPT两个机房，其中VNPT机房中三网直连国内的机房，他家的产品优势就是100Mbps不限流量。目前，VinaHost商家发布了新的优惠，购买虚拟主机、邮箱、云服务器、VPS超过三个月都有赠送相应的时长，最高送半年...
这几个Vultr VPS主机商家的优点造就商家的用户驱动力
目前云服务器市场竞争是相当的大的，比如我们在年中活动中看到各大服务商都找准这个噱头的活动发布各种活动，有的甚至就是平时的活动价格，只是换一个说法而已。可见这个行业确实竞争很大，当然我们也可以看到很多主机商几个月就消失，也有看到很多个人商家捣鼓几个品牌然后忽悠一圈跑路的。当然，个人建议在选择服务商的时候尽量选择老牌商家，这样性能更为稳定一些。近期可能会准备重新整理Vultr商家的一些信息和教程。以前...
SugarHosts新增Windows云服务器sugarhosts六折无限流量云服务器六折优惠
SugarHosts糖果主机商我们较早的站长们肯定是熟悉的，早年是提供虚拟主机起家的，如今一直还在提供虚拟主机，后来也有增加云服务器、独立服务器等。数据中心涵盖美国、德国、香港等。我们要知道大部分的海外主机商都只提供Linux系统云服务器。今天，糖果主机有新增SugarHosts夏季六折的优惠，以及新品Windows云服务器/云VPS上线。SugarHosts Windows系统云服务器有区分限制...
ivybridge为你推荐
-
vc组合有一首歌好像是什么昆虫组合？跟青春有关好像。叫什么了云计算什么是云计算？地图应用用哪个地图导航最好最准阿丽克丝·布莱肯瑞吉行尸走肉第六季女演员甲骨文不满赔偿公司倒闭员工不满一年怎么赔偿曲妙玲张婉悠香艳版《白蛇传》是电影还是写真集？杰景新特萨克斯吉普特500是台湾原产的吗xyq.163.cbg.com梦幻西游里，CBG是什么？在那里，能帮忙详细说一下吗porndao单词prondao的汉语是什么www.vtigu.com初三了，为什么考试的数学题都那么难，我最多也就135，最后一道选择，填空啊根本没法做，最后几道大题倒