EOI_INDUCEDivybridge
ivybridge 时间:2021-03-28 阅读:()
MessagePassingWorkloadsinKVMDavidMatlack,dmatlack@google.
com1MessagePassingWorkloadsLoopbackTCP_RRIPIandHLTDISCLAIMER:x86andIntelVT-xHaltPollingInterruptsandquestionsarewelcome!
Overview2Usually,anythingthatfrequentlyswitchesbetweenrunningandidle.
Event-drivenworkloadsMemcacheLAMPserversRedisMultithreadedworkloadsusinglowlatencywait/signalprimitivesforcoordination.
WindowsEventObjectspthread_cond_wait/pthread_cond_signalInter-processcommunicationTCP_RR(benchmark)MessagePassingWorkloads3Intuition:Workloadswhichdon'tinvolveIOvirtualizationshouldrunatnearnativeperformance.
Reality:MessagePassingWorkloadsmaynotinvolveanyIObutwillstillperformnXworsethannative.
(loopback)Memcache:2xhigherlatency.
WindowsEventObjects:3-4xhigherlatency.
MessagePassingWorkloads4MessagePassingWorkloads2.
Receive1bytefromclient.
Send1byteback.
1.
Send1bytetoserver.
3.
Receive1bytefromserver.
Microbenchmark:LoopbackTCP_RRClientandServerping-pong1-byteofdataoveranestablishedTCPconnection.
Loopback:Nonetworkingdevices(realorvirtual)involved.
Performance:Latencyofeachtransaction.
Onetransaction:(idle)(idle)(idle)ClientServer5LoopbackTCP_RRPerformance6Host:IvyBridge3.
11KernelGuest:DebianWheezyBackports(3.
16Kernel)3xhigherlatency25usslowerMessagePassingon1CPUContextSwitchMessagePassingon>1CPUInterprocessor-InterruptsWhat'sgoingonunderthehoodVMEXITsareagoodplacetostartlooking.
KVMhasbuilt-inVMEXITcountersandtimers.
perf-kvm(1)VirtualOverheadsofTCP_RR7VirtualOverheadsofTCP_RRTotalNumberofVMEXITsVMEXITs/Transaction1VCPU2VCPU1VCPU2VCPUEXTERNAL_INTERRUPT16705123710.
020.
07MSR_WRITE259917043340.
009.
58IO_INSTRUCTION17867620.
000.
00EOI_INDUCED613250.
000.
00EXCEPTION_NMI289310.
000.
00CPUID2521120.
000.
00CR_ACCESS1712720.
000.
00HLT343543930.
001.
99EPT_VIOLATION200.
000.
00PAUSE_INSTRUCTION020140.
000.
012HLTperTransaction10MSR_WRITEperTransaction8HLTsofTCP_RR2HLTCPUinstruction.
StopexecutinginstructionsonthisCPUuntilaninterruptarrives.
VCPUwishestostopexecutinginstructions.
GuestOShasdecidedthatthereisnothingtodo.
Nothingtodo==idle.
Messagepassingworkloadsswitchbetweenrunningandidle.
.
.
910MSR_WRITE"WritetoModelSpecificRegister"instructionexecutedintheguest.
8APICTimer"InitialCount"Register(MSR838)Writtentostartaper-CPUtimer.
"Startcountingdownandfireaninterruptwhenyougettozero.
"ArtifactofNOHZguestkernel.
2APICInterruptCommandRegister(MSR830)Usedtosendinterprocessor-interrupts(IPI).
Usedtodeliver"messages"betweenclient/serverprocessesrunningonseparateCPUs.
MSR_WRITEsofTCP_RR10VMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU111HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERVMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU112HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERCriticalPath8pertransaction4onthecriticalpathNOHZ(ticklessguestkernel)"Disable"scheduler-tickuponenteringidle.
"Enable"scheduler-tickuponleavingidle.
scheduler-tick==APICTimer(couldalsobeTSCDeadlineTimer)Why2writespertransitioninto/outofidlehrtimer_cancelhrtimer_startAdds3-5ustoround-triplatency.
APICTimer"InitialCount"Register13HLT:x86Instruction.
CPUstopsexecutinginstructionsuntilaninterruptarrives.
ThispartofHLTisnotonthecriticalpath!
HowitworksinKVMPlaceVCPUthreadonawaitqueue.
YieldtheCPUtoanotherthread.
HLTkvm_vcpu_block->schedule()VMEXITHLTcontextswitchtoanotherusertask,kernelthread,oridleVCPU(guest)PCPU(KVM)14kvm_sched_outSendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
IPI+HLTWRMSR:APICInterruptCommandRegisterkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)guesthostkvm_sched_in*VMEXITandVMRESUMEimplementedinHardware.
time15SendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
Sameoperationonbaremetalisentirelyimplementedinhardware.
HowmuchoverheadfromvirtualizationUnlikeAPIC_TMICT,can'tjusttimeVMEXITs.
Wecancomparewiththesameoperationonphysicalhardware.
IPI+HLT16KVMversusHardwareRing0Microbenchmark(kvm-unit-tests)1.
VCPU0:HLT.
2.
~100usdelay3.
VCPU1:A=RDTSC4.
VCPU1:SendIPIto[V]CPU0.
5.
VCPU0:B=RDTSC(firstinstructionofIPIISR).
6.
Latency=B-A7.
Repeat.
RuninKVMguestandonbare-metal.
Compare!
17VMRESUMEWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVCPU1VCPU0(HLT-ed)guesthostkvm_sched_intimeKVMversusHardwareA=RDTSCB=RDTSC18Median:KVMis12xslowerPathologicalcase(witnessed):KVMis400xslowerBestcase(witnessed):KVMis11xslowerKVM:5.
7us;Hardware:0.
5usKVMversusHardwareCyclesKVMHardwareMin137001200Average15800120050%ile14900120090%ile16000130099%ile249001300Max5210001400Host:SandyBridge@2.
6GHz3.
11KernelKVMperformanceissimilaronIvyBridge(5.
6us)andHaswell(4.
9us).
19Notesaboutthisbenchmark:NoguestFPUtosave/restore.
Hostotherwiseidle(VCPUcontextswitchestoidleonHLT).
Hostpowermanagementnottheculprit.
KVMversusHardware20KVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRvmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()kvm_vcpu_kick21VMRESUMEvmx_vcpu_runkvm_vcpu_kickKVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRIPIISRVMEXITVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()22RDTSCRDTSCRDTSCRDTSCRDTSCKVMHLTInternalsWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEMin(cycles):400600730032001300VCPU1VCPU0guesthostVT-xKVMSchedulerkvm_sched_in:4924001200850034001400Median(cycles):23Unsurprisingly,theschedulertakessometimetoruntheVCPUSlowevenintheuncontended,cache-hot,case.
ImagineiftheVCPUiscontendingforCPUtimewithotherthreads.
Experiment:Don'tscheduleonHLT.
JustpollfortheIPIinkvm_vcpu_block.
KVMHLTInternals24Whathappenswhenyoudon'tscheduleonHLTKVM(Alwaysschedule)5.
7usKVM(Neverschedule)1.
7usHardware(SandyBridge)0.
5usNeverschedule!
CyclesKVM(Alwaysschedule)KVM(Neverschedule)HardwareMin1380040001200Average158004400120050%ile149004300120090%ile160004500130099%ile2490069001300Max52100050000140025SimilarimprovementsonIvyBridge(5.
6us->1.
6us)Haswell(4.
9us->1.
5us).
Neverschedule!
WRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEAlwaysschedule:4001200850034001400VCPU1VCPU0guesthostVT-xKVMSchedulerNeverschedule:300130011004001200(mediancycles)26Neverschedule!
WeeliminatealmostallofthelatencyoverheadbynotschedulingonHLT.
Schedulingisoftentherightthingtodo.
LetotherthreadsrunorsavehostCPUpower.
Mostofthetimeimprovesguestperformance(lettheIOthreadsrun!
).
Canhurtperformance.
Seemicrobenchmark.
SeeTCP_RR.
27Halt-PollingStep1:PollForuptoXnanoseconds:IfataskiswaitingtorunonourCPU,gotoStep2.
Checkifaguestinterruptarrived.
Ifso,wearedone.
Repeat.
Step2:schedule()Scheduleoutuntilit'stimetocomeoutofHLT.
Pros:WorksonshortHLTs(Cons:IncreasesCPUusage(~1%foridleVCPUsifX=200,000ns)Doesnotappeartonegativelyaffectturboofactivecores.
28Halt-PollingMemcache:1.
5xlatencyimprovementWindowsEventObjects:2xlatencyimprovementReducemessagepassinglatencyby10-15us(includingnetworklatency).
29Halt-PollingMergedintothe4.
0kernel[PATCH]kvm:addhalt_poll_nsmoduleparameterThankstoPaoloBonziniUsetheKVMmoduleparameterhalt_poll_nstocontrolhowlongtopolloneachHLT.
Futureimprovements:Automaticpolltoggling(removeidleCPUoverheadbyturningpollingoff).
Automatichalt_poll_nsKVMwillset(andvary)halt_poll_nsdynamically.
Howtodothisisanopenquestion.
.
.
ideasLazyContextSwitchingEquivalentfeature,butavailableforanykernelcomponenttouse.
30ConclusionMessagePassingEvenloopbackmessagepassingrequiresvirtualization.
Beingidle(asaLinuxguest)requiresvirtualization.
Cross-CPUcommunicationrequiresvirtualization.
Halt-Pollingsaves10-15usonmessagepassinground-triplatency.
Remaininground-triplatency:4MSRwritestotheAPICtimer(3-5us)IPIsend(~2us)HLTwakeup(evenwithhalt-polling,stilladds~3us!
)31
com1MessagePassingWorkloadsLoopbackTCP_RRIPIandHLTDISCLAIMER:x86andIntelVT-xHaltPollingInterruptsandquestionsarewelcome!
Overview2Usually,anythingthatfrequentlyswitchesbetweenrunningandidle.
Event-drivenworkloadsMemcacheLAMPserversRedisMultithreadedworkloadsusinglowlatencywait/signalprimitivesforcoordination.
WindowsEventObjectspthread_cond_wait/pthread_cond_signalInter-processcommunicationTCP_RR(benchmark)MessagePassingWorkloads3Intuition:Workloadswhichdon'tinvolveIOvirtualizationshouldrunatnearnativeperformance.
Reality:MessagePassingWorkloadsmaynotinvolveanyIObutwillstillperformnXworsethannative.
(loopback)Memcache:2xhigherlatency.
WindowsEventObjects:3-4xhigherlatency.
MessagePassingWorkloads4MessagePassingWorkloads2.
Receive1bytefromclient.
Send1byteback.
1.
Send1bytetoserver.
3.
Receive1bytefromserver.
Microbenchmark:LoopbackTCP_RRClientandServerping-pong1-byteofdataoveranestablishedTCPconnection.
Loopback:Nonetworkingdevices(realorvirtual)involved.
Performance:Latencyofeachtransaction.
Onetransaction:(idle)(idle)(idle)ClientServer5LoopbackTCP_RRPerformance6Host:IvyBridge3.
11KernelGuest:DebianWheezyBackports(3.
16Kernel)3xhigherlatency25usslowerMessagePassingon1CPUContextSwitchMessagePassingon>1CPUInterprocessor-InterruptsWhat'sgoingonunderthehoodVMEXITsareagoodplacetostartlooking.
KVMhasbuilt-inVMEXITcountersandtimers.
perf-kvm(1)VirtualOverheadsofTCP_RR7VirtualOverheadsofTCP_RRTotalNumberofVMEXITsVMEXITs/Transaction1VCPU2VCPU1VCPU2VCPUEXTERNAL_INTERRUPT16705123710.
020.
07MSR_WRITE259917043340.
009.
58IO_INSTRUCTION17867620.
000.
00EOI_INDUCED613250.
000.
00EXCEPTION_NMI289310.
000.
00CPUID2521120.
000.
00CR_ACCESS1712720.
000.
00HLT343543930.
001.
99EPT_VIOLATION200.
000.
00PAUSE_INSTRUCTION020140.
000.
012HLTperTransaction10MSR_WRITEperTransaction8HLTsofTCP_RR2HLTCPUinstruction.
StopexecutinginstructionsonthisCPUuntilaninterruptarrives.
VCPUwishestostopexecutinginstructions.
GuestOShasdecidedthatthereisnothingtodo.
Nothingtodo==idle.
Messagepassingworkloadsswitchbetweenrunningandidle.
.
.
910MSR_WRITE"WritetoModelSpecificRegister"instructionexecutedintheguest.
8APICTimer"InitialCount"Register(MSR838)Writtentostartaper-CPUtimer.
"Startcountingdownandfireaninterruptwhenyougettozero.
"ArtifactofNOHZguestkernel.
2APICInterruptCommandRegister(MSR830)Usedtosendinterprocessor-interrupts(IPI).
Usedtodeliver"messages"betweenclient/serverprocessesrunningonseparateCPUs.
MSR_WRITEsofTCP_RR10VMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU111HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERVMEXITsofTCP_RRVMEXITSAPICTimerRegisterAPICInterruptCommandRegister(IPI)HLTclientclientidleserveridleidle1.
Send1bytetoserver.
Waitforresponse.
2.
Receive1bytefromclient.
Send1byteback.
3.
Receive1bytefromserver.
VCPU0VCPU112HLTHLTHLTIPIIPIAPICTIMERAPICTIMERAPICTIMERAPICTIMERCriticalPath8pertransaction4onthecriticalpathNOHZ(ticklessguestkernel)"Disable"scheduler-tickuponenteringidle.
"Enable"scheduler-tickuponleavingidle.
scheduler-tick==APICTimer(couldalsobeTSCDeadlineTimer)Why2writespertransitioninto/outofidlehrtimer_cancelhrtimer_startAdds3-5ustoround-triplatency.
APICTimer"InitialCount"Register13HLT:x86Instruction.
CPUstopsexecutinginstructionsuntilaninterruptarrives.
ThispartofHLTisnotonthecriticalpath!
HowitworksinKVMPlaceVCPUthreadonawaitqueue.
YieldtheCPUtoanotherthread.
HLTkvm_vcpu_block->schedule()VMEXITHLTcontextswitchtoanotherusertask,kernelthread,oridleVCPU(guest)PCPU(KVM)14kvm_sched_outSendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
IPI+HLTWRMSR:APICInterruptCommandRegisterkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)guesthostkvm_sched_in*VMEXITandVMRESUMEimplementedinHardware.
time15SendinganIPItowakeupaHLT-edCPU.
Onthecriticalpath!
Sameoperationonbaremetalisentirelyimplementedinhardware.
HowmuchoverheadfromvirtualizationUnlikeAPIC_TMICT,can'tjusttimeVMEXITs.
Wecancomparewiththesameoperationonphysicalhardware.
IPI+HLT16KVMversusHardwareRing0Microbenchmark(kvm-unit-tests)1.
VCPU0:HLT.
2.
~100usdelay3.
VCPU1:A=RDTSC4.
VCPU1:SendIPIto[V]CPU0.
5.
VCPU0:B=RDTSC(firstinstructionofIPIISR).
6.
Latency=B-A7.
Repeat.
RuninKVMguestandonbare-metal.
Compare!
17VMRESUMEWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVCPU1VCPU0(HLT-ed)guesthostkvm_sched_intimeKVMversusHardwareA=RDTSCB=RDTSC18Median:KVMis12xslowerPathologicalcase(witnessed):KVMis400xslowerBestcase(witnessed):KVMis11xslowerKVM:5.
7us;Hardware:0.
5usKVMversusHardwareCyclesKVMHardwareMin137001200Average15800120050%ile14900120090%ile16000130099%ile249001300Max5210001400Host:SandyBridge@2.
6GHz3.
11KernelKVMperformanceissimilaronIvyBridge(5.
6us)andHaswell(4.
9us).
19Notesaboutthisbenchmark:NoguestFPUtosave/restore.
Hostotherwiseidle(VCPUcontextswitchestoidleonHLT).
Hostpowermanagementnottheculprit.
KVMversusHardware20KVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRvmx_vcpu_runIPIISRVMEXITVMRESUMEVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()kvm_vcpu_kick21VMRESUMEvmx_vcpu_runkvm_vcpu_kickKVMHLTInternalsSoKVMisslowatdeliveringIPIsand/orcomingoutofHLT.
ButwhyPossibleculprits:WRMSRIPIISRVMEXITVCPU1VCPU0(HLT-ed)kvm_sched_intimereturnfromschedule()inkvm_vcpu_block()22RDTSCRDTSCRDTSCRDTSCRDTSCKVMHLTInternalsWRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEMin(cycles):400600730032001300VCPU1VCPU0guesthostVT-xKVMSchedulerkvm_sched_in:4924001200850034001400Median(cycles):23Unsurprisingly,theschedulertakessometimetoruntheVCPUSlowevenintheuncontended,cache-hot,case.
ImagineiftheVCPUiscontendingforCPUtimewithotherthreads.
Experiment:Don'tscheduleonHLT.
JustpollfortheIPIinkvm_vcpu_block.
KVMHLTInternals24Whathappenswhenyoudon'tscheduleonHLTKVM(Alwaysschedule)5.
7usKVM(Neverschedule)1.
7usHardware(SandyBridge)0.
5usNeverschedule!
CyclesKVM(Alwaysschedule)KVM(Neverschedule)HardwareMin1380040001200Average158004400120050%ile149004300120090%ile160004500130099%ile2490069001300Max52100050000140025SimilarimprovementsonIvyBridge(5.
6us->1.
6us)Haswell(4.
9us->1.
5us).
Neverschedule!
WRMSRkvm_vcpu_kickreturnfromschedule()inkvm_vcpu_block()vmx_vcpu_runIPIISRVMEXITVMRESUMEAlwaysschedule:4001200850034001400VCPU1VCPU0guesthostVT-xKVMSchedulerNeverschedule:300130011004001200(mediancycles)26Neverschedule!
WeeliminatealmostallofthelatencyoverheadbynotschedulingonHLT.
Schedulingisoftentherightthingtodo.
LetotherthreadsrunorsavehostCPUpower.
Mostofthetimeimprovesguestperformance(lettheIOthreadsrun!
).
Canhurtperformance.
Seemicrobenchmark.
SeeTCP_RR.
27Halt-PollingStep1:PollForuptoXnanoseconds:IfataskiswaitingtorunonourCPU,gotoStep2.
Checkifaguestinterruptarrived.
Ifso,wearedone.
Repeat.
Step2:schedule()Scheduleoutuntilit'stimetocomeoutofHLT.
Pros:WorksonshortHLTs(
28Halt-PollingMemcache:1.
5xlatencyimprovementWindowsEventObjects:2xlatencyimprovementReducemessagepassinglatencyby10-15us(includingnetworklatency).
29Halt-PollingMergedintothe4.
0kernel[PATCH]kvm:addhalt_poll_nsmoduleparameterThankstoPaoloBonziniUsetheKVMmoduleparameterhalt_poll_nstocontrolhowlongtopolloneachHLT.
Futureimprovements:Automaticpolltoggling(removeidleCPUoverheadbyturningpollingoff).
Automatichalt_poll_nsKVMwillset(andvary)halt_poll_nsdynamically.
Howtodothisisanopenquestion.
.
.
ideasLazyContextSwitchingEquivalentfeature,butavailableforanykernelcomponenttouse.
30ConclusionMessagePassingEvenloopbackmessagepassingrequiresvirtualization.
Beingidle(asaLinuxguest)requiresvirtualization.
Cross-CPUcommunicationrequiresvirtualization.
Halt-Pollingsaves10-15usonmessagepassinground-triplatency.
Remaininground-triplatency:4MSRwritestotheAPICtimer(3-5us)IPIsend(~2us)HLTwakeup(evenwithhalt-polling,stilladds~3us!
)31
- EOI_INDUCEDivybridge相关文档
- slightivybridge
- Revivybridge
- 1.0ivybridge
- 采购ivybridge
- Extreme3ivybridge
- subgroupivybridge
[6.18]DogYun：充100送10元,态云7折,经典云8折,独立服务器月省100元,幸运大转盘最高5折
DogYun是一家2019年成立的国人主机商，提供VPS和独立服务器租用等，数据中心包括中国香港、美国洛杉矶、日本、韩国、德国、荷兰等，其中VPS包括常规VPS(经典云)和按小时计费VPS(动态云)，使用自行开发的面板和管理系统，支持自定义配置，动态云各个硬件独立按小时计费，带宽按照用户使用量计费(不使用不计费)或者购买流量包，线路也可以自行切换。目前商家发布了6.18促销方案，新购动态云7折，经...
Boomer.Host(年付3.5美)休斯敦便宜VPS
Boomer.Host是一家比较新的国外主机商，虽然LEB自述 we’re now more than 2 year old，商家提供虚拟主机和VPS，其中VPS主机基于OpenVZ架构，数据中心为美国得克萨斯州休斯敦。目前，商家在LET发了两款特别促销套餐，年付最低3.5美元起，特别提醒：低价低配，且必须年付，请务必自行斟酌确定需求再入手。下面列出几款促销套餐的配置信息。CPU：1core内存：...
Hostiger发布哥伦布日提供VPS主机首月七折优惠 月费2.79美元
Hostiger商家我们可能以前也是有见过的，以前他们的域名是Hostigger，后来进行微调后包装成现在的。而且推出Columbus Day哥伦布日优惠活动，提供全场的VPS主机首月7折月付2.79美元起的优惠。这里我们普及一下基础知识，Columbus Day ，即为每年10月12日，是一些美洲国家的节日，纪念克里斯托弗·哥伦布在北美登陆，为美国的联邦假日。Hostiger 商家是一个成立于2...
ivybridge为你推荐
-
firetrap我发现好多外贸店都卖其乐的原单，有怎么多原单吗嘉兴商标注册我在濮院想注册一个羊毛衫商标？该怎么做？百度关键词价格查询百度关键字如何设定竟价价格?原代码什么叫源代码，源代码有什么作用陈嘉垣电视剧《反黑》里面，雷太太女儿扮演者是谁？杰景新特萨克斯吉普特500是台湾原产的吗冯媛甑夏如芝是康熙来了的第几期?罗伦佐娜维洛娜毛周角化修复液治疗毛周角化有用吗？谁用过？能告诉我吗？同一ip网站如何用不同的IP同时登陆一个网站avtt4.comCOM1/COM3/COM4是什么意思？？/