Memory迈拓kvm

DevelopmentsinKVMonPowerPaulMackerras,IBMLTCOzLabspaulus@au1.
ibm.
com221October20132013IBMOutlineIntroductionLittle-endiansupportOpenStackNestedvirtualizationGuesthotplugHardwareerrordetectionandrecovery321October20132013IBMIntroductionWewillbereleasingPOWERmachineswithKVM–AnnouncementbyArvindKrishna,IBMexecutivePOWER8processordisclosedatHotChipsconference–12coresperchip,8threadspercore–96kBL1cache,512kBL2cache,8MBL3cachepercoreonchipqemuKVMHostLinuxKernelSAPPHIREPOWERhardwareGuestOSGuestOSFSPqemuhostOSprocesshostOSprocesshostOSprocess421October20132013IBMIntroduction"Sapphire"firmwarebeingdevelopedforthesemachines–TeamledbyBenHerrenschmidt–SuccessortoOPALProvidesinitializationandbootservicesforhostOS–Loadfirst-stageLinuxkernelfromflash–Probethemachineandsetupdevicetree–Petitbootbootloadertoloadandrunthehostkernel(viakexec)Provideslow-levelrun-timeservicestohostkernel–Communicationwiththeserviceprocessor(FSP)ConsolePowerandrebootcontrolNon-volatilememoryTimeofdayclockErrorloggingfacilities–Somelow-levelerrordetectionandrecoveryservices521October20132013IBMLittle-endianSupportModernPOWERCPUshavealittle-endianmode–Instructionsandmulti-bytedataoperandsinterpretedinlittle-endianbyteorderLowest-numberedbyteisleastsignificant,ratherthanmostsignificant–"True"littleendian,notaddressswizzlingasonold32-bitPowerPCprocessorsEnabledbyanMSR(machinestateregister)bit–HypervisorregistercontrolsMSR[LE]settingoninterruptdeliveryLittle-endianmodehaslittleornoperformanceimpact–Somemisalignedloads/storestraponolderprocessors(POWER6,POWER7)GrowinginterestinrunningentireOSinlittle-endianmode–Easeportingofprogramsfromotherarchitectures–EaseportingofprogramswhichaccessfilescontainingLEbinarydata–EasecommunicationwithGPUsNewOpenPowerConsortium–IBM,Google,Tyan,Nvidia,MellanoxWanttobeabletorunlittle-endianOSasKVMguest–Host-sidechangessurprisinglyminor–Hostalwaysbig-endianfornow621October20132013IBMLittle-endianSupport"Bi-endian"support–KVMguestscanswitchendiannessatwill–Currentexecutionmodeunderdirectguestcontrol–InterruptdeliverymodecontrolledvianewH_SET_MODEhypercallPAPRparavirtualizationinterfaceisexplicitlybig-endian–MemoryoperandsforPAPRhypercallsarebig-endian,thereforeneedtobebyte-swappedbyLEguestkernels–Valuesinregistersdon'tneedbyteswapping:registersdon'thaveendianness–Memoryareassharedbetweenhostandguest(VirtualProcessorAreas)remainBEInstructionemulationrequiresbyte-swappingbyKVM–OnlyoccursforMMIOemulation–Byte-swapinstructionsafterreadingthemfromtheguest–Byte-swapmulti-bytedatavaluesfornormalload/stores,notforbyte-reversingloads/storesVirtiodatastructuresareinguestendianorder–Newvirtiospecificationwillspecifylittle-endian–Forcurrentguests,QEMUandKVMhavetobyte-swapforlittle-endianguests–Guestendianmodesampledatvirtiodeviceresettime721October20132013IBMLittle-endianSupportGuestsstartoutinbig-endianmode–Reverttobig-endianonrebootSLOF(guestbootfirmware)runsinbig-endianmode–WillbemodifiedtobeabletoloadbothBEandLEimagesLEkernelscheckcurrentmode,switchtoLEifnecessary–Usesinstructionthatisno-opinLEmode,branchinBEmode4800000cb.
+120c000048twi0,r0,72(trapnever)–SetMSR[LE]anddoH_SET_MODEifnecessaryNodifferencebetweenhowLEguestsandBEguestsarestartedChoiceofLEvs.
BEisaquestionofwhatimagegetsdeployedintheguest–Cataloguingproblematthesamelevelaschoiceofdistro–Allthesamearchitectureasfaraslibvirtandmanagementtoolsareconcerned.
POWER8addssplitlittle-endianmode–Allowsinstructionanddataendiannesstobedifferent821October20132013IBMOpenStackOpenStackisimportantasmanagementstackforKVMonPowermachinesUpstreamunmodifiedOpenStackcannowmanagePowercomputenodeswithKVM–Necessaryfixesareupstreamlibvirt:somex86-centricassumptionslibguestfs:buginpartitiontableparsing–MayneedextensionstoincludeLE/BEindicationinimagecatalogsRequirementfornestedvirtualization–NeededtoparticipateinOpenStack'scontinuousintegrationprocessRequirementforguestPCIhotplug–Virtualdiskandnetworkadapters921October20132013IBMNestedVirtualizationOpenStackCItestsproposedpatchesinvirtualcluster–Computenodesofvirtualclusterneedtobeabletorunguests–NodesareKVMguests,thereforedon'thaveaccesstohypervisormode–Twooptions:fullemulation,or"PR"styleKVMPRKVM,developedbyAlexGraf,runstheguestentirelyinusermode("PR"oblemstate)andemulatesallprivilegedinstructionsandtheMMUFullemulationhasproblems–Veryslow–QEMUdoesnotimplementalltheinstructionsinPOWER6/7/8–SomeLinuxdistributionsprovidepackagesoptimizedforPOWER7Fedora.
ppc64p7.
rpmpackagessinceFedora18PRKVMisourproposedsolutionfornestedvirtualization–Notasfastas"HV"styleKVM,butalotfasterthanfullemulation–Doesn'tcurrentlysupportallthefeaturesofPowerprocessorsDatabreakpoint(watchpoint)supportPerformancemonitorunitNewPOWER8featuressuchastransactionalmemorySupportingthesefeaturesisamatterofcoding–NotcurrentlypossibletocompilebothPRandHVKVMinonekernel1021October20132013IBMNestedVirtualizationWanttomakePRandHVKVMbothavailableinonekernel–Distroswon'tmaketwokernelbuildsavailable,sowillpickoneortheotherNeitherisasupersetoftheother–HVisfasterthanPR,assumingnecessaryhardwaresupportisavailable–HVKVMrequiresaparavirtualizedguestkernelHardwarenotdesignedtosupportfullvirtualization;guestaccesstohypervisorfacilitiestrapstotheguest,notthehost–HVKVMdoesn'tsupportemulationofancient,embeddedor32-bitprocessorsHardwarecompatibilitymodeforemulationofPOWER6andPOWER7MyproposalfromearlyAugust:–ModifybothPRandHVsothatbothcanbecompiledintoonekernel–EachVMhasanassociatedtype:PR,HVorunknown–ChangetypetoHVwhenPAPRcapabilityenabled(ifhardwareiscapable)–ChangetypetoPRwhenfirstvcpuisrunotherwise–Someproblems/objectionsUsersmightunexpectedlygetlower-performanceoptionthantheyexpectedAneeshKumar'spatches(earlyOctober)–Splitmoduleintothree:HV,PRandcore–UserspacechoosestypeatVMcreationtime1121October20132013IBMGuestPCIHotplugPrimarilyforvirtiodevicesratherthanrealPCIadapters–VirtiodevicesappearasemulatedPCIadapters–OpenStacktypicallybootsguestswithminimalconfigurationandaddsdisksandnetworkadapterswithhotplugPAPRincludesarchitectureforhotplug–Allsortsofresources:CPUs,memory,PCIdevices,PCIhostbridges–ReferredtoasDynamicLogicalPartitioning(DLPAR)–DesignedforPowerVMenvironmentOperationinitiatedfrommanagementconsole,nottheguestProprietaryclosed-sourcedaemonintheguest,talkingviasockettomanagementconsoleusingproprietaryprotocolDaemonperformsnecessaryfirmwareandsystemcallsExistingguestOSesdon'tautomaticallyhavesupportforhotplug–Eveniftheydoincludetheproprietarydaemon,wecan'tanddon'twanttouseitAlternativeapproachbeingdeveloped–Extendexistingopen-sourceeventloggingdaemon(rtas_errd)–Defineneweventsindicatingaddition/removalofPCIadapters–ModifyQEMUtogeneratetheseeventsandhandleresultingRTASfirmwarecalls(patchesbeingdevelopedbyMikeRoth,MikeDayandNathanFontenot)1221October20132013IBMHardwareErrorDetectionandRecoveryExploitReliability,AvailabilityandServiceability(RAS)featuresofthehardware–Hardwarehasalotoferrorcheckingandrecoveryfacilities–ParityorECConalmosteverything–Micro-checkpointingofthecore,rollbackontransienterrors–Don'thavePowerVMtoprovidesoftwaresupportErrordetection–CPU-generatedMachineCheckinterruptUseofdatawithuncorrectableerrorsAccesstonon-responsivephysicaladdressParityerrorsinSLBorTLBDuplicateSLBentries(canbegeneratedbyguest)–CPU-generatedHypervisorMaintenanceinterrupt–FSPscansforothertransient,correctederrorsandgenerateseventlogs–EnhancedErrorHandling(EEH)inPCIhostbridgesIsolatesPCIadapterswhenerrordetectedtopreventpropagationofbaddataErrorsincludeattemptstoaccessoutsideofpermittedbusaddressrangeaswellasparityerrorsandtimeouts1321October20132013IBMHardwareErrorDetectionandRecoveryHostmachinecheckhandler–PatchespostedbyMaheshSalgaonkar–AttempttocorrectMMU-relatederrorsinrealmodePotentiallystillinguestMMUcontextatthispoint–ThentransferstoguestexitcodeifthemachinecheckoccurredwhileinaKVMguestKVMhastodeliveramachinechecktotheguestinthiscasesinceSRR0/1registersmayhavebeenlive–Foruseofdatawithuncorrecteddata,exploithwpoisoninfrastructureEEHsupportforPCIpass-throughtoguests–EEHisolationeventscanbecausedbyguestmis-programmingofadapter,oradapterfailure–NeedtonotifyguestofeventviaRTASevent-loginfrastructureasspecifiedinPAPR–NeedtoimplementRTASfirmwarecallstoresetandde-isolateadapterOtherhost-sideRASfeaturesdon'timpactKVM–Daemon/databaseforloggingandretrievingerrorsandotherevents–Hostplatformdumps–Systemcatalog/VPDtools–Firmwareupdatetools–system,FSP,I/Oadapters1421October20132013IBMLegalStatementThisworkrepresentstheviewoftheauthoranddoesnotnecessarilyrepresenttheviewofIBM.
IBM,IBM(logo),AIX,POWER,POWER6,POWER7,POWER8andPowerVMaretrademarksorregisteredtrademarksofInternationalBusinessMachinesCorporationintheUnitedStatesand/orothercountries.
LinuxisaregisteredtrademarkofLinusTorvalds.
Othercompany,productandservicenamesmaybetrademarksorservicemarksofothers.