requestfedora16

XAPP1177(v1.
0)November15,2013www.
xilinx.
com1Copyright2013Xilinx,Inc.
Xilinx,theXilinxlogo,Artix,ISE,Kintex,Spartan,Virtex,Zynq,andotherdesignatedbrandsincludedhereinaretrademarksofXilinxintheUnitedStatesandothercountries.
PCI,PCIExpress,PCIe,andPCI-XaretrademarksofPCI-SIG.
Allothertrademarksarethepropertyoftheirrespectiveowners.
SummaryThisapplicationnotedemonstratestheSingleRootI/OVirtualization(SR-IOV)capabilityoftheXilinxVirtex-7FPGAPCIExpressGen3IntegratedBlock.
ItexplainsthekeyconceptsofSR-IOVandprovidesdetailsaboutconfiguringtheSR-IOVcapability.
ThisdocumentillustrateshowtocreateaPCIExpressx8Gen3EndpointdesignconfiguredfortwoPhysicalFunctions(PF)andsixVirtualFunctions(VF)asshowninFigure1.
ThereferencedesignistargetedataXilinxVirtex-7FPGAVC709ConnectivityKitandhasbeenhardware-validatedonasystemwithSR-IOVcapabilitybyperformingProgrammedI/Oreadsandwritestoallfunctionsusingthesupplieddrivers.
TheSR-IOVreferencedesignandtheLinuxkernel-modedriversourcecodeareincludedwiththisapplicationnote.
ApplicationNote:Virtex-7FamilyXAPP1177(v1.
0)November15,2013DesigningwithSR-IOVCapabilityofXilinxVirtex-7PCIExpressGen3IntegratedBlockAuthor:VivekSurabhiX-RefTarget-Figure1Figure1:SR-IOVSystemOverviewW/>>/EhyK^sDsDsDsDsDsDW&Z/sZy/>/Eys/Zdy&W''E/Ed'Zd>Ksd/KDDhZKySR-IOVSupportXAPP1177(v1.
0)November15,2013www.
xilinx.
com4TheregisterfieldsTotalVFs(0Eh)andInitialVFs(0Ch)areread-onlyfields(setbycoreattributes)andindicatethemaximumnumberofVFsassociatedwiththePF.
EachPFcanhavezeroormoreVFsassociatedwiththem.
BecausetheVirtex-7FPGAPCIExpressGen3IntegratedBlocksupportsamaximumofsixVFsbetweentwoPFs,thelargestvalueintheTotalVFsfield(0Eh)canbe6.
Basedonthevaluesinthesefieldsalongwithknowledgeofthesystemcapabilities,thesystemsoftwarewritestheappropriatevalueintotheNumVFsfield(10h)whichdeterminesthenumberofVFsvisibleinthePCIelogicaftertheVFsareenabled.
TheregisterfieldsVFStride(16h)andVFOffset(14h)arefixedfortheVirtex-7FPGAPCIExpressGen3IntegratedBlock.
VFOffsetisfixedto64(decimal)andVFStrideisfixedto1,whichmeansthatallVirtualFunctionsareinthefunctionnumberrangeof64to69.
Forexample,IfPF0hasfourVFsattachedandPF1hastwoVFsattached,theassociatedVFfunctionnumbersareasshowninTable1.
IfPF0hastwoVFsattachedandPF1hastwoVFsattached,theassociatedVFfunctionnumbersareasshowninTable2.
AllVirtualFunctionsassociatedwithaPhysicalFunctionsupportacommonDeviceIDandDeviceType.
TheDeviceTypeoftheVFisdependentontheassociatedPFwhereastheDeviceIDfield(1Ah)isconfigurable(setbycoreattributes).
TheSupportedPageSizesfield(1Ch)indicatesallthepagesizessupportedbythePFand,asrequiredbytheSR-IOVspecification,theVirtex-7FPGAPCIExpressGen3IntegratedBlocksupports4KB,8KB,64KB,256KB,1MBand4MBpagesizes.
BasedontheSupportedPageSizefield,thesystemsoftwaresetstheSystemPageSizefield(20h)whichisusedtomaptheVFBARmemoryaddresses.
EachVFBARaddressisalignedtothesystempageboundary.
Forexample,iftheVFBAR0sizeissetto4KBandsystempagesizeissetbythesystemsoftwareto8KBinaconfigurationthatsupportsfourVFs,theaddressmappingisasshowninFigure6.
Eventhougheachfunctionneedsonly4KBaddressspace,thestartaddressesarealignedto8KBboundaries.
Table1:VFFunctionNumberAssignmentPhysicalFunctionVirtualFunctionFunctionNumberPF0VF164PF0VF265PF0VF366PF0VF467PF1VF5(VF1toPF1)68PF1VF6(VF2forPF1)69Table2:VFFunctionNumberAssignmentPhysicalFunctionVirtualFunctionFunctionNumberPF0VF164PF0VF265PF1VF3(VF1toPF1)66PF1VF4(VF2forPF1)67SupportedVFCapabilitiesXAPP1177(v1.
0)November15,2013www.
xilinx.
com5EachVirtualFunctionsupportsuptosix32-bitBARsorthree64-bitBARs.
VirtualFunctionBARscanbeconfiguredwithoutanydependencyonthesettingsoftheassociatedPhysicalFunctionsBARs,butallvirtualfunctionsassociatedwiththephysicalfunctionsharethesamesettings.
VirtualFunctionBARsdonotsupportI/Ospaceandhencehavetobeconfiguredtomaptomemoryspace.
VFBARscanbeeither64-bitor32-bitandcanbeprefetchable:WhenconfiguringthecoreasanEndpointforPCIe,64-bitaddressingissupportedforallSR-IOVBARs(exceptBAR5)thathavetheprefetchablebitset.
32-bitaddressingispermittedforallSR-IOVBARsthatdonothavetheprefetchablebitset.
WhenaBARissetas64bits,itusesthenextBARfortheextendedaddressspaceandmakesthenextBARinaccessible.
TheFunctionDependencyLinkdescribesthedependenciesbetweenPFs.
BecausethefunctionssupportedbytheVirtex-7FPGAPCIExpressGen3IntegratedBlockareindependentofeachother,thisfieldshouldbesetbythecoreattributestocontainitsownfunctionnumber.
TheVirtex-7FPGAPCIExpressGen3IntegratedBlockdoesnotsupportVFMigrationforSR-IOV.
Hence,allfieldsintheSR-IOVcapabilitystructureassociatedwithVFMigrationareread-onlyfieldswithvalueofzero.
SupportedVFCapabilitiesAllvirtualfunctionshavetheirownconfigurationspaceincludingthePCIeCapabilityStructure.
XilinxVFssupportMSIandMSI-Xinterrupts.
ThevirtualfunctionsinterruptsupportcanbesetindependentlyoftheassociatedPF.
EachVFsupportsMSI,MSI-Xorbothbasedontheirconfiguration.
Virtualfunctionssupport:PendingBitArraystructureforMSI-X.
ForMSI-X,thetableoffsetandPBAoffsetisrelativetotheVFmemoryaddressspace.
However,VFsdonotsupporttheper-vectormaskingcapabilitystructureMSI.
PowerManagementcapabilitywithD1andD3hotsupport.
IfaVFdoesnotimplementthePowerManagementCapability,thentheVFbehavesasifithadbeenprogrammedintotheequivalentpowerstateofitsassociatedPF.
IfaVFimplementsthePowerManagementCapability,thefunctionalityisdefinedinthePCIExpressBaseSpecificationRev3.
0[Ref3].
Baselineerrorreportingcapability.
Advancederrorreporting(AER)issupportedonlyiftheassociatedPFsupportsAER.
Also,theVFsusetheVirtualchannelsoftheassociatedPFs.
Assuch,VFsdonothaveanyvirtualchannelcapabilities.
X-RefTarget-Figure6Figure6:BARAddressMappingWithRespecttoSystemPageSizes&ZD^s&ZD^s&ZD^s&ZD^yReferenceDesignOverviewXAPP1177(v1.
0)November15,2013www.
xilinx.
com10WhentheEndpointforPCIereceivesaMemoryWriteTLP,theTLPdestinationaddressandtransactiontypearecomparedwiththevaluesinthecoreBARs.
IftheTLPpassesthiscomparisoncheck,thecorepassestheTLPovertheCompleterRequesterAXI4-StreaminterfaceofthePIOdesign.
TheCompleterRequestDescriptorisshowninFigure8andhastheappropriateBARID(bits[114:112])andTargetFunction(bits[111:104])toindicatethespecificfunction(PF/VF)anditsBARthatmatchedtheincomingTLP.
Onreception,thereferencedesignRX_ENGINEprocessestheincomingWriteTLPandextractsthedataandrelevantaddressfieldssothatitcanpassthisalongtotheMEM_ACCESSblocktobestoredintheinternalBlockRAM.
InthecaseofaMemoryReadTLP,theRX_ENGINEprocessestheincomingReadTLPandextractsthedataandrelevantaddressfieldssothatitcanpassthisalongtotheTX_ENGINEblock.
TheTX_ENGINEblockforwardsthereadrequesttotheMEM_ACCESSblock.
OnreceivingthereaddatafromMEM_ACCESS,theTX_ENGINEblocksendstheCompletionwiththeDataovertheCompleterCompletionAXI4-Streaminterface.
TheCompleterCompletionDescriptorisshowninFigure9andhastheDevice/Functionfieldtoindicatethespecificfunction(PF/VF)generatingtheCompletion.
Note:TheCompleterIDEnablebitmustalwaysbesettozero.
X-RefTarget-Figure8Figure8:CompleterRequestDescriptorннннtнddннннtнd&ZdZ/ZZZ&Z/ннннtнddннннtнyX-RefTarget-Figure9Figure9:CompleterCompletionDescriptorFormatZннннtнннннtнddннннtн&W^//&ZZ&Z/>ZZZdyReferenceDesignDriversandApplicationXAPP1177(v1.
0)November15,2013www.
xilinx.
com11ThelocationintheBlockRAMisindicatedusingtherd_addr/wr_addrinputstotheMEM_ACCESScontroller.
CompleterRequestDescriptorbits[10:2]definetheDwordalignedaddresswithinthe2KBspace.
ThetargetfunctionisdefinedbytheCompleterRequestDescriptorbits[111:104].
Thetargetfunctionsettingsare:PF0:0hPF1:1hVFs:from40hto45h.
Forthisreason,{m_axis_cq_tdata[110],m_axis_cq_tdata[106:104]}areusedastheupper4bitsoftheread/writeaddress.
TheMEM_ACCESScontrollerusestheseupper4bitstodecodethereads/writestargetedatdifferentPhysicalandVirtualFunctions.
Forexample,ifaMem32WriteRequestisreceivedbythecoretargetingaddress0x04ofBAR0ofVF1.
ThewriteisreceivedontheCompleterRequesterinterfacewithBARIDsetto3'b0andTargetFunctionto40h{6'b010000}.
TheRX_ENGINEextractstheloweraddressbitsandthedatafieldfromtheTLPandinstructstheMEM_ACCESScontrollertoperformawriteattheaddress{1'b1,3'b000,8'b00000001}.
Thelower8bitsindicatetheaddressis0x1,Dwordaligned.
Theupper4bits{1'b1,3'b000}indicatethetransactionistargetedtowardsVF1.
Whilethewrite/readisbeingperformed,theRX_ENGINEstatemachinedeassertsm_axis_cq_treadycausingtheCompleterRequesterAXI4-StreaminterfacetodelayreceivinganyfurtherTLPs.
De-assertingm_axis_cq_treadyinthiswayisnotrequiredforalldesignsusingthecore.
TheSR-IOVreferencedesignusesthismethodtosimplifythecontrollogicoftheRX_ENGINE.
ReferenceDesignDriversandApplicationTheSR-IOVdriversaredeliveredasapartofthexapp1177.
zipfileandconsistof:Linux_Driver_PF0Linux_Driver_PF1Linux_Driver_VFThePF0andPF1driverswerebuiltandtestedwiththeRHELversion6.
2,andtheVFdriverwastestedwithFedora16.
TocompileandinstallthedriversmakesurethedeviceandvendorIDinthedriversmatchthedeviceundertest.
Ifneeded,editthexpcie.
cfileandchangethesetwolinestomatchthedeviceandvendorIDinuse:#definePCI_VENDOR_ID_XILINX0x10ee#definePCI_DEVICE_ID_XILINX_PCIE0x7038BydefaultthedriverDeviceIDsaresetas:PF0:0x7038PF1:0x7138VF:0x7238Also,thePF0andPF1drivershavethefieldtoenabletheVFsandthenumberofVFstobeenabledcanbemodifiedbychangingthisfieldinthexpcie.
cfile:#defineNUMBER_OF_VFS0x4BydefaultthePF0hasthisfieldsetto4toenableallfourassociatedVFs.
PF1hasthisfieldsetto2.
GeneratingandCustomizingtheCoreXAPP1177(v1.
0)November15,2013www.
xilinx.
com12Installationofthedriverrequiresrootprivileges.
Toinstallthedriverandrunthetest,gotothedirectorycontainingthedriverfiles,andtypethesecommands:1.
makeTheexpectedoutputisasfollows:[root@localhostLinux_Driver_PF0]#makemake-C/lib/modules/2.
6.
32-220.
el6.
x86_64/buildM=/root/Linux_Driver_PF0modulesmake[1]:Enteringdirectory'/usr/src/kernels/2.
6.
32-220.
el6.
x86_64'Buildingmodules,stage2.
MODPOST1modulesmake[1]:Leavingdirectory'/usr/src/kernels/2.
6.
32-220.
el6.
x86_64'g++met.
cpp-omet2.
.
/make_deviceTheexpectedoutputisasfollows:[root@localhostLinux_Driver_PF0]#.
/make_devicecrw-r--r--.
1rootroot240,1Sep2518:16/dev/xpcie3.
insmodxpcie.
koThereisnonoticeableoutputafterrunninginsmod.
Note:Withoutrootprivileges,insmodfailstoloadthedriver.
4.
.
/metThisrunstheapplication.
Theexpectedoutputisasfollows:Pass#[0]Pass#[1000]Pass#[2000]Pass#[3000]Pass#[4000]Pass#[5000]Pass#[6000]Pass#[7000]Pass#[8000]Pass#[9000]Pass#[10000]TheREADMEfileprovidedwiththedriverscontainsdescriptionsfortheprecedingcommands.
GeneratingandCustomizingtheCoreThissectionillustrateshowtogeneratetheVirtex-7FPGAPCIExpressGen3IntegratedBlockusingSR-IOVcapability.
TogeneratethecoreinVivadoDesignSuite2013.
3:1.
LaunchtheVivadoInteractiveDesignEnvironment1.
CreateanewRTLProjectthattargetsthexc7vx690tffg1761-2part.
2.
OpentheVivadoIPcatalog.
3.
IntheIPcatalog,expandtheStandardBusInterfaces>PCIExpresshierarchy.
4.
Right-clickVirtex-7FPGAPCIExpressGen3IntegratedBlockandselectCustomizeIP(seeFigure10).
GeneratingandCustomizingtheCoreXAPP1177(v1.
0)November15,2013www.
xilinx.
com13IntheCustomizeIPdialogbox,followthesesteps:5.
IntheBasictab(Figure11):a.
SelecttheAdvancedsettingfromtheModelistbox.
b.
SetLaneWidthtoX8,MaximumLinkSpeedto8.
0GT/s,andXilinxDevelopmentBoardtoVC709.
Note:AnAXI-STinterfacewidthof256bitsisrequiredbecausetheexamplereferencedesignforSR-IOVonlysupportsa256-bitinterface.
Theonlysupportedconfigurationsarex8at8.
0GT/sor5.
0GT/s.
6.
IntheCapabilitiestab,clickEnablePhysicalFunction1andSR-IOVCapabilitytoenablethesefeatures.
X-RefTarget-Figure10Figure10:CustomizingtheIPX-RefTarget-Figure11Figure11:BasicTabGeneratingandCustomizingtheCoreXAPP1177(v1.
0)November15,2013www.
xilinx.
com147.
InthePF0IDstab,usethedefaultvalues.
8.
InthePF1IDstab,enteraDeviceIDof7138,andleavetheotherfieldsasdefaults.
9.
InthePF0BARtab,selectBar0,theBartypeMemory,andsize4Kilobytes.
10.
InthePF1BARtab,selectBar0,theBartypeMemory,andsize4Kilobytes.
11.
IntheSR-IOVConfigtab,Fig.
,entervaluesasspecifiedinTable6.
Note:Grayed-outfieldsareinaccessibleandcannotbemodified.
12.
InthePF0SR-IOVBARtab,selectBar0,theBartypeMemory,andsize4Kilobytes.
13.
InthePF1SR-IOVBARtab,selectBar0,theBartypeMemory,andsize4Kilobytes.
Note:SelectaminimumBARsizeofatleast4KBwhichcorrespondstotheminimumsupportedpagesize.
Valuesinthesetwotabsareusedtoconfigurethesix32-bitBARsorthree64-bitBARssupportedbyVFsandcorrespondstosections24h-38hoftheSR-IOVcapabilitystructure.
14.
IntheLegacy/MSICaptab(Figure13):a.
UnderLegacyInterruptSettings,selectNONEforPF0InterruptPin,andNONEforPF1InterruptPin.
b.
CheckPF0EnableMSICapabilityStructureandPF1EnableMSICapabilityStructureandsetvaluesasshowninFigure13.
X-RefTarget-Figure12Figure12:SR-IOVConfigTabTable6:SR-IOVConfigSettingsFieldPF0SRIOVConfigPF1SRIOVConfigDescriptionCapVersion11CorrespondstotheCapabilityVersionofSR-IOVExtendedCapabilityHeader.
NumberofPF0VFs42CorrespondstotheInitialVFs(0Ch)andtheTotalVFsfield(OEh)oftheSR-IOVcapabilitystructure.
ThisnumbermustbesettoassignVFstothePF.
PFDependencyLink01CorrespondstotheFunctionDependencyLink(12h)oftheSR-IOVcapabilitystructure.
ThereisnodependencybetweenPF0andPF1.
FirstVFOffset6468CorrespondstoFirstVFOffsetfield(14h).
Thefieldisfixedandcannotbemodified.
TheFirstVFoffsetforPF1canbecalculatedusingtheexamplesfromTable1andTable2,page4VFDeviceID72387338CorrespondstotheVFDeviceIDfield(1Ah)oftheSR-IOVcapabilitystructure.
SupportedPageSize553553CorrespondstoSupportedPageSizefield(1Ch).
Thevaluesetcannotbemodifiedanditindicatesthat4-KB,8-KB,64-KB,256-KB,1-MBand4-MBpagesizesaresupportedasrequiredbytheSR-IOVspecification.
IntegratingtheReferenceDesignwiththeExampleDesignProjectXAPP1177(v1.
0)November15,2013www.
xilinx.
com1515.
IntheMSIxCaptab,noMSIxcapabilitiesareselectedforPF0orPF1forthisexampledesign.
16.
InthePowerManagementtab,usethedefaultssettings.
17.
IntheExtendedCapabilities1tab,checkEnableARICapability(PF0)andEnableARICapability(PF1).
18.
IntheExtendedCapabilities2tab,checkSR-IOVCapEnable.
19.
ClickOKtofinishthecoreconfiguration.
20.
Afterfinishingtheconfiguration,clickGeneratetogeneratetheoutputproducts.
21.
Right-clickthegeneratedIPfileandselectOpenIPExampleDesign.
22.
Selectthedestinationfolder,andclickOK.
Thisopensanewscreenwiththeexampledesignproject.
IntegratingtheReferenceDesignwiththeExampleDesignProjectThissectionillustrateshowtointegratetheSR-IOVreferencedesignfilesprovidedwiththeapplicationnote.
Thefilesareprovidedwiththegeneratedexampledesignprojectinthesriov_reference_designfolder.
1.
Unzipthexapp1177.
zipfileandcopythesriov_reference_designdirectoryintothelocalareawheretheexampleprojectfileexists.
2.
IntheSourcestaboftheexampleprojectintheVivadoIDE,selectallfilesunderthepcie_app_7vx_ihierarchy.
Right-clickthehighlightedfilenamesandselectRemoveFilefromProject(Figure14).
X-RefTarget-Figure13Figure13:Legacy/MCICapTabX-RefTarget-Figure14Figure14:Removepcie_app_7vxInstanceFilesHostSystemSetupXAPP1177(v1.
0)November15,2013www.
xilinx.
com163.
Right-clickpcie_app_7vx_i,andselectAddSources.
4.
Toaddfilesfromthesriov_reference_designdirectory,browsetothesriov_reference_designfolderandaddallprovideddesignfiles.
5.
Afterthefilesareadded,intheFlowNavigatorpane,clickGenerateBitstreamtostartimplementation.
HostSystemSetupThissectiondescribeshowtoenablevirtualizationcapabilityonthehostserver,andhowtosetupvirtualmachinesusingthedefaulthypervisorofthehostOS.
Figure15showsthehostsystemforthisexample.
1.
IntheBIOSoptionsofthemotherboardmakesurethatVT-disenabled.
TheVT-doptionisusuallyfoundinBIOS>AdvanceSettings>SystemAgent>EnableVT-d.
AsdiscussedinSystemRequirementsforSR-IOV,page6,RedHadEnterpriseLinuxversion6.
2isusedastheHostOperatingSystem.
TousevirtualizationonRedHatEnterpriseLinux,theqemu-kvmandqemu-imgpackagesareneeded.
ThesepackagesprovidetheKVMemulatoranddiskimagemanagerfunctions.
2.
SelecttheSoftwareDevelopmentWorkstationoptionwhileinstallingRHEL.
Thisensuresthatallrequiredvirtualizationpackages(qemu-kvmandqemu-img)arepre-installed.
X-RefTarget-Figure15Figure15:HostSystemSetup>/EhyK^sDsDsDsDsDsDW&Z/sZW&Z/sZ/Ed>sd/KDDhSystemtools.
6.
ClicktheCreateanewvirtualmachinebuttonasshowninFigure17.
7.
FollowtheinstructionsinthesubsequenttabstocreateaVirtualMachinewithFedora16astheGuestOS.
SelecttheSoftwareDevelopmentWorkstationoptionwhileinstallingFedora16sothatalllibrariesrequiredtocompilethedriversarepre-installed.
8.
AfterthefirstVirtualMachineiscreated,right-clickVirtualMachine,andselectthecloneoptiontocreate5othervirtualmachines.
Onlyoneclonecanbecreatedatatime.
AfterallVirtualmachinesarecreated,theVirtualMachineManagershouldresembleFigure18.
X-RefTarget-Figure16Figure16:Editgrub.
confX-RefTarget-Figure17Figure17:CreateNewVirtualMachineReferenceDesignValidationinHardwareXAPP1177(v1.
0)November15,2013www.
xilinx.
com18Foradditionaldetailsandtroubleshootinginformation,seetheRedHatEnterpriseLinuxVirtualizationHostConfigurationandGuestInstallationGuide[Ref6].
ReferenceDesignValidationinHardwareThissectiondescribesthehardwarevalidationoftheSR-IOVreferencedesignandprovidesdetailsonVFenumeration,attachingVFstoVirtualMachinesandrunningPIOtrafficontheattachedVFs.
1.
Withthehostsystempoweredoff,plugtheVC709boardintoaGen3PCIex8slotonthehostsystem.
2.
UsetheexternalACpoweradaptersuppliedandpower-ontheboard.
Caution!
DonotusethePCIepowerconnectorfromtheATXpowersupplyofthehostmachine.
SeetheVC709EvaluationBoardfortheVirtex-7FPGAUserGuide[Ref7]forproperprocedures.
3.
ConnectaUSBType-AtoMicro-BcablefromyourlocalmachinetotheUSBJTAG(Digilent)connectorontheVC709board,asshowninFigure19.
4.
IntheVivadoIDEFlowNavigatorpane,selectOpenHardwareSessionunderProgramandDebugtoloadthebitstream.
5.
SelectOpenNewHardwareTargetandusethedefaultVivadoCSEServer.
X-RefTarget-Figure18Figure18:AllVirtualMachinesX-RefTarget-Figure19Figure19:VC709BoardUSBJTAGConnectorReferenceDesignValidationinHardwareXAPP1177(v1.
0)November15,2013www.
xilinx.
com196.
SelecttheappropriatetargetasshowninFigure20.
7.
Selectthedefaultclockfrequency,andclickFinishtoviewtheHardwareDevice.
8.
IntheHardwarewindow,right-clicktheXC7VX690_Tdevice,andselectAssignProgrammingFileasshowninFigure21.
Theprogrammingfileisassociatedwiththehardwaredevice.
9.
Right-clickthedeviceandselectProgramDevicetoprogramthehardwaredevice.
10.
Poweronthehostmachineandloginastherootuser.
11.
Openaterminal,andtypethesecommands:lspci|grep-ixilinxTheexpectedoutputis:03:00.
0Memorycontroller:XilinxCorporationDevice703803:00.
1Memorycontroller:XilinxCorporationDevice7138lspci-s03:00.
0-vvvThiscommandliststheconfigurationspaceofPF0asshowninFigure22.
PF0ARIcapabilitiesareenabledandarelocatedatoffset0x140.
SR-IOVcapabilitiesarelocatedatX-RefTarget-Figure20Figure20:SelectHardwareTargetX-RefTarget-Figure21Figure21:AssignProgrammingFileReferenceDesignValidationinHardwareXAPP1177(v1.
0)November15,2013www.
xilinx.
com20address0x200.
PF0hasfourassociatedVFs,whichhavenotyetbeenenabled.
EachVFhasa4KBAR0enabled.
lspci-s03:01.
0-vvvSimilartothepreviouscommand,thiscommandliststhecapabilitiesofPF1asshowninFigure23.
X-RefTarget-Figure22Figure22:PF0CapabilitiesEnabledReferenceDesignValidationinHardwareXAPP1177(v1.
0)November15,2013www.
xilinx.
com2112.
CopytheLinux_Driver_PF0directorytothehostmachine.
13.
Modifythexpcie.
cfilesothatthevaluesinthefieldsshownreflectthelspcioutputofPF0:#definePCI_VENDOR_ID_XILINX0x10ee#definePCI_DEVICE_ID_XILINX_PCIE0x7038#defineNUMBER_OF_VFS0x414.
RunthesecommandsfromReferenceDesignDriversandApplication,page11,toinstallthedriversforPF0:a.
makeb.
.
/make_devicec.
insmodxpcie.
ko15.
Performstep12,step13,andstep14forthePF1driver.
16.
Inthexpcie_pf1.
cfile,verifythatthefieldsshownhavethecorrectvalue:#definePCI_VENDOR_ID_XILINX0x10ee#definePCI_DEVICE_ID_XILINX_PCIE0x7138#defineNUMBER_OF_VFS0x2X-RefTarget-Figure23Figure23:PF1CapabilitiesEnabledReferenceDesignValidationinHardwareXAPP1177(v1.
0)November15,2013www.
xilinx.
com2217.
Afterbothdriversareinstalled,typethecommand:lspci|grep-ixilinxAllVFsshouldbeenumeratedasshown:03:00.
0Memorycontroller:XilinxCorporationDevice703803:00.
1Memorycontroller:XilinxCorporationDevice713803:08.
0Memorycontroller:XilinxCorporationDevice723803:08.
1Memorycontroller:XilinxCorporationDevice723803:08.
2Memorycontroller:XilinxCorporationDevice723803:08.
3Memorycontroller:XilinxCorporationDevice723803:08.
4Memorycontroller:XilinxCorporationDevice733803:08.
5Memorycontroller:XilinxCorporationDevice733818.
UsethevirshcommandswhichareapartoftheRHELvirtualizationpackagetoprobethevirtualfunctions.
ToverifythatdevicesexistwithvirshandtoensurethatvirtualizationispresentandfunctionalontheHostOS,typethecommand:virshnodedev-list|grep0303representsthebusnumberonwhichtheXilinxPCIExpressEndpointispresent.
Theexpectedoutputis:pci_0000_00_03_0pci_0000_03_00_0pci_0000_03_00_1pci_0000_03_08_0pci_0000_03_08_1pci_0000_03_08_2pci_0000_03_08_3pci_0000_03_08_4pci_0000_03_08_519.
ToviewtheVirtualfunctiondetails,typethecommand:virshnodedev-dumpxmlpci_0000_03_08_0InFigure24,observethatthisvirtualfunctionisonbus0x03,slot0x08,andfunction0x0.
Thephysicalfunctionassociatedwiththisvirtualfunctionisonbus0x03,slot0x00andfunction0x0(PF0).
20.
Todetachthedevices,typethecommand:virshnodedev-dettachpci_0000_03_08_021.
Performstep20forallVFsasshowninFigure25.
X-RefTarget-Figure24Figure24:VirtualFunctionVF0DetailsReferenceDesignValidationinHardwareXAPP1177(v1.
0)November15,2013www.
xilinx.
com2322.
ToensurethatthePF0devicehasbeendetachedfromthehostsystem,typethecommand:readlink/sys/bus/pci/devices/0000\:03\:08.
0/driverTheVFsaredetachedandareconsumedbypci-stubasshowninFigure26.
23.
ToenablemanagementofthePCIExpressDevice(VirtualFunction)fromtheVirtualmachine,settheSELinuxBoolean.
Thisenablesthedevicestobemanagedbythevirtualmachines.
Typethiscommand:setsebool-Pvirt_use_sysfs124.
NowtheVFsarereadytobeattachedtotheGuestOS.
OpentheVirtualMachineandclicktheAddHardwarebuttontoselectthephysicaldevicetoaddtothevirtualmachine.
25.
SelectPCIHostDeviceandaddtheappropriateHostDevice(VirtualFunction)asshowninFigure27.
Note:WhenaVFisattachedtoavirtualmachineitcannotbeattachedtoanothervirtualmachineunlessitisdetachedfirst.
X-RefTarget-Figure25Figure25:DetachVFsfromHostOSX-RefTarget-Figure26Figure26:VFsDetachedReferenceDesignValidationinHardwareXAPP1177(v1.
0)November15,2013www.
xilinx.
com2426.
Afterthesetupiscomplete,powerontheguestvirtualmachine.
27.
CopytheLinux_Driver_VFdirectoryontotheguestvirtualmachineandperformthestepsshownfromReferenceDesignDriversandApplication,page11toinstallthedriversandrunPIOtrafficontheVirtualFunction:a.
makeb.
.
/make_devicec.
insmodxpcie.
ko28.
Performstep24throughstep27forallVirtualMachinestoattachallsixVFstothevirtualmachines.
Figure28showsPIOtrafficrunningsimultaneouslyonall6virtualmachines.
X-RefTarget-Figure27Figure27:SelectVirtualFunctiontoAttachtoVirtualMachineReferencesXAPP1177(v1.
0)November15,2013www.
xilinx.
com25ReferencesThesedocumentsprovidesupplementalmaterialusefulwiththisapplicationnote:1.
XilinxVirtex-7FPGAGen3IntegratedBlockforPCIExpressv2.
2(PG023)2.
XilinxVirtex-7FPGAIntegratedBlockforPCIExpress–ReleaseNotes(AR54645)3.
PCIExpressBaseSpecificationRev3.
0onPCI-SIGwebsite4.
SingleRootIOVirtualizationandSharingSpecificationv1.
1onPCI-SIGwebsite5.
PCI-SIGSR-IOVPrimer:AnIntroductiontoSR-IOVTechnology;PaperbyIntel6.
RedHatEnterpriseLinuxVirtualizationHostConfigurationandGuestInstallationGuide7.
VC709EvaluationBoardfortheVirtex-7FPGAUserGuide(UG887)8.
IntelVirtualizationTechnologyforDirectedI/OArchitectureSpecificationRevisionHistoryThefollowingtableshowstherevisionhistoryforthisdocument.
X-RefTarget-Figure28Figure28:SimultaneousPIOTrafficonAllVirtualMachinesDateVersionDescriptionofRevisions11/15/20131.
0InitialXilinxrelease.
NoticeofDisclaimerXAPP1177(v1.
0)November15,2013www.
xilinx.
com26NoticeofDisclaimerTheinformationdisclosedtoyouhereunder(the"Materials")isprovidedsolelyfortheselectionanduseofXilinxproducts.
Tothemaximumextentpermittedbyapplicablelaw:(1)Materialsaremadeavailable"ASIS"andwithallfaults,XilinxherebyDISCLAIMSALLWARRANTIESANDCONDITIONS,EXPRESS,IMPLIED,ORSTATUTORY,INCLUDINGBUTNOTLIMITEDTOWARRANTIESOFMERCHANTABILITY,NON-INFRINGEMENT,ORFITNESSFORANYPARTICULARPURPOSE;and(2)Xilinxshallnotbeliable(whetherincontractortort,includingnegligence,orunderanyothertheoryofliability)foranylossordamageofanykindornaturerelatedto,arisingunder,orinconnectionwith,theMaterials(includingyouruseoftheMaterials),includingforanydirect,indirect,special,incidental,orconsequentiallossordamage(includinglossofdata,profits,goodwill,oranytypeoflossordamagesufferedasaresultofanyactionbroughtbyathirdparty)evenifsuchdamageorlosswasreasonablyforeseeableorXilinxhadbeenadvisedofthepossibilityofthesame.
XilinxassumesnoobligationtocorrectanyerrorscontainedintheMaterialsortonotifyyouofupdatestotheMaterialsortoproductspecifications.
Youmaynotreproduce,modify,distribute,orpubliclydisplaytheMaterialswithoutpriorwrittenconsent.
CertainproductsaresubjecttothetermsandconditionsoftheLimitedWarrantieswhichcanbeviewedathttp://www.
xilinx.
com/warranty.
htm;IPcoresmaybesubjecttowarrantyandsupporttermscontainedinalicenseissuedtoyoubyXilinx.
Xilinxproductsarenotdesignedorintendedtobefail-safeorforuseinanyapplicationrequiringfail-safeperformance;youassumesoleriskandliabilityforuseofXilinxproductsinCriticalApplications:http://www.
xilinx.
com/warranty.
htm#critapps.