分片云爆发

Copyright2016,Oracleand/oritsaffiliates.
Allrightsreserved.
|Oracle12cR2ShardingAugust8,2016WeiHu胡维VicePresident副总裁HighAvailabilityTechnologiesOracleConfidential–Internal/Restricted/HighlyRestrictedCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirection.
Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.
Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.
Thedevelopment,release,andtimingofanyfeaturesorfunctionalitydescribedforOracle'sproductsremainsatthesolediscretionofOracle.
Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|数据库分片Sharding概览Overview部署Deployment查询路由QueryRouting生命周期管理LifecycleManagement云Cloud客户Customers123456Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|什么是数据库分片WhatisDatabaseShardingRAC和DataGuard对于超过99%的应用而言都能够满足要求并且是应用透明的RACandDataGuardmeetneedsofover99%ofapplicationswhilepreservingapplicationtransparency一些运行在世界范围内的应用需要独立的数据库群SomeWorldScaleapplicationswantfarmofindependentdatabases-数据库分片databasesharding–避免特大型的单一数据库的可扩展性和可用性问题Avoidscalabilityoravailabilityedgecasesofagiganticsinglesystemimagedatabase–愿意修改应用,把负载路由到数据库群中的特定的数据库上Willingtomodifyapplicationstohelprouteworkloadstospecificdatabasesinthefarm–对所有的重要网站使用标准的部署方式Standarddeploymentpatternforallmajorwebsites–NoSQL容易部署成分片的方式,现在Oracle数据库自带的分片功能也很容易部署SQLdatabasesmadeiteasytodeploySharding,nowOracleDatabaseNativeShardingalsomakesiteasy4ShardDB#1BADCFEHGShardDB#2ShardDB#3JILKBADCFEHGJILKOneGiantDBtoManySmallDBsTablePartitionsCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|巨大的可扩展性&整体的故障隔离MassiveScalability&TotalFaultIsolationOracleConfidential–Internal5Oracle自带的分片功能介绍IntroducingOracleDatabaseNativeSharding水平分区数据到最多1000个Shared-nothing硬件上的数据库中,每个分片有自己的CPU,内存和磁盘Horizontallypartitiondataupto1,000Oracledatabases(shards)Shared-nothinghardware.
EachshardhasitsownCPU,memoryanddisk每个分片存储数据的一个子集Eachshardstoresasubsetofdata使用数据复制用于可用性和附加的可扩展性Replicatedforavailabilityandadditionalscalability非常适合内存处理和本地flash存储Greatforin-memoryandlocalflashstorage不使用共享存储,非常适合公有云基础架构Greatforpubliccloudinfrastructuresthatmostlydonothavesharedstorage线性的可扩展性–数据,负载和用户Linearscalability-data,workloadandusers最大的故障隔离Maximumfaultisolation–Outageorslowdownaffectsonly1ofNshards–Mixeddatabasereleasesacrossshards单个逻辑数据库分片到N个物理的数据库上SinglelogicalDBshardedintoNphysicalDatabasesShardDB01PartitionsBADCPartitionsFEHGShardDB02ShardDB03PartitionsJILKDBServerADBServerBDBServerCCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|数据库分片Sharding概览Overview部署Deployment查询路由QueryRouting生命周期管理LifecycleManagement云Cloud客户Customers123456Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|7自动分配AutomatedDistributionEnhancedSQLDDLforSharding增强SQLDDL用于分片…CREATESHARDEDTABLECustomers(CustIdVARCHAR2(60)NOTNULL,FirstNameVARCHAR2(60),LastNameVARCHAR2(60),…PRIMARYKEY(CustId),)PARTITIONBYCONSISTENTHASH(CustId)…使用标准的SQL创建一个分片表StandardSQLforcreatingshardedtables不像NoSQL要使用专用的APINotproprietaryAPIsaswithNoSQL建立分片表后自动地在分片范围分区数据Creationofashardedtableautomaticallypartitionsdataacrossshards选择分片方式Choiceofshardingmethods:系统管理分片-一致性哈希Systemmanaged-consistenthash用户自定义-范围,列表Userdefined-range,list组合分区–范围-哈希,列表-哈希Composite-range-hash,list-hash常用的参考数据自动地复制到所有的分片Commonreferencedata(e.
g.
stockitems)isautomaticallyduplicatedonallshardsOracleConfidential–InternalCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|系统管理分片SystemManagedSharding数据开始被分成许多小的分区Dataisfirstsplitintomanysmallpartitions使用一致性哈希算法把分区分到分片PartitionsarethenhashedtoshardsusingCONSISTENTHASH+自动平衡数据分布Automaticbalanceddatadistribution-用户不控制的数据的位置Userhasnocontrolonlocationofdata8…Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|组合分片CompositeSharding两级数据组织Two-levelsofdataorganization应用使用2个分片键Applicationuses2shardingkeys(86-1065386688)分区首先是列表或者范围PartitionsarefirstdistributedbyLISTorRANGE(specifiedbytheuser)在同一个组内数据自动哈希Withinagroup,dataisautomaticallyhashed+地理位置分布Geo-distribution+混合云Hybridcloud9…………ChinaUSEuropeAsia…………BeiJingShangHaiJiangSuZheJiangCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|ShardDirector集中的自动化Schema管理CentralizedandAutomaticSchemaManagement10Shard1Shard2ShardnShardCatalogconnecttoGDS$CATALOGservicealtersessionenableshardddl;createtablespaceset…createuser.
.
.
createshardedtable….
.
Createduplicatedtable…Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|OracleConfidential–Internal如果一个分片出了故障呢Whatifashardfails自动配置的复制用于分片HAAutomaticConfigurationofReplicationforShardHA11DataGuardwithFast-StartFailover-Default*GoldenGateactive-activereplicationwithAutoConflictDetection/Resolution**AutomateddeploymentPartition_FPartition_FPartition_APartition_COptionally–complementreplicationwithOracleRACforserverHACopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|使用OracleRAC解决高可用性HighAvailabilityusingOracleRACPartition_FPartition_APartition_CShardDirectorsShardDirectorsCatalog/CoordinatorStandbyDGThreePrimaryShardsThreeStandbyShardsDGDGDG在数据中心中每个分片使用RAC做HAEachshardusesRACforHAwithindatacenterDataGuard用于跨数据中心的容灾和保护RAC的故障DataGuardusedacrossdatacentersfordisasterrecoveryandprotectionagainstRACoutageCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|使用ActiveDataGuard解决高可用性HighAvailabilityusingActiveDataGuard使用ActiveDataGuard把每一个分片复制到1个或者更多个备份库上Eachshardisreplicatedto1ormorestandbysusingActiveDataGuard用同步复制技术实现自动的故障保护Canusesynchronousreplicationusingautomaticfailover自动配置Automaticconfiguration12.
2最多使用1000个分片Upto1,000shardsin12.
2ShardDirectorsCatalog/CoordinatorShardDirectorsPrimaryShardsStandbyShardsStandbyADGADGADGDGCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|使用GoldenGate解决高可用性HighAvailabilityusingGoldenGate每一个chunk都使用GoldenGate复制EachchunkisreplicatedusingGoldenGate能配置复制的数量CanconfigurenumberofreplicasActive/Active复制Active/activereplication自动配置Automaticconfiguration12.
2最多复制32个分片Upto32shardsin12.
2ShardDirectorsCatalog/CoordinatorShardDirectorsShardsStandbyDGOGGOGGOGGCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|OutageofaShardhasNoImpactonOtherShards容错FaultTolerantApplicationThroughput应用在其它分片上保持可用性ApplicationremainsavailableonothershardsShardOutageAutomaticfailoverquicklyrestoresservicefortheshardimpactedbyoutageCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|分片能提高可靠性ShardingcanImproveAvailability现有的HA架构可以通过使用切换到其它的RAC实例上或者DataGuard的另外一个数据库上实现故障保护CurrentHAarchitectureshandlefailuresbyfailingovertoanotherRACinstance,oranotherdatabasewithDataGuard–故障保护会很快Failovercanbefast–但是这个过程可能回被扰乱Butcanbedisruptive–需要重连,会话可能会暂停needtoreconnect,sessionsmaybepaused假设使用10个数据库分片,一个分片故障保护只会扰乱10%的用户IfIshardacross10databases,afailoverwilldisruptonly10%ofmyusers假设使用100个数据库分片,一个分片故障保护只会扰乱1%的用户IfIshardacross100databases,afailoverwilldisruptonly1%ofmyusers99%的用户可以完全不受影响99%ofmyusersarecompletelyunaffected16Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|数据库分片Sharding概览Overview部署Deployment查询路由QueryRouting生命周期管理LifecycleManagement云Cloud客户Customers123456Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|使用Oracle分区的查询路由QueryRoutingwithOracleSharding使用sharding_key的直接路由DirectRoutingusingsharding_key–在连接过程中对于OLTP的负载使用特定的sharding_keyForOLTPworkloadsthatspecifysharding_key(e.
g.
customer_id)duringconnect一个事物在单个分区进行Atransactionhappensonsingleshard–在智能中间层连接池和驱动中激活Enabledbysmartmid-tierconnectionpoolsanddriversJDBC/UCP,OCI,andODP.
NET可以认出分片键JDBC/UCP,OCI,andODP.
NETrecognizeshardingkeys–PHP,Python,Perl,andNode.
jsuseOCIsupportforsharding使用coordinatorDB实现代理路由ProxyRoutingviacoordinatorDB–应用不能确定sharding_keyForapplicationthatcannotspecifysharding_key(aspartofconnection)–报表和批处理业务Appliestoreporting,batchjobs–跨片的并行查询Queriesexecuteinparallelacrossshards18Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|ApplicationTierRoutingTierDatabaseTier使用shardingkey直接路由DirectRoutingviaShardingKeyApplicationServerShardDirectors连接池维护一个分片拓扑的缓冲区Connectionpools(UCP,OCI,.
.
)maintainshardtopologycache;amappingofkeyrangestoshards数据库请求在缓冲区中获得key后直接连接分片数据库DBrequestsforakeyinacachedrangegodirectlytotheshard(i.
e.
,bypassessharddirector)19ShardKeyRangesChunkNameShards1--10Chunk1Shard1,Shard210--20Chunk2Shard1,Shard220--30Chunk3Shard3,Shard430–40Chunk4Shard3,Shard4Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|线性的可扩展性-容易LinearScalability–Easy02000004000006000008000001000000OLTPThroughputScale-outOLTPWorkloadfrom10to50Shards#ofShardsTPM11020304050在本地测试过1000个分片Testedupto1,000shardson-premises在Oracle云中也测试过AlsotestinginOracleCloud因为没有共享的硬件和软件所以容易实现EasybecausethereisnosharedhardwareorsoftwareCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|CoordinatorDBApplicationTierRoutingTierDatabaseTier21代理路由ProxyRoutingNon-ShardKeyAccess&Cross-ShardQueries如果没有shardingkey,连接会连到定位数据库中Ifnoshardingkey,thentheconnectionismadetothecoordinatordatabase定位库解析SQL然后代理/路由连接请求到正确的分片上CoordinatorparsesSQLandproxy/routerequesttocorrectshard多分片查询使用相同的流Sameflowformulti-shardqueriesSQL在每个分片上处理SQLprocessingdonebytheshards开发省事但是性能下降Fordeveloperconvenienceandnotforhighperformance定位库可以使用RAC,DataGuard和GoldenGate实现HACoordinatorcanuseRAC,DataGuard,andGoldenGateforHAApplicationServerShardDirectorsOracleConfidential–InternalCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|数据库分片Sharding概览Overview部署Deployment查询路由QueryRouting生命周期管理LifecycleManagement云Cloud客户Customers123456Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|什么是再分片Whataboutresharding分片的一个挑战是增加和减少分片的时候需要重新分配数据Oneofthechallengesofshardingistheneedtoredistributedatawhenyouaddorremoveshards谷歌的经验Google'sexperience(OSDI2012)–AdWords最开始是基于MySQL数据库的,使用了手工分区技术This[AdWords]backendwasoriginallybasedonaMySQLdatabasethatwasmanuallysharded…–伴随着客户数目的增长和数据库中非常昂贵的数据一直进行的重分区关键收入数据库.
最后重分区用了两年时间…Reshardingthisrevenue-criticaldatabaseasitgrewinthenumberofcustomersandtheirdatawasextremelycostly.
Thelastreshardingtookovertwoyears…23PainpointofshardedarchitecturesCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|自动化分配AutomatedDistributionOnlineAddandRebalanceShards24移动是自动地初始化或手工进行Moveisinitiatedautomaticallyormanually(byDBA)自动化Automated–使用RMAN的增量备份和可传输的表空间技术UsesRMANIncrementalBackup&TransportableTablespaceAddshardOracleConfidential–InternalCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|25自动化地分配AutomatedDistributionOnlineAdditionandRebalancingofShards移动是自动地初始化或手工进行Moveisinitiatedautomaticallyormanually(byDBA)自动化Automated使用RMAN的增量备份和可传输的表空间技术UsesRMANIncrementalBackup&TransportableTablespace在重分片的过程中应用不需要改变ApplicationdoesnotneedtochangeforreshardingAutoRebalanceOracleConfidential–InternalCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|别的数据库也经常提到错误的分片技术FalseShardingisalsoPromotedbyotherDatabases分片技术目前很热门,许多数据库声称支持分片Shardingishot.
Manydatabasesclaimtohavesharding通常的方法是前端数据库路由请求到其它的数据库Commonwayisafront-endDBroutingrequeststootherdatabases和真正的分片不同,增加后端的数据库不会增加读/写的吞吐量因为前端数据库会成为瓶颈Unlikerealsharding,addingbackenddatabasesdoesnotincreaseread/writethroughputbecausefront-endDBbecomesbottleneck和真正的分片不同,这些分片没有完整的故障隔离.
如果前端数据库死机,所有的会话断连.
如果前端数据库变慢,所有的会话都会受到影响Unlikerealsharding,donothavecompletefaultisolation.
Iffront-enddies,allsessionsdisconnected.
Iffront-endslowsdown,allsessionsaffectedOracle直接连接到分片查询的能力可以消除这些问题Oracledirect-to-shardquerycapabilityeliminatestheseissues另外,Oracle分片支持集中化的分片建立,增加/移除分片,自动化配置Plus,OracleShardingsupportscentralizedshardcreation,adding/removingshards,automaticconfiguration,…26Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|数据库分片Sharding概览Overview部署Deployment查询路由QueryRouting生命周期管理LifecycleManagement云Cloud客户Customers123456Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|在云中分片技术ShardingontheCloudExadata和分片是两个在云端处理海量数据的方法ExadataandShardingwillbethe2approachesforhugedatabaseworkloadsinthecloudExadata提供了云端RAC的最大的可伸缩性ExadataoffersmostscalableRACinthecloud(application-transparent)Oracle本地数据库分片技术用于能用到分片技术的应用OracleNativeDatabaseShardingforshardedapplications由于oracle在云端和在本地用了相同的架构,你可以在你的数据中心也这么做BecauseofOracle'scommonarchitectureinthecloudandon-premises,youcandothisinyourowndatacentersalso……MassiveScalabilityintheCloudCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|混合云的分片ShardingonHybridCloud在缺乏本地系统的地区全球化地部署应用Globaldeploymentofapplicationsingeographieswhereyoulackanon-premisespresence在云端容灾Disasterrecoveryonthecloud使用"云爆发"技术增加本地系统的能力Cloudburstingtoaugmenton-premisecapacityUse-Cases……Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|云爆发使用Oracle分片技术CloudBurstingwithOracleSharding季节性的需求扩展系统能力Scalecapacityforseasonaldemand–更多的transations,客户和数据moretransactions,customersanddata为新的客户扩展系统能力用于新的产品发布和新提供的服务Scalecapacityfornewcustomersduetonewproductlaunchesornewserviceofferings…SteadyStateBursttoCloudScaleBack…ShardedDatabaseCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|数据库分片Sharding概览Overview部署Deployment查询路由QueryRouting生命周期管理LifecycleManagement云Cloud客户Customers123456Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|Oracle分片技术和客户一起开发OracleShardingdevelopedwithCustomers和有需求的大客户一起工作Workedwithlargecustomersonrequirements–互联网支付公司InternetPaymentCompany–互联网零售商InternetRetailer–机票订购系统AirlineReservationSystem–电信公司TelecomCompany–手机制造商Cellphonemanufacturer–银行Bank–社交媒体公司SocialMediaCompanySeptember2016On-siteBetaatOracleHQ–11个客户(2个来自中国)使用betasoftware做了hands-on11customers(2fromChina)workedhands-onwithbetasoftware–2个客户同意在OOW2015演讲2customersagreedtopresentatOOW201532OracleSHARDING2015LinkedInCorporation.
AllRightsReserved.
VinothGovindarajwww.
linkedin.
com/in/vgovindarajWhereLinkedInWantsToBe-Oracle12cSharding2015LinkedInCorporation.
AllRightsReserved.
34….
.
….
.
ShardGroup1-Repfactor-1DataGuardStandby….
.
ShardGroup2-Repfactor-1ShardGroup3-Repfactor-1APPAPPAPP….
.
….
.
Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|BuzzonsocialmediaatOOW"我听说其他几个人到处走动谈论12.
2新分片的能力.
对我来说,这是一个重点,也很可能是OOW15highlights列表最顶上的一个要点.
""Iheardafewotherpeoplewalkingaroundtalkingabout12.
2'snewshardingability.
Tome,thisisamajorpointandcouldprobablybeatthetopofthelistofhighlightsfromOOW15.
""在社交媒体上的新闻里都是Oracle12c的下一带有分片功能的版本即将到来.
这是非常令人兴奋,让我们看看他们在这方面将带来什么.
""ThenewsoutthereonsocialmediaisthatOracle12cnextversioniscomingupwithShardingoption.
Thatisprettyexcitingandlet'sseewhattheycomeupinthisregard.
""最佳的新特性之一Oneofthebestnewfeatures.
""Oracle12.
2上的分片数据库选项在demoground中是真正酷的,一直到它明年发布""ShardingdatabaseoptiononOracle12.
2,reallycoolondemogroundsonly#oow15untilitgoespublicnextyear""Oracle分片技术是一个新的留下词,基于分区技术,因此不仅仅是扩展…""Oracleshardingisthenewtrendingword,basedonpartitioning.
Sonotonlyscaling.
.
.
#oow15@AMISnl""OracleDB12.
2引进了shared-nothing分片,支持多达1000个节点""OracleDB12.
2introducedshared-nothingsharding,supportsupto1000nodes#oow15""多达1000个分片,不共享存储,将来支持混合数据库版本""Upto1000shardsin12.
2,nosharedstorage,andsupportformixeddatabasereleasesgoingforward#oow15""太好了,我正在寻找web-scale,Oracle正好将要在12cR2中提供了…"""WellifI'mlookingforweb-scale,Oracleisnowgoingtoprovideitwith12cR2.
Oracle12cbroughtnativeJSONsupport,whichisgreat.
Oracle12.
2willnowletmecreateashardedtableandenjoythatweb-scale-abilitythatMongoDBprovides.
MyapplicationscanconnecttoanOracledatabaseandenjoyshardedandnon-shardedtableswiththesamedatabaseconnection.
Idon'tloseanyoftheotherstuffI'vegrownsofondof.
353rdHighestAttendedDemoBoothAcrossallofOracleDatabaseCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|分片技术全球范围内的试用World-wideShardingTestDrivesJapan–December17-18,2015OracleHQ–January28,2016London–February11,2016Beijing–Feb25,2016(mostattendees)NewYork–March10,2016广泛的用户群:互联网公司,金融公司,制造业公司,,政府,制造和零售Widerangeofcustomers:Internetcompanies,Financialcompanies,manufacturingcompanies,government,manufacturing,andretailersCopyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|…37Oracle数据库分片–总结OracleDatabaseSharding-Summary线性可扩展容错地理分布在线增加分片提高数据库的吞吐能力.
在线分拆重平衡Addshardsonlinetoincreasedatabasethroughput.
Onlinesplitandrebalance.
不共享硬件和软件以隔离故障,分片可能运行不同Oracle版本Nosharedhardwareorsoftwaretoisolatefaults.
ShardsmayrundifferentOraclereleases.
基于性能,可用性,DR的考虑或者满足一般性的需求,用户自定义数据的位置Userdefineddataplacementforperformance,availability,DRortomeetregulatoryrequirements.
…Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|38Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|OracleConfidential–Internal/Restricted/HighlyRestricted39Copyright2015,Oracleand/oritsaffiliates.
Allrightsreserved.
|OracleConfidential–Internal/Restricted/HighlyRestricted40