propertyMapFromFile403forbidden

DataIngestionServiceUserGuideIssue01Date2020-07-07HUAWEITECHNOLOGIESCO.
,LTD.
CopyrightHuaweiTechnologiesCo.
,Ltd.
2020.
Allrightsreserved.
NopartofthisdocumentmaybereproducedortransmittedinanyformorbyanymeanswithoutpriorwrittenconsentofHuaweiTechnologiesCo.
,Ltd.
TrademarksandPermissionsandotherHuaweitrademarksaretrademarksofHuaweiTechnologiesCo.
,Ltd.
Allothertrademarksandtradenamesmentionedinthisdocumentarethepropertyoftheirrespectiveholders.
NoticeThepurchasedproducts,servicesandfeaturesarestipulatedbythecontractmadebetweenHuaweiandthecustomer.
Allorpartoftheproducts,servicesandfeaturesdescribedinthisdocumentmaynotbewithinthepurchasescopeortheusagescope.
Unlessotherwisespecifiedinthecontract,allstatements,information,andrecommendationsinthisdocumentareprovided"ASIS"withoutwarranties,guaranteesorrepresentationsofanykind,eitherexpressorimplied.
Theinformationinthisdocumentissubjecttochangewithoutnotice.
Everyefforthasbeenmadeinthepreparationofthisdocumenttoensureaccuracyofthecontents,butallstatements,information,andrecommendationsinthisdocumentdonotconstituteawarrantyofanykind,expressorimplied.
Issue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
iContents1IAMPermissionsManagement.
11.
1CreatingaUserandGrantingPermissions.
12CreatingaDISStream.
33GettingStarted.
83.
1GeneralProcedure.
83.
2Step2:PreparingaDISApplicationDevelopmentEnvironment.
83.
3Step3:SendingDatatoDIS.
133.
4Step4:ObtainingDatafromDIS.
143.
5ObtainingAuthenticationInformation.
143.
6InterconnectingwithOBS.
153.
7CreatinganIAMAgency.
154ManagingDISStreams.
184.
1ListingDISStreams.
184.
2ViewingStreamMonitoringMetrics.
184.
3ChangingaSourceDataType.
204.
4ManagingaSourceDataSchema.
214.
5ManagingStreamTags.
244.
6ScalingUp/DownaStream.
264.
7DeletingaStream.
285UsingDIS.
295.
1CheckingandConfiguringDNSInformation.
295.
2UploadingDatabyUsingAgent.
305.
2.
1DISAgentOverview.
305.
2.
2PreparingforInstallingDISAgent.
315.
2.
3InstallingDISAgent.
335.
2.
4ConfiguringDISAgent.
335.
2.
5StartingDISAgent.
415.
2.
6TestingDISAgent.
415.
2.
7StoppingDISAgent.
445.
3UsingDISFlumePlugintoUploadandDownloadData445.
3.
1Overview.
455.
3.
2PreparingforInstallingaDISFlumePlugin.
45DataIngestionServiceUserGuideContentsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
ii5.
3.
3InstallingaDISFlumePlugin.
465.
3.
4ConfiguringtheDISFlumePlugin.
475.
3.
5TestingaDISFlumePlugin.
555.
3.
6(Optional)UninstallingaDISFlumePlugin.
575.
4UsingaDISLogstashPlugintoUploadandDownloadData585.
4.
1Overview.
585.
4.
2PreparingforInstallingaDISLogstashPlugin.
595.
4.
3InstallingDISLogstashPluginOnline.
605.
4.
4InstallingDISLogstashPluginOffline.
615.
4.
5ConfiguringtheDISLogstashPlugin.
625.
4.
6TestingtheDISLogstashPlugin.
665.
4.
7(Optional)UninstallingtheDISLogstashPlugin.
675.
5UsingSparkStreamingSDKtoDownloadData.
675.
5.
1Overview.
685.
5.
2PreparingaDISSparkStreamingEnvironment.
685.
5.
3CustomizingaSparkStreamingJob.
696ManagingaDumpTask.
766.
1CreatingaDumpTask.
766.
2DumpingDatatoOBS.
796.
3DumpingDatatoDLI.
886.
4DumpingDatatoMRS.
897ManagingEnterpriseProjects.
938NotifyingEvents.
958.
1EventNotificationOverview.
958.
2SettingNotification.
968.
3ViewingEvents.
989Monitoring.
1009.
1SupportedMetrics.
1009.
2SettingAlarmRules.
1029.
3QueryingMetrics.
102AChangeHistory.
104DataIngestionServiceUserGuideContentsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
iii1IAMPermissionsManagement1.
1CreatingaUserandGrantingPermissionsThischapterdescribeshowtouseIAMtoimplementfine-grainedpermissionscontrolforyourDISresources.
WithIAM,youcan:CreateIAMusersforemployeesusingtheHUAWEICLOUDaccountbasedontheorganizationalstructureofyourenterprise.
EachIAMuserhastheirownsecuritycredentials,providingaccesstoDISresources.
Grantonlythepermissionsrequiredforuserstoperformatask.
EntrustaHUAWEICLOUDaccountorcloudservicetoperformprofessionalandefficientO&MonyourDISresources.
IfyourHUAWEICLOUDaccountdoesnotneedindividualIAMusers,thenyoumayskipoverthischapter.
Thissectiondescribestheprocedureforgrantingpermissions.
Figure1-1showstheprocedure.
PrerequisitesLearnaboutthepermissions(see)supportedbyDISandchoosepoliciesorrolesaccordingtoyourrequirements.
Forthepermissionsofotherservices,see.
DataIngestionServiceUserGuide1IAMPermissionsManagementIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
1ProcessFlowFigure1-1ProcessforgrantingCDMpermissions1.
Createausergroupandassignpermissionstoit.
CreateausergroupontheIAMconsoleandassigntheDISOperatorpolicytothegroup.
2.
CreateanIAMuser.
CreateauserontheIAMconsoleandaddtheusertothegroupcreatedin1.
3.
Loginandverifypermissions.
Logintotheconsolebyusingtheusercreated,andverifythattheuserhasthegrantedpermissions.
–ChooseServiceList>DataIngestionService.
OntheDISconsole,createastream.
Ifnomessageappearsindicatinginsufficientpermissionstoperformtheoperation,theDISOperatorpolicyhasalreadytakeneffect.
–ChooseanyotherserviceinServiceList.
Ifamessageappearsindicatingthatyouhaveinsufficientpermissionstoaccesstheservice,theDISOperatorpolicyhasalreadytakeneffect.
DataIngestionServiceUserGuide1IAMPermissionsManagementIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
22CreatingaDISStreamYoucancreateaDISstreamontheDISmanagementconsole.
ProcedureStep1UsetheaccounttologintotheDISconsole.
Step2Clickintheupperleftcornerofthepageandselectaregionandproject.
Step3ClickBuyStreamandsetrelatedparameters.
Table2-1StreamparametersParameterDescriptionExampleBillingModePay-per-usePay-per-useRegionPhysicallocationofthecloudservice.
Youcanselectadifferentregionfromthedrop-downlist.
-BasicInformationStreamNameNameoftheDISstreamtobecreated.
Astreamnameis1to64characterslong.
Onlyletters,digits,hyphens(-),andunderscores(_)areallowed.
dis-TidoStreamTypeCommon:Eachpartitionsupportsamaximumreadspeedof2MB/sandamaximumwritespeedof1MB/s.
Advanced:Eachpartitionsupportsamaximumreadspeedof10MB/sandamaximumwritespeedof5MB/s.
-DataIngestionServiceUserGuide2CreatingaDISStreamIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
3ParameterDescriptionExamplePartitionsPartitionsarethebasethroughputunitofaDISstream.
ForacommonDISstream,thevalueofPartitionsisanintegerfrom1to50.
Atenantcancreateamaximumof50partitions.
ForanadvancedDISstream,thevalueofPartitionsisanintegerfrom1to10.
Atenantcancreateamaximumof10partitions.
5PartitionCalculatorCalculatorusedtocalculatetheestimatednumberofpartitionsbasedontheinformationyouentered.
1.
ClickPartitionCalculator.
2.
InthePartitionCalculatordialogbox,configuretheAverageRecordSize(KB),Max.
RecordsWritten,andConsumerApplicationsparameters.
TheEstimatedPartitionsfieldthendisplaystherecommendednumberofpartitions.
Thevalueofthisfieldcannotbemodified.
NOTEPartitioncalculationformulas:–Basedonthetraffic(thefinalvaluemustberoundedup):Commonstream:Averagerecordsizex(1+20%)xMaximumrecordswritten/(1x1024KB)(20%isthereservedpartitionpercentage.
)Advancedstream:Averagerecordsizex(1+20%)xMaximumrecordswritten/(5x1024KB)(20%isthereservedpartitionpercentage.
)–Basedontheconsumerprogramquantity(thefinalvaluemustberoundedup):(Numberofconsumerprograms/2)xNumberofpartitionscalculatedbasedonthetraffic(Theresultofthenumberofconsumerprograms/2mustreservetwodecimals.
)Thelargestvalueamongthevaluescalculatedbasedonthepreviousthreeformulasisconsideredastheestimatedpartitionvalue.
3.
ClickUseEstimatedValue.
TheestimatedvalueisautomaticallyusedasthevalueofPartitions.
-DataRetention(hours)ThemaximumnumberofhoursforwhichdatacanbepreservedinDIS.
Datawillbedeletedwhentheretentionperiodexpires.
Valuerange:anintegerrangingfrom24to72.
24DataIngestionServiceUserGuide2CreatingaDISStreamIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
4ParameterDescriptionExampleSourceDataTypeBLOB:acollectionofbinarydatastoredasasingleentityinadatabasemanagementsystem.
IfSourceDataTypeissettoBLOB,thesupportedDumpDestinationcanbeOBSorMRS.
JSON:anopen-standardfileformatthatuseshuman-readabletexttotransmitdataobjectsconsistingofattribute–valuepairsandarraydatatypes.
IfSourceDataTypeissettoJSON,thesupportedDumpDestinationcanbeOBS,MRS,DLI,CloudTable,orDWS.
CSV:asimpletextformatforstoringtabulardatainaplaintextfile.
IfSourceDataTypeissettoCSV,thesupportedDumpDestinationcanbeOBS,MRS,DLI,orDWS.
JSONAuto-ScalingYoucanchoosetoenableordisableauto-scalingwhencreatingastream.
Youcanclickortodisableorenableauto-scaling.
NOTEYoucanchoosewhethertoenableauto-scalingwhencreatingastream.
Youcanalsomodifytheauto-scalingattributesforacreatedstream.
Auto-ScaleDownToLowerlimitforautomaticscale-down.
Thenumberoftargetpartitionsforautomaticscale-downmustbegreaterthanorequaltothelowerlimit.
-Auto-ScaleUpToUpperlimitforautomaticscale-up.
Thenumberoftargetpartitionsforautomaticscale-upmustbesmallerthanthelowerlimit.
-SchemaSpecifieswhethertocreateaschemawhencreatingastream.
Youcanclickortodisableorenabletheschemaconfiguration.
NOTEIfyoudonotcreateadataschemawhencreatingastream,youcangotothemanagementpagetocreateoneafterthestreamissuccessfullycreated.
Fordetails,seeManagingaSourceDataSchema.
YoucancreateaschemaonlywhenthesourcedatatypeissettoJSONorCSV.
DataIngestionServiceUserGuide2CreatingaDISStreamIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
5ParameterDescriptionExampleSourceDataSchemaYoucanenterorimportsourcedatasamplesinJSONorCSVformat.
Fordetails,seeManagingaSourceDataSchema.
1.
Inthelefttextbox,enteraJSONorCSVsourcedatasampleorclicktoimportasourcedatasample.
2.
Inthelefttextbox,clicktodeleteyourenteredorimportedsourcedatasample.
3.
Inthelefttextbox,clicktogenerateanAvroschemaintherighttextboxaccordingtothesourcedatasample.
4.
Intherighttextbox,clicktodeletethegeneratedAvroschema.
5.
Intherighttextbox,clicktomodifythegeneratedAvroschema.
ThisparameterismandatoryonlywhenSchemaissettoEnable.
EnterpriseProjectConfiguretheenterpriseprojecttowhichstreamsbelong.
YoucanconfigurethisparameteronlywhentheEnterpriseManagementserviceisenabled.
Thedefaultvalueisdefault.
Anenterpriseprojectfacilitatesproject-levelmanagementandgroupingofcloudresourcesandusers.
Youcanselectthedefaultenterpriseproject(default)orotherexistingenterpriseprojects.
Tocreateanenterpriseproject,logintotheEnterpriseManagementconsole.
Fordetails,seetheEnterpriseManagementUserGuide.
-ConfigureClickConfigurenow.
TheTagparameterisdisplayed.
Fordetailsabouthowtoaddatag,seeManagingStreamTags.
-SkipNoadvancedsettingsneedtobeconfigured.
-TagIdentifierofthestream.
Addingtagstostreamscanhelpyouidentifyandmanageyourstreamresources.
-Step4ClickNext.
TheDetailspageisdisplayed.
DataIngestionServiceUserGuide2CreatingaDISStreamIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
6Step5ClickSubmit.
----EndDataIngestionServiceUserGuide2CreatingaDISStreamIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
73GettingStarted3.
1GeneralProcedureThefollowingisthegeneralprocedureforusingDIS:CreatingaDISStreamYouneedtocreateastreambeforeusingDIS.
Step2:PreparingaDISApplicationDevelopmentEnvironmentBeforedevelopingaDISapplication,installanapplicationdevelopmenttool,andimportyourSDKpackageandsampleprojectintothedevelopmentenvironment.
Step3:SendingDatatoDISWriteaproducerapplicationandrunittosenddatatothecloud.
TheDISstreaminformationcanbeviewedontheDISconsole.
Step4:ObtainingDatafromDISWriteaconsumerapplicationandrunittoretrievedatafromthecloud.
3.
2Step2:PreparingaDISApplicationDevelopmentEnvironmentBeforedevelopingDISapplications,prepareanapplicationdevelopmentenvironment,andthenobtainasoftwaredevelopmentkit(SDK)andsampleprojectandimportthemtothedevelopmentenvironment.
PrerequisitesJDK1.
8orlaterhasbeeninstalled.
Eclipsehasbeeninstalled.
DataIngestionServiceUserGuide3GettingStartedIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
8ProcedureStep1ConfigureaJDKusingEclipse.
1.
StartEclipseandchooseWindow>Preferences.
ThePreferencesdialogboxisdisplayed.
2.
Inthenavigationtree,chooseJava.
OntheJavapage,configuregeneralsettingsforJavadevelopmentandthenclickOK.
Figure3-1Preferences3.
Inthenavigationtree,chooseJava>InstalledJREs.
–EnsurethatconfiguredJDKenvironmentalvariablesaredisplayedontheInstalledJREspage.
ThengotoStep1.
3.
a.
–ToconfiguredifferentvariablesfordifferentversionsofJDK,performStep1.
3.
btoStep1.
3.
d.
DataIngestionServiceUserGuide3GettingStartedIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
9Figure3-2InstalledJREsa.
SelecttheinstalledJDKandclickOK.
b.
ClickAdd.
TheAddJREdialogboxisdisplayed.
Figure3-3JRETypec.
SelectaJREtypeandclickNext.
DataIngestionServiceUserGuide3GettingStartedIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
10Figure3-4JREDefinitiond.
ConfigurethebasicinformationaboutJDKandclickFinish.
JREhome:JDKinstallationpath.
DefaultVMarguments:JDKrunningparameters.
Step2Downloadthehuaweicloud-sdk-dis-java-X.
X.
X.
zippackagefromDISSDKanddecompressit.
Step3ImporttheEclipseproject.
1.
StartEclipse.
ChooseFile>Import.
TheImportdialogboxisdisplayed.
2.
ChooseMaven>ExistingMavenProjects,andclickNext.
TheImportdialogboxisdisplayed.
3.
ClickBrowseandselectasavelocationforthedis-sdk-demosampleproject.
IntheProjectsarea,selectasampleproject.
DataIngestionServiceUserGuide3GettingStartedIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
11Figure3-5Importingaproject4.
ClickFinishtoimporttheproject.
Step4Configurethedemoproject.
1.
SettheprojectcodetoUTF-8.
a.
Inthenavigationtree,right-clicktherequiredprojectunderProjectExplorerandchoosePropertiesfromtheshortcutmenu.
ThePropertiesfordis-sdk-demodialogboxisdisplayed.
b.
Inthenavigationtree,chooseResource.
TheResourcepageisdisplayedintherightpane.
c.
IntheOtherdrop-downlist,selectUTF-8.
d.
ClickApplyandClose.
2.
AddtheJDK.
a.
Inthenavigationpane,chooseProjectExplorer.
Right-clickthechosenprojectandchoosePropertiesfromtheshortcutmenu.
b.
Inthenavigationtree,chooseJavaBuildPath.
TheJavaBuildPathpageisdisplayedintherightpane.
c.
ClicktheLibrariestab,andthenclickAddLibrary.
TheAddLibrarydialogboxisdisplayed.
d.
SelectJRESystemLibraryandclickNext.
VerifythattheversionofWorkspacedefaultJREisjdk1.
8orlater.
e.
ClickFinishtoexittheAddLibrarydialogbox.
f.
ClickApplyandClose.
DataIngestionServiceUserGuide3GettingStartedIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
12Step5InitializeaDISclientsample.
Fordetailsaboutendpoint,ak,sk,region,andprojectId,seeObtainingAuthenticationInformation.
----End3.
3Step3:SendingDatatoDISFunctionLocaldataiscontinuouslyuploadedtoDIS.
DatacanbestoredinMRS,DIS,OBS,andDLI.
Fordetailsabouthowtoconfigureastoragelocation,seeCreatingaDumpTask.
ThemaximumnumberofdaysforDIStopreservedatacannotexceedDataRetention(days).
SampleCodeTheexamplecodefileistheProducerDemo.
javafileinthe\dis-sdk-demo\src\main\java\com\bigdata\dis\sdk\demodirectorydecompressedfromthehuaweicloud-sdk-dis-java-X.
X.
X.
zippackage.
ThecompressionpackageisdownloadedinStep2:PreparingaDISApplicationDevelopmentEnvironment.
RunningtheProducerProgramRight-clicktheproducerapplicationandchooseRunAs>1JavaApplicationfromtheshortcutmenu.
Figure3-6RunningaproducerapplicationWhiledataisbeingsenttoDIS,theDISconsoledisplaysDISstreaminformation.
Ifinformationsimilartothefollowingisdisplayed,thedatahasbeensuccessfullysenttoDIS:DataIngestionServiceUserGuide3GettingStartedIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
1314:40:20.
090[main]INFOcom.
bigdata.
dis.
sdk.
DISConfig-getfromclassLoader14:40:20.
093[main]INFODEMOTBEGINPUT14:40:21.
186[main]INFOcom.
bigdata.
dis.
sdk.
util.
config.
ConfigurationUtils-getfromclassLoader14:40:21.
187[main]INFOcom.
bigdata.
dis.
sdk.
util.
config.
ConfigurationUtils-propertyMapFromFilesize:214:40:22.
092[main]INFOcom.
bigdata.
dis.
sdk.
demo.
ProducerDemo-Put3records[3successful/0failed].
14:40:22.
092[main]INFOcom.
bigdata.
dis.
sdk.
demo.
ProducerDemo-[helloworld.
]putsuccess,partitionId[shardId-0000000000],partitionKey[964885],sequenceNumber[0]14:40:22.
092[main]INFOcom.
bigdata.
dis.
sdk.
demo.
ProducerDemo-[helloworld.
]putsuccess,partitionId[shardId-0000000000],partitionKey[910960],sequenceNumber[1]14:40:22.
092[main]INFOcom.
bigdata.
dis.
sdk.
demo.
ProducerDemo-[helloworld.
]putsuccess,partitionId[shardId-0000000000],partitionKey[528377],sequenceNumber[2]14:40:22.
092[main]INFOcom.
bigdata.
dis.
sdk.
demo.
ProducerDemoPUTOVER3.
4Step4:ObtainingDatafromDISFunctionYoucanretrievedatafromDISwhenneeded.
SampleCodeTheexamplecodefileistheConsumerDemo.
javafileinthe\dis-sdk-demo\src\main\java\com\bigdata\dis\sdk\demodirectorydecompressedfromthehuaweicloud-sdk-dis-java-X.
X.
X.
zippackage.
ThecompressionpackageisdownloadedinStep2:PreparingaDISApplicationDevelopmentEnvironment.
RunningtheConsumerApplicationIfinformationsimilartothefollowingappears,datahasbeensuccessfullyretrievedfromDIS:14:55:42.
954[main]INFOcom.
bigdata.
dis.
sdk.
DISConfig-getfromclassLoader14:55:44.
103[main]INFOcom.
bigdata.
dis.
sdk.
util.
config.
ConfigurationUtils-getfromclassLoader14:55:44.
105[main]INFOcom.
bigdata.
dis.
sdk.
util.
config.
ConfigurationUtils-propertyMapFromFilesize:214:55:45.
235[main]INFOcom.
bigdata.
dis.
sdk.
demo.
ConsumerDemo-GetstreamstreamName[partitionId=0]cursorsuccess:eyJnZXRJdGVyYXRvclBhcmFtIjp7InN0cmVhbS1uYW1lIjoiZGlzLTEzbW9uZXkiLCJwYXJ0aXRpb24taWQiOiIwIiwiY3Vyc29yLXR5cGUiOiJBVF9TRVFVRU5DRV9OVU1CRVIiLCJzdGFydGluZy1zZXF1ZW5jZS1udW1iZXIiOiIxMDY4OTcyIn0sImdlbmVyYXRlVGltZXN0YW1wIjoxNTEzNjY2NjMxMTYxfQ14:55:45.
305[main]INFOcom.
bigdata.
dis.
sdk.
demo.
ConsumerDemo-GetRecord[helloworld.
],partitionKey[964885],sequenceNumber[0].
14:55:45.
305[main]INFOcom.
bigdata.
dis.
sdk.
demo.
ConsumerDemo-GetRecord[helloworld.
],partitionKey[910960],sequenceNumber[1].
14:55:46.
359[main]INFOcom.
bigdata.
dis.
sdk.
demo.
ConsumerDemo-GetRecord[helloworld.
],partitionKey[528377],sequenceNumber[2].
3.
5ObtainingAuthenticationInformationObtainingAK/SKAccessKeyID/SecretAccessKey(AK/SK)iscreatedonIdentityandAccessManagement(IAM)toauthenticatecallstoapplicationprogramminginterfaces(APIs)onthepubliccloud.
ToobtainAK/SK,chooseMyCredential>AccessKeys.
DataIngestionServiceUserGuide3GettingStartedIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
14ObtainingProjectIDAprojectisagroupoftenantresources.
ToviewtheprojectIDsofdifferentregions,chooseMyCredential>Projects.
ObtainingRegionInformationandEndpointInformationFordetailsaboutregionsandendpoints,seeRegionsandEndpoints.
3.
6InterconnectingwithOBSIntroductionDIScanuploaddatatoObjectStorageService(OBS).
PrerequisitesAnIAMagencyhasbeencreatedbyfollowingtheprocedureinCreatinganIAMAgency.
ThisIAMagencyentrustsDIStoaccessyourOBSresources.
DataDumpingInsectionCreatingaDumpTask,youcanselectalocationtowhichdatafromDISstreamswillbedumped.
IfDumpDestinationissettoOBS,DISperiodicallyimportsdatafromDISstreamstoOBS.
3.
7CreatinganIAMAgencyIntroductionIfyouchoosetodumpdatafromDIStoOBS,MRS,orDLI,createanIAMagencythatgrantsDISpermissionstoaccessOBS,MRS,orDLI.
CreatinganIAMAgencyStep1Logintothemanagementconsole.
Step2ClickServiceList.
UnderManagement&Deployment,selectIdentifyandAccessManagement.
Step3SelectAgenciesinthenavigationtreepane,andclickCreateAgency.
Step4ConfigureagencyparametersandclickOK.
Table3-1AgencyparameterdescriptionParameterDescriptionAgencyNameNameoftheagencytobecreated.
Thevalueofthisparameteris1to64characterslongandcannotbeleftunspecified.
DataIngestionServiceUserGuide3GettingStartedIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
15ParameterDescriptionAgencyTypeTypeoftheagencytobecreated.
ThisparametermustbesettoCloudservice.
CloudServiceClickSelectnexttoCloudService.
IntheSelectCloudServicedialogbox,selectDISandclickOK.
ValidityPeriodSelectPermanent.
NOTECurrently,thisparametermustbesettoPermanent.
Usinganothervaluemayresultinauthorizationfailures.
DescriptionAgencydescription.
Theentereddescriptioncannotexceed255characters.
DataIngestionServiceUserGuide3GettingStartedIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
16ParameterDescriptionPermissionsIfDumpDestinationisOBS,policysettingsareasfollows:Region:GlobalserviceProject:OBSPolicy:TenantAdministratorIfDumpDestinationisMRS,policysettingsareasfollows:NOTEBothofthefollowingpermissionsneedtobeconfigured.
–Region:GlobalserviceProject:OBSPolicy:TenantAdministrator–Region:theregiontowhichthechosenMRSclusterbelongs;Project:theprojecttowhichthechosenMRSclusterbelongs;Policy:MRSAdministrator,ServerAdministrator,andTenantGuestIfDumpDestinationisDLI,policysettingsareasfollows:NOTEBothofthefollowingpermissionsneedtobeconfigured.
–Region:GlobalserviceProject:OBSPolicy:TenantAdministrator–Region:theregiontowhichthechosenDLItablebelongs;Project:theprojecttowhichthechosenDLItablebelongs;Policy:DLIServiceUserIfDumpDestinationisDWS,policysettingsareasfollows:NOTEBothofthefollowingpermissionsneedtobeconfigured.
–Region:GlobalserviceProject:OBSPolicy:TenantAdministrator–Region:theregiontowhichthechosenDWSclusterbelongs;Project:theprojecttowhichthechosenDWSclusterbelongs;Policy:DWSAdministrator,ServerAdministrator,andTenantGuestTomodifyagencypolicies,clickModifyintheOperationcolumn.
IntheAvailablePoliciesarea,selectyourrequiredpolicyandclickOK.
NOTEAfteranagencyiscreated,itspoliciescannotbemodified.
----EndDataIngestionServiceUserGuide3GettingStartedIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
174ManagingDISStreams4.
1ListingDISStreamsTheStreamManagementpagedisplaysallDISstreamscreated.
Afterclickingastream,youcanviewthefollowinginformationaboutthisstream:Name/ID:UniquenameoftheDISstreamtobecreated.
Astreamnameis1to64characterslong.
Onlyletters,digits,hyphens(-),andunderscores(_)areallowed.
Status:Streamstatus.
StreamType:CommonandAdvanced.
–Common:Eachpartitionsupportsamaximumreadspeedof2MB/sandamaximumwritespeedof1MB/sor1,000records/s.
–Advanced:Eachpartitionsupportsamaximumreadspeedof10MB/sandamaximumwritespeedof5MB/sor2,000records/s.
Partitions:ThenumberofpartitionsintowhichdatarecordsinthenewlycreatedDISstreamwillbedistributed.
Multiplepartitionsofastreamcanconcurrentlytransmitdatatoimproveefficiency.
SourceDataType:BLOB,JSON,andCSV.
DataRetention(hours):ThemaximumnumberofhoursforDIStopreservedata.
Datawillbedeletedwhentheretentionperiodexpires.
Valuerange:anintegerrangingfrom24to72.
Unit:hourCreated:TimeatwhichtheDISstreamiscreated.
Thecreationtimeisintheyyyy/MM/ddHH:mm:ssGMTformat.
Forexample,2017/05/0908:00:00GMT+08:00.
BillingMode:Onlypay-per-useissupported.
Operation:SupportedoperationsincludeChangeSourceDataType,Delete,andViewDumpTask.
4.
2ViewingStreamMonitoringMetricsYoucanviewstreammonitoringinformationontheconsoleandmonitorthedataconsumedbyapplicationsinthestream.
DataIngestionServiceUserGuide4ManagingDISStreamsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
18Step1UsetheaccounttologintotheDISconsole.
Step2ClickintheupperleftcornerandselectRegionandProject.
Step3Inthenavigationtreeontheleft,chooseStreamManagementStreamManagement.
Step4Inthestreamlist,clickthenameoftheDISstreamwhosemonitoringmetricswillbeviewed.
Themonitoringpageisdisplayed.
Step5OntheMonitoringpage,clicktheStreamsorPartitionstabtoviewstreamorpartitionmonitoringmetrics.
Table4-1describesthemonitoringparameters.
Fordetailsaboutbasicstreaminformation,seeStep3.
Table4-1DISmonitoringinformationParameterDescriptionTimeRangeMonitoringtimerange.
Values:–1h–2h–3hYoucancustomizethetimerangeforviewingmonitoringinformation.
–ClicknexttoCustom,andthensetthestarttimeandendtime.
–Theendtimecannotbelaterthanthecurrentsystemtime.
–Thedifferencebetweenthestarttimeandendtimecannotexceed72hours.
PartitionsPartitionIDIDofthepartition.
Itstartsfrom0bydefault.
SelectanyofthefollowingvaluesfromthePartitionIDdrop-down.
DataRate(KB/s)Ratesatwhichdataissenttoandretrievedfromthechosenpartitionwithinthespecifiedtimerange.
Unit:KB/sRecordsPerSecondThenumberofrecordssenttoandretrievedfromthechosenpartitionwithinthespecifiedtimerange.
StreamsDataRate(KB/s)RatesatwhichdataissenttoandretrievedfromthechosenDISstreamwithinthespecifiedtimerange.
Unit:KB/sDataIngestionServiceUserGuide4ManagingDISStreamsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
19ParameterDescriptionRecordsPerSecondThenumberofrecordssenttoandretrievedfromthechosenDISstreamwithinthespecifiedtimerange.
SuccessfulRequestsPerSecondThenumberofPutRecordsandGetRecordsrequestssuccessfullyfulfilledwithinthespecifiedtimerange.
ThrottledRequestsPerSecondThenumberofPutRecordsandGetRecordsrequestsrejectedwithinthespecifiedtimerangeduetoflowcontrol.
AverageRequestProcessingTime(ms)TheaverageamountoftimespentinprocessingaPutRecordsorGetRecordsrequest.
Step6Intheupperrightcornerofthegraph,clicktoenlargethegraphforviewingdetails.
----End4.
3ChangingaSourceDataTypeThesourcedataschemaisrequiredfordataconversionofaspecificdumptaskinastream.
Ifitisincorrectlyconfigured,dataconversionfailsandthedumptaskisabnormal.
Youcanconfigurethesourcedataschemawhencreatingastreamorwhencreatingadumptask.
Youcanmodifytheconfiguredsourcedataschemaonthestreamdetailspage.
Step1UsetheaccounttologintotheDISconsole.
Step2Clickintheupperleftcornerofthepageandselectaregionandproject.
Step3Inthenavigationtreeontheleft,chooseStreamManagementStreamManagement.
1.
Clickthenameofastreamthatyouwanttoview.
TheStreamManagementpageisdisplayed.
2.
ClicknexttoSourceDataTypeandselectadesiredsourcedatatypefromthedrop-downlist.
Alternatively,intheOperationcolumnofthestreamforwhichyouwanttochangeitssourcedatatype,chooseMore>ChangeSourceDataType.
Inthedisplayeddialogbox,changethesourcedatatype.
YoucanchangethesourcedatatypeforastreamwhoseSourceDataTypeisBLOB,JSON,orCSVonlywhenthestreamhasnodumptasks.
Afteryouchangethesourcedatatypeofastreamforwhichyouhaveconfiguredsourcedataschema,theexistingsourcedataschemawillbecomeinvalidandcannotberecovered.
Youneedtoreconfigurethesourcedataschemaforthestream.
----EndDataIngestionServiceUserGuide4ManagingDISStreamsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
204.
4ManagingaSourceDataSchemaAsourcedataschemaisauser'sJSONorCSVdatasampleusedtodescribetheJSONorCSVdataformat.
Forexample,DIScangenerateanAvroschemabasedontheJSONdatasampleandconverttheJSONdatauploadedtoastreamtotheParquetorCarbonDataformat.
Threeentrancesareavailableforcreatingasourcedataschema:EnableSchemawhencreatingastream.
Fordetails,seeFigure4-1.
KeepSchemadisabledwhencreatingastream.
Afterthestreamiscreated,chooseStreamManagementinthenavigationtreeandclickthecreatedstream.
ClickCreateSourceDataSchemanexttoSourceDataType.
Fordetails,seeFigure4-2.
KeepSchemadisabledwhencreatingastream.
Afterthestreamiscreated,chooseStreamManagementinthenavigationtreeandclickthecreatedstream.
OntheDumpManagementtabpage,clickCreateDumpTask.
Onthedisplayedpage,createasourcedataschema.
Fordetails,seeFigure4-3.
Figure4-1Method1forcreatingaschemaFigure4-2Entrance2forcreatingaschemaDataIngestionServiceUserGuide4ManagingDISStreamsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
21Figure4-3Entrance3forcreatingaschemaCreatingaSchemaforSourceDatabyImportingFilesUsethefollowingmethodtocreateasourcedataschema:1.
WhenconfiguringSourceDataSchema,clickImportFile.
2.
Inthelefttextbox,enteraJSONorCSVsourcedatasampleorclicktoimportasourcedatasample.
Whenimportingsourcedatasamples,youcanimportonly.
txt,.
json,.
csv,and.
javafiles.
3.
Todeletethesourcedatasample,click.
4.
Inthelefttextbox,clicktogenerateanAvroschemaintherighttextboxaccordingtothesourcedatasample.
5.
Intherighttextbox,clicktomodifytheAvroschema.
6.
Toformattheparseddata,clickFormatandUnformat.
CreatingaSchemaforSourceDatabyCreatingaSchemaTreeUsethefollowingmethodtocreateasourcedataschema:1.
WhenconfiguringSourceDataSchema,clickCreateSchemaTree.
2.
Afterconfiguringanattributenameanddatatype,clickAddtoaddarootnode,asshowninFigure4-4.
Figure4-4Addingtherootnode3.
Selectthecreatedrootnodeandconfigureanattributenameanddatatypeinthesamewaytoaddsubnodes.
DataIngestionServiceUserGuide4ManagingDISStreamsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
22Figure4-5CreatingasubnodeTodeleteanode,selectthecheckboxofthenodeandclickDelete.
Toedittheattributesofanode,selectthecheckboxofthenodeandclickEdit.
Todeleteallnodes,clickReset.
4.
ClickSubmit.
ModifyingaSourceDataSchemaDonotmodifythesourcedataschemaofastreamifthestreamhasdumptasks.
Step1UsetheaccounttologintotheDISconsole.
Step2Clickintheupperleftcornerofthepagetoselectaregion.
Step3Inthenavigationtreeontheleft,chooseStreamManagementStreamManagement.
1.
Clickastreamname.
TheStreamManagementpageisdisplayed.
2.
ClickViewExistingSourceDataSchemanexttoSourceDataType.
3.
IntheSourceDataSchematextbox,clicktomodifythesourcedataschema.
Figure4-6ModifyingthesourcedataschemaIfastreamhasdumptasks,modifyingthesourcedataschemaofthestreamwillcausesomedataunabletobesuccessfullydumped.
DataIngestionServiceUserGuide4ManagingDISStreamsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
234.
Afterthemodificationiscomplete,clickSubmit.
ClickCanceltogiveupmodifyingthesourcedataschema.
----End4.
5ManagingStreamTagsAtagisanidentifierofastream.
Addingtagstostreamscanhelpyouidentifyandmanageyourstreamresources.
Youcanaddamaximumof10tagstoastreamwhencreatingthestreamoraddthemonthedetailspageofthecreatedstream.
Atagconsistsofatagkeyandatagvalue.
Table4-2describestherulesfornamingthetagkeyandvalue.
Table4-2NamingrulesforatagkeyandvalueParameterRuleExampleKeyAtagkeycannotbeleftblank.
Atagkeymustbeuniqueforastream.
Atagkeycontainsamaximumof36characters.
Atagvaluecannotcontainspecialcharactersorstartorendwithaspace.
OrganizationTagvalueAtagvaluecontainsamaximumof43characters.
Atagvaluecannotcontainspecialcharactersorstartorendwithaspace.
Thisparametercanbeleftblank.
ApacheAddingaTagtoaStreamYoucanaddatagtoastreamontheBuyStreampage.
1.
Logintothemanagementconsole.
2.
ChooseEnterpriseIntelligence>DataIngestionService.
3.
OntheDISmanagementconsole,clickBuyStream.
DataIngestionServiceUserGuide4ManagingDISStreamsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
244.
OntheAdvancedSettingstabpage,selectConfigure.
Enterthekeyandvalueofatagtobeadded.
Youcanaddamaximumof10tagstothestreamanduseintersectionsoftagstosearchforthetargetstream.
Youcanalsoaddtagstoexistingstreams.
Fordetails,seeManagingTags.
SearchingforaTargetStreamYoucansearchforatargetstreambytagontheStreamManagementpage.
1.
Logintothemanagementconsole.
2.
ChooseEIEnterpriseIntelligent>DataIngestionService.
3.
Inthenavigationtree,chooseIngestionManagement>StreamManagement.
Intheupperrightcornerofthepage,clickSearchbyTag.
4.
Enterthetagkeyandvalueofthestreamyouaresearchingfor.
Youcanselectatagkeyortagvaluefromitsdrop-downlist.
Whenthetagkeyortagvalueisexactlymatched,thesystemcanautomaticallylocatethetargetstream.
Ifyouentermultipletags,theirintersectionsareusedtosearchforthestream.
5.
ClickSearch.
Thesystemsearchesforthetargetstreambytagkeyorvalue.
ManagingTagsYoucanadd,delete,modify,andviewtagsontheTagstabpageofastream.
1.
Logintothemanagementconsole.
2.
ChooseEIEnterpriseIntelligent>DataIngestionService.
3.
Inthenavigationtree,chooseIngestionManagement>StreamManagement.
Clickastreamtowhichthetagstobemanagedbelongto.
Thestreamdetailspageisdisplayed.
4.
ClicktheTagstabandadd,deleted,modify,andviewtags.
–ViewOntheTagstabpage,youcanviewdetailsabouttagsofthestream,includingthenumberoftagsandthekeyandvalueofeachtag.
–AddClickAddTagintheupperleftcorner.
InthedisplayedAddTagdialogbox,enterthekeyandvalueofthetagtobeadded,andclickOK.
–ModifyIntheOperationcolumnofatag,clickEdit.
InthedisplayedEditTagpage,enteranewtagkeyandvalueandclickOK.
–DeleteIntheOperationcolumnofthetag,clickDelete.
Afterconfirmation,clickOKonthedisplayedDeleteTagpage.
DataIngestionServiceUserGuide4ManagingDISStreamsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
254.
6ScalingUp/DownaStreamAfterastreamiscreatedsuccessfully,youcanscaleupordownastreambyaddingorreducingpartitionstomeetcapacitychangerequirements.
Streamscanbescaledautomaticallyormanually.
FunctionScalingrulesWhenscalingupastream,ensurethatthenumberoftargetpartitionsisgreaterthanthenumberofthecurrentpartitionsbutisnotgreaterthanthetotalnumberoftheremainingquotasandcurrentpartitions.
Whenscalingupastream,restoretheDELETEDandEXPIREDpartitionstotheACTIVEpartitionstomakethepartitionsaccessible.
Ifthestreamcapacityisstillinsufficient,thesystemwillcreatenewpartitions.
Whenscalingdownastream,ensurethatthenumberoftargetpartitionsissmallerthanthenumberofcurrentpartitionsbutisnotsmallerthan1.
Thepartitionsthathavebeenscaleddownwillnotbechargedoroccupythequota.
DataonthosepartitionsisreadablebutisnotwritableduringDataRetentionconfiguredinCreatingaDISStream.
OnceDataRetentionisexpired,thedatawillbecomeinaccessible.
Amaximumof10automaticscale-up,10manualscale-up,and1scale-downoperationscanbeperformedforeachstreamwithinonehour.
Theintervalbetweenperformingautomaticscale-upandscale-downmustbegreaterthan1minute.
Automaticscale-downcannotbetriggeredwithin2minutesafterautomaticormanualscale-upisperformed,andmeanwhileautomaticscale-upcannotbetriggeredwithin2minutesafterautomaticormanualscale-downisperformed.
PrecautionsWhenuploadingdata,youneedtosetPartitionKeybasedonyourspecificdemandssothatDIScanautomaticallyuploaddatatoallpartitionsaccordingtoyourrequirements.
Whendownloadingdata,youneedtoperiodicallyusethedescriptStreamAPItocheckthepartitionquantitychangesothatDIScandownloaddatafromallpartitions.
ManuallyScalingUp/DownaStreamStep1UsetheaccounttologintotheDISconsole.
Step2Clickintheupperleftcornerofthepageandselectaregionandproject.
Step3Performeitherofthefollowingoperationstomanuallyscaleupordownastream:Inthenavigationtreeontheleft,choose.
a.
OntheStreamManagementpage,clickthenameofthestreamtobescaled.
DataIngestionServiceUserGuide4ManagingDISStreamsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
26b.
Intheupperrightcornerofthedisplayedpage,clickScaleUp/Down.
TheScaleUp/DownStreamdialogboxisdisplayed.
c.
ChangethenumberoftargetpartitionsandclickYes.
Inthenavigationtreeontheleft,choose.
a.
IntheOperationcolumnofthestreamtobescaled,clickMoreandchooseScaleUp/Downfromthedrop-downlist.
b.
TheScaleUp/DownPartitiondialogboxisdisplayed.
c.
ChangethenumberoftargetpartitionsandclickYes.
----EndAutomaticallyScalingUp/DownaStreamStep1UsetheaccounttologintotheDISconsole.
Step2Clickintheupperleftcornerofthepageandselectaregionandproject.
Step3Performthefollowingstepstoautomaticallyscaleupordownastream:Inthenavigationtreeontheleft,choose.
1.
OntheStreamManagementpage,clickthenameofthestreamtobescaled.
2.
ClickEditnexttoAuto-Scaling.
3.
InthedisplayedEditAuto-ScalingParameterdialogbox,enableAuto-Scaling.
Figure4-7EditAuto-ScalingParameter4.
SettheupperandlowerthresholdsforautomaticscalingandclickOK.
----EndViewingScale-Up/DownLogsStep1UsetheaccounttologintotheDISconsole.
DataIngestionServiceUserGuide4ManagingDISStreamsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
27Step2Clickintheupperleftcornerofthepageandselectaregionandproject.
Step3Inthenavigationtreeontheleft,choose.
Step4Clickthenameofastreamthatyouwanttoview.
TheStreamManagementpageisdisplayed.
Step5Onthedisplayedpage,clicktheScale-Up/DownLogstab.
Viewthescalingdetailsofthestream.
----End4.
7DeletingaStreamNOTICEAdeletedstreamwillnotbechargedandcannotberestored.
Exercisecautionwhenperformingthisoperation.
Step1UsetheaccounttologintotheDISconsole.
Step2Clickintheupperleftcornerofthepageandselectaregionandproject.
Step3Inthenavigationtreeontheleft,choose.
Step4ClickDeleteintheOperationcolumnofthestreamthatyouwanttodelete.
TheDeleteStreamdialogboxisdisplayed.
Step5ClickOKtodeletetheselectedstream.
----EndDataIngestionServiceUserGuide4ManagingDISStreamsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
285UsingDIS5.
1CheckingandConfiguringDNSInformationBydefault,anElasticCloudServer(ECS)isconfiguredwithtwoexternaldomainnamesystem(DNS)servers.
#GeneratedbyNetwordManagersearchopenstacklocalnameserver114.
114.
114.
114nameserver114.
114.
115.
115IftheECSyouwillusedoesnothaveanelasticIPaddress(EIP)oryoudonotwanttouseanEIPtotransmitthetrafficloadgeneratedduringtheuseofAEI_Register.
shandfisclientprograms,addaDNSserverinconfigurationfile/etc/resolv.
conf.
XXX.
XXX.
XXX.
XXXistheIPaddressoftheDNSserver.
#GeneratedbyNetwordManagersearchopenstacklocalnameserverXXX.
XXX.
XXX.
XXXnameserver114.
114.
114.
114nameserver114.
114.
115.
115ThenewDNSserveraddressmustbeplacedabovealltheexistingDNSserveraddresses.
TheDNSconfigurationtakeseffectimmediatelyaftermodificationstothe/etc/resolv.
conffilearesaved.
Modificationstothe/etc/resolv.
conffilebecomeinvalidaftertheECSrestarts.
Inthiscase,youneedtoreconfigurethefile.
IfyoudonotwanttoreconfigurethefileeachtimeafterrestartingtheECS,performthefollowingstepstomodifythesubnetinformationoftheVirtualPrivateCloud(VPC)towhichtheECSbelongsandaddDNSserveraddressestothesubnettowhichtheECSbelongs.
ProcedureStep1LogintotheECSconsole.
DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
29Step2InthenavigationtreeoftheECSConsole,chooseElasticCloudServer.
Step3OntheECSdetailspage,clicktheNICtab.
ExpanddetailsofachosenNICandviewthenameofthesubnettowhichtheECSbelongs.
Step4OntheECSdetailspage,clicktheVPCnameorID.
TheNetworkConsoleisthenlaunched.
Step5IntherightpaneoftheNetworkConsole,clicktheVPCnameorID.
ApagewithdetailsoftheVPCisthendisplayed.
Step6OntheVPCdetailspage,clicktheSubnettab.
OntheSubnettabpage,clickthenameorIDofthesubnettowhichtheECSbelongs.
Apagewithdetailsofthesubnetisthendisplayed.
Step7Onthesubnetdetailspage,clickModifynexttoDNSServerAddress.
Step8EntertheIPaddressoftheDNSserverinDNSServerAddress1andclickOK.
Step9RestarttheECSandcheckthatthe/etc/resolv.
conffilecontainsthenewDNSserveraddresswhichisplacedinthefrontofotherDNSserveraddresses.
#GeneratedbyNetwordManagersearchopenstacklocalnameserverXXX.
XXX.
XXX.
XXXnameserver114.
114.
115.
115ModifyingthesubnetinformationofaVPCaffectsallECSsinthissubnet.
----End5.
2UploadingDatabyUsingAgent5.
2.
1DISAgentOverviewDISAgentisaclient-sideprogramprovidedbytheDataIngestionService(DIS)tofulfillthefollowingtasks:Monitortextfilescontinuously,collectincrementaldatainrealtime,parsethedatabydelimiteranduploadittoDISstreams.
ThesourcedatatypeofastreamcanbeBLOB,JSON,andCSV.
Figure5-1depictstheprocessforinstallingtheDISagent.
DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
30Figure5-1Installationflowchart5.
2.
2PreparingforInstallingDISAgentCheckingDependenciesStep1Checktheservertype.
Linuxx86-64(64-bit)server,forexample,EulerOS,Ubuntu,Debian,CentOS,orOpenSUSEWindows7oralaterversionStep2EnsurethatJava1.
8.
0orlaterhasbeeninstalled.
TodownloadJRE,gotohttps://www.
java.
com/en/download/manual.
jsp.
IfJava1.
8.
0orlaterisnotinstalled,performthefollowingstepstoinstallitonaLinuxserver:1.
Astherootuser,runthefollowingcommandtonavigatetothe/optdirectory:cd/optDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
312.
RunthefollowingcommandtocreatetheJDKinstallationdirectoryjre:mkdir-pjre3.
Runthefollowingcommandtoassignpermission755totheJDKinstallationdirectory:chmod-R777jre/4.
DownloadtheJavaruntimeenvironment(JRE)installationpackagefromthefollowingwebsite:tar-zxvf'JREinstallationpackagename'.
tar.
gz5.
Modifythe/etc/profilefile.
a.
Runthefollowingcommandtoopenthe/etc/profilefile:vim/etc/profileb.
AddthefollowingJDKinstallationdirectoryinformationtotheconfigurationoptionJAVA_HOMEinthe/etc/profilefile:exportJAVA_HOME=pathtothejrefolderexportPATH=$PATH:$JAVA_HOME/binexportCLASSPATH=.
:$JAVA_HOME/lib/rt.
jar:$JAVA_HOME/lib/extc.
Runthefollowingcommandtosavethemodificationandexit::wq6.
RunthefollowingcommandtovalidatetheJDKconfiguration:source/etc/profile----EndCheckingDISStreamsStep1UsetheaccounttologintotheDISconsole.
Step2ClickintheupperleftcornerandselectRegionandProject.
Step3Inthenavigationtree,chooseStreamManagement.
Step4EnsurethatatleastoneDISstreamisintheRunningstateandreadytoreceiveincomingdata.
----EndCheckingAuthenticationInformationAK/SKfileAccessKeyID/SecretAccessKey(AK/SK)filesarecreatedbytheIdentityandAccessManagement(IAM)servicetoauthenticatecallstoapplicationprogramminginterfaces(APIs)onthecloud.
ToobtainAK/SK,chooseMyCredentials>AccessKeys.
ProjectIDAprojectisagroupoftenantresources.
Atenantcanhavemultipleprojects,oneforeachregion.
EachregionhasauniqueProjectID.
ToviewtheprojectIDsofdifferentregions,choose.
DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
32ObtainingDISAgentPackageObtainthedis-agent-X.
X.
X.
zippackagefromhttps://dis-publish.
obs-website.
cn-north-1.
myhwclouds.
com/.
5.
2.
3InstallingDISAgentPrerequisitesPuTTYhasbeeninstalled.
InstallingDISAgentonaLinuxServerStep1StartPuTTYandlogintotheserveronwhichlogsreside,thatis,theserverthatexperiencesadependencycheckinsectionCheckingDependencies.
Step2UploadtheDISAgentpackagedis-agent-X.
X.
X.
zipobtainedinsectionObtainingDISAgentPackagetothe/optdirectory.
Step3Runthefollowingcommandtodecompressthedis-agent-X.
X.
XdirectoryfromtheDISAgentpackagedis-agent-X.
X.
X.
zip:unzipdis-agent-X.
X.
X.
zipStep4Runthefollowingcommandtonavigatetothedis-agent-X.
X.
Xdirectory:cddis-agent-X.
X.
X----EndInstallingDISAgentonaWindowsServerStep1Savethedis-agent-X.
X.
X.
zippackageobtainedinObtainingDISAgentPackagetothelocalhost.
Step2Decompressthedis-agent-X.
X.
X.
zippackagetothecurrentdirectory.
----End5.
2.
4ConfiguringDISAgentTheDISAgentconfigurationfileisintheYAMLformat.
Configurationparametersandvaluesmustbeseparatedbycolon(:)andspace.
Table5-1describestheconfigurationparametersinthefile.
Aftertheconfigurationiscomplete,deleteunnecessaryexamplesfromflowsinagent.
ymloruse#tocommentoutthem.
Forexample,ifonlyoneDISStreamisconfigured,deleteorcommentoutthefollowingCustomFileStreamandotherDISStreammodules.
ConfiguringDISAgentonaLinuxServerStep1StartPuTTYandlogintotheLinuxserveronwhichtheDISAgentisinstalled.
Step2Runthecd/opt/dis-agent-X.
X.
X/commandtoopenthedis-agent-X.
X.
Xdirectory.
DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
33Step3Runthevimconf/agent.
ymlcommandtoopentheDISAgentconfigurationfileagent.
yml.
Modifyparametervaluesinthefiletomeetspecificrequirements.
Table5-1describestheconfigurationparametersinthefile.
Table5-1Parametersintheagent.
ymlfileParameterMandatoryDescriptionDefaultValueregionYesRegionwhereDISisdeployed.
NOTEFordetailsabouthowtoobtaintheregionwhereDISisdeployed,seeRegionsandEndpoints.
-akYesUser'sAK.
FordetailsabouthowtoobtainanAK,seeCheckingAuthenticationInformation.
-skYesUser'sSK.
FordetailsabouthowtoobtainanSK,seeCheckingAuthenticationInformation.
-projectIdYesProjectIDspecifictoyourregion.
FordetailsabouthowtoobtainaprojectID,seeCheckingAuthenticationInformation.
-endpointYesDISgatewayaddress.
Format:https://DISterminalnodeNOTEFordetailsabouthowtoobtaintheDISterminalnode,seeRegionsandEndpoints.
-DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
34ParameterMandatoryDescriptionDefaultValuebody.
serialize.
typeNoFormatoftheDISdatapackagetobeuploaded(non-originaldataformat).
json:TheDISdatapacketisencapsulatedintheformatofJSON.
protobuf:TheDISdatapacketisencapsulatedinthebinaryformat.
Afterbeingencapsulated,thevolumeofthedatapacketisreducedby1/3.
Thisformatisrecommendedwhenamassiveamountofdataisgenerated.
jsonbody.
compress.
enabledNoSpecifieswhethertoenabledatacompression.
falsebody.
compress.
typeNoDatacompressionformatselectedwhencompressionisenabled.
Currently,thefollowingcompressionformatsaresupported:lz4:acompressionalgorithmwithafastcompressionspeedandhighcompressionefficiencyzstd:anewlosslesscompressionalgorithmwithafastcompressionspeedandhighcompressionratiolz4PROXY_HOSTNoProxyIPaddress.
Thisparameterismandatorywhenrequestsaresentthroughtheproxyserver.
-PROXY_PORTNoProxyport.
80PROXY_PROTOCOLNoProxyprotocol.
httpandhttpsaresupported.
httpPROXY_USERNAMENoProxyusername.
-DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
35ParameterMandatoryDescriptionDefaultValuePROXY_PASSWORDNoProxypassword.
-[flows]The[flows]sectionpresentsinformationaboutthefilesthatwillbeuploadedtoDIS.
Thefollowinguploadmodeissupported:DISStream:DISAgentmonitorstextfilescontinuously,collectsincrementaldatainrealtime,parsesthedatabydelimiter,anduploadsittoDISstreams(sourcedatatype:BLOB,JSON,andCSV).
Table5-2describesconfigurationparameters.
Theagent.
ymlfileprovidesexampleparametersettings.
Table5-2DISStreamconfigurationparametersParameterMandatoryDescriptionDefaultValueDISStreamYesNameoftheDISstream.
ParsesthefilecontentmatchingfilePatternbydelimiteranduploadsthefiletothestream.
-filePatternYesFilemonitoringpath.
Filesinonlyonedirectorycanbemonitored.
Directoriescannotbemonitoredrecursively.
Tomonitormultipledirectories,configuremultipleDISstreamsinflows.
Thefilenamescanbematchedbyasterisk(*)/tmp/*.
log:Matchesallfileswhosenamesendwith.
loginthe/tmpdirectory.
/tmp/access-*.
log:Matchesallfileswhosenamesstartwithaccess-andendwith.
loginthe/tmpdirectory.
InWindows,theexamplepathisD:\logs\*.
log.
-DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
36ParameterMandatoryDescriptionDefaultValuedirectoryRecursionEnabledNoSpecifieswhethertosearchforasubdirectory.
Possiblevalues:false:Nottosearchforsubdirectoriesrecursivelyandmatchonlyfilesintherootdirectory.
true:Searchforallsubdirectoriesrecursively.
Forexample,iffilePatternissetto/tmp/*.
log,/tmp/one.
log,/tmp/child/two.
log,and/tmp/child/child/three.
logcanbematched.
falseinitialPositionNoInitialpositionfromwhichthefilestartedtobemonitored.
Possiblevalues:END_OF_FILE:Aftermonitoringstarts,thesystemdoesnotparsethefilesthatmatchfilePattern.
Instead,thenewlyaddedfileorfilecontentwillbeparsedbydelimiteranduploadedtoDIS.
START_OF_FILE:AllthefilesthatmatchfilePatternwillbeparsedbydelimiteranduploadedtoDISbasedonthefilemodificationtime(fromtheearliestmodifiedtothelatestmodified).
START_OF_FILEmaxBufferAgeMillisNoThemaximumnumberofmillisecondsthatmustelapsebeforedatacanbeuploadedtotheDIS.
Unit:msIfthebufferisfullwithdatawaitingtobeuploaded,datawillbeimmediatelyuploadedtotheDIS.
Iftherecordqueueisnotfull,fileswillbeuploadedtoDISonlyafterthespecifiedperiodoftimeisreached.
5000maxBufferSizeRecordsNoThemaximumnumberofrecordsforwhichtheagentbuffersdatabeforesendingittoDIS.
Ifthenumberofrecordsinaqueuereachesthevalue,thedatawillbeuploadedtoDISimmediately.
500DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
37ParameterMandatoryDescriptionDefaultValuepartitionKeyOptionNoMethodforgeneratingthepartitionkey.
Eachrecordcarriesapartitionkey.
Recordswiththesamepartitionkeyareallocatedtothesamepartition.
Possiblevalues:RANDOM_INT:Thepartitionkeyisarandomnumericstring.
Recordswithsuchakeyareevenlydistributedtoeachpartition.
FILE_NAME:Thepartitionkeyisafilenamestring.
Recordswithsuchakeyisdistributedtoaspecificpartition.
FILE_NAME,RANDOM_INT:Thepartitionkeyisacombinationofafilenamestringandarandomnumericstring,whichareseparatedbycomma(,).
Recordswithsuchakeycarriesfilenamesandareevenlydistributedtoallpartitions.
RANDOM_INTrecordDelimiterNoDelimiterusedtoseparaterecords.
Valuerange:anycharacterthatisenclosedindoublequotationmarks.
Thevaluecannotbeempty.
Thatis,thisparametercannotbesetto"".
NOTEIfthevalueisaspecialcharacter,useabackslash(\)toescape.
Forexample,ifthevalueisaquotationmark("),setthisparameterto\".
Ifthevalueisabackslash(\),setthisparameterto\\.
Ifthevalueisacontrolcharacter,forexample,STX,setthisparameterto\u0002.
"\n"isRemainRecordDelimiterNoSpecifieswhetheradelimiteriscontainedinrecordstobeuploaded.
Possiblevalues:true:Thedelimiteriscontainedinrecordstobeuploaded.
false:Thedelimiterisnotcontainedinrecordstobeuploaded.
falseDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
38ParameterMandatoryDescriptionDefaultValueisFileAppendableNoSpecifieswhetherthefilecontainsadditionalcontent.
Possiblevalues:true:Thefilemaycontainadditionalcontent.
Agentcontinuouslymonitorsfiles.
Ifcontentisaddedtoafile,AgentparsesthefilebyrecordDelimiteranduploadsrecords.
Inthiscase,ensurethatthefileendswithrecordDelimiter.
Otherwise,AgentconsidersthatthecontenthasnotbeenaddedtothefileandwaitsforrecordDelimitertobewritten.
false:Thefilewillnotcontainadditionalcontent.
IfthelastrowofthefiledoesnotendwithrecordDelimiter,Agentstilluploadsthefileasthelastrecord.
Aftertheuploadiscomplete,AgentwilldeleteorrenamethefilebasedontheconfigurationofdeletePolicyandfileSuffix.
truemaxFileCheckingMillisNoMaximumtimeforcheckingfilechanges.
Ifthefilesize,modificationtime,andfileIDdonotchangewithinthisperiodoftime,acompletefileisgeneratedandstartstobeuploaded.
Setthisparameterbasedontheactualfilechangefrequencytopreventanincompletefilefrombeinguploaded.
Ifthefileischangedafterbeinguploaded,itwillbefullyuploadedagain.
Unit:msNOTEThisparameterisavailableonlywhenisFileAppendableissettofalse.
5000deletePolicyNoPolicyfordeletingafileafterthefilecontentisuploaded.
Possiblevalues:never:Thefilewillnotbedeletedafterthefilecontentisuploaded.
immediate:Thefilewillbedeletedafterthefilecontentisuploaded.
NOTEThisparameterisavailableonlywhenisFileAppendableissettofalse.
neverDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
39ParameterMandatoryDescriptionDefaultValuefileSuffixNoSuffixofthefilenamethatisaddedafterthefilecontentisuploaded.
Iftheoriginalfilenameisx.
txtandfileSuffixissetto.
COMPLETED,thenameoftheuploadedfileisx.
txt.
COMPLETED.
NOTEThisparameterisavailableonlywhenisFileAppendableissettofalseanddeletePolicyissettonever.
.
COMPLETEDsendingThreadSizeNoThenumberofsenderthreads.
Bydefault,thereisonlyonesenderthread.
NOTICEIfmultiplethreadsareused,thefollowingproblemsmayoccur:Datamaynotbesentinorder.
Somedataislostaftertheprogramstopsabnormallyandrestarts.
1fileEncodingNoFileencodingformat.
Possiblevalues:UTF8,GBK,GB2312,andISO-8859-1.
UTF8resultLogLevelNoLevelofthecallingresultloggeneratedeachtimewhentheDISdatasendingAPIiscalled.
OFF:EachAPIcallingresultisnotlogged.
INFO:EachAPIcallingresultisloggedattheINFOlevel.
WARN:EachAPIcallingresultisloggedattheWARNlevel.
ERROR:EachAPIcallingresultisloggedattheERRORlevel.
INFO----EndConfiguringDISAgentonaWindowsServerStep1Useafilemanagertoopenthedirectory(forexample,C:\dis-agent-X.
X.
X)wheretheinstallationpackageisdecompressed.
Step2Opentheagent.
ymlfileusinganeditorandmodifyparametervaluesinthefiletomeetspecificrequirements.
Table5-1describestheconfigurationparametersinthefile.
Theagent.
ymlfileisintheLinuxformat.
YouareadvisedtousetheNotepad++tooltoeditthefile.
----EndDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
405.
2.
5StartingDISAgentStartingDISAgentonaLinuxServerStep1StartPuTTYandlogintotheserveronwhichlogsreside.
Step2RunthefollowingcommandtonavigatetotheDISAgentinstallationdirectory:cd/opt/dis-agent-x.
x.
x/x.
x.
xindicatestheversionnumber.
Step3RunthefollowingscripttostarttheDISAgent:bashbin/start-dis-agent.
shIfmultipleDISAgentprocessesneedtobestarted,youneedtouse-ctospecifytheconfigurationfileand-ntospecifythenameofthenewAgentprocess.
bashbin/start-dis-agent.
sh-cconfig/anotherAgent.
yml-nanotherAgentNOTICERunthebashscripttostarttheDISAgent.
Runninganotherscript,suchasshor.
/mayfailtostarttheDISAgent.
Ifinformationsimilartothefollowingappears,theDISAgenthasstartedsuccessfully:SuccesstostartDISAgent[xxxxx].
IfnoJavavariablescanbefoundaftertheDISAgentstarts,runthefollowingcommandtorestarttheDISAgent:source/etc/profile----EndStartingDISAgentonaWindowsServerStep1NavigatetothebindirectoryofDISAgent,forexample,C:\dis-agent-X.
X.
X\bin.
Step2Double-clickstart-dis-agent.
battostarttheDISAgent.
Ifinformationsimilartothefollowingappearsinlogfiles,theDISAgenthasstartedsuccessfully:[INFO](main)com.
bigdata.
dis.
agent.
AgentAgent:StartupcompletedinXXXms.
----End5.
2.
6TestingDISAgentTestingDISAgentonaLinuxServerStep1StartPuTTYandlogintotheserveronwhichlogsreside.
Step2RunthefollowingcommandtonavigatetothedirectoryinwhichtheDISAgentstoreslogs:DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
41cd/opt/dis-agent-X.
X.
X/logsStep3Runthefollowingcommandtoviewlogs:tail-100fdis-agent.
logIfinformationsimilartothefollowingappears,theDISAgentisrunningproperly.
Agent:StartupcompletedinxxmsIfanyofthefollowinginformationappears,theDISAgentisnotrunningproperlyandyouneedtotroubleshoottheDISAgent:–HttpClientErrorException:400BadRequestPossiblecause:ThevalueofDISStreamorprojectIdinConfiguringDISAgentisincorrect.
Solution:StoptheDISAgentprocessandcorrecttheparametervalues.
–HttpClientErrorException:403ForbiddenPossiblecause:TheDISgatewayblackliststheIPaddressoftheLinuxserveronwhichtheDISAgentisinstalledandconsequentlytherequesttoviewlogsisblocked.
Usually,theLinuxserverIPaddressisblacklistedbecauseaDISAPIisrepeatedlycalledusingincorrectconfiguration.
Solution:StoptheDISAgentprocess,modifytheagent.
ymlfile(seeConfiguringDISAgent).
RestarttheDISAgentprocess30minutesafteritstops.
–UnknownHttpStatusCodeException:Unknownstatuscode[441]Possiblecause:AK/SKconfigurationisincorrect.
Solution:StoptheDISAgentprocessandcorrecttheAK/SKconfiguration.
–ConnectTimeoutException:ConnecttoDOMAIN[DOMAIN/IP]failed:connecttimedoutPossiblecause:TheconnectionbetweentheLinuxserveronwhichtheDISAgentisinstalledandtheDISgatewaytimesout.
Solution:EnsurethattheLinuxserveronwhichtheDISAgentisinstalledcanconnecttotheInternet.
Step4CheckwhethertheDISAgentcanuploadlogs.
Ifthemonitoringdirectoryconfiguredinagent.
ymlcontainsmatchedfiles,logswillrecordtheinformationsimilartothefollowing,indicatingthat[N1records(B1bytes)/N2files(B2bytes)]areparsedand[N3records/N4files]aresuccessfullyuploaded.
Agent:Progress:[N1records(B1bytes)/N2files(B2bytes)]parsed,and[N3records/N4files]sentsuccessfullytodestinations.
Uptime:30146msIfnofilesmatchthefilePattern,runthefollowingcommandtogeneratelogfilesrequiredfortestingloguploading:echo"`date`Helloworld.
">>/tmp/test.
logStep5LogintotheDISconsoleandviewmonitoringdataoftheDISstreamspecifiedbytheDISStreamorCustomFileStreamparameterinConfiguringDISAgent.
IfthestreammonitoringdatashowsthatdatahasbeenuploadedtotheDISstream,theDISAgenthasbeensuccessfullyinstalled.
----EndDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
42TestingDISAgentonaWindowsServerStep1Useafilemanagertonavigatetothelogsdirectory.
Step2Openthedis-agent.
logfileusinganeditor.
Viewlogsinthefile.
Ifinformationsimilartothefollowingappears,theDISAgentisrunningproperly.
Agent:StartupcompletedinxxmsIfanyofthefollowinginformationappears,theDISAgentisnotrunningproperlyandyouneedtotroubleshoottheDISAgent:–HttpClientErrorException:400BadRequestPossiblecause:ThevalueofDISStreamorprojectIdinConfiguringDISAgentisincorrect.
Solution:StoptheDISAgentprocessandcorrecttheparametervalues.
–HttpClientErrorException:403ForbiddenPossiblecause:TheDISgatewayblackliststheIPaddressoftheLinuxserveronwhichtheDISAgentisinstalledandconsequentlytherequesttoviewlogsisblocked.
Usually,theLinuxserverIPaddressisblacklistedbecauseaDISAPIisrepeatedlycalledusingincorrectconfiguration.
Solution:StoptheDISAgentprocess,modifytheagent.
ymlfile(seeConfiguringDISAgent).
RestarttheDISAgentprocess30minutesafteritstops.
–UnknownHttpStatusCodeException:Unknownstatuscode[441]Possiblecause:AK/SKconfigurationisincorrect.
Solution:StoptheDISAgentprocessandcorrecttheAK/SKconfiguration.
–ConnectTimeoutException:ConnecttoDOMAIN[DOMAIN/IP]failed:connecttimedoutPossiblecause:TheconnectionbetweentheLinuxserveronwhichtheDISAgentisinstalledandtheDISgatewaytimesout.
Solution:EnsurethattheLinuxserveronwhichtheDISAgentisinstalledcanconnecttotheInternet.
Step3CheckwhethertheDISAgentcanuploadlogs.
Ifthemonitoringdirectoryconfiguredinagent.
ymlcontainsmatchedfiles,logswillrecordtheinformationsimilartothefollowing,indicatingthat[N1records(B1bytes)/N2files(B2bytes)]areparsedand[N3records/N4files]aresuccessfullyuploaded.
Agent:Progress:[N1records(B1bytes)/N2files(B2bytes)]parsed,and[N3records/N4files]sentsuccessfullytodestinations.
Uptime:30146msIfnofilesmatchthefilePattern,runthefollowingcommandtogeneratelogfilesrequiredfortestingloguploading:echo%date%time%Helloworld.
>>C:\test.
logStep4LogintotheDISconsoleandviewmonitoringdataoftheDISstreamspecifiedbytheDISStreamorCustomFileStreamparameterinConfiguringDISAgent.
IfthestreammonitoringdatashowsthatdatahasbeenuploadedtotheDISstream,theDISAgenthasbeensuccessfullyinstalled.
----EndDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
435.
2.
7StoppingDISAgentStoppingDISAgentonaLinuxServerStep1StartPuTTYandlogintotheserveronwhichlogsreside.
Step2RunthefollowingcommandtonavigatetotheDISAgentinstallationdirectory:cd/opt/dis-agent-X.
X.
X/Step3RunthefollowingscripttostoptheDISAgent:bashbin/stop-dis-agent.
shNOTICERunthebashscripttostoptheDISAgent.
Runninganotherscript,suchasshor.
/mayfailtostoptheDISAgent.
Ifinformationsimilartothefollowingappears,theDISAgentisstopping.
xxxxxindicatestheprocessID.
StoppingAgent[xxxxx].
.
.
.
.
Ifinformationsimilartothefollowingappears,theDISAgenthasstopped:StoppingAgent[xxxxx]Successfully.
ToforciblystoptheDISAgentprocess,performthefollowingsteps:1.
RunthecommandtoobtaintheDISAgentprocessID(PID):ps-ef|grepdis-agent|grep-vgrepThesecondfieldinthecommandoutputindicatesthePID.
2.
RunthefollowingcommandtoforciblystoptheDISAgentprocess:kill-9PID----EndStoppingDISAgentonaWindowsServerStep1PressCtrl+ContheCMDterminal.
Thefollowinginformationappears:[INFO](AgentSTOPPING)com.
bigdata.
dis.
agent.
AgentAgent:Shuttingdown.
.
.
Step2Whenthefollowinginformationappears,itindicatesthattheDISAgenthasstopped.
EnterYandthenpressEntertoexit.
Terminatebatchjob(Y/N)----End5.
3UsingDISFlumePlugintoUploadandDownloadDataDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
445.
3.
1OverviewADISFlumePluginisaFlumepluginprovidedbyDISandconsistsofDISSourceandDISSink.
DISSourceisusedtodownloaddatafromDIStoFlumeChannel;DISSinkisusedtouploaddatafromFlumeChanneltoDIS.
Figure5-2depictstheprocessforinstallingaDISFlumePlugin.
Figure5-2ProcessforinstallingaDISFlumePlugin5.
3.
2PreparingforInstallingaDISFlumePluginCheckingDependenciesStep1CheckthatFlumeisrunningproperly.
Step2IntheFlumeinstallationdirectory,runthefollowingcommandtocheckthattheFlumeversionis1.
4.
0orlater:$bin/flume-ngversion|grepFlumeStep3RunthefollowingcommandtocheckthattheJavaruntimeenvironment(JRE)versionis1.
8.
0orlater:java-version----EndDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
45CheckingDISStreamsStep1UsetheaccounttologintotheDISconsole.
Step2ClickintheupperleftcornerandselectRegionandProject.
Step3Inthenavigationtree,chooseStreamManagement.
Step4EnsurethatatleastoneDISstreamisintheRunningstateandreadytoreceiveincomingdata.
----EndCheckingAuthenticationInformationAK/SKfileAccessKeyID/SecretAccessKey(AK/SK)filesarecreatedbytheIdentityandAccessManagement(IAM)servicetoauthenticatecallstoapplicationprogramminginterfaces(APIs)onthepubliccloud.
ToobtainAK/SK,choose.
ProjectIDAprojectisagroupoftenantresources.
Atenantcanhavemultipleprojects,oneforeachregion.
EachregionhasauniqueProjectID.
ToviewtheprojectIDsofdifferentregions,chooseMyCredential>Projects.
ObtainingaDISFlumePluginPackageObtainthedis-flume-Plugin-X.
X.
X.
zippackagefromhttps://dis-publish.
obs-website.
cn-north-1.
myhwclouds.
com/.
5.
3.
3InstallingaDISFlumePluginPrerequisitesPuTTYhasbeeninstalled.
ProcedureStep1LogintotheremoteFlumeserverusingPuTTYoranotherremotelogintool.
Step2RunthefollowingcommandtonavigatetotheFlumeinstallationdirectory:cd${FLUME_HOME}Step3Uploadthedis-flume-plugin-X.
X.
X.
zipinstallationpackagetothisdirectory.
Step4Decompresstheinstallationpackage.
unzipdis-flume-plugin-X.
X.
X.
zipStep5Runthefollowingcommandtonavigatetothedirectorywherethepackageisdecompressed.
cddis-flume-pluginStep6Runthefollowingcommandtoexecutetheinstallationprogram.
DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
46bashinstall.
shIfthefollowinginformationappears,theDISFlumePluginhasbeensuccessfullyinstalledinthe${FLUME_HOME}/plugin.
d/dis-flume-plugindirectory:Installdis-flume-pluginsuccessfully.
----End5.
3.
4ConfiguringtheDISFlumePluginADISFlumePluginconsistsofDISSourceandDISSink.
Thedis-flume-plugin.
conf.
templatefileintheinstallationpackageliststheconfigurationmethods.
ThissectiondescribestheconfigurationitemsofDISSourceandDISSink.
Dis-flume-plugin.
conf.
templateisaconfigurationsampleforDISplug-inandisnotaconfigurationfilethatwillbeaccessedwhenFlumeisrun.
FlumeprovidesaconfigurationsamplefileinFLUME_HOME/conf/flume-conf.
properties.
template,whereFLUME_HOMEistheinstallationpathofFlume.
Youcanmodifytheconfigurationfilebasedonsiterequirements.
ConfiguringDISSourceTable5-3DISSourceconfigurationparametersParameterMandatoryDescriptionDefaultValuechannelsYesNameoftheFlumechannel.
-typeYesDISSourcetype.
com.
huaweicloud.
dis.
adapter.
flume.
source.
DISSourcestreamsYesDISstreamname.
TheenteredDISstreamnamemustbethesameasthestreamnamespecifiedwhenyouarecreatingaDISstreamontheDISconsole.
akYesUser'sAK.
FordetailsabouthowtoobtainanAK,seeCheckingAuthenticationInformation.
-DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
47ParameterMandatoryDescriptionDefaultValueskYesUser'sSK.
FordetailsabouthowtoobtainanSK,seeCheckingAuthenticationInformation.
-regionYesRegioninwhichtheDISislocated.
-projectIdYesProjectIDspecifictoyourregion.
FordetailsabouthowtoobtainaprojectID,seeCheckingAuthenticationInformation.
-endpointYesDataAPIaddressoftheregionwhereDISresides.
-group.
idYesApplicationname,whichisusedtoidentifyaconsumergroupandconsistsofletters,digits,hyphens(-),andunderscores(_).
-ConfiguringDISSinkTable5-4DISSinkconfigurationparametersParameterMandatoryDescriptionDefaultValuechannelYesNameoftheFlumechannel.
-typeYesSinktype.
com.
huaweicloud.
dis.
adapter.
flume.
sink.
DISSinkDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
48ParameterMandatoryDescriptionDefaultValuestreamNameYesNameoftheDISstream.
TheenteredDISstreamnamemustbethesameasthestreamnamespecifiedwhenyouarecreatingaDISstreamontheDISconsole.
akYesUser'sAK.
FordetailsabouthowtoobtainanAK,seeCheckingAuthenticationInformation.
-skYesUser'sSK.
FordetailsabouthowtoobtainanSK,seeCheckingAuthenticationInformation.
-regionYesRegioninwhichtheDISislocated.
-projectIdYesProjectIDspecifictoyourregion.
FordetailsabouthowtoobtainaprojectID,seeCheckingAuthenticationInformation.
-endpointYesDataAPIaddressoftheregionwhereDISresides.
-partitionNumberNoThenumberofpartitionsthatthechosenDISstreamhas.
ThevalueisusedtocalculatebatchSize.
1DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
49ParameterMandatoryDescriptionDefaultValuebatchSizeNoNumberofdatarecordsthatcanbebatchprocessedinasingleFlumetransaction.
batchSize=partitionNumber*250sendingThreadSizeNoThenumberofsenderthreads.
Bydefault,thereisonlyonesenderthread.
NOTEIfmultiplesenderthreadsareused,thefollowingsituationswilloccur:Thereisnoguaranteeontheorderinwhichdatawillbesent.
CertaindatawillberesentiftheFlumeapplicationrestartsafteritstopsabruptly.
1DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
50ParameterMandatoryDescriptionDefaultValuesendingRecordSizeNoNumberofdatarecordsthatcanbesentinasinglecalltotheDISAPIthatisusedtoputdataintoDISstreams.
NOTEbatchSizeindicatesthenumberofdatarecordsthatcanbebatchprocessedinasingleFlumetransaction,whereassendingRecordSizeindicatesthenumberofdatarecordsthatcanbebatchprocessedinasingleAPIcall.
Forexample,ifbatchSizeis1000andsendingRecordSizeis250,itindicatesthatfourAPIcallswillbemadetocompletetheFlumetransaction.
AFlumetransactioniscompletedandsubmittedonlyafterthebatchSizeamountofdataissuccessfullysent.
IftheapplicationrestartsbeforeaFlumetransactionissubmitted,datawillberesent.
IfsendingThreadSizeissetto1,itindicatesthatsendingRecordSizeandbatchSizewillhavethesamevalue.
Thispreventsunnecessarydataresending.
250DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
51ParameterMandatoryDescriptionDefaultValueretrySizeNoThemaximumnumberoftimesthattheDISFlumeSinkcanretrytocallaDISAPIwhentheinitialcalltotheDISAPIfails.
Thedefaultvalue2147483647isrecommended,indicatingthatSinkcanretrytheAPIcallforanunlimitednumberoftimes.
Exponentialbackoffisusedtoincrementallyincreasethewaitbetweenretryattemptsinordertoreduceserverloadandincreasethelikelihoodthatrepeatedrequestswillsucceed.
2147483647DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
52ParameterMandatoryDescriptionDefaultValueresultLogLevelNoTheleveloflogsgeneratedtoprintoutthelatestsequenceNumberattheendofeachDISAPIcall.
Loglevelsarelistedintheorderoffromlowtohigh:OFFAccessKeys.
ProjectIDAprojectisagroupoftenantresources.
Atenantcanhavemultipleprojects,oneforeachregion.
EachregionhasauniqueProjectID.
ToviewtheprojectIDsofdifferentregions,choose.
ObtainingaDISLogstashPluginPackageObtainthedis-logstash-Plugin-X.
X.
X.
zippackagefromhttps://dis-publish.
obs-website.
cn-north-1.
myhwclouds.
com/.
5.
4.
3InstallingDISLogstashPluginOnlineDISLogstashPlugincanbeinstalledonlineoroffline.
Foronlineinstallation,youdonotneedtodownloadtheDISLogstashPluginpackagebutdirectlyconnecttothepublicnetwork.
PrerequisitesPuTTYhasbeeninstalled.
Installinglogstash-input-disStep1StartPuTTYoranotherremotelogintooltologintotheserverwhereLogstashisinstalled.
Step2RunthefollowingcommandtonavigatetotheLogstashinstallationdirectory:cd${LOGSTASH_HOME}Step3Runthefollowingcommandtoinstalllogstash-input-dis:bin/logstash-plugininstalllogstash-input-disIfinformationsimilartothefollowingisdisplayed,theinstallationissuccessful:Validatinglogstash-input-disInstallinglogstash-input-disInstallationsuccessful----EndDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
60Installinglogstash-output-disStep1StartPuTTYoranotherremotelogintooltologintotheserverwhereLogstashisinstalled.
Step2RunthefollowingcommandtonavigatetotheLogstashinstallationdirectory:cd${LOGSTASH_HOME}Step3Runthefollowingcommandtoinstalllogstash-output-dis:bin/logstash-plugininstalllogstash-output-disIfinformationsimilartothefollowingisdisplayed,theinstallationissuccessful:Validatinglogstash-output-disInstallinglogstash-output-disInstallationsuccessful----End5.
4.
4InstallingDISLogstashPluginOfflineDISLogstashPlugincanbeinstalledonlineoroffline.
Forofflineinstallation,youneedtoobtaintheDISLogstashPluginpackageandexecutetheinstallationscript.
PrerequisitesPuTTYhasbeeninstalled.
ProcedureStep1StartPuTTYoranotherremotelogintooltologintotheserverwhereLogstashisinstalled.
Step2RunthefollowingcommandtonavigatetotheLogstashinstallationdirectory:cd${LOGSTASH_HOME}Step3Uploadthedis-logstash-plugins-X.
X.
X.
zipinstallationpackagetothisdirectory.
Step4Decompresstheinstallationpackage.
unzipdis-logstash-plugins-X.
X.
X.
zipStep5Runthefollowingcommandtonavigatetothedirectorywherethepackageisdecompressed:cdlogstash-pluginsStep6Runthefollowingcommandtoruntheinstallationprogram:bashinstall.
sh–p${LOGSTASH_HOME}Ifinformationsimilartothefollowingisdisplayed,theinstallationissuccessful:Installdis-logstash-pluginssuccessfully.
----EndDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
615.
4.
5ConfiguringtheDISLogstashPluginDISLogstashPluginsconsistoftheInputandOutputplugins.
Thissectiondescribestheconfigurationitemsoftheplugins.
ConfiguringDISLogstashInputTheconfigurationtemplate(usedtodownloaddatafromaDISstreamtoalocalfile)isasfollows:input{dis{streams=>["YOUR_DIS_STREAM_NAME"]endpoint=>"https://dis.
${region}.
myhuaweicloud.
com"ak=>"YOUR_ACCESS_KEY_ID"sk=>"YOUR_SECRET_KEY_ID"region=>"YOUR_Region"project_id=>"YOUR_PROJECT_ID"group_id=>"YOUR_APP_ID"client_id=>"YOUR_CLIENT_ID"auto_offset_reset=>"earliest"}}output{file{path=>["/tmp/test.
log"]}}Table5-5DISLogstashInputconfigurationparametersParameterMandatoryDescriptionDefaultValuestreamYesDISstreamname.
TheenteredDISstreamnamemustbethesameasthestreamnamespecifiedwhenyouarecreatingaDISstreamontheDISconsole.
akYesUser'sAK.
FordetailsabouthowtoobtainanAK,seeCheckingAuthenticationInformation.
-skYesUser'sSK.
FordetailsabouthowtoobtainanSK,seeCheckingAuthenticationInformation.
-DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
62ParameterMandatoryDescriptionDefaultValueregionYesRegioninwhichDISislocated.
-project_idYesProjectIDspecifictoyourregion.
FordetailsabouthowtoobtainaprojectID,seeCheckingAuthenticationInformation.
-client_idNoClientID,whichidentifiesaconsumerinaconsumergroup.
IfmultiplepipelinesorLogstashinstancesarestartedforconsumption,setthisparametertodifferentvalues.
Forexample,thevalueofinstance1isclient1,andthevalueofinstance2isclient2.
logstashendpointYesDataAPIaddressoftheregionwhereDISresides.
-group_idYesDISAppname,usedtoidentifyaconsumergroup.
Thevaluecanbeanycharacterstring.
-DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
63ParameterMandatoryDescriptionDefaultValueauto_offset_resetNoPositionwheredatastartstobeconsumedfromthestream.
Theoptionsareasfollows:earliest:Dataisconsumedfromtheearliestone.
latest:Dataisconsumedfromthelatestone.
latestConfiguringDISLogstashOutputTheconfigurationtemplate(usedtoreaddatafromalocalfileanduploadittoaDISstream)isasfollows:input{file{path=>["/tmp/test.
log"]type=>"log4j"start_position=>"beginning"}}output{dis{stream=>["YOUR_DIS_STREAM_NAME"]endpoint=>"https://dis.
${region}.
myhuaweicloud.
com"ak=>"YOUR_ACCESS_KEY_ID"sk=>"YOUR_SECRET_KEY_ID"region=>"YOUR_Region"project_id=>"YOUR_PROJECT_ID"}}Table5-6DISLogstashOutputconfigurationparametersParameterMandatoryDescriptionDefaultValuestreamYesDISstreamname.
TheenteredDISstreamnamemustbethesameasthestreamnamespecifiedwhenyouarecreatingaDISstreamontheDISconsole.
DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
64ParameterMandatoryDescriptionDefaultValueakYesUser'sAK.
FordetailsabouthowtoobtainanAK,seeCheckingAuthenticationInformation.
-skYesUser'sSK.
FordetailsabouthowtoobtainanSK,seeCheckingAuthenticationInformation.
-regionYesRegioninwhichDISislocated.
-project_idYesProjectIDspecifictoyourregion.
FordetailsabouthowtoobtainaprojectID,seeCheckingAuthenticationInformation.
-body_compress_enabledNoSpecifieswhethertoenabledatacompression.
NoDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
65ParameterMandatoryDescriptionDefaultValuebody_compress_typeNoDatacompressiontype.
Thefollowingcompressionalgorithmsaresupported:lz4:acompressionalgorithmwithafastcompressionspeedandhighcompressionefficiencysnappy:acompressionalgorithmwithahighcompressionspeedandareasonablecompressionrate,notforachievingmaximumcompressionorbeingcompatiblewithothercompressionformatszstd:anewlosslesscompressionalgorithmwithafastcompressionspeedandhighcompressionratiolz45.
4.
6TestingtheDISLogstashPluginTestingDISLogstashInputStep1StartPuTTYandlogintotheserveronwhichLogstashisinstalled.
Step2StarttheLogstashprogram.
bin/logstash-fdis_to_local.
confIntheprecedingcommand,-findicatesthepathoftheconfigurationfile.
Step3CheckthatDISInputcansuccessfullydownloaddatafromDIS.
UploaddatatoastreamtowhichDISInputpoints.
IfLogstashdoesnotreportanerrorandDISOutputcanobtaindata,thedownloadissuccessful.
DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
66Step4LogintotheDISconsole.
Twominuteslater,checkmonitoringdataoftheDISstreamspecifiedinTable5-5.
Ifdatadownload(bluelines)isdisplayed,DISLogstashInputisrunningsuccessfully.
----EndTestingDISLogstashOutputStep1StartPuTTYandlogintotheserveronwhichLogstashisinstalled.
Step2StarttheLogstashprogram.
bin/logstash-flocal_to_dis.
confIntheprecedingcommand,-findicatesthepathoftheconfigurationfile.
Step3CheckthatDISLogstashOutputcansuccessfullyuploaddatatoDIS.
IngestdatatotheinputendofLogstash.
IfLogstashdoesnotreportanerrorandcanuploaddatatothespecifiedstream,theuploadissuccessful.
Step4LogintotheDISconsole.
Twominuteslater,checkmonitoringdataoftheDISstreamspecifiedinTable5-6.
Ifdataupload(greenlines)isdisplayed,DISLogstashOutputisrunningsuccessfully.
----End5.
4.
7(Optional)UninstallingtheDISLogstashPluginProcedureStep1StartPuTTYandlogintotheserveronwhichLogstashisinstalled.
Step2StoptheLogstashprogram.
Step3GotothedirectorywheretheDISLogstashPluginislocated.
cd${LOGSTASH_HOME}cdlogstash-pluginsStep4RunthefollowingcommandtouninstalltheDISLogstashPlugin:bashuninstall.
sh–p${LOGSTASH_HOME}Ifinformationsimilartothefollowingappears,theDISLogstashPluginhasbeensuccessfullyuninstalled:Uninstalldis-logstash-pluginssuccessfully.
----End5.
5UsingSparkStreamingSDKtoDownloadDataDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
675.
5.
1OverviewDISSparkStreamingisasoftwaredevelopmentkit(SDK)providedbyDIStocreateaDStreamthatusestheDISasadatasourceandrunswithSparkStreaming.
Figure5-4depictstheprocessforusingDISSparkStreaming.
Figure5-4DISSparkStreamingusageprocess5.
5.
2PreparingaDISSparkStreamingEnvironmentPreparingaDISApplicationDevelopmentEnvironmentStep1SetupDISapplicationdevelopmentenvironmentsbyfollowingtheprocedureinCreatingaDISStream.
Step2InstallMaven.
Configurealocalrepositoryaddress.
Step3Installscala-sdk.
----EndConfiguringtheDISSparkStreamingDependencyTheDISSparkStreamingdependencycanbeintroducedtotheprojectthroughthefollowingconfiguration:com.
huaweicloud.
dishuaweicloud-dis-spark-streaming_2.
111.
2.
1compileCheckingAuthenticationInformationAK/SKfileDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
68AccessKeyID/SecretAccessKey(AK/SK)filesarecreatedbytheIdentityandAccessManagement(IAM)servicetoauthenticatecallstoapplicationprogramminginterfaces(APIs)onthecloud.
ToobtainAK/SK,chooseMyCredentials>AccessKeys.
ProjectIDAprojectisagroupoftenantresources.
Atenantcanhavemultipleprojects,oneforeachregion.
EachregionhasauniqueProjectID.
ToviewtheprojectIDsofdifferentregions,choose.
5.
5.
3CustomizingaSparkStreamingJobObtainingtheDISSparkStreamingDemoStep1Obtainthedis-spark-streaming-X.
X.
X.
zippackagefromhttps://dis-publish.
obs-website.
cn-north-1.
myhwclouds.
com/.
Decompressthedis-spark-streaming-X.
X.
X.
zippackagetoobtainthefollowingdirectorythatcontainsaMavenprojectsample:dis-spark-streaming-demo:Thedis-spark-streaming-demodirectorycontainsamavenprojectsample.
----EndBuildingaSparkStreamingJobThefollowingusestheIntelliJIDEAcommunityversionasanexampletodescribehowtobuildaSparkStreamingjob.
MakesurethattheSparkStreamingjobhasbeenconfiguredontheIDEA.
JDK1.
8+Scala-sdk-2.
11Maven3.
3.
*Step1StartIntelliJIDEAandchooseFile>Open.
Selectthedis-spark-streaming-demodirectoryanddouble-clickpom.
xml.
DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
69Step2Whenthefollowingdialogboxisdisplayed,selectOpenasProject.
Step3ClickNewWindowtoopentheprojectinanewwindow.
Step4InthedisplayedIDEAwindow,chooseFile>Settings.
Step5Entermaveninthesearchboxandfindthemavenconfiguration.
EnsurethatMavenhomedirectory(maveninstallationpath),Usersettingsfile(settings.
xmlfilelocation),andLocalrepository(localrepositoryaddress)arecorrectlyconfigured.
Ifconfigurationsareincorrect,modifythem.
Otherwise,theSDKinstalledinStep2cannotbefound.
DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
70Step6OpentheDISSparkStreamingExamplefile.
IftheIDEAdisplaysthemessage"NoScalaSDKinmodule",clickSetupScalaSDKnexttoitandselectonefromtheScalaSDKlist.
IfnoScalaSDKisavailable,createoneandassociateitwiththeScalapath.
Version2.
11canbeselected.
Step7Right-clickpom.
xmlandchooseMaven>Reimportfromtheshortcutmenutoimportthemavendependencylibraryagain.
Step8IfthereisnoerrorintheDISSparkStreamingExamplefileopenedbyIDEA,thedevelopmentenvironmentisconfiguredsuccessfully.
ThelogicofthisfileistoreadthedatainDISstreamsandcounttheoccurrencetimesofeachword.
DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
711.
DISSparkStreamingExampleisanexampleofusingtheAssignmode.
Itdoesnothavethecapabilitytostartfromthelaststop.
TheSDKconstructionmethodisasfollows:ConsumerStrategies.
Assign[String,String](streamName,params,startingOffsets)StreamName:aDISstreamname.
params:asetofparametermaps.
Itincludesatleastendpoint(DISgatewayaddress),region(regionwhereDISresides),ak(userAK),sk(userSK),andprojectId(userprojectID).
StartingOffsets:startpositionforreadingDISdata.
LATESTindicatesreadfromthelatestdata.
EARLIESTindicatesreadingfromtheoldestdata.
Iftheexactstartpositionofeachpartitionneedstobespecified,thevaluecanbeaJSONcharacterstring.
Forexample,{"0":23,"1":-1,"2":-2},whichindicatesthatthestartpositionofpartition0is23,thefirstpartitionstartsfromthelocationofthelatestdata,andthesecondpartitionstartsfromthepositionoftheearliestdata.
Ifthereisnospecifiedlocationforapartition,thesystemstartsfromthelatestdatapositionbydefault.
2.
DISSparkStreamingExample:anexampleofusingtheSubscribemode.
Ithasthecapabilitytostartfromthelaststop.
TheSDKconstructionmethodisasfollows:ConsumerStrategies.
Subscribe[String,String](Array(streamName),params)StreamName:aDISstreamname.
params:asetofparametermaps.
Itincludesatleastendpoint(DISgatewayaddress),region(regionwhereDISresides),ak(userAK),sk(userSK),projectId(userprojectID),andgroup.
id(appname,indicatingaconsumergroup).
Itcanalsoincludeauto.
offset.
reset.
ThemeaningofthisparameteristhesameasthatofstartingOffsetsinAssignmode.
Ifenable.
auto.
commitissettotrue,thesystemautomaticallysubmitstheoffsetevery5000ms(whichcanbemodifiedbysettingauto.
commit.
interval.
ms).
Iftheparameterissettofalse,thesystemdoesnotautomaticallysubmittheoffset.
YoucanmanuallyinvokecommitAsynctosubmittheoffset.
Fordetails,seethefollowingpartinthesamplecode:stream.
foreachRDD{rdd=>valoffsetRanges=rdd.
asInstanceOf[HasOffsetRanges].
offsetRanges//commitoffsettoDISasync.
stream.
asInstanceOf[CanCommitOffsets].
commitAsync(offsetRanges)}----EndTestingaSparkStreamingJobInareal-worldscenario,theSparkStremingjobneedstobesubmittedtotheSparkcluster.
However,thisdemonstrationonlydescribeshowtotestthejobinthelocalIDEtounderstandthebasicusageoftheSDK.
Afterthetestiscomplete,youcancreateclusters(suchasCSclusters)andsubmitajobforverification.
Step1UsetheaccounttologintotheDISconsole.
Step2ClickintheupperleftcornerandselectRegionandProject.
DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
72Step3CreateaDISstreambyreferringtoCreatingaDISStreamandcontinuouslyuploaddatatothenewlycreatedDISstream.
Inthisexample,thecontenttobeuploadedishelloworld.
Step4Openthepom.
xmlfile,pressCtrl+/tocommentouttheprovidedrow,andsavethesetting.
Step5Right-clickpom.
xmlandchooseMaven>Reimportfromtheshortcutmenutoimportthedependencypackageagain.
Step6Right-clickintheDISSparkStreamingExamplefileandchooseCreate'DISSparkStreamingExample'fromtheshortcutmenu.
Step7Onthedisplayedconfigurationpage,enter-Dspark.
master=local[*]inVMoptions,indicatingthataSparkjobrunsinlocalmode.
EnterrunningparametersinProgramargumentsinthefollowingformat:DISgatewayaddressRegionnameAKSKProjectIDStreamnameStartpositionStreamingbatchtimeForexample,ifatestisperformedinNorthChina-Beijing1,theparameterexampleisasfollows:https://dis.
${region}.
myhwclouds.
com${region}YOU_AKYOU_SKYOU_PROJECTIDYOU_STREAM_NAMElatest10Theparametersequenceandmeaningareavailableinthesamplecode.
Fordetails,seethefollowinginformation:DataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
73ThefinalconfigurationsoftheIDEAareasfollows.
Afterconfirmingthatallconfigurationsarecorrect,clickOKtoclosethewindow.
Step8Right-clickintheDISSparkStreamingExamplefileandchooseRun'DISSparkStreamingExample'fromtheshortcutmenutostartthejob.
Step9Duringthestartup,anerrormessage"hadoopbinarypath"isdisplayed,whichcanbeignored.
18/08/2810:26:10ERRORShell:Failedtolocatethewinutilsbinaryinthehadoopbinarypathjava.
io.
IOException:Couldnotlocateexecutablenull\bin\winutils.
exeintheHadoopbinaries.
Step10Iftherearenoothererrors,thejobrunsabatchatafixedinterval,readsdatafromthebatch,andexportsthereadresult.
Thefollowinggivesanexample:Time:1535423650000msDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
74(hello,30)(world.
,30)Step11Afterverifyingthatthejobcanrunlocallywithouterror,removethecommenttagfromtheprovidedrowinpom.
xmltopreventtheSparkdependencyfrombeingpackaged.
Thenstopthedatauploadprogram.
----EndDataIngestionServiceUserGuide5UsingDISIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
756ManagingaDumpTask6.
1CreatingaDumpTaskIfadumptaskiscreatedforaDISstream,datasenttotheDISstreamcanbeautomaticallydumpedtotheselectedtargetspecifiedinthedumptask.
Step1UsetheaccounttologintotheDISconsole.
Step2Clickintheupperleftcornerofthepageandselectaregionandproject.
Step3Inthenavigationtreeontheleft,choose.
Step4Clickthenameofastreamthatyouwanttoview.
Onthedisplayedpage,clicktheDumpManagementtab.
Alternatively,intheOperationcolumnofastreamthatyouwanttoview,clickMoreandchooseViewDumpTaskfromthedrop-downlist.
Step5ClickAddDumpTask.
OntheAddDumpTaskpage,configuredumpparameters.
Dumptaskparametersaredescribedin.
Amaximumoffivedumptaskscanbecreatedforeachstream.
Step6ClickCreateNow.
Step7IntheOperationcolumnofthecorrespondingTaskName,clickMore>ViewDumpLogtoviewthedumptaskdetailsofthestream.
Table6-1describesthedumplogparameters.
DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
76Table6-1DumplogparametersParameterDescriptionStartTimeTimewhenthedumplogiscreated.
Format:YYYY/MM/ddHH:mm:ssGTMYYYY:year.
MM:month.
dd:date.
HH:hour.
mm:minute.
ss:second.
GMT:timezone.
EndTimeTimewhenyoufinishcreatingthedumplog.
Format:YYYY/MM/ddHH:mm:ssGTMYYYY:year.
MM:month.
dd:date.
HH:hour.
mm:minute.
ss:second.
GMT:timezone.
StatusDumpstatus.
SucceededFailedAbnormalDumpFileNameNameofthefilethatisdumpedtothetargetservice.
Theuserrecordsreadfromthestreamarewrittenintothefileandthendumpedtothetargetservice(suchasOBS)inthefileformat.
RecordsNumberoftherecordsuploadedbetweenthetimewhenyoustarttocreateadumplogtothetimewhenyoufinishcreatingit.
DataAmount(bytes)Amountofthedatauploadedbetweenthetimewhenyoustarttocreatethedumplogtothetimewhenyoufinishcreatingit.
Unit:byteDataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
77ParameterDescriptionOperationDumpfailuredetails.
IfStatusisSucceeded,thecolumnisnotoperable.
IfStatusisFailed,clickViewDetailstoviewdumpdetails.
IfStatusisAbnormal,clickViewDetailstoviewdumpdetails.
----EndModifyingandEnablingDumpTasksAftercreatingastreamandaddingadumptasksuccessfully,youcanmodifytheattributesofthecreatedstream.
Step1LogintotheDISconsole.
Step2Clickintheupperleftcornerofthepageandselectaregionandproject.
Step3Inthenavigationtree,chooseStreamManagement.
Step4Clickthenameofastreamthatyouwanttoview.
Onthedisplayedpage,clicktheDumpManagementtab.
Alternatively,intheOperationcolumnofastreamthatyouwanttoview,clickMoreandchooseViewDumpTaskfromthedrop-downlist.
Step5IntheOperationcolumnofthestreamforwhichadumptaskhasbeenadded,performthefollowingoperations:1.
ChooseMore>Modifytomodifythedumptask.
2.
ChooseMore>Starttostartthedumptask.
3.
ChooseMore>Pausetopausethedumptask.
----EndDataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
786.
2DumpingDatatoOBSDumpingtheJSON,BLOB,andCSVDatatotheTextDataTable6-2ParametersforconfiguringaTextdumpfileParameterDescriptionValueTaskNameNameofthedumptask.
Thenamesofdumptaskscreatedforthesamestreammustbeunique.
Adumptasknameis1to64characterslong.
Onlyletters,digits,hyphens(-),andunderscores(_)areallowed.
-DumpBucketNameoftheOBSbucketusedtostoredatafromtheDISstream.
ThebucketnameiscreatedwhenyoucreateabucketinOBS.
-FileDirectoryDirectorycreatedinOBStostorefilesfromtheDISstream.
Thisdirectorynameis0to50characterslong.
Bydefault,thisparameterisleftunspecified.
-TimeDirectoryFormatDatawillbesavedaccordingtothetimeformatinthefiledirectoryoftheOBSbucket.
Forexample,ifthetimedirectoryisaccuratetoday,thesavepathwillbeintheformatofbucketname/filedirectory/year/month/day.
Possiblevaluesareasfollows:N/A:Ifthisfieldisleftunspecified,thetimedirectoryformatwillnotbeused.
yyyy:year.
yyyy/MM:yearandmonth.
yyyy/MM/dd:year,month,andday.
yyyy/MM/dd/HH:year,month,day,andhour.
yyyy/MM/dd/HH/mm:year,month,day,hour,andminute.
Youcanonlyselectbutnotenteravalueinthisfield.
-DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
79ParameterDescriptionValueRecordDelimiterDelimiterusedtoseparatedifferentdumprecords.
Possiblevaluesareasfollows:Comma(,)Semicolon(;)Verticalbar(|)Newline(\n)NULLYoucanonlyselectbutnotenteravalueinthisfield.
-OffsetLatest:Maximumoffset,indicatingthatthelatestdatawillberead.
Earliest:Minimumoffset,indicatingthattheearliestdatawillberead.
LatestDumpInterval(s)IntervalatwhichdatafromtheDISstreamwillbeimportedintodumpdestination,suchasOBS,MRS,DLI,DWS,andCloudTable.
IfnodatawaspushedtotheDISstreamduringthetimespecifiedhere,thedumpfilewillnotbegenerated.
Valuerange:30sto900sUnit:secondDefaultvalue:300s-DumpingtheJSONDatatotheCSVDataTable6-3ParametersforconfiguringaCSVdumpfileParameterDescriptionValueTaskNameNameofthedumptask.
Thenamesofdumptaskscreatedforthesamestreammustbeunique.
Adumptasknameis1to64characterslong.
Onlyletters,digits,hyphens(-),andunderscores(_)areallowed.
-DumpBucketNameoftheOBSbucketusedtostoredatafromtheDISstream.
ThebucketnameiscreatedwhenyoucreateabucketinOBS.
-DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
80ParameterDescriptionValueFileDirectoryDirectorycreatedinOBStostorefilesfromtheDISstream.
Thisdirectorynameis0to50characterslong.
Bydefault,thisparameterisleftunspecified.
-TimeDirectoryFormatDatawillbesavedaccordingtothetimeformatinthefiledirectoryoftheOBSbucket.
Forexample,ifthetimedirectoryisaccuratetoday,thesavepathwillbeintheformatofbucketname/filedirectory/year/month/day.
Possiblevaluesareasfollows:N/A:Ifthisfieldisleftunspecified,thetimedirectoryformatwillnotbeused.
yyyy:year.
yyyy/MM:yearandmonth.
yyyy/MM/dd:year,month,andday.
yyyy/MM/dd/HH:year,month,day,andhour.
yyyy/MM/dd/HH/mm:year,month,day,hour,andminute.
Youcanonlyselectbutnotenteravalueinthisfield.
-OffsetLatest:Maximumoffset,indicatingthatthelatestdatawillberead.
Earliest:Minimumoffset,indicatingthattheearliestdatawillberead.
LatestDumpInterval(s)User-definedintervalatwhichdataisimportedfromthecurrentDISstreamintothetargetplace.
IfnodataispushedtotheDISstreamduringthecurrentinterval,nodumpfilepackagewillbegenerated.
Valuerange:30sto900sUnit:secondDefaultvalue:300s-DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
81DumpingtheJSONandCSVDatatotheParquetDataTable6-4liststhedifferentiatedparametersthatneedtobesetwhenthesourcedatatypeisJSONorCSV,thedumpdestinationisOBS,andthedumpfileformatisParquet.
Fordetailsabouthowtoconfigureothercommonparameters,seeTable6-2.
Table6-4ParametersforconfiguringaParquetdumpfileParameterDescriptionValueSourceDataSchemaJSONorCSVdataexample,usedtodescribetheJSONorCSVdataformat.
DIScangenerateanAvroschemabasedontheJSONorCSVdatasampleandconverttheuploadedJSONorCSVdatatotheParquetformat.
-DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
82ParameterDescriptionValueFileDirectoryDirectorycreatedinOBStostorefilesfromtheDISstream.
Thisdirectorynameis0to100characterslong.
Bydefault,thisparameterisleftunspecified.
NOTEWhenthesourcedatatypeisJSON,ELandbuilt-infunctionsaresupported.
ELexampleSourcedata:{"name":"Andy","city":"","time":1556323141582}Thedumpfiledirectoryisconfiguredasfollows:dis/basePath/app_key_p=${name}Thefinaldirectorystructureisasfollows:{$Bucketname}/dis/basePath/app_key_p=AndyExampleofbuilt-infunctionsSourcedata:{"name":"Andy","city":"","time":1556323141582}Thedumpfiledirectoryisconfiguredasfollows:dis/basePath/date_p=toDate(${time},"yyMMdd")Thefinaldirectorystructureisasfollows:{$Bucketname}/dis-basePath/date_p=20190427Supportedbuilt-infunctionstoDate(timestamp,format):Convertthetimestamptoaspecifiedtimeformat.
Forexample:toDate(1556323141582,'yymmdd')DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
83ParameterDescriptionValueCustomTimeDirectoryYoucanclickortodisableorenablethecustomtimedirectory.
Ifthecustomtimestampisdisabled,thedirectorywheretheobjectfilewrittentoOBSresidesisnamedafterthecreationtimeofthedumpfile.
Forexample,ifadumpfilewascreatedonOctober16,2018andthetimedirectoryformatisaccuratetoday,thefilewillbesavedinOBSbucketname/dumpfiledirectory/2018/10/16.
Ifthecustomtimestampisdisabled,thedirectorywheretheobjectfilewrittentoOBSresidesisnamedinthetimeformatspecifiedinthesourcedata.
Forexample,ifadumpfilewascreatedonOctober16,2018,thetimedirectoryformatisaccuratetoday,andthetimeformatspecifiedinthedatasourceis2017/09/0811:01:01,thefilewillbesavedinOBSbucketname/dumpfiledirectory/2017/09/08.
Thestoragedirectoryisdefinedbasedonthetimeformatdefinedinthesourcedatainsteadofthetimewhenthedumpfileiscreated.
Example1:DumpsimpleJSONdata.
Sourcedata:{"id":"1","date":"2018/10/1611:00:05"}Theconfigurationisasfollows:Setthetimestampattributenametodate,datatypetoString,andtimestampformattoyyyy/MM/ddHH:mm:ssbasedonthesourcedatatypeofthedatatobedumped.
Afterthedataisdumpedsuccessfully,thestoragedirectorystructuredependsonthesourcedatatimestampandthetimedirectoryformat.
Inthisexample,thetimedirectoryformatisaccuratetoday.
Therefore,thefinaldatastoragedirectoryisOBSbucketname/dumpfiledirectory/2018/10/16.
Example2:DumpmultiplylayersofnestedJSONdata.
Sourcedata:{"id":"1","detail":{"detID":"05790110000000000103#567fd3cb13a4493eaa43076953253eed","endTime":"2018/10/0713:26:35"}}Theconfigurationisasfollows:Setthetimestampattributenametodetail.
endTime,datatypetoString,andtimestampformattoyyyy/MM/ddHH:mm:ssDataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
84ParameterDescriptionValueSourceDataTimestampAttributenameofthetimestamp.
NOTEEnterthefieldnamecorrespondingtothetimestampdefinedinthesourcedatatobeuploaded.
Timestampformat.
Possiblevaluescanbe:yyyy/MM/ddHH:mm:ssMM/dd/yyyyHH:mm:ssdd/MM/yyyyHH:mm:ssyyyy-MM-ddHH:mm:ssMM-dd-yyyyHH:mm:ssdd-MM-yyyyHH:mm:ssDatatype.
Possiblevaluescanbe:–String–TimestampNOTEIfthetypeofthesourcedatatobeuploadedisTimestamp,thevaluemustbeaccuratetomilliseconds.
basedonthesourcedatatypeofthedatatobedumped.
Afterthedataisdumpedsuccessfully,thestoragedirectorystructuredependsonthesourcedatatimestampandthetimedirectoryformat.
Inthisexample,thetimedirectoryformatisaccuratetoday.
Therefore,thefinaldatastoragedirectoryisOBSbucketname/dumpfiledirectory/2018/10/07.
Example3:DumpCSVdata.
Sourcedata:a,2010-10-1211:00:00,b,2011-10-1211:00:10Theconfigurationisasfollows:Selecttimestamp2010-10-1211:00:00basedonthesourcedatatobedumped.
AfterdataisconvertedtotheParquetformat,thecorrespondingattributefieldnameisfield_1.
Whencreatingadumptask,setthetimestampattributetofield_1,datatypetoString,andtimestampformattoyyyy/MM/ddHH:mm:ss.
Afterthedataisdumpedsuccessfully,thestoragedirectorystructuredependsonthesourcedatatimestampandthetimedirectoryformat.
Inthisexample,thetimedirectoryformatisaccuratetoday.
Therefore,thefinaldatastoragedirectoryisOBSDataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
85ParameterDescriptionValuebucketname/dumpfiledirectory/2010/10/12.
DumpingtheJSONandCSVDatatotheCarbonDataDataTable6-5liststhedifferentiatedparametersthatneedtobesetwhenthesourcedatatypeisJSONorCSV,thedumpdestinationisOBS,andthedumpfileformatisCarbonData.
Fordetailsabouthowtoconfigureothercommonparameters,seeTable6-2.
Table6-5ParametersforconfiguringaCarbonDatadumpfileParameterDescriptionValueSourceDataSchemaJSONorCSVdatasample,usedtodescribetheJSONorCSVdataformat.
DIScangenerateanAvroschemabasedontheJSONorCSVdatasampleandconverttheuploadedJSONorCSVdatatotheCarbonDataformat.
-DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
86ParameterDescriptionValueCarbonDataRetrievalAttributeAttributeofthecarbontable,usedtocreateacarbonwriter.
Thefollowingkeysaresupported:table_blocksize:Sizeofatableblock.
Thevaluerangesfrom1MBto2048MB.
Thedefaultvalueis1024MB.
table_blocklet_size:Sizeoftheblockletinafile.
Thedefaultvalueis64MB.
local_dictionary_thresholdlocal_dictionary_enable:Possiblevaluescanbetrueorfalse.
Thedefaultvalueisfalse.
sort_columns:Specifiestheindexcolumn.
Multi-levelindexcolumnsareseparatedbycommas(,).
sort_scope:Specifiesthescopewheredataissortedduringloading.
Currently,thefollowingtypesaresupported:–local_sort:Defaultvalue,indicatingthatdataissortedinanode.
–no_sort:Dataisnotsorted.
Itisusedwhendataneedstobesavedtoadatabasequickly.
Afterthedataissavedtothedatabase,youcanusetheCompactioncommandtocreateanindexwhenthesystemisidle.
–batch_sort:ACarbonDatafileisgeneratedafterthememoryissortedinanodeandnofullsortingisperformedonthenode.
Thisconfigurationimprovestheloadingspeed,butthequeryperformanceisinferiortothatofLOCAL_SORT.
long_string_columns-DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
876.
3DumpingDatatoDLISourceDataType:JSONandCSVTable6-6DumpparametersParameterDescriptionValueTaskNameNameofthedumptask.
Thenamesofdumptaskscreatedforthesamestreammustbeunique.
Adumptasknameis1to64characterslong.
Onlyletters,digits,hyphens(-),andunderscores(_)areallowed.
-DLIDatabaseClickSelect.
IntheSelectDLIDatabasedialogbox,selectadatabase.
Youcanonlyselectbutnotenteravalueinthisfield.
-DLITableClickSelect.
IntheSelectDLITabledialogbox,selectadatatable.
OnlytablesinDLIaresupported.
BeforeselectingaDLItable,ensurethatyouhavethepermissiontoinsertdataintothetable.
Youcanonlyselectbutnotenteravalueinthisfield.
ThisparameterisavailableonlyafteryouselectaDLIdatabase.
OffsetLatest:Maximumoffset,indicatingthatthelatestdatawillberead.
Earliest:Minimumoffset,indicatingthattheearliestdatawillberead.
LatestDumpInterval(s)IntervalatwhichdatafromtheDISstreamwillbeimportedintodumpdestination,suchasOBS,MRS,DLI,DWS,andCloudTable.
IfnodatawaspushedtotheDISstreamduringthetimespecifiedhere,thedumpfilewillnotbegenerated.
Valuerange:30sto900sUnit:secondDefaultvalue:300s-TemporaryBucketOBSbucketinwhichadirectoryiscreatedfortemporarilystoringuserdata.
Thedatainthedirectoryisdeletedafterbeingdumpedtoaspecificdestination.
-DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
88ParameterDescriptionValueTemporaryDirectoryDirectoryinthechosenTemporaryBucketfortemporarilystoringdata.
Thedatainthedirectoryisdeletedafterbeingdumpedtoaspecificdestination.
Ifthisfieldisleftblank,thedataisstoreddirectlytotheTemporaryBucket.
-6.
4DumpingDatatoMRSSourceDataType:JSON,BLOB,andCSV;DumpFileFormat:TextTable6-7ParametersforconfiguringaTextdumpfileParameterDescriptionValueTaskNameNameofthedumptask.
Thenamesofdumptaskscreatedforthesamestreammustbeunique.
Adumptasknameis1to64characterslong.
Onlyletters,digits,hyphens(-),andunderscores(_)areallowed.
-MRSClusterClickSelect.
IntheSelectMRSClusterdialogbox,selectanMRScluster.
DataisdumpedonlytoanMRSclusterthatisnotauthenticatedbyKerberos.
Youcanonlyselectbutnotenteravalueinthisfield.
-HDFSPathClickSelect.
IntheSelectHDFSPathdialogbox,selectanHDFSpath.
Youcanonlyselectbutnotenteravalueinthisfield.
ThisparameterisavailableonlyafteryouselectanMRScluster.
-FileDirectoryDirectorycreatedinOBStostorefilesfromtheDISstream.
Thisdirectorynameis0to50characterslong.
Bydefault,thisparameterisleftunspecified.
-OffsetLatest:Maximumoffset,indicatingthatthelatestdatawillberead.
Earliest:Minimumoffset,indicatingthattheearliestdatawillberead.
LatestDataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
89ParameterDescriptionValueDumpInterval(s)IntervalatwhichdatafromtheDISstreamwillbeimportedintodumpdestination,suchasOBS,MRS,DLI,DWS,andCloudTable.
IfnodatawaspushedtotheDISstreamduringthetimespecifiedhere,thedumpfilewillnotbegenerated.
Valuerange:30sto900sUnit:secondDefaultvalue:300s-TemporaryBucketOBSbucketinwhichadirectoryiscreatedfortemporarilystoringuserdata.
Thedatainthedirectoryisdeletedafterbeingdumpedtoaspecificdestination.
-TemporaryDirectoryDirectoryinthechosenTemporaryBucketfortemporarilystoringdata.
Thedatainthedirectoryisdeletedafterbeingdumpedtoaspecificdestination.
Ifthisfieldisleftblank,thedataisstoreddirectlytotheTemporaryBucket.
-SourceDataType:JSONandCSV;DumpFileFormat:ParquetTable6-8liststhedifferentiatedparametersthatneedtobesetwhenthesourcedatatypeisJSONorCSV,thedumpdestinationisMRS,andthedumpfileformatisParquet.
Fordetailsabouthowtoconfigureothercommonparameters,seeTable6-7.
Table6-8ParametersforconfiguringaParquetdumpfileParameterDescriptionValueSourceDataSchemaJSONorCSVdataexample,usedtodescribetheJSONorCSVdataformat.
DIScangenerateanAvroschemabasedontheJSONorCSVdatasampleandconverttheuploadedJSONorCSVdatatotheParquetformat.
-DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
90SourceDataType:JSONandCSV;DumpFileFormat:CarbonDataTable6-9liststhedifferentiatedparametersthatneedtobesetwhenthesourcedatatypeisJSONorCSV,thedumpdestinationisMRS,andthedumpfileformatisCarbonData.
Fordetailsabouthowtoconfigureothercommonparameters,seeTable6-7.
Table6-9ParametersforconfiguringaCarbonDatadumpfileParameterDescriptionValueSourceDataSchemaJSONorCSVdataexample,usedtodescribetheJSONorCSVdataformat.
DIScangenerateanAvroschemabasedontheJSONorCSVdatasampleandconverttheuploadedJSONorCSVdatatotheCarbonDataformat.
-DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
91ParameterDescriptionValueCarbonDataRetrievalAttributeAttributeofthecarbontable,usedtocreateacarbonwriter.
Thefollowingkeysaresupported:table_blocksize:Sizeofatableblock.
Thevaluerangesfrom1MBto2048MB.
Thedefaultvalueis1024MB.
table_blocklet_size:Sizeoftheblockletinafile.
Thedefaultvalueis64MB.
local_dictionary_thresholdlocal_dictionary_enable:Possiblevaluescanbetrueorfalse.
Thedefaultvalueisfalse.
sort_columns:Specifiestheindexcolumn.
Multi-levelindexcolumnsareseparatedbycommas(,).
sort_scope:Specifiesthescopewheredataissortedduringloading.
Currently,thefollowingtypesaresupported:–local_sort:Defaultvalue,indicatingthatdataissortedinanode.
–no_sort:Dataisnotsorted.
Itisusedwhendataneedstobesavedtoadatabasequickly.
Afterthedataissavedtothedatabase,youcanusetheCompactioncommandtocreateanindexwhenthesystemisidle.
–batch_sort:ACarbonDatafileisgeneratedafterthememoryissortedinanodeandnofullsortingisperformedonthenode.
Thisconfigurationimprovestheloadingspeed,butthequeryperformanceisinferiortothatofLOCAL_SORT.
long_string_columns-DataIngestionServiceUserGuide6ManagingaDumpTaskIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
927ManagingEnterpriseProjectsTheenterpriseprojectisacloudresourcemanagementmode.
EnterpriseManagementprovidescomprehensivemanagementforcloud-basedresources,personnel,permissions,andfinance.
Differentfromcommonmanagementconsolesthatfeatureindependentcontrolandconfigurationofcloudproducts,theEnterpriseManagementconsoleisorientedtoresourcemanagement,helpingenterprisesmanagecloud-basedresources,personnel,permissions,andfinanceinahierarchicalmanner,suchasmanagementofcompanies,departments,andprojects.
UserswhohaveenabledEnterpriseManagementcanuseittomanagecloudserviceresourcesonHUAWEICLOUD.
BindinganEnterpriseProjectWhencreatingaDISstream,youcanselectanenterpriseprojectforthestreamtoassociatethestreamwiththeenterpriseproject.
Fordetails,seeCreatingaDISStream.
TheEnterpriseProjectdrop-downlistdisplaystheprojectsyoucreated.
Inaddition,thesystemhasabuilt-inenterpriseproject(default).
Ifyoudonotselectanenterpriseproject,thedefaultprojectisused.
Duringstreamcreation,ifthestreamissuccessfullyboundtoanenterpriseproject,thestreamissuccessfullycreated.
Ifthebindingfails,thesystemsendsanalarmandthestreamfailstobecreated.
AfteryoudeleteaDISstream,theassociationbetweenthestreamanditsenterpriseprojectisautomaticallydeleted.
ViewingEnterpriseProjectsAfterastreamiscreated,youcanviewtheenterpriseprojectsassociatedwiththestreamonthestreamlistandbasicstreaminformationpage.
Youcanqueryonlythestreamresourcesoftheprojectonwhichyouhavetheaccesspermission.
IntheEnterpriseProjectcolumnoftheStreamspage,viewenterpriseprojectstowhichstreamsbelong.
Figure7-1ViewingenterpriseprojectsDataIngestionServiceUserGuide7ManagingEnterpriseProjectsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
93OntheStreamspage,clickastreamname.
Inthestreamdetailsarea,viewtheenterpriseprojectassociatedwiththestream.
ClicktheenterpriseprojectnametoviewandedititontheEnterpriseManagementconsole.
Figure7-2ViewingenterpriseprojectsWhenqueryingtheresourcelistofaspecifiedprojectontheEnterpriseManagementconsole,youcanalsoquerytheDISresources.
SearchingforStreambyEnterpriseProjectLogintotheDISconsoleandchooseStreamManagement.
Inthedrop-downlistabovethestreams,selectarequiredprojectnametoviewallstreamsassociatedwiththeproject.
MigratingStreamsintooroutofEnterpriseProjectsOneDISstreamcanbeassociatedwithonlyoneenterpriseproject.
Afterastreamiscreated,youcanmigrateitfromitscurrententerpriseprojecttoanotheroneontheEnterpriseManagementconsole,ormigratethestreamfromanotherenterpriseprojecttoaspecifiedenterpriseproject.
Afterthemigration,thestreamisassociatedwiththenewenterpriseproject.
Theassociationbetweenthemthestreamandtheoriginalenterpriseprojectisautomaticallyreleased.
Fordetails,seeResourceManagement>ManagingEnterpriseProjectResourcesintheEnterpriseManagementUserGuide.
DataIngestionServiceUserGuide7ManagingEnterpriseProjectsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
948NotifyingEvents8.
1EventNotificationOverviewOverviewDISusesSimpleMessageNotification(SMN)tosendnotificationsofDISevents.
Inasubscription,youneedtospecifyoneormoreeventfilteringconditions.
Whenaneventthatmatchesallfilteringconditionsoccurs,DISsendsanotificationbasedonthesubscription.
ThefilterconditionsincludetheEventType(forexample,Management,Monitoring,orSecurity),EventLevel(forexample,NormalorWarning),andEventSource(forexample,Stream,Dumptask,orUser).
SupportedEventTypesandEventsEventsarerecordsofchangesinthetenant'sstreamstatus.
Itcanbetriggeredbyauseroperation(forexample,anauditevent),ormaybecausedbyastreamstatuschange(forexample,adumptaskisabnormaloradumptaskisrecovered).
ThefollowingtableslisttheeventsandeventtypessupportedbyDIS:ThefollowingtableliststheeventswhoseEventSourceisStream.
Table8-1EventswhoseEventSourceisStreamEventSourceEventLevelEventStreamWarningTrafficlimitedStreamWarningAutomaticstreamscalingsucceededStreamWarningAutomaticstreamscalingfailedStreamWarningStreamtrafficabnormalStreamWarningStreamtrafficrestoredDataIngestionServiceUserGuide8NotifyingEventsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
95ThefollowingtableliststheeventswhoseEventSourceisEvent.
Table8-2EventswhoseEventSourceisUserEventSourceEventLevelEventUserWarningQuotainsufficientThefollowingtableliststheeventswhoseEventSourceisDumptask.
Table8-3EventswhoseEventSourceisDumptaskEventSourceEventLevelEventDumptaskNormalDumptaskrestoredDumptaskWarningDumptaskabnormal8.
2SettingNotificationAfterenablingNotificationforsubscribedevents,youwillreceivenotificationsbyemailorSMSwhenmanagement,monitoring,orsecurityeventsoccurinaspecificclusterordumptask.
CreatingaSubscriptionStep1UsetheaccounttologintotheDISconsole.
Step2ClickEventManagement.
Step3OntheEventManagementpage,chooseSubscription>CreateSubscription.
Step4IntheSubscriptionSettingsarea,setbasicsubscriptioninformationandeventfiltering.
TheSubscribedEventListareadisplaystheeventsfilteredbythesystembasedonthesubscriptionsettings.
Table8-4SubscriptionparametersParameterDescriptionNotificationEnableordisableeventsubscription.
indicatesthatnotificationisenabledandindicatesthatnotificationisdisabled.
Bydefault,notificationisenabled.
Afternotificationisdisabled,thesystemstopssendingnotificationsofsubscribedeventsbutdoesnotdeletethesubscription.
DataIngestionServiceUserGuide8NotifyingEventsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
96ParameterDescriptionSubscriptionNameEnterthenameofasubscription.
Thenamecancontainletters(upperorlowercase),digits,hyphens(-),andunderscores(_)andmuststartwithaletterordigit.
Thenamemustbebetween1and256charactersinlength.
SubscriptionStreamSpecifieswhethertoenablesubscriptiontoalarmsofaspecifiedstream.
indicatesthatsubscriptiontoalarmsofaspecifiedstreamisenabled.
indicatesthatsubscriptiontoalarmsofaspecifiedchannelisdisabled.
Bydefault,thefunctionisdisabled.
Afterthisfunctionisenabled,thealarmssentfromyourspecifiedstreamarereceived,andsentfromotherstreamswillnotbereceived.
SubscriptionTypeTwosubscriptiontypesaresupported:SMNandDIS.
NOTEIfSubscriptionTypeissettoSMN,gotoStep6toselectanSMNtopic.
IfSubscriptionTypeissettoDIS,gotoStep5toselectastream.
Step5SelectaDISstream.
Step6SelectamessagenotificationtopicfromtheSMNTopicdrop-downlist.
Youcanperformthefollowingoperationstocreateamessagenotificationtopicasrequired.
1.
ClickCreateSMNTopic.
TheTopicspageofSMNisdisplayed.
YoucanclickCreateTopicintheupperrightcornertocreateatopic.
Fordetails,seesectionCreatingaTopicintheSimpleMessageNotificationUserGuide.
2.
IntheOperationcolumnofatopic,chooseMore>ConfigureTopicPolicyandselectDISunderServicesthatcanpublishmessagestothistopictoenableSMNtopublishDIStopics.
3.
Intherowcontainingthecreatedtopic,clickAddSubscriptiontoaddasubscriptiontothetopic.
Fordetails,seeAddingaSubscriptionintheSimpleMessageNotificationUserGuide.
Step7ClickSubmit.
----EndModifyingtheSubscriptionStep1OntheEventManagementpage,clickSubscription.
DataIngestionServiceUserGuide8NotifyingEventsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
97Step2IntheOperationcolumnofaspecifiedsubscriptionname,clickMore>Modify.
Step3OntheSubscriptionSettingspage,settheparametersthatyouwanttomodify.
Fordetails,seeStep4toStep7insectionCreatingaSubscription.
----EndDeletingtheSubscriptionStep1OntheEventManagementpage,clickSubscription.
Step2IntheOperationcolumnofaspecifiedsubscriptionname,clickMore>Delete.
TheDeleteSubscriptiondialogboxisdisplayed.
Figure8-1ConfirmingdeletionStep3ClickOK.
----End8.
3ViewingEventsThissectiondescribeshowtosearchforeventsthatoccurinastreamordumptask.
Step1Inthenavigationtree,chooseEventManagement>Events.
Alleventsaredisplayedbydefault.
Step2Selectdifferentfiltercriteriafromthedrop-downlistintheupperrightcorneroftheeventlisttosearchforevents.
Eventscanbefilteredbasedontheeventleveloreventsource.
IntheEventLevelsdrop-downlist,selectAll,Normal,orWarningIntheEventsourcedrop-downlist,selectEventsource.
Inthesearchbox,enterastreamnameoradumptaskname,forexample,demo.
Step3Click.
Thefilteredeventqueryresultsaredisplayed.
DataIngestionServiceUserGuide8NotifyingEventsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
98Step4ClickontherightofEventandselectaneventname,forexample,Dumptaskrestoredtofilterthecorrespondingevents.
----EndDataIngestionServiceUserGuide8NotifyingEventsIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
999Monitoring9.
1SupportedMetricsDescriptionThissectiondescribesmetricsreportedbyDIStoCloudEyeaswellastheirnamespacesanddimensions.
YoucanuseCloudEyetoquerymetricinformationgeneratedforDIS.
NamespaceSYS.
DAYUMetricsTable9-1liststheDISmetrics.
Table9-1DISmetricsMetricNameMeaningValueRangeMonitoredObjectMonitoringPeriod(RawData)TotalInputTrafficTheamountofdatauploadedthroughastreamduringaspecificperiod.
Unit:byte/s≥0bytes/sStream1minuteTotalOutputTrafficTheamountofdatadownloadedthroughastreamduringaspecificperiod.
Unit:byte/s≥0bytes/sStream1minuteDataIngestionServiceUserGuide9MonitoringIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
100MetricNameMeaningValueRangeMonitoredObjectMonitoringPeriod(RawData)TotalInputRecordsThenumberofrecordsuploadedthroughastreamduringaspecificperiod.
Unit:count/s≥0Count/sStream1minuteTotalOutputRecordsThenumberofrecordsdownloadedthroughastreamduringaspecificperiod.
Unit:count/s≥0Count/sStream1minuteSuccessfulUploadRequestsThenumberofsuccessfulrequestsforuploadingdatathroughastreamduringaspecificperiod.
Unit:count/s≥0Count/sStream1minuteSuccessfulDownloadRequestsThenumberofsuccessfulrequestsfordownloadingdatathroughastreamduringaspecificperiod.
Unit:count/s≥0Count/sStream1minuteAverageProcessingTimeofUploadRequestsAverageuploadrequestdelayofastreamduringaspecificperiod.
Unit:ms0to50msStream1minuteAverageProcessingTimeofDownloadRequestsAveragedownloadrequestdelayofastreamduringaspecificperiod.
Unit:ms0to50msStream1minuteThrottledUploadRequestsThenumberofrejecteduploadrequestsduetoflowcontrol.
Unit:count/s0to1Stream1minuteThrottledDownloadRequestsThenumberofrejecteddownloadrequestsduetoflowcontrol.
Unit:count/s0to1Stream1minuteDataIngestionServiceUserGuide9MonitoringIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
101DimensionKeyValuestream_idReal-timedataingestion9.
2SettingAlarmRulesScenarioYoucancustomizethemonitoredobjectsandnotificationpoliciesbysettingalarmrulestolearnDISrunningstatusinatimelymanner.
ADISalarmruleincludesthealarmrulename,monitoredobject,metric,threshold,monitoringinterval,andwhethertosendanotification.
ThissectiondescribeshowtosetDISalarmrules.
ProcedureStep1Logintothemanagementconsole.
Step2ChooseAllServices>Management&Deployment>CloudEye.
Step3Intheleftnavigationpane,chooseAlarmManagement>AlarmRules>CreateAlarmRule.
Step4SetalarmrulesfortheDISstreamasprompted.
Currently,onlycustomalarmrulesaresupported.
Step5Afterthesettingiscomplete,clickCreate.
Whenanalarmthatmeetstheruleisgenerated,thesystemautomaticallysendsanotification.
FormoreinformationaboutDISalarmrules,seetheCloudEyeUserGuide.
----End9.
3QueryingMetricsScenarioCloudEyemonitorstherunningstatusoftheDISstream.
YoucanobtainthemonitoringmetricsofCDMontheCloudEyemanagementconsole.
Monitoreddatarequiresaperiodoftimefortransmissionanddisplay.
ThestatusofCDMdisplayedontheCloudEyepageisthestatusobtained5to10minutesbefore.
YoucanviewthemonitoreddataofanewlycreatedDISstream5to10minuteslater.
DataIngestionServiceUserGuide9MonitoringIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
102PrerequisitesTheDISstreamisrunningproperly.
CloudEyewilldeleteadeletedstreamfromthemonitoringlistandwillnotmonitoritanymore.
However,youneedtomanuallyclearitsalarmrules.
AlarmruleshavebeenconfiguredontheCloudEyepage.
Fordetails,seeSettingAlarmRules.
ProcedureStep1UsetheaccounttologintotheDISconsole.
Step2Clickintheupperleftcornerofthepageandselectaregionandproject.
Step3Inthenavigationtreeontheleft,chooseStreamManagement.
Step4Inthestreamlist,clickthenameoftheDISstreamwhosemonitoringmetricswillbeviewed.
Themonitoringpageisdisplayed.
Step5OntheStreamstabpage,clickViewdetailstoswitchtotheCloudEyeconsole.
Step6OntheDISmonitoringpage,youcanviewthegraphsofallmonitoringmetrics.
Figure9-1ViewingDISmonitoringmetricsStep7Clickintheupperrightcornerofthegraphstozoomoutthegraphs.
Youcanviewtherawmonitoringdatacurvesofdifferentmetricsinthelast1hour,3hours,and12hours.
YoucandeterminewhethertoenableAutoRefresh.
CloudEyeprovidesanautomaticrefreshintervalof60seconds.
----EndDataIngestionServiceUserGuide9MonitoringIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
103AChangeHistoryReleaseDateWhat'sNew2020-06-24Thisissueisthethirdofficialrelease.
InterconnectedwithEnterpriseManagement.
Fordetails,seeManagingEnterpriseProjectsandCreatingaDISStream.
UpdatedtheIAMpermissionconfiguration.
Fordetails,seeIAMPermissionsManagement.
SupportedinterconnectwithCloudEye.
Fordetails,seeSupportedMetricstoViewingStreamMonitoringMetrics.
UpdatedtheIAMpermissionconfiguration.
Fordetails,seeCreatingaUserandGrantingPermissions.
Modifiedthedataretention.
Fordetails,seeCreatingaDISStreamandListingDISStreams.
2019-07-05Thisissueisthesecondofficialrelease.
Modifiedthepartitioncalculationmethod.
Fordetails,seeCreatingaDISStream.
SupportedELandbuilt-infunctionsinthedumppathconfiguration.
Fordetails,seeDumpingDatatoOBS.
2019-05-06Thisisthefirstofficialrelease.
DataIngestionServiceUserGuideAChangeHistoryIssue01(2020-07-07)CopyrightHuaweiTechnologiesCo.
,Ltd.
104