合并查看源代码
查看源代码 时间:2021-03-21 阅读:(
)
Hadoop元数据合并异常及解决方法这几天观察了一下StandbyNN上面的日志,发现每次Fsimage合并完之后,StandbyNN通知ActiveNN来下载合并好的Fsimage的过程中会出现以下的异常信息:2014-04-2314:42:54,964ERRORorg.
apache.
hadoop.
hdfs.
server.
namenode.
ha.
StandbyCheckpointer:ExceptionindoCheckpointjava.
net.
SocketTimeoutException:Readtimedoutatjava.
net.
SocketInputStream.
socketRead0(NativeMethod)atjava.
net.
SocketInputStream.
read(SocketInputStream.
java:152)atjava.
net.
SocketInputStream.
read(SocketInputStream.
java:122)atjava.
io.
BufferedInputStream.
fill(BufferedInputStream.
java:235)atjava.
io.
BufferedInputStream.
read1(BufferedInputStream.
java:275)atjava.
io.
BufferedInputStream.
read(BufferedInputStream.
java:334)atsun.
net.
www.
http.
HttpClient.
parseHTTPHeader(HttpClient.
java:687)atsun.
net.
www.
http.
HttpClient.
parseHTTP(HttpClient.
java:633)atsun.
net.
www.
protocol.
http.
HttpURLConnection.
getInputStream(HttpURLConnection.
java:1323)atjava.
net.
HttpURLConnection.
getResponseCode(HttpURLConnection.
java:468)atorg.
apache.
hadoop.
hdfs.
server.
namenode.
TransferFsImage.
doGetUrl(TransferFsImage.
java:268)atorg.
apache.
hadoop.
hdfs.
server.
namenode.
TransferFsImage.
getFileClient(TransferFsImage.
java:247)atorg.
apache.
hadoop.
hdfs.
server.
namenode.
TransferFsImage.
uploadImageFromStorage(TransferFsImage.
java:162)atorg.
apache.
hadoop.
hdfs.
server.
namenode.
ha.
StandbyCheckpointer.
doCheckpoint(StandbyCheckpointer.
java:174)atorg.
apache.
hadoop.
hdfs.
server.
namenode.
ha.
StandbyCheckpointer.
access$1100(StandbyCheckpointer.
java:53)atorg.
apache.
hadoop.
hdfs.
server.
namenode.
ha.
StandbyCheckpointer$CheckpointerThread.
doWork(StandbyCheckpointer.
java:297)atorg.
apache.
hadoop.
hdfs.
server.
namenode.
ha.
StandbyCheckpointer$CheckpointerThread.
access$300(StandbyCheckpointer.
java:210)atorg.
apache.
hadoop.
hdfs.
server.
namenode.
ha.
StandbyCheckpointer$CheckpointerThread$1.
run(StandbyCheckpointer.
java:230)atorg.
apache.
hadoop.
security.
SecurityUtil.
doAsLoginUserOrFatal(SecurityUtil.
java:456)atorg.
apache.
hadoop.
hdfs.
server.
namenode.
ha.
StandbyCheckpointer$CheckpointerThread.
run(StandbyCheckpointer.
java:226)1/5上面的代码贴出来有点乱啊,可以看下下面的图片截图:StandbyCheckpointer于是习惯性的去Google了一下,找了好久也没找到类似的信息.
只能自己解决.
我们通过分析日志发现更奇怪的问题,上次Checkpoint的时间一直都不变(一直都是StandbyNN启动的时候第一次Checkpoint的时间),如下:2014-04-2314:50:54,429INFOorg.
apache.
hadoop.
hdfs.
server.
namenode.
ha.
StandbyCheckpointer:Triggeringcheckpointbecauseithasbeen70164secondssincethelastcheckpoint,whichexceedstheconfiguredinterval600难道这是Hadoop的bug于是我就根据上面的错误信息去查看源码,经过仔细的分析,发现上述的问题都是由StandbyCheckpointer类输出的:privatevoiddoWork(){//Resetcheckpointtimesothatwedon'talwayscheckpoint//onstartup.
lastCheckpointTime=now();while(shouldRun){try{Thread.
sleep(1000*checkpointConf.
getCheckPeriod());}catch(InterruptedExceptionie){}if(!
shouldRun){break;}try{//Wemayhavelostourticketsincelastcheckpoint,loginagain,//justincaseif(UserGroupInformation.
isSecurityEnabled()){UserGroupInformation.
getCurrentUser().
checkTGTAndReloginFromKeytab();}longnow=now();longuncheckpointed=countUncheckpointedTxns();longsecsSinceLast=(now-lastCheckpointTime)/1000;2/5booleanneedCheckpoint=false;if(uncheckpointed>=checkpointConf.
getTxnCount()){LOG.
info("Triggeringcheckpointbecausetherehavebeen"+uncheckpointed+"txnssincethelastcheckpoint,which"+"exceedstheconfiguredthreshold"+checkpointConf.
getTxnCount());needCheckpoint=true;}elseif(secsSinceLast>=checkpointConf.
getPeriod()){LOG.
info("Triggeringcheckpointbecauseithasbeen"+secsSinceLast+"secondssincethelastcheckpoint,which"+"exceedstheconfiguredinterval"+checkpointConf.
getPeriod());needCheckpoint=true;}synchronized(cancelLock){if(now0){connection.
setConnectTimeout(timeout);connection.
setReadTimeout(timeout);}if(connection.
getResponseCode()!
=HttpURLConnection.
HTTP_OK){thrownewHttpGetFailedException("Imagetransferservletat"+url+"failedwithstatuscode"+connection.
getResponseCode()+"\nResponsemessage:\n"+connection.
getResponseMessage(),connection);}DFS_IMAGE_TRANSFER_TIMEOUT_KEY这个时间是由dfs.
image.
transfer.
timeout参数所设置的,默认值为10*60*1000,单位为毫秒.
然后我看了一下这个属性的解释:Timeoutforimagetransferinmilliseconds.
Thistimeoutandtherelateddfs.
image.
transfer.
bandwidthPerSecparametershouldbeconfiguredsuchthatnormalimagetransfercancompletewithinthetimeout.
Thistimeoutpreventsclienthangswhenthesender4/5failsduringimagetransfer,whichisparticularlyimportantduringcheckpointing.
Notethatthistimeoutappliestotheentiretyofimagetransfer,andisnotasockettimeout.
这才发现问题,这个参数的设置和dfs.
image.
transfer.
bandwidthPerSec息息相关,要保证ActiveNN在dfs.
image.
transfer.
timeout时间内把合并好的Fsimage从StandbyNN上下载完,要不然会出现异常.
然后我看了一下我的配置dfs.
image.
transfer.
timeout60000dfs.
image.
transfer.
bandwidthPerSec104857660秒超时,一秒钟拷贝1MB,而我的集群上的元数据有800多MB,显然是不能在60秒钟拷贝完,后来我把dfs.
image.
transfer.
timeout设置大了,观察了一下,集群再也没出现过上述异常信息,而且以前的一些异常信息也由于这个而解决了.
.
本博客文章除特别声明,全部都是原创!
原创文章版权归过往记忆大数据(过往记忆)所有,未经许可不得转载.
本文链接:【】()PoweredbyTCPDF(www.
tcpdf.
org)5/5
官方网站:点击访问火数云活动官网活动方案:CPU内存硬盘带宽流量架构IP机房价格购买地址4核4G50G 高效云盘20Mbps独享不限openstack1个九江287元/月立即抢购4核8G50G 高效云盘20Mbps独享不限openstack1个九江329元/月立即抢购2核2G50G 高效云盘5Mbps独享不限openstack1个大连15.9元/月立即抢购2核4G50G 高效云盘5Mbps独享不限...
racknerd发表了2021年美国独立日的促销费用便宜的vps,两种便宜的美国vps位于洛杉矶multacom室,访问了1Gbps的带宽,采用了solusvm管理,硬盘是SSDraid10...近两年来,racknerd的声誉不断积累,服务器的稳定性和售后服务。官方网站:https://www.racknerd.com多种加密数字货币、信用卡、PayPal、支付宝、银联、webmoney,可以付...
greencloudvps怎么样?greencloudvps是一家国外主机商,VPS数据中心多,之前已经介绍过多次了。现在有几款10Gbps带宽的特价KVM VPS,Ryzen 3950x处理器,NVMe硬盘,性价比高。支持Paypal、支付宝、微信付款。GreenCloudVPS:新加坡/美国/荷兰vps,1核@Ryzen 3950x/1GB内存/30GB NVMe空间/1TB流量/10Gbps...
查看源代码为你推荐
有机zz怎么看不了呢有机zz怎么进不去了kaixin.com开心网。www.kaixin.com是什么时间出来的。老虎数码我想买个一千左右的数码相机!最好低于一千五!再给我说一下像素是多少?www.765.com下载小说地址www.7788k.comwww.6601txq.com.有没有这个网站www.03ggg.comwww.tvb33.com这里好像有中国性戏观看吧??www.bbb551.com广州欢乐在线551要收费吗?www.6vhao.com有哪些电影网站www.henhenlu.com有一个两位数,十位数字是个位数字的二分之一,将十位数字与个位数字对调,新的两位数比原来大36,这个两位数www.45gtv.com登录农行网银首页www.abchina.com,
directspace 10t等于多少g iisphpmysql 42u标准机柜尺寸 ssh帐号 华为云主机 hnyd 新天域互联 北京双线机房 789电视网 国外代理服务器地址 可外链相册 免费全能主机 河南移动网 免费phpmysql空间 鲁诺 idc查询 爱奇艺会员免费试用 drupal安装 超级服务器 更多