hadoop - Deleting files from HDFS does not free up disk space -

-

after upgrading our small cloudera hadoop cluster cdh 5, deleting files no longer frees available storage space. though delete more data add, file system keeps filling up.

cluster setup

we running 4 node cluster on physical, dedicated hardware, 110 tb of total storage capacity. on april 3, upgraded cdh software 5.0.0-beta2 version version 5.0.0-1.

we used put log data on hdfs in plain text format @ rate of approximately 700 gb/day. on april 1 switched importing data .gz files instead, lowered daily ingestion rate 130 gb.

since want retain data age, there nightly job delete obsolete files. result of used visible in hdfs capacity monitoring chart, can no longer seen.

sine import 570 gb less data delete every day, 1 expect capacity used go down. instead our reported hdfs use has been growing since cluster software upgraded.

problem description

running hdfs hadoop fs -du -h / gives following output:

0       /system 1.3 t   /tmp 24.3 t  /user

this consistent expect see, given size of imported files. using replication factor of 3, should correspond physical disk usage of 76.8 tb.

when instead running hdfs dfsadmin -report result different:

configured capacity: 125179101388800 (113.85 tb) present capacity: 119134820995005 (108.35 tb) dfs remaining: 10020134191104 (9.11 tb) dfs used: 109114686803901 (99.24 tb) dfs used%: 91.59% under replicated blocks: 0 blocks corrupt replicas: 0 missing blocks: 0

here, dfs used reported 99.24 tb, see in monitoring chart. did data come from?

what have tried

the first thing suspected automatic emptying of trash not working, not seem case. deleted files in trash, , automatically disappear after day.

our issue seem similar happen if hdfs metadata upgrade performed not finalized. don't think needed when upgrading between these versions, have still performed both steps 'just in case'.

on dn storage volumes in local file system, there lot of data under `previous/finalized'. have little knowledge of implementation details of hdsf know if significant, indicate finalization out of synch.

we run out of disk space on cluster, appreciated.

i found similar issue on our cluster, stemmed failed upgrade.

first make sure finalize upgrade on namenode

hdfs dfsadmin -finalizeupgrade

what found datanodes reason did not finalize directories @ all.

on datanode, should see following directory layout

/[mountpoint}/dfs/dn/current/{blockpool}/current

and

/[mountpoint}/dfs/dn/current/{blockpool}/previous

if have not finalized previous directory contains data created before update. if delete not remove - hence storage never reduces.

actually simplest solution sufficient

restart namenode

watch log of datanode, should see this

info org.apache.hadoop.hdfs.server.common.storage: finalizing upgrade storage directory

afterwards directories cleared in background , storage reclaimed.


Comments

Post a Comment

Popular posts from this blog

windows - Single EXE to Install Python Standalone Executable for Easy Distribution -

-
i used pyinstaller create standalone exe file pyqt project. did not use single file option because exe created single file option takes long open. therefore, pyinstaller created dist folder contains bunch of files including program.exe file. create setup.exe file place contents of dist folder in program directory , create shortcut in start menu , on desktop. want super simple user. maybe setup.exe files when download program cnet. found inno-setup, looks promising. however, not know if there special considerations because program standalone program create python/pyqt program. thanks! have experience task? there program using task not know about? inno-setup or nsis easiest use. tell them files include , put them , create nice installer you. wrote short tutorial on experiences using innosetup might find helpful: http://www.blog.pythonlibrary.org/2008/08/27/packaging-wxpymail-for-distribution/ note tutorial based around wxpython app wrote, concepts same.
Read more

c# - Access objects in UserControl from MainWindow in WPF -

-
i have simple xaml, how can change text property of textblock in usercontrol1 mainwindow ? <window x:class="refactorxaml.mainwindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:refactorxaml="clr-namespace:refactorxaml" title="mainwindow" height="350" width="525"> <grid> <grid.rowdefinitions> <rowdefinition height="100"></rowdefinition> <rowdefinition height="100"></rowdefinition> </grid.rowdefinitions> <grid> <refactorxaml:usercontrol1></refactorxaml:usercontrol1> </grid> <grid grid.row="1"> <button>change text</button> </grid> </grid> </window> <usercon...
Read more

javascript - How to name a jQuery function to make a browser's back button work? -

-
i'm trying write function ajaxyfy web site, works perfect, except function button. that's have now: function loading() { if (typeof history.pushstate !== "undefined") { var historycount = 0; $('.menu-item, .logo').on('click','a',function(e){ e.preventdefault(); $(this).data('clicked', true); if ($(this).is('.logo a')) { var homepage = $(this).attr('href'); function1(homepage); history.pushstate(null, null, homepage); } else if ($(this).is('.projects a')) { var projects = $(this).attr('href'); function2(projects); history.pushstate(null, null, projects); } else { var pages = $(this).attr('href'); function3(pages); history.pushstate(null, null, pages); ...
Read more