hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-6963) Fix FileUtil.getDU. It should not include the size of the directory or follow symbolic links
Date Fri, 06 Apr 2012 12:43:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248274#comment-13248274

Hudson commented on HADOOP-6963:

Integrated in Hadoop-Hdfs-0.23-Build #219 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/219/])
    HADOOP-6963. In FileUtil.getDU(..), neither include the size of directories nor follow
symbolic links.  (Ravi Prakash via bobby) (Revision 1310018)

     Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310018
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

> Fix FileUtil.getDU. It should not include the size of the directory or follow symbolic
> --------------------------------------------------------------------------------------------
>                 Key: HADOOP-6963
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6963
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions:, 0.23.1
>            Reporter: Owen O'Malley
>            Assignee: Ravi Prakash
>            Priority: Critical
>             Fix For: 1.0.3, 0.23.3
>         Attachments: HADOOP-6963.branch-1.0.2.patch, HADOOP-6963.branch-1.0.2.patch,
HADOOP-6963.branch-1.patch, HADOOP-6963.branch-23.patch, HADOOP-6963.branch-23.patch, HADOOP-6963.branch-23.patch
> The getDU method should not include the size of the directory. The Java interface says
that the value is undefined and in Linux/Sun it gets the 4096 for the inode. Clearly this
isn't useful.
> It also recursively calls itself. In case the directory has a symbolic link forming a
cycle, getDU keeps spinning in the cycle. In our case, we saw this in the org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects
call. This prevented other tasks on the same node from committing, causing the TT to become
effectively useless (because the JT thinks it already has enough tasks running)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message