Windows上的Hadoop:不是有效的DFS文件名

2019年4月20日 6点热度 0条评论 来源: Tanuj B

我在Windows上配置了Hadoop 2.7.2,我可以看到名称节点,数据节点,资源管理器和节点管理器正常运行,当我尝试运行其中一个作为例子提供的地图缩小程序。Windows上的Hadoop:不是有效的DFS文件名

请找到下面的命令,我正在
c:\hdp\bin\yarn jar c:\hdp\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.7.2.jar wordcount c:\hdp\LICENSE.txt /out

我可以看到所有的文件都存在于所期望的位置。 请找到堆栈跟踪如下:

C:\WINDOWS\system32>c:\hdp\bin\yarn jar c:\hdp\share\hadoop\mapreduce\hadoop-map 
reduce-examples-2.7.2.jar wordcount c:\hdp\LICENSE.txt /out 
16/02/03 16:50:55 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0 
:8032 
16/02/03 16:50:56 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp 
/hadoop-yarn/staging/tbhakre/.staging/job_1454492091405_0006 
java.lang.IllegalArgumentException: Pathname /c:/hdp/LICENSE.txt from hdfs://0.0 
.0.0:19000/c:/hdp/LICENSE.txt is not a valid DFS filename. 
     at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedF 
ileSystem.java:197) 
     at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFi 
leSystem.java:106) 
     at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFil 
eSystem.java:1305) 
     at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFil 
eSystem.java:1301) 
     at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkRes 
olver.java:81) 
     at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(Distribute 
dFileSystem.java:1301) 
     at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) 
     at org.apache.hadoop.fs.Globber.glob(Globber.java:252) 
     at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1674) 
     at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedL 
istStatus(FileInputFormat.java:294) 
     at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File 
InputFormat.java:265) 
     at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI 
nputFormat.java:387) 
     at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter. 
java:301) 
     at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.jav 
a:318) 
     at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitt 
er.java:196) 
     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) 
     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at javax.security.auth.Subject.doAs(Subject.java:415) 
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma 
tion.java:1657) 
     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) 
     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) 
     at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. 
java:57) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces 
sorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:606) 
     at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(Progra 
mDriver.java:71) 
     at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) 
     at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. 
java:57) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces 
sorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:606) 
     at org.apache.hadoop.util.RunJar.run(RunJar.java:221) 
     at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 

===========解决方案如下:

LICENSE.TXT(对单词计数第一个参数)应设在HDFS,不是你的本地计算机上。

    原文作者:Tanuj B
    原文地址: https://stackoverflow.com/q/35176094
    本文转自网络文章,转载此文章仅为分享知识,如有侵权,请联系管理员进行删除。