搭建Mapreduce（YARN）环境

• 修改yarn-site.xml文件：

cd /usr/local/hadoop/etc/hadoop/
vim yarn-site.xml

如图：

具体内容：

<!-- Site specific YARN configuration properties -->
<configuration>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>
  </property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class </name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

其中名称yarn.resourcemanager.hostname指的是ResourceManager机器所在的节点位置；名称yarn.nodemanager.aux-services在hadoop2.2.0版本中是mapreduce_shuffle.

测试YARN环境

• 启动hdfs
  start-dfs.sh
• 启动hdfs
  Start-yarn.sh
  使用浏览器打开页面：

http://master:8088/
或http://192.168.50.100:8088/

测试Mapreduce环境（运行Wordcount示例）

find /usr/local/hadoop -name *example*.jar
 //查找示例程序,目录名需根据自己环境适当调整。

• 在HDFS上创建input目录
  hdfs dfs -mkdir input
• 在HDFS上创建output目录
  hdfs dfs -mkdir output
• 创建运行示例文档CountWord.txt并进行编写：
  

例：

helol
123
hello world
hello java
hello html
hello css
hello c
hello c#
hello world
hello css
hello c
hello c#
hello world
hello css
hello css
hello css
hello c
hello c#
hello world

hello c
hello c#
hello world
hello c
hello c#
hello world
hello css
hello c
hello c#
hello world
hello css
hello c

• 将CountWord.txt上传到HDFS中input目录下：

hdfs dfs -put CountWord.txt /input

• 查看：
  hdfs dfs -ls /input
  
• 运行Wordcount示例程序：
  输入为：/input/CountWord.txt ，运行结果输出目录为： /output

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input/CountWord.txt  /output

出现错误（FileAlreadyExistsException: Output directory hdfs://master:9000/output already exists）：

原因：
上次运行的输出目录也是output，没有进行删除。（好像每次运行的输出目录不能存在，运行过程中自动创建，若存在则运行失败，也有可能是我重复执行同一个文件的原因）
解决方法：
将/output目录删除:
hdfs dfs -rm -r /output

• 再次运行，成功：

• 

• 查看运行结果：
  hdfs dfs -text /output/part-r-00000

注意事项：

每次运行前，请确保运行结果的输出目录不存在

[root@master mapreduce]# hdfs dfs -rm -r /output

执行的txt文件是需要自己上传到HDFS后才能运行的。

[root@master ~]# hdfs dfs -put CountWord.txt /input

运行成功详细信息：

[root@master ~]# hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /inpu/tCountWord.txt  /output
21/06/08 11:42:49 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
21/06/08 11:42:49 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
21/06/08 11:42:50 INFO mapreduce.JobSubmitter: Cleaning up the staging area file:/var/hadoop/mapred/staging/root284026624/.staging/job_local284026624_0001
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://master:9000/inpu/tCountWord.txt
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:323)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:265)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:387)
	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301)
	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
	at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
	at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
	at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
[root@master ~]# hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input/CountWord.txt  /output
21/06/08 11:43:35 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
21/06/08 11:43:35 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
21/06/08 11:43:36 INFO input.FileInputFormat: Total input paths to process : 1
21/06/08 11:43:36 INFO mapreduce.JobSubmitter: number of splits:1
21/06/08 11:43:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local764084505_0001
21/06/08 11:43:36 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
21/06/08 11:43:36 INFO mapreduce.Job: Running job: job_local764084505_0001
21/06/08 11:43:36 INFO mapred.LocalJobRunner: OutputCommitter set in config null
21/06/08 11:43:36 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
21/06/08 11:43:36 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
21/06/08 11:43:36 INFO mapred.LocalJobRunner: Waiting for map tasks
21/06/08 11:43:36 INFO mapred.LocalJobRunner: Starting task: attempt_local764084505_0001_m_000000_0
21/06/08 11:43:37 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
21/06/08 11:43:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
21/06/08 11:43:37 INFO mapred.MapTask: Processing split: hdfs://master:9000/input/CountWord.txt:0+426
21/06/08 11:43:37 INFO mapreduce.Job: Job job_local764084505_0001 running in uber mode : false
21/06/08 11:43:37 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
21/06/08 11:43:37 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
21/06/08 11:43:37 INFO mapred.MapTask: soft limit at 83886080
21/06/08 11:43:37 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
21/06/08 11:43:37 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
21/06/08 11:43:37 INFO mapreduce.Job:  map 0% reduce 0%
21/06/08 11:43:37 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
21/06/08 11:43:38 INFO mapred.LocalJobRunner: 
21/06/08 11:43:38 INFO mapred.MapTask: Starting flush of map output
21/06/08 11:43:38 INFO mapred.MapTask: Spilling map output
21/06/08 11:43:38 INFO mapred.MapTask: bufstart = 0; bufend = 715; bufvoid = 104857600
21/06/08 11:43:38 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214080(104856320); length = 317/6553600
21/06/08 11:43:38 INFO mapred.MapTask: Finished spill 0
21/06/08 11:43:38 INFO mapred.Task: Task:attempt_local764084505_0001_m_000000_0 is done. And is in the process of committing
21/06/08 11:43:38 INFO mapred.LocalJobRunner: map
21/06/08 11:43:38 INFO mapred.Task: Task 'attempt_local764084505_0001_m_000000_0' done.
21/06/08 11:43:38 INFO mapred.LocalJobRunner: Finishing task: attempt_local764084505_0001_m_000000_0
21/06/08 11:43:38 INFO mapred.LocalJobRunner: map task executor complete.
21/06/08 11:43:38 INFO mapred.LocalJobRunner: Waiting for reduce tasks
21/06/08 11:43:38 INFO mapred.LocalJobRunner: Starting task: attempt_local764084505_0001_r_000000_0
21/06/08 11:43:38 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
21/06/08 11:43:38 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
21/06/08 11:43:38 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5eb258cc
21/06/08 11:43:38 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
21/06/08 11:43:38 INFO reduce.EventFetcher: attempt_local764084505_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
21/06/08 11:43:39 INFO mapreduce.Job:  map 100% reduce 0%
21/06/08 11:43:39 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local764084505_0001_m_000000_0 decomp: 97 len: 101 to MEMORY
21/06/08 11:43:39 INFO reduce.InMemoryMapOutput: Read 97 bytes from map-output for attempt_local764084505_0001_m_000000_0
21/06/08 11:43:39 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 97, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->97
21/06/08 11:43:39 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
21/06/08 11:43:39 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
21/06/08 11:43:39 INFO mapred.LocalJobRunner: 1 / 1 copied.
21/06/08 11:43:39 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
21/06/08 11:43:39 INFO mapred.Merger: Merging 1 sorted segments
21/06/08 11:43:39 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 91 bytes
21/06/08 11:43:39 INFO reduce.MergeManagerImpl: Merged 1 segments, 97 bytes to disk to satisfy reduce memory limit
21/06/08 11:43:39 INFO reduce.MergeManagerImpl: Merging 1 files, 101 bytes from disk
21/06/08 11:43:39 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
21/06/08 11:43:39 INFO mapred.Merger: Merging 1 sorted segments
21/06/08 11:43:39 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 91 bytes
21/06/08 11:43:39 INFO mapred.LocalJobRunner: 1 / 1 copied.
21/06/08 11:43:39 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
21/06/08 11:43:39 INFO mapred.Task: Task:attempt_local764084505_0001_r_000000_0 is done. And is in the process of committing
21/06/08 11:43:39 INFO mapred.LocalJobRunner: 1 / 1 copied.
21/06/08 11:43:39 INFO mapred.Task: Task attempt_local764084505_0001_r_000000_0 is allowed to commit now
21/06/08 11:43:39 INFO output.FileOutputCommitter: Saved output of task 'attempt_local764084505_0001_r_000000_0' to hdfs://master:9000/output/_temporary/0/task_local764084505_0001_r_000000
21/06/08 11:43:39 INFO mapred.LocalJobRunner: reduce > reduce
21/06/08 11:43:39 INFO mapred.Task: Task 'attempt_local764084505_0001_r_000000_0' done.
21/06/08 11:43:39 INFO mapred.LocalJobRunner: Finishing task: attempt_local764084505_0001_r_000000_0
21/06/08 11:43:39 INFO mapred.LocalJobRunner: reduce task executor complete.
21/06/08 11:43:40 INFO mapreduce.Job:  map 100% reduce 100%
21/06/08 11:43:40 INFO mapreduce.Job: Job job_local764084505_0001 completed successfully
21/06/08 11:43:40 INFO mapreduce.Job: Counters: 35
	File System Counters
		FILE: Number of bytes read=592184
		FILE: Number of bytes written=1149995
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=852
		HDFS: Number of bytes written=61
		HDFS: Number of read operations=13
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=4
	Map-Reduce Framework
		Map input records=44
		Map output records=80
		Map output bytes=715
		Map output materialized bytes=101
		Input split bytes=103
		Combine input records=80
		Combine output records=9
		Reduce input groups=9
		Reduce shuffle bytes=101
		Reduce input records=9
		Reduce output records=9
		Spilled Records=18
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=46
		Total committed heap usage (bytes)=242360320
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=426
	File Output Format Counters 
		Bytes Written=61
[root@master ~]# hdfs dfs -text /output/part-r-00000

123	1
c	9
c#	9
css	9
hello	39
helol	1
html	1
java	1
world	10