解决running beyond virtual memory limits. Current usage: 35.5 MB of 1 GB physical memory used; 16.8 G-程序员宅基地

技术标签： jvm hadoop 集群大数据平台开发笔记（hadoop|storm|spark）

1、刚在公司搭建好的一个集群，然后运行wordcount测试看是否能正常使用，发现报如下错误（我在自己电脑上也是用同一版本，并没有报错）

[root@S1PA124 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.2.0.jar wordcount /input /output
14/08/20 09:51:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/08/20 09:51:35 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/08/20 09:51:36 INFO input.FileInputFormat: Total input paths to process : 1
14/08/20 09:51:36 INFO mapreduce.JobSubmitter: number of splits:1
14/08/20 09:51:36 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/08/20 09:51:36 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/08/20 09:51:36 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/08/20 09:51:37 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/08/20 09:51:37 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/08/20 09:51:37 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/08/20 09:51:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1408499127545_0001
14/08/20 09:51:37 INFO impl.YarnClientImpl: Submitted application application_1408499127545_0001 to ResourceManager at /0.0.0.0:8032
14/08/20 09:51:37 INFO mapreduce.Job: The url to track the job: http://S1PA124:8088/proxy/application_1408499127545_0001/
14/08/20 09:51:37 INFO mapreduce.Job: Running job: job_1408499127545_0001
14/08/20 09:51:44 INFO mapreduce.Job: Job job_1408499127545_0001 running in uber mode : false
14/08/20 09:51:44 INFO mapreduce.Job:  map 0% reduce 0%
14/08/20 09:51:49 INFO mapreduce.Job:  map 100% reduce 0%
14/08/20 09:51:54 INFO mapreduce.Job: Task Id : attempt_1408499127545_0001_r_000000_0, Status : FAILED
Container [pid=26042,containerID=container_1408499127545_0001_01_000003] is running beyond virtual memory limits. Current usage: 35.5 MB of 1 GB physical memory used; 16.8 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1408499127545_0001_01_000003 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 26047 26042 26042 26042 (java) 36 3 17963216896 8801 /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_0 3 
        |- 26042 25026 26042 26042 (bash) 0 0 65409024 276 /bin/bash -c /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_0 3 1>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000003/stdout 2>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000003/stderr  

Container killed on request. Exit code is 143

14/08/20 09:52:00 INFO mapreduce.Job: Task Id : attempt_1408499127545_0001_r_000000_1, Status : FAILED
Container [pid=26111,containerID=container_1408499127545_0001_01_000004] is running beyond virtual memory limits. Current usage: 100.3 MB of 1 GB physical memory used; 16.8 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1408499127545_0001_01_000004 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 26116 26111 26111 26111 (java) 275 8 18016677888 25393 /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_1 4 
        |- 26111 25026 26111 26111 (bash) 0 0 65409024 275 /bin/bash -c /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_1 4 1>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000004/stdout 2>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000004/stderr  

Container killed on request. Exit code is 143

14/08/20 09:52:06 INFO mapreduce.Job: Task Id : attempt_1408499127545_0001_r_000000_2, Status : FAILED
Container [pid=26185,containerID=container_1408499127545_0001_01_000005] is running beyond virtual memory limits. Current usage: 100.4 MB of 1 GB physical memory used; 16.8 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1408499127545_0001_01_000005 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 26190 26185 26185 26185 (java) 271 7 18025807872 25414 /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_2 5 
        |- 26185 25026 26185 26185 (bash) 0 0 65409024 276 /bin/bash -c /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_2 5 1>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000005/stdout 2>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000005/stderr  

Container killed on request. Exit code is 143

14/08/20 09:52:13 INFO mapreduce.Job:  map 100% reduce 100%
14/08/20 09:52:13 INFO mapreduce.Job: Job job_1408499127545_0001 failed with state FAILED due to: Task failed task_1408499127545_0001_r_000000
Job failed as tasks failed. failedMaps:0 failedReduces:1

14/08/20 09:52:13 INFO mapreduce.Job: Counters: 32
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=80425
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=895
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=3
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=0
        Job Counters 
                Failed reduce tasks=4
                Launched map tasks=1
                Launched reduce tasks=4
                Rack-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=3082
                Total time spent by all reduces in occupied slots (ms)=11065
        Map-Reduce Framework
                Map input records=56
                Map output records=56
                Map output bytes=1023
                Map output materialized bytes=1141
                Input split bytes=96
                Combine input records=56
                Combine output records=56
                Spilled Records=56
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=25
                CPU time spent (ms)=680
                Physical memory (bytes) snapshot=253157376
                Virtual memory (bytes) snapshot=18103181312
                Total committed heap usage (bytes)=1011875840
        File Input Format Counters 
                Bytes Read=799

2、mapred-site.xml配置文件配置如下

<configuration>
         <property>
                <name>mapreduce.cluster.local.dir</name>
                <value>/root/install/hadoop/mapred/local</value>
        </property>
        <property>
                <name>mapreduce.cluster.system.dir</name>
                <value>/root/install/hadoop/mapred/system</value>
        </property>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>S1PA124:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>S1PA124:19888</value>
        </property>
<!--
        <property>
                 <name>mapred.child.java.opts</name>
                 <value>-Djava.awt.headless=true</value>
        </property>
        <property>
                 <name>yarn.app.mapreduce.am.command-opts</name>
                 <value>-Djava.awt.headless=true -Xmx1024m</value>
        </property>
        <property>
                 <name>yarn.app.mapreduce.am.admin-command-opts</name>
                 <value>-Djava.awt.headless=true</value>
         </property>

-->
</configuration>

3、解决办法

我把mapred-site.xml配置文件里配置与JVM运行内存空间的那几行配置注释掉，然后重新启动集群就解决了。具体原因暂时还没有时间来研究，大概知道是与机器JVM的分配情况有关。

本文链接：https://blog.csdn.net/panguoyuan/article/details/38703111

原作者删帖不实内容删帖广告或垃圾文章投诉

智能推荐

leetcode 172. 阶乘后的零-程序员宅基地

文章浏览阅读63次。题目给定一个整数 n，返回 n! 结果尾数中零的数量。解题思路每个0都是由2 * 5得来的，相当于要求n!分解成质因子后2 * 5的数目，由于n中2的数目肯定是要大于5的数目，所以我们只需要求出n!中5的数目。C++代码class Solution {public: int trailingZeroes(int n) { ...

Day15-【Java SE进阶】IO流（一）：File、IO流概述、File文件对象的创建、字节输入输出流FileInputStream FileoutputStream、释放资源。_outputstream释放-程序员宅基地

文章浏览阅读992次，点赞27次，收藏15次。UTF-8是Unicode字符集的一种编码方案，采取可变长编码方案，共分四个长度区:1个字节，2个字节，3个字节，4个字节。文件字节输入流：每次读取多个字节到字节数组中去，返回读取的字节数量，读取完毕会返回-1。注意1:字符编码时使用的字符集，和解码时使用的字符集必须一致，否则会出现乱码。定义一个与文件一样大的字节数组，一次性读取完文件的全部字节。UTF-8字符集:汉字占3个字节，英文、数字占1个字节。GBK字符集:汉字占2个字节，英文、数字占1个字节。GBK规定:汉字的第一个字节的第一位必须是1。_outputstream释放

jeecgboot重新登录_jeecg 登录自动退出-程序员宅基地

文章浏览阅读1.8k次，点赞3次，收藏3次。解决jeecgboot每次登录进去都会弹出请重新登录问题，在utils文件下找到request.js文件注释这段代码即可_jeecg 登录自动退出

数据中心供配电系统负荷计算实例分析-程序员宅基地

文章浏览阅读3.4k次。我国目前普遍采用需要系数法和二项式系数法确定用电设备的负荷，其中需要系数法是国际上普遍采用的确定计算负荷的方法，最为简便;而二项式系数法在确定设备台数较少且各台设备容量差..._数据中心用电负荷统计变压器

HTML5期末大作业：网页制作代码网站设计——人电影网站(5页) HTML+CSS+JavaScript 学生DW网页设计作业成品 dreamweaver作业静态HTML网页设计模板_网页设计成品百度网盘-程序员宅基地

文章浏览阅读7k次，点赞4次，收藏46次。HTML5期末大作业：网页制作代码网站设计——人电影网站(5页) HTML+CSS+JavaScript 学生DW网页设计作业成品 dreamweaver作业静态HTML网页设计模板常见网页设计作业题材有个人、美食、公司、学校、旅游、电商、宠物、电器、茶叶、家居、酒店、舞蹈、动漫、明星、服装、体育、化妆品、物流、环保、书籍、婚纱、军事、游戏、节日、戒烟、电影、摄影、文化、家乡、鲜花、礼品、汽车、其他等网页设计题目, A+水平作业_网页设计成品百度网盘

【Jailhouse 文章】Look Mum, no VM Exits_jailhouse sr-iov-程序员宅基地

文章浏览阅读392次。jailhouse 文章翻译，Look Mum, no VM Exits!_jailhouse sr-iov

随便推点

chatgpt赋能python：Python怎么删除文件中的某一行_python 删除文件特定几行-程序员宅基地

文章浏览阅读751次。本文由chatgpt生成，文章没有在chatgpt生成的基础上进行任何的修改。以上只是chatgpt能力的冰山一角。作为通用的Aigc大模型，只是展现它原本的实力。对于颠覆工作方式的ChatGPT，应该选择拥抱而不是抗拒，未来属于“会用”AI的人。AI职场汇报智能办公文案写作效率提升教程专注于AI+职场+办公方向。下图是课程的整体大纲下图是AI职场汇报智能办公文案写作效率提升教程中用到的ai工具。_python 删除文件特定几行

Java过滤特殊字符的正则表达式_java正则表达式过滤特殊字符-程序员宅基地

文章浏览阅读2.1k次。【代码】Java过滤特殊字符的正则表达式。_java正则表达式过滤特殊字符

CSS中设置背景的7个属性及简写background注意点_background设置背景图片-程序员宅基地

文章浏览阅读5.7k次，点赞4次，收藏17次。css中背景的设置至关重要，也是一个难点，因为属性众多，对应的属性值也比较多，这里详细的列举了背景相关的7个属性及对应的属性值，并附上演示代码，后期要用的话，可以随时查看，那我们坐稳开车了······1: background-color 设置背景颜色2:background-image来设置背景图片- 语法：background-image:url(相对路径);-可以同时为一个元素指定背景颜色和背景图片，这样背景颜色将会作为背景图片的底色，一般情况下设置背景..._background设置背景图片

Win10 安装系统跳过创建用户，直接启用 Administrator_windows10msoobe进程-程序员宅基地

文章浏览阅读2.6k次，点赞2次，收藏8次。Win10 安装系统跳过创建用户，直接启用 Administrator_windows10msoobe进程

PyCharm2021安装教程-程序员宅基地

文章浏览阅读10w+次，点赞653次，收藏3k次。Windows安装pycharm教程新的改变功能快捷键合理的创建标题，有助于目录的生成如何改变文本的样式插入链接与图片如何插入一段漂亮的代码片生成一个适合你的列表创建一个表格设定内容居中、居左、居右SmartyPants创建一个自定义列表如何创建一个注脚注释也是必不可少的KaTeX数学公式新的甘特图功能，丰富你的文章UML 图表FLowchart流程图导出与导入导出导入下载安装PyCharm1、进入官网PyCharm的下载地址：http://www.jetbrains.com/pycharm/downl_pycharm2021

《跨境电商——速卖通搜索排名规则解析与SEO技术》一一1.1　初识速卖通的搜索引擎...-程序员宅基地

文章浏览阅读835次。本节书摘来自异步社区出版社《跨境电商——速卖通搜索排名规则解析与SEO技术》一书中的第1章，第1.1节，作者：冯晓宁，更多章节内容可以访问云栖社区“异步社区”公众号查看。1.1　初识速卖通的搜索引擎1.1.1　初识速卖通搜索作为速卖通卖家都应该知道，速卖通经常被视为“国际版的淘宝”。那么请想一下，普通消费者在淘宝网上购买商品的时候，他的行为应该..._跨境电商速卖通搜索排名规则解析与seo技术 pdf