第一步：安装jdk

由于hadoop是java开发的，所以需要JDK来运行代码。这里安装的是jdk1.6.

jdk的安装见

第二步：创建独立的用户

useradd hadooppasswd hadoop

有些机器不能设置空密码的时候

passwd -d hadoop

这里的用户名为hadoop，如果你要调试的时候要注意名字。

比如我用windows调试linux的集群，这个名字应该是windows系统的用户名（否则你没有权限提交作业到hadoop）。

第三步：设置用户无密码登陆

su - hadoopssh-keygen -t rsacat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keyschmod 0600 ~/.ssh/authorized_keysexit

第四步：下载hadoop

mkdir /opt/hadoopcd /opt/hadoop/wget http://apache.mesi.com.ar/hadoop/common/hadoop-1.2.0/hadoop-1.2.0.tar.gztar -xzf hadoop-1.2.0.tar.gzmv hadoop-1.2.0 hadoopchown -R hadoop /opt/hadoopcd /opt/hadoop/hadoop/

第五步：配置hadoop

vi conf/core-site.xml

       
      
       hadoop.tmp.dir
        
      
       /app/hadoop/tmp
        
      
       A base for other temporary directories.
      
     
       
      
       fs.default.name
        
      
       hdfs://10.53.132.52:54310
        
      
       The name of the default file system.  A URI whose  scheme and authority determine the FileSystem implementation.  The  uri's scheme determines the config property (fs.SCHEME.impl) naming  the FileSystem implementation class.  The uri's authority is used to  determine the host, port, etc. for a filesystem.
      
     
         
      
       dfs.permissions
          
      
       false
      
     

 

vi conf/hdfs-site.xml

       
      
       dfs.replication
        
      
       1
        
      
       Default block replication.  The actual number of replications can be specified when the file is created.  The default is used if replication is not specified in create time.  
      
     

　　

vi conf/mapred-site.xml

       
      
       mapred.job.tracker
        
      
       10.53.132.52:54311
        
      
       The host and port that the MapReduce job tracker runs  at.  If "local", then jobs are run in-process as a single map  and reduce task.  
      
     

 

第六步：开启hadoop

bin/hadoop namenode -format

bin/start-all.sh

关闭是

bin/stop-all.sh

验证开启是

jps

26049 SecondaryNameNode25929 DataNode26399 Jps26129 JobTracker26249 TaskTracker25807 NameNode

 

第七步：下载并设置eclipse的hadoop插件。

插件文件是：hadoop-eclipse-plugin-1.2.0.jar

放到eclipse的plugins目录下即可。

 

第八步：打开eclipse创建map/reduce项目。

修改map/reduce和hdfs的地址和端口

 

第九步：调试hadoop

package org.apache.hadoop.examples;import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class WordCount {	public static class TokenizerMapper	extends Mapper
     
      {		private final static IntWritable one = new IntWritable(1);		private Text word = new Text();		public void map(Object key, Text value, Context context)		throws IOException, InterruptedException {			StringTokenizer itr = new StringTokenizer(value.toString());            while (itr.hasMoreTokens()) {            	word.set(itr.nextToken());            	context.write(word, one);            	}		}	}	  	public static class IntSumReducer	extends Reducer
      
        {		private IntWritable result = new IntWritable();		public void reduce(Text key, Iterable
       
         values,Context context)		throws IOException, InterruptedException {			int sum = 0;			for (IntWritable val : values) {				sum += val.get();				}			result.set(sum);			context.write(key, result);		}	}	  	  public static void main(String[] args) throws Exception {		    Configuration conf = new Configuration();		    conf.set("mapred.job.tracker", "10.53.132.52:54311");		    		    //conf.addResource(new Path("\\soft\\hadoop\\conf\\core-site.xml"));		    //conf.addResource(new Path("\\soft\\hadoop\\conf\\hdfs-site.xml"));		    		    String[] ars=new String[]{"input","output"};		    String[] otherArgs = new GenericOptionsParser(conf, ars).getRemainingArgs();		    if (otherArgs.length != 2) {		      System.err.println("Usage: wordcount  ");		      System.exit(2);		    }		    Job job = new Job(conf, "word count");		    job.setJarByClass(WordCount.class);		    job.setMapperClass(TokenizerMapper.class);		    job.setCombinerClass(IntSumReducer.class);		    job.setReducerClass(IntSumReducer.class);		    job.setOutputKeyClass(Text.class);		    job.setOutputValueClass(IntWritable.class);		    FileInputFormat.addInputPath(job, new Path(otherArgs[0]));		    FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));		    System.exit(job.waitForCompletion(true) ? 0 : 1);	  }}
       
      
     

（这里是吧作业提交到远端的hadoop）

调试

 

结果

13/09/17 17:50:32 INFO input.FileInputFormat: Total input paths to process : 213/09/17 17:50:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable13/09/17 17:50:33 WARN snappy.LoadSnappy: Snappy native library not loaded13/09/17 17:50:33 INFO mapred.JobClient: Running job: job_201309171747_000213/09/17 17:50:34 INFO mapred.JobClient:  map 0% reduce 0%13/09/17 17:50:39 INFO mapred.JobClient:  map 100% reduce 0%13/09/17 17:50:47 INFO mapred.JobClient:  map 100% reduce 33%13/09/17 17:50:48 INFO mapred.JobClient:  map 100% reduce 100%13/09/17 17:50:49 INFO mapred.JobClient: Job complete: job_201309171747_000213/09/17 17:50:49 INFO mapred.JobClient: Counters: 2913/09/17 17:50:49 INFO mapred.JobClient:   Job Counters 13/09/17 17:50:49 INFO mapred.JobClient:     Launched reduce tasks=113/09/17 17:50:49 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=611513/09/17 17:50:49 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=013/09/17 17:50:49 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=013/09/17 17:50:49 INFO mapred.JobClient:     Launched map tasks=213/09/17 17:50:49 INFO mapred.JobClient:     Data-local map tasks=213/09/17 17:50:49 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=870213/09/17 17:50:49 INFO mapred.JobClient:   File Output Format Counters 13/09/17 17:50:49 INFO mapred.JobClient:     Bytes Written=4113/09/17 17:50:49 INFO mapred.JobClient:   FileSystemCounters13/09/17 17:50:49 INFO mapred.JobClient:     FILE_BYTES_READ=7913/09/17 17:50:49 INFO mapred.JobClient:     HDFS_BYTES_READ=28613/09/17 17:50:49 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=17401513/09/17 17:50:49 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=4113/09/17 17:50:49 INFO mapred.JobClient:   File Input Format Counters 13/09/17 17:50:49 INFO mapred.JobClient:     Bytes Read=5013/09/17 17:50:49 INFO mapred.JobClient:   Map-Reduce Framework13/09/17 17:50:49 INFO mapred.JobClient:     Map output materialized bytes=8513/09/17 17:50:49 INFO mapred.JobClient:     Map input records=213/09/17 17:50:49 INFO mapred.JobClient:     Reduce shuffle bytes=8513/09/17 17:50:49 INFO mapred.JobClient:     Spilled Records=1213/09/17 17:50:49 INFO mapred.JobClient:     Map output bytes=8213/09/17 17:50:49 INFO mapred.JobClient:     Total committed heap usage (bytes)=60299673613/09/17 17:50:49 INFO mapred.JobClient:     CPU time spent (ms)=202013/09/17 17:50:49 INFO mapred.JobClient:     Combine input records=813/09/17 17:50:49 INFO mapred.JobClient:     SPLIT_RAW_BYTES=23613/09/17 17:50:49 INFO mapred.JobClient:     Reduce input records=613/09/17 17:50:49 INFO mapred.JobClient:     Reduce input groups=513/09/17 17:50:49 INFO mapred.JobClient:     Combine output records=613/09/17 17:50:49 INFO mapred.JobClient:     Physical memory (bytes) snapshot=55517593613/09/17 17:50:49 INFO mapred.JobClient:     Reduce output records=513/09/17 17:50:49 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=192679936013/09/17 17:50:49 INFO mapred.JobClient:     Map output records=8