Hadoop 3.2.2 + JDK 21 Windows 开发环境搭建：从零到运行首个MapReduce作业

Hadoop 3.2.2 JDK 21 Windows开发环境全流程实战指南对于需要在Windows环境下进行大数据开发的Java/Scala开发者来说搭建一个本地Hadoop开发环境是入门的第一步。本文将带你从零开始完成Hadoop 3.2.2与JDK 21的环境搭建并实现一个完整的MapReduce作业开发流程。1. 环境准备与安装1.1 JDK 21安装配置首先需要安装JDK 21作为Hadoop运行的基础环境从Oracle官网下载JDK 21 Windows x64安装包运行安装程序建议安装路径不要包含空格如C:\Java\jdk-21配置系统环境变量新建JAVA_HOME变量值为JDK安装路径在Path变量中添加%JAVA_HOME%\bin验证安装是否成功java -version应显示类似java version 21的版本信息。1.2 Hadoop 3.2.2安装Hadoop原生是为Linux设计的在Windows上运行需要额外配置从Apache官网下载hadoop-3.2.2.tar.gz解压到不含空格的目录如D:\hadoop-3.2.2下载Windows专用工具从GitHub获取winutils.exe和hadoop.dll放入Hadoop的bin目录配置Hadoop环境变量新建HADOOP_HOME指向安装目录在Path中添加%HADOOP_HOME%\bin2. Hadoop核心配置2.1 配置文件修改进入%HADOOP_HOME%\etc\hadoop目录修改以下关键配置文件core-site.xml- 定义HDFS默认地址configuration property namefs.defaultFS/name valuehdfs://localhost:9000/value /property /configurationhdfs-site.xml- 配置HDFS参数configuration property namedfs.replication/name value1/value /property property namedfs.namenode.name.dir/name value/D:/hadoop/data/namenode/value /property property namedfs.datanode.data.dir/name value/D:/hadoop/data/datanode/value /property /configurationmapred-site.xml- 指定使用YARN框架configuration property namemapreduce.framework.name/name valueyarn/value /property /configurationyarn-site.xml- 配置YARN参数configuration property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property /configuration2.2 初始化与启动格式化NameNodehdfs namenode -format启动HDFS服务start-dfs.cmd启动YARN服务start-yarn.cmd验证服务是否正常运行HDFS Web界面http://localhost:9870YARN Web界面http://localhost:80883. 开发首个MapReduce作业3.1 WordCount示例代码创建一个标准的Maven项目添加Hadoop依赖dependency groupIdorg.apache.hadoop/groupId artifactIdhadoop-client/artifactId version3.2.2/version /dependencyMapper实现public class WordCountMapper extends MapperLongWritable, Text, Text, IntWritable { private final static IntWritable one new IntWritable(1); private Text word new Text(); public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String[] words value.toString().split(\\s); for (String w : words) { word.set(w); context.write(word, one); } } }Reducer实现public class WordCountReducer extends ReducerText, IntWritable, Text, IntWritable { public void reduce(Text key, IterableIntWritable values, Context context) throws IOException, InterruptedException { int sum 0; for (IntWritable val : values) { sum val.get(); } context.write(key, new IntWritable(sum)); } }3.2 作业提交与执行Driver类配置public class WordCountDriver { public static void main(String[] args) throws Exception { Configuration conf new Configuration(); Job job Job.getInstance(conf, word count); job.setJarByClass(WordCountDriver.class); job.setMapperClass(WordCountMapper.class); job.setCombinerClass(WordCountReducer.class); job.setReducerClass(WordCountReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }执行流程将项目打包为wordcount.jar准备输入文件并上传到HDFShdfs dfs -mkdir /input hdfs dfs -put localfile.txt /input提交MapReduce作业hadoop jar wordcount.jar WordCountDriver /input /output查看结果hdfs dfs -cat /output/part-r-000004. 开发环境优化技巧4.1 常见问题解决DataNode启动失败检查hdfs-site.xml中dfs.datanode.data.dir路径权限确保路径存在且可写尝试删除data目录后重新格式化端口冲突修改默认端口配置core-site.xml中的fs.defaultFShdfs-site.xml中的dfs.namenode.http-address4.2 性能调优建议对于本地开发环境可以调整以下参数提升性能参数推荐值说明mapreduce.map.memory.mb1024Map任务内存mapreduce.reduce.memory.mb2048Reduce任务内存yarn.nodemanager.resource.memory-mb4096NodeManager总内存mapreduce.map.java.opts-Xmx768mMap任务JVM参数4.3 IDE集成开发在IntelliJ IDEA中配置Hadoop开发环境添加Hadoop依赖配置运行参数主类WordCountDriver程序参数input output启用Delegate IDE build/run actions to Maven5. 进阶开发实践5.1 使用HDFS API除了命令行还可以通过Java API操作HDFSConfiguration conf new Configuration(); FileSystem fs FileSystem.get(conf); // 创建目录 fs.mkdirs(new Path(/user/test)); // 上传文件 fs.copyFromLocalFile(new Path(localfile.txt), new Path(/user/test/localfile.txt)); // 列出文件 RemoteIteratorLocatedFileStatus files fs.listFiles(new Path(/), true); while (files.hasNext()) { System.out.println(files.next().getPath()); }5.2 调试MapReduce作业本地模式调试技巧设置mapreduce.framework.name为local直接使用本地文件路径而非HDFS路径在Mapper/Reducer中添加日志输出context.getCounter(Custom, Processed).increment(1); LOG.info(Processing key: key);5.3 结合其他大数据工具Hadoop生态系统的其他组件也可以在本地开发环境中集成Hive本地元数据存储用于SQL查询Spark本地模式运行Spark作业HBase单机模式运行这些工具的本地配置可以极大提升开发效率避免频繁部署到集群测试。