基于Docker Compose编排Hadoop集群-一键部署实战

1. 为什么选择Docker Compose部署Hadoop集群在传统Hadoop集群部署中我们需要手动配置多台物理机或虚拟机安装JDK、Hadoop、配置SSH免密登录、修改大量XML配置文件。这个过程不仅耗时耗力而且环境难以复用。我在实际项目中遇到过多次因环境差异导致的问题比如不同机器上的JDK版本不一致、配置文件路径不同等。Docker Compose的声明式编排方式完美解决了这些问题。通过一个YAML文件就能定义整个集群的拓扑结构包括服务依赖关系NameNode先于DataNode启动网络配置自定义子网和端口映射存储卷持久化HDFS数据健康检查自动监测服务可用性实测下来用Docker Compose部署Hadoop集群比传统方式快5倍以上。去年我在客户现场演示时原本需要半天的部署工作用Compose只用了不到30分钟就完成了3节点集群的搭建。2. 环境准备与工具安装2.1 基础环境要求确保你的开发机满足以下条件操作系统Linux/macOS/Windows(WSL2)Docker版本≥20.10支持Compose V3语法资源分配建议至少4核CPU/8GB内存验证Docker环境是否正常docker --version # 应显示Docker版本≥20.10 docker-compose --version # 需显示Compose V22.2 镜像选择建议经过多次测试对比我推荐以下官方镜像组合基础镜像eclipse-temurin:8-jdk比OpenJDK更稳定Hadoop镜像apache/hadoop:3.3.6如果网络条件有限可以使用阿里云镜像加速mkdir -p /etc/docker echo {registry-mirrors:[https://你的ID.mirror.aliyuncs.com]} /etc/docker/daemon.json systemctl restart docker3. 编写Docker Compose文件3.1 集群拓扑设计这是我们即将部署的3节点集群架构hadoop-master: NameNode ResourceManager hadoop-worker1: DataNode NodeManager hadoop-worker2: DataNode NodeManager3.2 完整compose.yaml配置创建hadoop-cluster/docker-compose.yaml文件version: 3.8 services: namenode: image: apache/hadoop:3.3.6 container_name: hadoop-master hostname: hadoop-master ports: - 9870:9870 # HDFS UI - 8088:8088 # YARN UI volumes: - ./data/namenode:/hadoop/dfs/name - ./config/core-site.xml:/opt/hadoop/etc/hadoop/core-site.xml - ./config/hdfs-site.xml:/opt/hadoop/etc/hadoop/hdfs-site.xml environment: - HDFS_NAMENODE_USERroot - YARN_RESOURCEMANAGER_USERroot healthcheck: test: [CMD, curl, -f, http://localhost:9870] interval: 30s timeout: 10s retries: 3 datanode1: image: apache/hadoop:3.3.6 hostname: hadoop-worker1 depends_on: namenode: condition: service_healthy volumes: - ./data/datanode1:/hadoop/dfs/data - ./config/core-site.xml:/opt/hadoop/etc/hadoop/core-site.xml - ./config/hdfs-site.xml:/opt/hadoop/etc/hadoop/hdfs-site.xml environment: - HDFS_DATANODE_USERroot - YARN_NODEMANAGER_USERroot datanode2: image: apache/hadoop:3.3.6 hostname: hadoop-worker2 depends_on: namenode: condition: service_healthy volumes: - ./data/datanode2:/hadoop/dfs/data - ./config/core-site.xml:/opt/hadoop/etc/hadoop/core-site.xml - ./config/hdfs-site.xml:/opt/hadoop/etc/hadoop/hdfs-site.xml environment: - HDFS_DATANODE_USERroot - YARN_NODEMANAGER_USERroot networks: default: driver: bridge ipam: config: - subnet: 172.20.0.0/16关键配置说明volumes将配置文件和数据目录挂载到容器内healthcheck确保NameNode完全启动后再启动DataNodesubnet固定IP段方便后续扩展4. Hadoop核心配置详解4.1 配置文件生成在hadoop-cluster/config/目录下创建配置文件core-site.xmlconfiguration property namefs.defaultFS/name valuehdfs://hadoop-master:9000/value /property property namehadoop.tmp.dir/name value/hadoop/tmp/value /property /configurationhdfs-site.xmlconfiguration property namedfs.replication/name value2/value /property property namedfs.namenode.name.dir/name value/hadoop/dfs/name/value /property property namedfs.datanode.data.dir/name value/hadoop/dfs/data/value /property /configuration4.2 权限问题解决Hadoop默认会检查用户权限在容器环境中建议关闭!-- 在hdfs-site.xml中添加 -- property namedfs.permissions.enabled/name valuefalse/value /property5. 集群启动与验证5.1 一键启动集群cd hadoop-cluster docker-compose up -d # 后台启动 docker-compose ps # 查看服务状态5.2 初始化HDFSdocker exec hadoop-master hdfs namenode -format # 格式化 docker exec hadoop-master start-dfs.sh # 启动HDFS docker exec hadoop-master start-yarn.sh # 启动YARN5.3 验证服务打开浏览器访问HDFS管理界面http://localhost:9870YARN管理界面http://localhost:8088执行测试命令docker exec hadoop-master hdfs dfs -mkdir /test docker exec hadoop-master hdfs dfs -put /opt/hadoop/README.txt /test/ docker exec hadoop-master hdfs dfs -ls /test6. 常见问题排查6.1 端口冲突问题如果遇到BindException可能是端口被占用netstat -tulnp | grep 8088 # 查找占用进程解决方案修改docker-compose.yaml中的端口映射或者停止占用端口的服务6.2 内存不足问题在docker-compose.yaml中限制容器资源datanode1: deploy: resources: limits: cpus: 1 memory: 2G6.3 数据持久化建议定期备份./data目录tar -czvf hadoop-data-$(date %Y%m%d).tar.gz ./data7. 生产环境优化建议7.1 安全加固措施启用Kerberos认证修改默认SSH端口配置防火墙规则7.2 监控方案# 在compose中添加Prometheus监控 monitor: image: prom/prometheus ports: - 9090:9090 volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml7.3 扩展集群规模要新增节点只需# 在docker-compose.yaml中添加 datanode3: image: apache/hadoop:3.3.6 hostname: hadoop-worker3 # ...其他配置与datanode1相同然后执行docker-compose up -d --scale datanode3