前置条件ntp服务安装防火墙关闭执行以下优化代码systemctl stop firewalld.service echo * soft nofile 128000 /etc/security/limits.conf echo * hard nofile 128000 /etc/security/limits.conf echo * soft nproc 128000 /etc/security/limits.conf echo * hard nproc 128000 /etc/security/limits.conf cat /etc/security/limits.conf sed -i s/4096/unlimited/ /etc/security/limits.d/20-nproc.conf cat /etc/security/limits.d/20-nproc.conf sed -i s/4096/unlimited/ /etc/security/limits.d/90-nproc.conf cat /etc/security/limits.d/90-nproc.conf ulimit -SHn 128000 ulimit -SHu 128000 echo vm.swappiness10 /etc/sysctl.conf sysctl -p sysctl -a|grep swappiness echo net.core.somaxconn 32768 /etc/sysctl.conf sysctl -p sysctl -a|grep somaxconn echo never /sys/kernel/mm/transparent_hugepage/defrag echo never /sys/kernel/mm/transparent_hugepage/enabledCDH6.2 下载信息https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_62_download.html yum安装https://archive.cloudera.com/cdh6/6.2.0/redhat7/yum/ parcel安装https://archive.cloudera.com/cdh6/6.2.0/parcels/这里我们讲解使用 parcel 方式安装ClouderaManager 安装包下载https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/enterprise-debuginfo-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0update181-1.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/allkeys.ascCDH 安装包下载https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha1 https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha256 https://archive.cloudera.com/cdh6/6.2.0/parcels/manifest.json安装 createrepo作用:createrepo 命令用于创建yum源软件仓库即为存放于本地特定位置的众多rpm包建立索引描述各包所需依赖信息并形成元数据。yum install -y createrepo在CM目录下执行 createrepo . 生成rpm源[rootcdh01 CM]# createrepo . Spawning worker 0 with 1 pkgs Spawning worker 1 with 1 pkgs Spawning worker 2 with 1 pkgs Spawning worker 3 with 1 pkgs Spawning worker 4 with 1 pkgs Spawning worker 5 with 1 pkgs Spawning worker 6 with 0 pkgs Spawning worker 7 with 0 pkgs Spawning worker 8 with 0 pkgs Spawning worker 9 with 0 pkgs Spawning worker 10 with 0 pkgs Spawning worker 11 with 0 pkgs Spawning worker 12 with 0 pkgs Spawning worker 13 with 0 pkgs Spawning worker 14 with 0 pkgs Spawning worker 15 with 0 pkgs Spawning worker 16 with 0 pkgs Spawning worker 17 with 0 pkgs Spawning worker 18 with 0 pkgs Spawning worker 19 with 0 pkgs Spawning worker 20 with 0 pkgs Spawning worker 21 with 0 pkgs Spawning worker 22 with 0 pkgs Spawning worker 23 with 0 pkgs Workers Finished Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Sqlite DBs complete安装 httpd 服务yum install -y httpd启动服务systemctl start httpd systemctl status httpd将CM 与 CDH 目录移动到目录 /var/www/html/ 下mv CM/ CDH/ /var/www/html/验证是否安装成功访问 http://ip/CM http://ip/CDH显示如下即可提示以下内容说明没有成功原因是SELinux未禁用成功在所有服务器执行setenforce 0 即可配置安装源vim /etc/yum.repos.d/cm.repo [cmrepo] name cm_repo baseurl http://ip/CM enable true gpgcheck false安装JDKyum install oracle-j2sdk1.8-1.8.0update181-1.x86_64安装 Cloudera Manager Serveryum install -y cloudera-manager-server到这里CM服务端安装完成先不要启动安装数据库安装MariaDBMariaDB数据库管理系统是MySQL的一个分支主要由开源社区在维护采用GPL授权许可 MariaDB的目的是完全兼容MySQL包括API和命令行使之能轻松成为MySQL的代替品。在存储引擎方面使用XtraDB英语XtraDB来代替MySQL的InnoDB。 MariaDB由MySQL的创始人Michael Widenius英语Michael Widenius主导开发他早前曾以10亿美元的价格将自己创建的公司MySQL AB卖给了SUN此后随着SUN被甲骨文收购MySQL的所有权也落入Oracle的手中。MariaDB名称来自Michael Widenius的女儿Maria的名字。MariaDB基于事务的Maria存储引擎替换了MySQL的MyISAM存储引擎它使用了Percona的 XtraDBInnoDB的变体分支的开发者希望提供访问即将到来的MySQL 5.4 InnoDB性能。这个版本还包括了 PrimeBase XT (PBXT) 和 FederatedX存储引擎。yum -y install mariadb yum -y install mariadb-server启动、配置MariaDBsystemctl start mariadb[rootcdh01 ~]# /usr/bin/mysql_secure_installation NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MariaDB to secure it, well need the current password for the root user. If youve just installed MariaDB, and you havent set the root password yet, the password will be blank, so you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MariaDB root user without the proper authorisation. Set root password? [Y/n] y #设置root密码 New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MariaDB installation has an anonymous user, allowing anyone to log into MariaDB without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] y #是否禁止匿名登录 ... Success! Normally, root should only be allowed to connect from localhost. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] n #是否禁止远程登录 ... skipping. By default, MariaDB comes with a database named test that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] y #删除测试数据库 - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] y #加载最新设置 ... Success! Cleaning up... All done! If youve completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB!登录mysql 创建数据库mysql -uroot -p123456 create database metastore default character set utf8; #创建数据库名称为metastore默认编码utf-8 CREATE USER hive% IDENTIFIED BY 123456; #创建用户名为hive的用户密码为123456可以在任何机器上访问 GRANT ALL PRIVILEGES ON metastore. * TO hive%; #授权hive用户metastore所有权限 FLUSH PRIVILEGES; #刷新配置 create database cm default character set utf8; CREATE USER cm% IDENTIFIED BY 123456; GRANT ALL PRIVILEGES ON cm. * TO cm%; FLUSH PRIVILEGES; create database hue default character set utf8; CREATE USER hue% IDENTIFIED BY 123456; GRANT ALL PRIVILEGES ON hue. * TO hue%; FLUSH PRIVILEGES; create database oozie default character set utf8; CREATE USER oozie% IDENTIFIED BY 123456; GRANT ALL PRIVILEGES ON oozie. * TO oozie%; FLUSH PRIVILEGES; create database sentry default character set utf8; CREATE USER sentry% IDENTIFIED BY 123456; GRANT ALL PRIVILEGES ON sentry. * TO sentry%; FLUSH PRIVILEGES;安装Mysql驱动此处不安装服务无法找到mysql驱动会导致服务失败集群所有服务都安装避免以后遇到问题[rootcdh01 cdh]# mkdir /usr/share/java [rootcdh01 cdh]# mv mysql-connector-java-5.1.30.jar /usr/share/java/ [rootcdh01 cdh]# chmod 777 /usr/share/java/mysql-connector-java-5.1.30.jar [rootcdh01 cdh]# ln -s /usr/share/java/mysql-connector-java-5.1.30.jar /usr/share/java/mysql-connector-java.jar [rootcdh01 cdh]# ll /usr/share/java/ 总用量 932 -rwxrwxrwx. 1 root root 954038 6月 19 11:28 mysql-connector-java-5.1.30.jar lrwxrwxrwx. 1 root root 47 9月 1 13:59 mysql-connector-java.jar - /usr/share/java/mysql-connector-java-5.1.30.jarCloudera Manager Server 初始化数据库注意这里 123456 是我们上一步创建的数据库密码[rootcdh01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm 123456 JAVA_HOME/usr/local/java/jdk1.8.0_191 Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/local/java/jdk1.8.0_191/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly!启动Cloudera Manager Server 服务systemctl start cloudera-scm-server systemctl status cloudera-scm-server启动可能需要半分钟左右可以查看日志启动状态提示如下内容就启动成功了日志目录/var/log/cloudera-scm-server2019-09-01 14:05:02,994 INFO WebServerImpl:org.eclipse.jetty.server.AbstractConnector: Started ServerConnector534fca49{HTTP/1.1,[http/1.1]}{0.0.0.0:7180} 2019-09-01 14:05:02,995 INFO WebServerImpl:org.eclipse.jetty.server.Server: Started 95217ms 2019-09-01 14:05:02,995 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.浏览器访问提示登录默认账号密码均为 adminhttp://ip:7180/cmf/login接受继续这里选择免费版本(也可以选择试用版到期后自动恢复成免费版本)开始安装集群选择我们刚刚安装的httpd服务链接agent 安装成功下一步安装 parcel 这个过程可能要等一段时间这里我们看下检查结果有两处异常根据提示修改配置sysctl vm.swappiness10 修改CDH建议值重新检测。解决了第一个问题设置一下两个配置echo never /sys/kernel/mm/transparent_hugepage/defrag echo never /sys/kernel/mm/transparent_hugepage/enabled再次查看全部正常根据需求自己分配角色安装成功
前置条件ntp服务安装防火墙关闭执行以下优化代码systemctl stop firewalld.service echo * soft nofile 128000 /etc/security/limits.conf echo * hard nofile 128000 /etc/security/limits.conf echo * soft nproc 128000 /etc/security/limits.conf echo * hard nproc 128000 /etc/security/limits.conf cat /etc/security/limits.conf sed -i s/4096/unlimited/ /etc/security/limits.d/20-nproc.conf cat /etc/security/limits.d/20-nproc.conf sed -i s/4096/unlimited/ /etc/security/limits.d/90-nproc.conf cat /etc/security/limits.d/90-nproc.conf ulimit -SHn 128000 ulimit -SHu 128000 echo vm.swappiness10 /etc/sysctl.conf sysctl -p sysctl -a|grep swappiness echo net.core.somaxconn 32768 /etc/sysctl.conf sysctl -p sysctl -a|grep somaxconn echo never /sys/kernel/mm/transparent_hugepage/defrag echo never /sys/kernel/mm/transparent_hugepage/enabledCDH6.2 下载信息https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_62_download.html yum安装https://archive.cloudera.com/cdh6/6.2.0/redhat7/yum/ parcel安装https://archive.cloudera.com/cdh6/6.2.0/parcels/这里我们讲解使用 parcel 方式安装ClouderaManager 安装包下载https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/enterprise-debuginfo-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0update181-1.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/allkeys.ascCDH 安装包下载https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha1 https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha256 https://archive.cloudera.com/cdh6/6.2.0/parcels/manifest.json安装 createrepo作用:createrepo 命令用于创建yum源软件仓库即为存放于本地特定位置的众多rpm包建立索引描述各包所需依赖信息并形成元数据。yum install -y createrepo在CM目录下执行 createrepo . 生成rpm源[rootcdh01 CM]# createrepo . Spawning worker 0 with 1 pkgs Spawning worker 1 with 1 pkgs Spawning worker 2 with 1 pkgs Spawning worker 3 with 1 pkgs Spawning worker 4 with 1 pkgs Spawning worker 5 with 1 pkgs Spawning worker 6 with 0 pkgs Spawning worker 7 with 0 pkgs Spawning worker 8 with 0 pkgs Spawning worker 9 with 0 pkgs Spawning worker 10 with 0 pkgs Spawning worker 11 with 0 pkgs Spawning worker 12 with 0 pkgs Spawning worker 13 with 0 pkgs Spawning worker 14 with 0 pkgs Spawning worker 15 with 0 pkgs Spawning worker 16 with 0 pkgs Spawning worker 17 with 0 pkgs Spawning worker 18 with 0 pkgs Spawning worker 19 with 0 pkgs Spawning worker 20 with 0 pkgs Spawning worker 21 with 0 pkgs Spawning worker 22 with 0 pkgs Spawning worker 23 with 0 pkgs Workers Finished Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Sqlite DBs complete安装 httpd 服务yum install -y httpd启动服务systemctl start httpd systemctl status httpd将CM 与 CDH 目录移动到目录 /var/www/html/ 下mv CM/ CDH/ /var/www/html/验证是否安装成功访问 http://ip/CM http://ip/CDH显示如下即可提示以下内容说明没有成功原因是SELinux未禁用成功在所有服务器执行setenforce 0 即可配置安装源vim /etc/yum.repos.d/cm.repo [cmrepo] name cm_repo baseurl http://ip/CM enable true gpgcheck false安装JDKyum install oracle-j2sdk1.8-1.8.0update181-1.x86_64安装 Cloudera Manager Serveryum install -y cloudera-manager-server到这里CM服务端安装完成先不要启动安装数据库安装MariaDBMariaDB数据库管理系统是MySQL的一个分支主要由开源社区在维护采用GPL授权许可 MariaDB的目的是完全兼容MySQL包括API和命令行使之能轻松成为MySQL的代替品。在存储引擎方面使用XtraDB英语XtraDB来代替MySQL的InnoDB。 MariaDB由MySQL的创始人Michael Widenius英语Michael Widenius主导开发他早前曾以10亿美元的价格将自己创建的公司MySQL AB卖给了SUN此后随着SUN被甲骨文收购MySQL的所有权也落入Oracle的手中。MariaDB名称来自Michael Widenius的女儿Maria的名字。MariaDB基于事务的Maria存储引擎替换了MySQL的MyISAM存储引擎它使用了Percona的 XtraDBInnoDB的变体分支的开发者希望提供访问即将到来的MySQL 5.4 InnoDB性能。这个版本还包括了 PrimeBase XT (PBXT) 和 FederatedX存储引擎。yum -y install mariadb yum -y install mariadb-server启动、配置MariaDBsystemctl start mariadb[rootcdh01 ~]# /usr/bin/mysql_secure_installation NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MariaDB to secure it, well need the current password for the root user. If youve just installed MariaDB, and you havent set the root password yet, the password will be blank, so you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MariaDB root user without the proper authorisation. Set root password? [Y/n] y #设置root密码 New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MariaDB installation has an anonymous user, allowing anyone to log into MariaDB without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] y #是否禁止匿名登录 ... Success! Normally, root should only be allowed to connect from localhost. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] n #是否禁止远程登录 ... skipping. By default, MariaDB comes with a database named test that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] y #删除测试数据库 - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] y #加载最新设置 ... Success! Cleaning up... All done! If youve completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB!登录mysql 创建数据库mysql -uroot -p123456 create database metastore default character set utf8; #创建数据库名称为metastore默认编码utf-8 CREATE USER hive% IDENTIFIED BY 123456; #创建用户名为hive的用户密码为123456可以在任何机器上访问 GRANT ALL PRIVILEGES ON metastore. * TO hive%; #授权hive用户metastore所有权限 FLUSH PRIVILEGES; #刷新配置 create database cm default character set utf8; CREATE USER cm% IDENTIFIED BY 123456; GRANT ALL PRIVILEGES ON cm. * TO cm%; FLUSH PRIVILEGES; create database hue default character set utf8; CREATE USER hue% IDENTIFIED BY 123456; GRANT ALL PRIVILEGES ON hue. * TO hue%; FLUSH PRIVILEGES; create database oozie default character set utf8; CREATE USER oozie% IDENTIFIED BY 123456; GRANT ALL PRIVILEGES ON oozie. * TO oozie%; FLUSH PRIVILEGES; create database sentry default character set utf8; CREATE USER sentry% IDENTIFIED BY 123456; GRANT ALL PRIVILEGES ON sentry. * TO sentry%; FLUSH PRIVILEGES;安装Mysql驱动此处不安装服务无法找到mysql驱动会导致服务失败集群所有服务都安装避免以后遇到问题[rootcdh01 cdh]# mkdir /usr/share/java [rootcdh01 cdh]# mv mysql-connector-java-5.1.30.jar /usr/share/java/ [rootcdh01 cdh]# chmod 777 /usr/share/java/mysql-connector-java-5.1.30.jar [rootcdh01 cdh]# ln -s /usr/share/java/mysql-connector-java-5.1.30.jar /usr/share/java/mysql-connector-java.jar [rootcdh01 cdh]# ll /usr/share/java/ 总用量 932 -rwxrwxrwx. 1 root root 954038 6月 19 11:28 mysql-connector-java-5.1.30.jar lrwxrwxrwx. 1 root root 47 9月 1 13:59 mysql-connector-java.jar - /usr/share/java/mysql-connector-java-5.1.30.jarCloudera Manager Server 初始化数据库注意这里 123456 是我们上一步创建的数据库密码[rootcdh01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm 123456 JAVA_HOME/usr/local/java/jdk1.8.0_191 Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/local/java/jdk1.8.0_191/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly!启动Cloudera Manager Server 服务systemctl start cloudera-scm-server systemctl status cloudera-scm-server启动可能需要半分钟左右可以查看日志启动状态提示如下内容就启动成功了日志目录/var/log/cloudera-scm-server2019-09-01 14:05:02,994 INFO WebServerImpl:org.eclipse.jetty.server.AbstractConnector: Started ServerConnector534fca49{HTTP/1.1,[http/1.1]}{0.0.0.0:7180} 2019-09-01 14:05:02,995 INFO WebServerImpl:org.eclipse.jetty.server.Server: Started 95217ms 2019-09-01 14:05:02,995 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.浏览器访问提示登录默认账号密码均为 adminhttp://ip:7180/cmf/login接受继续这里选择免费版本(也可以选择试用版到期后自动恢复成免费版本)开始安装集群选择我们刚刚安装的httpd服务链接agent 安装成功下一步安装 parcel 这个过程可能要等一段时间这里我们看下检查结果有两处异常根据提示修改配置sysctl vm.swappiness10 修改CDH建议值重新检测。解决了第一个问题设置一下两个配置echo never /sys/kernel/mm/transparent_hugepage/defrag echo never /sys/kernel/mm/transparent_hugepage/enabled再次查看全部正常根据需求自己分配角色安装成功
![[C++]内存管理：串顺序存储的内存回收](http://pic.xiahunao.cn/yaotu/[C++]内存管理：串顺序存储的内存回收)
