个人笔记
SongPinru 的小仓库
如何新增节点
前置条件
操作系统：ubuntu18.04
运维初始化机器，确保集群内可互访问，硬盘挂载成功
步骤
DNS新增节点
首先需要把新增节点的域名和ip加入我们的DNS中
DNS节点：10.11.20.51 、10.11.20.52
DNS使用的软件是bind9，其中10.11.20.51是主节点，10.11.20.52是从节点，配置文件在10.11.20.51上
- 登录10.11.20.51机器
- 切换root权限
- 进入
/etc/bind/zones目录 - 修改
db.10.11.20和db.optaim.com文件- 目前只支持10.11.20号段，且域名必须是optaim.com结尾
新增节点加入配置文件，按照格式新增即可
db.optaim.com正向解析：
...
bjuc49.optaim.com. IN A 10.11.20.49
bjuc50.optaim.com. IN A 10.11.20.50
bjuc51.optaim.com. IN A 10.11.20.51
bjuc52.optaim.com. IN A 10.11.20.52
bjuc53.optaim.com. IN A 10.11.20.53
bjuc54.optaim.com. IN A 10.11.20.54
bjuc55.optaim.com. IN A 10.11.20.55
pbj-cdh-20-69.optaim.com. IN A 10.11.20.69
pbj-cdh-20-70.optaim.com. IN A 10.11.20.70
pbj-cdh-20-71.optaim.com. IN A 10.11.20.71
pbj-cdh-20-72.optaim.com. IN A 10.11.20.72
pbj-cdh-20-73.optaim.com. IN A 10.11.20.73
pbj-cdh-20-101.optaim.com. IN A 10.11.20.101
pbj-cdh-20-102.optaim.com. IN A 10.11.20.102
pbj-cdh-20-103.optaim.com. IN A 10.11.20.103
pbj-cdh-20-104.optaim.com. IN A 10.11.20.104
pbj-cdh-20-114.optaim.com. IN A 10.11.20.114
pbj-cdh-20-115.optaim.com. IN A 10.11.20.115
pbj-cdh-20-116.optaim.com. IN A 10.11.20.116
...
db.10.11.20反向解析:
...
114 IN PTR pbj-cdh-20-114.optaim.com. ; 10.11.20.114
115 IN PTR pbj-cdh-20-115.optaim.com. ; 10.11.20.115
116 IN PTR pbj-cdh-20-116.optaim.com. ; 10.11.20.116
81 IN PTR pbj-gw-20-81.optaim.com. ; 10.11.20.81
212 IN PTR pbj-gw-20-212.optaim.com. ; 10.11.20.212
117 IN PTR pbj-clickhouse-20-117.optaim.com ; 10.11.20.117
118 IN PTR pbj-clickhouse-20-118.optaim.com ; 10.11.20.118
210 IN PTR pbj-clickhouse-20-210.optaim.com ; 10.11.20.210
211 IN PTR pbj-clickhouse-20-211.optaim.com ; 10.11.20.211
...
增加之后，需要重载bind9，使更改生效
sudo systemctl reload bind9
最后登录10.11.20.52，也重载一下bind9
修改成功之后可以使用ping或者nslookup指令来检测是否添加成功
新增节点配置DNS
登录要新增的节点，配置DNS为10.11.20.51和10.11.20.52
vim /etc/netplan/50-cloud-init.yaml
network:
bonds:
bond0:
addresses:
- 10.11.20.101/24
gateway4: 10.11.20.1
interfaces:
- eth0
- eth1
nameservers:
#增加或修改下main部分
addresses:
- 10.11.20.51
- 10.11.20.52
- 8.8.8.8
search:
- optaim.com
然后由于18.04的一点小问题，部分系统需要重新连接resolv.conf文件（可以通过ls查看）
rm /etc/resolv.conf
ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
然后执行
netplan apply
最后检查resolv.conf是否符合以下（顺序必须一致，不一致需要重启systemd-resolve）
nameserver 10.11.20.51
nameserver 10.11.20.52
nameserver 8.8.8.8
search optaim.com
修改成功之后可以使用ping或者nslookup指令来检测是否配置成功
安装CM
两种方法，脚本或者web UI
脚本：
#! /bin/bash
SOURCE="$(readlink -f ${BASH_SOURCE[0]})"
BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
environment='export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin
'
source_list='deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
'
soft_dir=$BIN_DIR
daemons_path="${soft_dir}/cloudera-manager-daemons_6.3.1~1466458.ubuntu1804_all.deb"
agnet_path="${soft_dir}/cloudera-manager-agent_6.3.1~1466458.ubuntu1804_amd64.deb"
# functions
function init(){
wget http://10.10.72.25:10086/cloudera/jdk1.8.0_181-cloudera.tar
wget http://10.10.72.25:10086/cloudera/cm/pool/contrib/e/enterprise/cloudera-manager-daemons_6.3.1~1466458.ubuntu1804_all.deb
wget http://10.10.72.25:10086/cloudera/cm/pool/contrib/e/enterprise/cloudera-manager-agent_6.3.1~1466458.ubuntu1804_amd64.deb
if [ ! -d /usr/java/ ];then
mkdir -p /usr/java/
fi
tar -xvf jdk1.8.0_181-cloudera.tar -C /usr/java/
}
function set_env(){
echo "$environment" > /etc/profile.d/environment.sh
}
function init_apt(){
echo "$source_list"> /etc/apt/sources.list
apt update
}
function install_cm_agent(){
apt-get install -yq libsasl2-modules-gssapi-mit
apt-get install -yq libssl-dev
apt-get install -yq rpcbind
apt-get install -yq python-psycopg2
apt-get install -yq python-mysqldb
apt-get install -yq apache2
apt-get install -yq iproute2
if [ -e cloudera-manager-daemons_6.3.1~1466458.ubuntu1804_all.deb ]&&[ -e cloudera-manager-agent_6.3.1~1466458.ubuntu1804_amd64.deb ];then
dpkg -i cloudera-manager-daemons_6.3.1~1466458.ubuntu1804_all.deb
dpkg -i cloudera-manager-agent_6.3.1~1466458.ubuntu1804_amd64.deb
fi
}
function start_agent(){
sed -i 's|server_host=localhost|server_host=bjuc49.optaim.com|' /etc/cloudera-scm-agent/config.ini
systemctl start cloudera-scm-agent
}
# main
if [ $(whoami) == 'root' ];then
echo 'i am root'
cd ${soft_dir}
init
set_env
init_apt
install_cm_agent
start_agent
else
echo 'error:not root,please use sudo'
fi
Web UI：略，根据提示即可，需要ssh连接
分配角色
分配角色前，可以使用主机模板来简化操作
修改主机模板，调整配置，然后批量添加即可，这里全是页面操作，略
分配角色完成后启动所有角色，节点增加完成