介绍 | w3xse7en

hyper-v 安装 centos

Mon, 01 Jan 0001 00:00:00 +0000

Centos

清华大学centos7.8.2003镜像

清华大学镜像站

虚拟机

宿主机i5-6200u 2c16g
每台机器均为2c4g配置

hyper-v

网络

安装完的centos是不能上网的
此处使用桥接模式进行网络连接
使用桥接模式使node和host处在同一网段更方便其他设备访问

桥接详情

配置Centos 静态ip地址
网段参考网桥ip

vi /etc/sysconfig/network-scripts/ifcfg-eth0

ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.1.201
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=192.168.1.1

service network restart

网络拓扑图

kubespray 安装 k8s

Mon, 01 Jan 0001 00:00:00 +0000

参考

kubespray

[ Kube 65.1 ] Kubespray - Kubernetes cluster provisioning

Deploying kubernetes using Kubespray

INSTALLING A MULTINODE KUBERNETES CLUSTER USING KUBESPRAY

it’s really really really hard run kubespray in china! #6207

使用kubespray安装kubernetes

使用kubespray搭建生产级高可用集群

使用Kubespray安装k8s集群

docker/kubernetes国内源/镜像源解决方式

配置免密码登录

ssh-keygen -t rsa
ssh-copy-id root@192.168.1.200
ssh-copy-id root@192.168.1.201
ssh-copy-id root@192.168.1.202

下载kubespray

# master分支还在更新中，此处使用当前最新release的版本v2.14.1
git clone --single-branch -b v2.14.1 https://github.com/kubernetes-sigs/kubespray.git

安装python3-pip

yum install python3-pip

安装kubespray依赖

pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/

使用ansible对每台机器进行批处理

配置ansible访问k8s集群hosts

vi /etc/ansible/hosts

[k8s]

192.168.1.200 ansible_ssh_user=root
192.168.1.201 ansible_ssh_user=root
192.168.1.202 ansible_ssh_user=root

测试

ansible k8s -m ping

更新yum

ansible k8s -m shell -a 'yum update -y'

安装ntp服务，统一集群时间

ansible k8s -m shell -a 'yum install -y ntp &&
systemctl enable ntpd &&
systemctl start ntpd &&
timedatectl set-timezone Asia/Shanghai &&
timedatectl set-ntp yes && 
ntpq -p'

配置ipv4转发

ansible k8s -m shell -a 'echo net.ipv4.ip_forward = 1 >> /etc/sysctl.conf && sysctl -p'

关闭防火墙

ansible k8s -m shell -a 'systemctl stop firewalld && systemctl disable firewalld'

生成kubespray所需配置

创建可自定义的配置

cp -rfp inventory/sample inventory/mycluster

IPS=(此处填写k8s集群的ip地址，注意空格分割)

declare -a IPS=(192.168.1.200 192.168.1.201 192.168.1.202)

CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

查看生成的hosts.yaml

cat inventory/mycluster/hosts.yaml

此处是 2个master 3个node 3个etcd 的高可用配置，可以根据喜好进行配置

all:
  hosts:
    node1:
      ansible_host: 192.168.1.200
      ip: 192.168.1.200
      access_ip: 192.168.1.200
    node2:
      ansible_host: 192.168.1.201
      ip: 192.168.1.201
      access_ip: 192.168.1.201
    node3:
      ansible_host: 192.168.1.202
      ip: 192.168.1.202
      access_ip: 192.168.1.202
  children:
    kube-master:
      hosts:
        node1:
        node2:
    kube-node:
      hosts:
        node1:
        node2:
        node3:
    etcd:
      hosts:
        node1:
        node2:
        node3:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

下载依赖

方案1
- 将所有源换成国内镜像加速
方案2
- 离线安装
方案3
- 设置代理
此处使用方案3

vi inventory/mycluster/group_vars/all/all.yml

http_proxy: "http://192.168.1.9:1080"
https_proxy: "http://192.168.1.9:1080"

在阿里云环境下使用代理
- 开启自己电脑上的http proxy server
- 使用frp将本地电脑http proxy port 映射至公网
- 配置http_proxy
- 直接在阿里云内使用代理可能会被警告，因此使用frp做一层中转
配置docker镜像源

vi inventory/mycluster/group_vars/all/docker.yml
## Add other registry,example China registry mirror.
docker_registry_mirrors:
  - https://mirror.aliyuncs.com
  - https://registry.docker-cn.com

提前下载k8s所需依赖，避免安装时报错

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml --tags container_engine

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml --tags download

若遇到报错，可尝试将http_proxy注释掉重新下载，重复试几次后一般都能下完
- 阿里云apt, yum等源设置的是内网地址，需要直连才可以访问

vi inventory/mycluster/group_vars/all/all.yml

# http_proxy: "http://192.168.1.9:1080"
# https_proxy: "http://192.168.1.9:1080"

frp-v2ray-代理下载

ubuntu 系统安装完毕后，请删除/etc/apt/apt.conf里的代理配置

(可选)开启helm

vi inventory/mycluster/group_vars/k8s-cluster/addons.yml
# Helm deployment
helm_enabled: true

安装

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml

安装花费21分钟

验证

k8s v1.18.9安装成功

重置

若在安装过程中遇到问题可用以下命令重置

ansible-playbook -i inventory/mycluster/hosts.yaml reset.yml

Dashboard

Kubespray并没有替你创建用户，所以需要创建用户，然后获得Token，使用Token登录。

添加admin-user用户

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system
EOF

获取admin-user的token

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

访问dashboard

https://192.168.1.200:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login

services "https:kubernetes-dashboard:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

遇到此问题请查看system:anonymous 无法访问 k8s dashboard

添加/删除节点

Mon, 01 Jan 0001 00:00:00 +0000

参考

kubespray

[ Kube 65.3 ] Kubespray - Adding & Removing Kubernetes nodes

添加节点

配置ansible访问新节点(192.168.1.203)

vi /etc/ansible/hosts

[k8s]

192.168.1.200 ansible_ssh_user=root
192.168.1.201 ansible_ssh_user=root
192.168.1.202 ansible_ssh_user=root
192.168.1.203 ansible_ssh_user=root

测试

ansible k8s -m ping

更新新节点功能(时区,防火墙等)

新增node4

vi inventory/mycluster/hosts.yaml

all:
  hosts:
    node1:
      ansible_host: 192.168.1.200
      ip: 192.168.1.200
      access_ip: 192.168.1.200
    node2:
      ansible_host: 192.168.1.201
      ip: 192.168.1.201
      access_ip: 192.168.1.201
    node3:
      ansible_host: 192.168.1.202
      ip: 192.168.1.202
      access_ip: 192.168.1.202
    node4:
      ansible_host: 192.168.1.203
      ip: 192.168.1.203
      access_ip: 192.168.1.203
  children:
    kube-master:
      hosts:
        node1:
        node2:
    kube-node:
      hosts:
        node1:
        node2:
        node3:
        node4:
    etcd:
      hosts:
        node1:
        node2:
        node3:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

下载依赖

假设node1与node4为同系统将node1 /tmp/releases copy至node4 /tmp/releases即可避免node4再次下载k8s依赖组件

scp /tmp/releases root@192.168.1.203:/tmp/releases

确定依赖已完善

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml --tags container_engine

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml --tags download

添加节点

ansible-playbook -i inventory/mycluster/hosts.yaml scale.yml

移除节点

ansible-playbook -i inventory/mycluster/hosts.yaml remove-node.yml --extra-vars "node=node4"

创建 service account

Mon, 01 Jan 0001 00:00:00 +0000

参考

Setting up a Kubernetes CI/CD Service Account

场景

权限划分，cicd限定namespace=dev使用kubectl apply -f部署服务

创建 namespace dev

cat <<EOF | kubectl apply -f -
{
  "apiVersion": "v1",
  "kind": "Namespace",
  "metadata": {
    "name": "dev",
    "labels": {
      "name": "dev"
    }
  }
}
EOF

创建 `service account` `cicd-user`

# 该脚本依赖 jq 组件
yum install jq -y

# [可修改]
SERVICE_ACCOUNT_NAME="cicd-user"

# [可修改]
NAMESPACE="dev"

# [可修改]
TARGET_FOLDER="/home/k8s/kube"

kubectl create sa "${SERVICE_ACCOUNT_NAME}" --namespace "${NAMESPACE}"

mkdir -p "${TARGET_FOLDER}"

SECRET_NAME=$(kubectl get sa "${SERVICE_ACCOUNT_NAME}" --namespace="${NAMESPACE}" -o json | jq -r .secrets[].name)

kubectl get secret --namespace "${NAMESPACE}" "${SECRET_NAME}" -o json | jq -r '.data["ca.crt"]' | base64 --decode > "${TARGET_FOLDER}/ca.crt"

USER_TOKEN=$(kubectl get secret --namespace "${NAMESPACE}" "${SECRET_NAME}" -o json | jq -r '.data["token"]' | base64 --decode)

KUBECFG_FILE_NAME="${TARGET_FOLDER}/k8s-${SERVICE_ACCOUNT_NAME}-${NAMESPACE}-conf"

context=$(kubectl config current-context)

CLUSTER_NAME=$(kubectl config get-contexts "$context" | awk '{print $3}' | tail -n 1)

ENDPOINT=$(kubectl config view \
-o jsonpath="{.clusters[?(@.name == \"${CLUSTER_NAME}\")].cluster.server}")

kubectl config set-cluster "${CLUSTER_NAME}" \
--kubeconfig="${KUBECFG_FILE_NAME}" \
--server="${ENDPOINT}" \
--certificate-authority="${TARGET_FOLDER}/ca.crt" \
--embed-certs=true

kubectl config set-credentials \
"${SERVICE_ACCOUNT_NAME}-${NAMESPACE}-${CLUSTER_NAME}" \
--kubeconfig="${KUBECFG_FILE_NAME}" \
--token="${USER_TOKEN}"


kubectl config set-context \
"${SERVICE_ACCOUNT_NAME}-${NAMESPACE}-${CLUSTER_NAME}" \
--kubeconfig="${KUBECFG_FILE_NAME}" \
--cluster="${CLUSTER_NAME}" \
--user="${SERVICE_ACCOUNT_NAME}-${NAMESPACE}-${CLUSTER_NAME}" \
--namespace="${NAMESPACE}"

kubectl config use-context "${SERVICE_ACCOUNT_NAME}-${NAMESPACE}-${CLUSTER_NAME}" \
--kubeconfig="${KUBECFG_FILE_NAME}"

为 `cicd-user` 配置权限

cat <<EOF | kubectl apply -f -
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: "${SERVICE_ACCOUNT_NAME}-role"
  namespace: "${NAMESPACE}"
rules:
- apiGroups:
  - ""
  - apps
  - extensions
  resources:
  - '*'
  verbs:
  - '*'
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: "${SERVICE_ACCOUNT_NAME}-role"
  namespace: "${NAMESPACE}"
subjects:
  - kind: ServiceAccount
    name: "${SERVICE_ACCOUNT_NAME}"
roleRef:
  kind: Role
  name: "${SERVICE_ACCOUNT_NAME}-role"
  apiGroup: rbac.authorization.k8s.io
EOF

其他实例使用 `cicd-user`

将${TARGET_FOLDER}生成的k8s-cicd-user-dev-conf文件copy至其他实例~/.kube路径下改名为config使用

注意查看k8s-cicd-user-dev-conf里cluster server地址是否合法

遇到证书ip问题请查看kubectl get pods 证书ip问题

solve k8s problem

Mon, 01 Jan 0001 00:00:00 +0000

介绍

记录在使用k8s过程中所遇到的问题，以及解决方案

kubectl get nodes 证书ip问题

Unable to connect to the server: x509: certificate is valid for 10.233.0.1, 172.19.157.57, 172.19.157.57, 10.233.0.1, 127.0.0.1, 172.19.157.57, 172.19.252.199, not 47.xxx.xx.x

场景

阿里云安装，全部使用私网地址，创建service account后使用kubectl从公网访问报错

解决方案 supplementary_addresses_in_ssl_keys 添加公网地址

vi inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

supplementary_addresses_in_ssl_keys: [47.xxx.xx.x]

# 重新安装部署，使配置生效
ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml

参考

Invalid x509 certificate for kubernetes master

经验证kubespray v2.14.1不需要执行rm /etc/kubernetes/pki/apiserver.*

system:anonymous 无法访问 k8s dashboard

`services "https:kubernetes-dashboard:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"`

解决方案添加system:anonymous的访问权限

cat <<EOF | kubectl apply -f -
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubernetes-dashboard-anonymous
rules:
- apiGroups: [""]
  resources: ["services/proxy"]
  resourceNames: ["https:kubernetes-dashboard:"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- nonResourceURLs: ["/ui", "/ui/*", "/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/*"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard-anonymous
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-dashboard-anonymous
subjects:
- kind: User
  name: system:anonymous
EOF

参考

Enable Access for Kubernetes Dashboard via external VIP or Floating IP

介绍 | w3xse7en

hyper-v 安装 centos

Centos

虚拟机

网络

kubespray 安装 k8s

参考

配置免密码登录

下载kubespray

安装python3-pip

安装kubespray依赖

使用ansible对每台机器进行批处理

生成kubespray所需配置

下载依赖

(可选)开启helm

安装

验证

重置

Dashboard

添加/删除节点

参考

添加节点

配置ansible访问新节点(192.168.1.203)

新增node4

下载依赖

添加节点

移除节点

创建 service account

参考

创建 service account cicd-user

为 cicd-user 配置权限

其他实例使用 cicd-user

solve k8s problem

介绍

kubectl get nodes 证书ip问题

system:anonymous 无法访问 k8s dashboard

创建 `service account` `cicd-user`

为 `cicd-user` 配置权限

其他实例使用 `cicd-user`