harvester · noahgildersleeve · May 3, 2022 · May 3, 2022
diff --git a/...content/manual/harvester-rancher/30-configure-harvester-loadbalancer-service.md b/...content/manual/harvester-rancher/30-configure-harvester-loadbalancer-service.md
@@ -4,35 +4,35 @@ title: 30-Configure Harvester LoadBalancer service
 Prerequisite: 
 Already provision RKE1/RKE2 cluster in previous test case
 
-- Open `Global Settings` in hamburger menu
-- Replace `ui-dashboard-index` to `https://releases.rancher.com/harvester-ui/dashboard/latest/index.html`
-- Change `ui-offline-preferred` to `Remote`
-- Refresh the current page (ctrl + r)
-- Open provisioned RKE2 cluster from hamburger menu
-- Drop down `Service Discovery`
-- Click `Services`
-- Click Create 
-- Select `Load Balancer`
+1. Open `Global Settings` in hamburger menu
+1. Replace `ui-dashboard-index` to `https://releases.rancher.com/harvester-ui/dashboard/latest/index.html`
+1. Change `ui-offline-preferred` to `Remote`
+1. Refresh the current page (ctrl + r)
+1. Open provisioned RKE2 cluster from hamburger menu
+1. Drop down `Service Discovery`
+1. Click `Services`
+1. Click Create 
+1. Select `Load Balancer`
 
 ![image.png](https://images.zenhubusercontent.com/61519853321ea20d65443929/f628094c-a195-4f99-9fb7-858d759dc019)
 
-- Given service name to make the load balancer name composed of the cluster name, namespace, svc name, and suffix(8 characters) more than 63 characters
-- Provide Listening port and Target port
+1. Given service name to make the load balancer name composed of the cluster name, namespace, svc name, and suffix(8 characters) more than 63 characters
+1. Provide Listening port and Target port
 
 ![image.png](https://images.zenhubusercontent.com/61519853321ea20d65443929/2c20c759-4769-438b-94ad-5b995ba66873)
 
-- Click `Add-on Config`
-- Select Health Check port
-- Select `dhcp` as IPAM mode
-- Provide Health Check Threshold
-- Provide Health Check Failure Threshold
-- Provide Health Check Period
-- Provide Health Check Timeout
-- Click Create button
+1. Click `Add-on Config`
+1. Select Health Check port
+1. Select `dhcp` as IPAM mode
+1. Provide Health Check Threshold
+1. Provide Health Check Failure Threshold
+1. Provide Health Check Period
+1. Provide Health Check Timeout
+1. Click Create button
 
 ![image.png](https://images.zenhubusercontent.com/61519853321ea20d65443929/a8d11df6-cc76-4897-8310-def670682775)
 
-- Create another load balancer service with the name  characters.
+1. Create another load balancer service with the name  characters.
 
 ## Expected Results
 - Can create the load balance service correctly

diff --git a/...tent/manual/harvester-rancher/31-specify-pool-ipam-mode-loadbalancer-service.md b/...tent/manual/harvester-rancher/31-specify-pool-ipam-mode-loadbalancer-service.md
@@ -8,6 +8,10 @@ Already provision RKE1/RKE2 cluster in previous test case
 1. Replace `ui-dashboard-index` to `https://releases.rancher.com/harvester-ui/dashboard/latest/index.html`
 1. Change `ui-offline-preferred` to `Remote`
 1. Refresh the current page (ctrl + r)
+1. Access Harvester dashboard UI
+1. Go to Settings
+1. Create a vip-pool in Harvester settings.
+  ![image](https://user-images.githubusercontent.com/29251855/158514040-bfcd9ff3-964a-4511-94d7-a497ef88848f.png)
 1. Open provisioned RKE2 cluster from hamburger menu
 1. Drop down `Service Discovery`
 1. Click `Services`
@@ -34,4 +38,4 @@ Already provision RKE1/RKE2 cluster in previous test case
 
 ## Expected Results
 1. Can create load balance service correctly
-1. Can operate and foward workload as expected
+1. Can operate and route to deployed service correctly
diff --git a/...-harvester-from-airgapped-rancher-rke1.md → ...harvester-from-airgapped-rancher-proxy.md b/...-harvester-from-airgapped-rancher-rke1.md → ...harvester-from-airgapped-rancher-proxy.md
@@ -1,5 +1,5 @@
 ---
-title: 57-Import airgapped harvester from airgapped rancher rke1
+title: 57-Import airgapped harvester from airgapped rancher with Proxy
 ---
 
 * Related task: [#1052](https://github.com/harvester/harvester/issues/1052) Test Air gap with Rancher integration

diff --git a/docs/content/manual/harvester-rancher/59-create-k3s-kubernetes-cluster.md b/docs/content/manual/harvester-rancher/59-create-k3s-kubernetes-cluster.md
@@ -0,0 +1,49 @@
+---
+title: 59-Create K3s Kubernetes Cluster
+---
+1. Click Cluster Management
+1. Click Cloud Credentials
+1. Click create and select `Harvester`
+1. Input credential name
+1. Select existing cluster in the `Imported Cluster` list
+1. Click Create
+
+![image.png](https://images.zenhubusercontent.com/61519853321ea20d65443929/4a2f6a52-dac7-4a27-84b3-14cbeb4156aa)
+
+1. Click Clusters 
+1. Click Create
+1. Toggle RKE2/K3s 
+1. Select Harvester
+1. Input `Cluster Name`
+1. Select `default` namespace
+1. Select ubuntu image 
+1. Select network `vlan1`
+1. Input SSH User: `ubuntu`
+![image](https://user-images.githubusercontent.com/29251855/166188165-588adc48-fb41-4a01-a59e-9b059eb06949.png)
+1. Click `Show Advanced`
+1. Add the following user data:
+    ```
+    password: 123456
+    chpasswd: { expire: false }
+    ssh_pwauth: true
+    ```
+    ![image](https://user-images.githubusercontent.com/29251855/166188400-2e5e3051-f5ce-4b40-8497-71d6ff3cfdfa.png)
+
+1. Click the drop down Kubernetes version list
+1. Select K3s kubernetes version
+    ![image](https://user-images.githubusercontent.com/29251855/165777245-6059f10d-da2f-49d3-9da3-3b72491f7051.png)
+
+1. Click `Advanced`
+1. Add `Arguments`
+1. Add `cloud-provider=external`
+    ![image](https://user-images.githubusercontent.com/29251855/166189212-d422a433-7ac7-4f26-80fd-452c6df966ae.png)
+1. Click Create
+1. Wait for K3s cluster provisioning complete
+
+## Expected Results
+1. Provision K3s cluster successfully with `Running` status
+![image](https://user-images.githubusercontent.com/29251855/166189728-d98b92a3-aa0d-44a8-951c-e637f9530031.png)
+
+1. Can acccess K3s cluster to check all resources and services
+![image](https://user-images.githubusercontent.com/29251855/166189812-2fa41514-e416-4a0d-91ec-537ecd9a3b00.png)
+
diff --git a/docs/content/manual/harvester-rancher/60-delete-k3s-kubernetes-cluster.md b/docs/content/manual/harvester-rancher/60-delete-k3s-kubernetes-cluster.md
@@ -0,0 +1,13 @@
+---
+title: 60-Delete K3s Kubernetes Cluster
+---
+1. Open Cluster Management
+1. Check provisioned K3s cluster 
+1. Click `Delete` from menu
+
+
+## Expected Results
+1. Can remove K3s Cluster and disapper on Cluster page
+1. K3s Cluster will be removed from rancher menu under explore cluster
+1. K3s virtual machine should be also be removed from Harvester 
+
diff --git a/...t/manual/harvester-rancher/61-deploy-harvester-cloud-provider-to-k3s-cluster.md b/...t/manual/harvester-rancher/61-deploy-harvester-cloud-provider-to-k3s-cluster.md
@@ -0,0 +1,39 @@
+---
+title: 61-Deploy Harvester cloud provider to k3s Cluster	
+---
+
+* Related task: [#1812](https://github.com/harvester/harvester/issues/1812) K3s cloud provider and csi driver support
+
+### Environment Setup
+1. Docker install rancher v2.6.4
+1. Create one node harvester with enough resource
+
+### Verify steps
+Follow step **1~13** in tets plan `59-Create K3s Kubernetes Cluster`
+
+1. Click the Edit yaml button
+  ![image](https://user-images.githubusercontent.com/29251855/166190410-47331a84-1d4e-4478-9d85-e68a3da91626.png)
+1. Set `disable-cloud-provider: true` to disable default k3s cloud provider.
+  ![image](https://user-images.githubusercontent.com/29251855/158510820-4d8a0021-1675-4c92-86b9-a6427f2e382b.png)
+1. Add `cloud-provider=external` to use harvester cloud provider.
+  ![image](https://user-images.githubusercontent.com/29251855/158511002-47a4a532-7f67-4eb0-8da4-074c6d9752e9.png)
+1. Create K3s cluster 
+![image](https://user-images.githubusercontent.com/29251855/158511706-1c0c6af5-8909-4b1d-bc2a-0fa2fa26e000.png)
+1. Download the [Generate addon configuration](https://github.com/harvester/cloud-provider-harvester/blob/master/deploy/generate_addon.sh) for cloud provider 
+1. Download Harvester kubeconfig and add into your local ~/.kube/config file 
+1. Generate K3s kubeconfig by running generate addon script 
+  ` ./deploy/generate_addon.sh <k3s cluster name> <namespace>`
+  e.g `./generate_addon.sh k3s-focal-cloud-provider default`
+1. Copy the kubeconfig content 
+1. ssh to K3s VM
+  ![image](https://user-images.githubusercontent.com/29251855/158534901-8fd22159-6a04-4592-ba25-ba4d73742a20.png)
+1. Add kubeconfig content to `/etc/kubernetes/cloud-config` file, remember to align the yaml layout
+1. Install Harvester cloud provider 
+  ![image](https://user-images.githubusercontent.com/29251855/158512528-42ff575a-87a6-4424-bfb5-fa7af94ea74d.png)
+  ![image](https://user-images.githubusercontent.com/29251855/158512667-18b0249c-f859-4ae4-96b7-42ce873cb97a.png)
+
+
+## Expected Results
+1. Can install the Harvester cloud provider on k3s cluster correctly
+  ![image](https://user-images.githubusercontent.com/29251855/158512758-d06df2f6-7094-4d41-b960-d50b26cd23fb.png)
+
diff --git a/docs/content/manual/harvester-rancher/62-configure-k3s-dhcp-loadbalancer.md b/docs/content/manual/harvester-rancher/62-configure-k3s-dhcp-loadbalancer.md
@@ -0,0 +1,46 @@
+---
+title: 62-Configure the K3s "DHCP" LoadBalancer service
+---
+Prerequisite: 
+Already provision K3s cluster and cloud provider on test plan 
+* 59-Create K3s Kubernetes Cluster 
+* 61-Deploy Harvester cloud provider to k3s Cluster 
+
+#### Create Nginx workload for testing
+1. Create a test-nginx deployment with image nginx:latest.
+  ![image](https://user-images.githubusercontent.com/29251855/158512919-a35a079a-aa75-4ce8-bac6-a79438a2e112.png)  
+1. Add pod label test: test.
+  ![image](https://user-images.githubusercontent.com/29251855/158513017-5afc909a-662a-4f4e-b867-2555241a2cbd.png)
+  ![image](https://user-images.githubusercontent.com/29251855/158513105-09ab472b-7cd4-4352-b4e1-84f673ee7088.png)
+
+#### Create a DHCP LoadBalancer
+1. Open Kubectl shell.
+1. Create `test-dhcp-lb.yaml` file.
+    ```
+    apiVersion: v1
+    kind: Service
+    metadata:
+        annotations:
+        cloudprovider.harvesterhci.io/ipam: dhcp
+        name: test-dhcp-lb
+        namespace: default
+    spec:
+        ports:
+        - name: http
+        nodePort: 30172
+        port: 8080
+        protocol: TCP
+        targetPort: 80
+        selector:
+        test: test
+        sessionAffinity: None
+        type: LoadBalancer
+    ```
+1. Run `k apply -f test-dhcp-lb.yaml` to apply it.
+  ![image](https://user-images.githubusercontent.com/29251855/158513659-3e0c487b-c819-492c-8c62-62fc644fd858.png)
+1. The Pool LoadBalancer should get an IP from vip-pool and work.
+  ![image](https://user-images.githubusercontent.com/29251855/158513800-70d7c0ba-5a4b-4462-90df-c8d05b5a389d.png)
+
+## Expected Results
+1. Can create load balance service correctly
+1. Can route workload to nginx deployment
diff --git a/docs/content/manual/harvester-rancher/63-configure-k3s-pool-loadbalancer.md b/docs/content/manual/harvester-rancher/63-configure-k3s-pool-loadbalancer.md
@@ -0,0 +1,48 @@
+---
+title: 63-Configure the K3s "Pool" LoadBalancer service
+---
+Prerequisite: 
+Already provision K3s cluster and cloud provider on test plan 
+* 59-Create K3s Kubernetes Cluster 
+* 61-Deploy Harvester cloud provider to k3s Cluster 
+
+#### Create Nginx workload for testing
+1. Create a test-nginx deployment with image nginx:latest.
+  ![image](https://user-images.githubusercontent.com/29251855/158512919-a35a079a-aa75-4ce8-bac6-a79438a2e112.png)  
+1. Add pod label test: test.
+  ![image](https://user-images.githubusercontent.com/29251855/158513017-5afc909a-662a-4f4e-b867-2555241a2cbd.png)
+
+#### Create a Pool LoadBalancer
+1. Modify vip-pool in Harvester settings.
+  ![image](https://user-images.githubusercontent.com/29251855/158514040-bfcd9ff3-964a-4511-94d7-a497ef88848f.png)
+
+1. Open Kubectl shell.
+1. Create `test-pool-lb.yaml` file.
+    ```
+    apiVersion: v1
+    kind: Service
+    metadata:
+      annotations:
+        cloudprovider.harvesterhci.io/ipam: pool
+      name: test-pool-lb
+      namespace: default
+    spec:
+      ports:
+      - name: http
+        nodePort: 32155
+        port: 8080
+        protocol: TCP
+        targetPort: 80
+      selector:
+        test: test
+      sessionAffinity: None
+      type: LoadBalancer
+    ```
+1. Run `k apply -f test-pool-lb.yaml` to apply it.
+1. The Pool LoadBalancer should get an IP from vip-pool and work.
+
+
+## Expected Results
+1. Can create `Pool` load balance service correctly
+1. Can route workload to nginx deployment
+  ![image](https://user-images.githubusercontent.com/29251855/158514315-1b570f64-fe18-400e-acc4-56d03bc30e61.png)
diff --git a/docs/content/manual/harvester-rancher/64-configure-k3s-dhcp-LB-healcheck.md b/docs/content/manual/harvester-rancher/64-configure-k3s-dhcp-LB-healcheck.md
@@ -0,0 +1,19 @@
+---
+title: 62-Configure the K3s "DHCP" LoadBalancer service
+---
+Prerequisite: 
+Already provision K3s cluster and cloud provider on test plan 
+* 59-Create K3s Kubernetes Cluster 
+* 61-Deploy Harvester cloud provider to k3s Cluster
+* 62-Configure the K3s "DHCP" LoadBalancer service
+
+1. A `Working` DHCP load balancer service created on K3s cluster
+1. Edit Load balancer config
+1. Check the "Add-on Config" tabs
+1. Configure `port`, `IPAM` and `health check` related setting on `Add-on Config` page
+![image](https://user-images.githubusercontent.com/29251855/141245366-799057f1-2aa7-4d7a-90d2-5e11541ddbc3.png)
+
+
+## Expected Results
+1. Can create load balance service correctly
+1. Can route workload to nginx deployment
diff --git a/docs/content/manual/harvester-rancher/65-configure-k3s-pool-LB-healthcheck.md b/docs/content/manual/harvester-rancher/65-configure-k3s-pool-LB-healthcheck.md
@@ -0,0 +1,19 @@
+---
+title: 65-Configure the K3s "Pool" LoadBalancer health check
+---
+Prerequisite: 
+Already provision K3s cluster and cloud provider on test plan 
+* 59-Create K3s Kubernetes Cluster 
+* 61-Deploy Harvester cloud provider to k3s Cluster
+* 63-Configure the K3s "Pool" LoadBalancer service
+
+1. A `Working` DHCP load balancer service created on K3s cluster
+1. Edit Load balancer config
+1. Check the "Add-on Config" tabs
+1. Configure `port`, `IPAM` and `health check` related setting on `Add-on Config` page
+![image](https://user-images.githubusercontent.com/29251855/141245366-799057f1-2aa7-4d7a-90d2-5e11541ddbc3.png)
+
+
+## Expected Results
+1. Can create load balance service correctly
+1. Can route workload to nginx deployment
diff --git a/...ntent/manual/harvester-rancher/66-deploy-harvester-csi-driver-to-k3s-cluster.md b/...ntent/manual/harvester-rancher/66-deploy-harvester-csi-driver-to-k3s-cluster.md
@@ -0,0 +1,53 @@
+---
+title: 66-Deploy Harvester csi driver to k3s Cluster	
+---
+
+* Related task: [#1812](https://github.com/harvester/harvester/issues/1812) K3s cloud provider and csi driver support
+
+### Environment Setup
+1. Docker install rancher v2.6.4
+1. Create one node harvester with enough resource
+
+### Verify steps
+Follow step **1~13** in tets plan `59-Create K3s Kubernetes Cluster`
+
+1. Download the [Generate addon configuration](https://github.com/harvester/harvester-csi-driver/blob/master/deploy/generate_addon.sh) for csi driver
+1. ssh to harvester node 1
+1. Execute `cat /etc/rancher/rke2/rke2.yaml`  
+1. change the server value from https://127.0.0.1:6443 to your node1 IP
+1. Copy the kubeconfig and add into your local ~/.kube/config file
+1. Generate K3s kubeconfig by running generate addon script
+` ./deploy/generate_addon.sh <k3s cluster name> <namespace>`
+e.g `./generate_addon.sh k3s-csi-cluster default`
+1. Copy the ks3 kubeconfig content start with `cloud-provider-config:` between `addons_include:`
+1. ssh to K3s VM
+  ![image](https://user-images.githubusercontent.com/29251855/158932912-38297b70-7546-4349-801f-6a4b8b973305.png)
+
+1. Add kubeconfig content to `/etc/kubernetes/cloud-config`file, remember to align the yaml layout
+1. Install Harvester csi driver
+  ![image](https://user-images.githubusercontent.com/29251855/158550983-61cff655-66a6-4a49-96bd-6e208f4fc9d8.png)
+  ![image](https://user-images.githubusercontent.com/29251855/158551034-090cee3d-9bd6-425c-84d8-16f6a66a5c64.png)
+  ![image](https://user-images.githubusercontent.com/29251855/158933473-a0172a62-5f7c-4e68-860c-c9cb38275791.png)
+  ![image](https://user-images.githubusercontent.com/29251855/158933756-d4198111-ae05-4a7d-8ea8-d21e3f3d2b87.png)
+
+
+
+
+## Expected Results
+1. Can deploy K3s cluster on harvester with kubernetes version `v1.23.4+k3s1`
+  ![image](https://user-images.githubusercontent.com/29251855/158935474-9d6f1c37-ea59-485d-83f5-6f1b19ebfa98.png)
+
+1. Can correctly install harvester csi driver on K3s cluster
+  ![image](https://user-images.githubusercontent.com/29251855/158933473-a0172a62-5f7c-4e68-860c-c9cb38275791.png)
+  ![image](https://user-images.githubusercontent.com/29251855/158933756-d4198111-ae05-4a7d-8ea8-d21e3f3d2b87.png)
+
+1. Can deploy nginx service with new PVC created
+  ![image](https://user-images.githubusercontent.com/29251855/158934469-6b050d39-a45c-493d-9e29-89e16c1cf23d.png)
+  ![image](https://user-images.githubusercontent.com/29251855/158934499-6c7d9525-f43a-4426-8786-1e4aec099964.png)
+  ![image](https://user-images.githubusercontent.com/29251855/158934531-9c42b704-67cc-4c91-b24e-fa29f7183f05.png)
+1. Can allocate size in nginx container 
+  ![image](https://user-images.githubusercontent.com/29251855/158934986-c08ddccc-3b33-4508-9861-6d10c5ded3c5.png)
+1. Can resize and delete volume with harvester storage class
+  ![image](https://user-images.githubusercontent.com/29251855/158935263-a8b6fa9e-1f4b-43ae-a687-6df3e34986a1.png)
+  ![image](https://user-images.githubusercontent.com/29251855/158935288-67409de3-d5d5-4c9d-af04-421f4153658b.png)
+