Here are the cluster definitions for apiVersion "vlabs":
| Name | Required | Description |
|---|---|---|
| apiVersion | yes | The version of the template. For "vlabs" the value is "vlabs" |
orchestratorProfile describes the orchestrator settings.
| Name | Required | Description |
|---|---|---|
| orchestratorType | yes | Specifies the orchestrator type for the cluster |
| orchestratorRelease | no | Specifies the orchestrator release for the cluster |
| orchestratorVersion | no | Specifies the orchestrator version for the cluster |
Here are the valid values for the orchestrator types:
Kubernetes- this represents the Kubernetes orchestrator.
To learn more about supported versions, run the get-versions command:
$ aks-engine get-versionskubernetesConfig describes Kubernetes specific configuration.
| Name | Required | Description |
|---|---|---|
| addons | no | Configure various Kubernetes addons configuration. See addons configuration below |
| apiServerConfig | no | Configure various runtime configuration for apiserver. See apiServerConfig below |
| cloudControllerManagerConfig | no | Configure various runtime configuration for cloud-controller-manager. See cloudControllerManagerConfig below |
| clusterSubnet | no | The IP subnet used for allocating IP addresses for pod network interfaces. The subnet must be in the VNET address space. With Azure CNI enabled, the default value is 10.240.0.0/12. Without Azure CNI, the default value is 10.244.0.0/16. |
| containerRuntime | no | The container runtime to use as a backend. The default is docker. The other options are kata-containers, and containerd |
| controllerManagerConfig | no | Configure various runtime configuration for controller-manager. See controllerManagerConfig below |
| customWindowsPackageURL | no | Configure custom windows Kubernetes release package URL for deployment on Windows that is generated by scripts/build-windows-k8s.sh. The format of this file is a zip file with multiple items (binaries, cni, infra container) in it. This setting will be depreciated in future release of aks-engine where the binaries will be pulled in the format of Kubernetes releases that only contain the kubernetes binaries. |
| WindowsNodeBinariesURL | no | Windows Kubernetes Node binaries can be provided in the format of Kubernetes release (example: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#node-binaries-1). This setting allows overriding the binaries for custom builds. |
| dnsServiceIP | no | IP address for kube-dns to listen on. If specified must be in the range of serviceCidr |
| mobyVersion | no (for development only) | Enables an explicit moby version, e.g. 3.0.3. Default is 3.0.5. This kubernetesConfig property is for development only, and applies only to cluster creation: aks-engine upgrade will always statically set mobyVersion to the default version at the time of upgrade, to ensure that upgraded clusters have the most recent, validated version of moby. |
| containerdVersion | no (for development only) | Enables an explicit containerd version, e.g. 1.1.4. Default is 1.1.5. This kubernetesConfig property is for development only, and applies only to cluster creation: aks-engine upgrade will always statically set containerdVersion to the default version at the time of upgrade, to ensure that upgraded clusters have the most recent, validated version of containerd. |
| dockerBridgeSubnet | no | The specific IP and subnet used for allocating IP addresses for the docker bridge network created on the kubernetes master and agents. Default value is 172.17.0.1/16. This value is used to configure the docker daemon using the --bip flag |
| enableAggregatedAPIs | no | Enable Kubernetes Aggregated APIs.This is required by Service Catalog. (boolean - default is true for k8s versions greater or equal to 1.9.0, false otherwise) |
| enableDataEncryptionAtRest | no | Enable kubernetes data encryption at rest.This is currently an alpha feature. (boolean - default == false) |
| enableEncryptionWithExternalKms | no | Enable kubernetes data encryption at rest with external KMS.This is currently an alpha feature. (boolean - default == false) |
| enablePodSecurityPolicy | no | Enable kubernetes pod security policy.This is currently a beta feature. (boolean - default == false for Kubernetes 1.14 and earlier. This setting is ignored and forced to true for Kubernetes v1.15+) |
| enableRbac | no | Enable Kubernetes RBAC (boolean - default == true) RBAC support is required for Kubernetes 1.15.0 and greater, so enableRbac=false is not an allowed configuration for clusters >= 1.15.0. If you upgrade a cluster to 1.15.0 or greater from a version less than 1.15, and RBAC is disabled, the cluster configuration will be statically modified to enable RBAC as a result of running aks-engine upgrade. |
| etcdDiskSizeGB | no | Size in GB to assign to etcd data volume. Defaults (if no user value provided) are: 256 GB for clusters up to 3 nodes; 512 GB for clusters with between 4 and 10 nodes; 1024 GB for clusters with between 11 and 20 nodes; and 2048 GB for clusters with more than 20 nodes |
| etcdEncryptionKey | no | Enryption key to be used if enableDataEncryptionAtRest is enabled. Defaults to a random, generated, key |
| etcdVersion | no (for development only) | Enables an explicit etcd version, e.g. 3.2.23. Default is 3.3.15. This kubernetesConfig property is for development only, and recommended only for ephemeral clusters. However, you may use aks-engine upgrade on a cluster with an api model that includes a user-modified etcdVersion value. If aks-engine upgrade determines that the user-modified version is greater than the current AKS Engine default, aks-engine upgrade will not replace the newer version with an older version. However, if aks-engine upgrade determines that the user-modified version is older than the current AKS Engine default, it will build the newly upgraded master node VMs with the newer, AKS Engine default version of etcd. |
| gcHighThreshold | no | Sets the --image-gc-high-threshold value on the kublet configuration. Default is 85. See kubelet Garbage Collection |
| gcLowThreshold | no | Sets the --image-gc-low-threshold value on the kublet configuration. Default is 80. See kubelet Garbage Collection |
| kubeletConfig | no | Configure various runtime configuration for kubelet. See kubeletConfig below |
| kubernetesImageBase | no | Specifies the default image base URL (everything preceding the actual image filename) to be used for all kubernetes-related containers such as hyperkube, cloud-controller-manager, pause, addon-manager, heapster, exechealthz etc. e.g., k8s.gcr.io/ |
| loadBalancerSku | no | Sku of Load Balancer and Public IP. Candidate values are: basic and standard. If not set, it will be default to basic. Requires Kubernetes 1.11 or newer. NOTE: VMs behind standard SKU load balancer will not be able to access the internet without an outbound rule configured with at least one frontend IP. We have created a loadbalancer with an outbound rule and with agent nodes added to the backend pool, as described in the Outbound NAT for internal Standard Load Balancer scenarios doc |
| networkPlugin | no | Specifies the network plugin implementation for the cluster. Valid values are:"azure" (default), which provides an Azure native networking experience "kubenet" for k8s software networking implementation. "flannel" for using CoreOS Flannel "cilium" for using the default Cilium CNI IPAM |
| networkPolicy | no | Specifies the network policy enforcement tool for the cluster (currently Linux-only). Valid values are:"calico" for Calico network policy."cilium" for cilium network policy (Lin), and "azure" (experimental) for Azure CNI-compliant network policy (note: Azure CNI-compliant network policy requires explicit "networkPlugin": "azure" configuration as well).See network policy examples for more information. |
| privateCluster | no | Build a cluster without public addresses assigned. See privateClusters below. |
| schedulerConfig | no | Configure various runtime configuration for scheduler. See schedulerConfig below |
| serviceCidr | no | IP range for Service IPs, Default is "10.0.0.0/16". This range is never routed outside of a node so does not need to lie within clusterSubnet or the VNET |
| useInstanceMetadata | no | Use the Azure cloudprovider instance metadata service for appropriate resource discovery operations. Default is true |
| useManagedIdentity | no | Includes and uses MSI identities for all interactions with the Azure Resource Manager (ARM) API. Instead of using a static service principal written to /etc/kubernetes/azure.json, Kubernetes will use a dynamic, time-limited token fetched from the MSI extension running on master and agent nodes. This support is currently alpha and requires Kubernetes v1.9.1 or newer. (boolean - default == false). When MasterProfile is using VirtualMachineScaleSets, this feature requires Kubernetes v1.12 or newer as we default to using user assigned identity. |
| azureCNIURLLinux | no | Deploy a private build of Azure CNI on Linux nodes. This should be a full path to the .tar.gz |
| azureCNIURLWindows | no | Deploy a private build of Azure CNI on Windows nodes. This should be a full path to the .tar.gz |
| maximumLoadBalancerRuleCount | no | Maximum allowed LoadBalancer Rule Count is the limit enforced by Azure Load balancer. Default is 250 |
| kubeProxyMode | no | kube-proxy --proxy-mode value, either "iptables" or "ipvs". Default is "iptables". See https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/ for further reference. |
| outboundRuleIdleTimeoutInMinutes | no | Specifies a value for IdleTimeoutInMinutes to control the outbound flow idle timeout of the agent standard loadbalancer. This value is set greater than the default Linux idle timeout (15.4 min): https://pracucci.com/linux-tcp-rto-min-max-and-tcp-retries2.html |
| cloudProviderBackoff | no | Use the Azure cloudprovider exponential backoff implementation when encountering retry-able errors from the Azure API. Defaults to true for Kubernetes v1.14.0 and greater. |
| cloudProviderBackoffMode | no | Which version of the Azure cloudprovider backoff implementation to use: the options are "v1" or "v2" (Kubernetes v1.14.0 or greater only). "v2" is a more recent backoff implementation which better honors Azure API HTTP headers to align backoff timings with the Azure API. Defaults to "v2" for Kubernetes v1.14.0 and greater, and "v1" for earlier versions of Kubernetes. |
| cloudProviderBackoffRetries | no | How many backoff retries before terminally failing the original Azure API operation. Defaults to 6. |
| cloudProviderBackoffJitter | no | Only available in the "v1" cloudProviderBackoffMode: how much random variation to inject in retry timings, to better distribute stacked retries. Defaults to 1 when using cloudProviderBackoffMode "v1", which effectively disables jitter. |
| cloudProviderBackoffDuration | no | The base duration, in seconds, in between retry attempts. Defaults to 5. |
| cloudProviderBackoffExponent | no | Only available in the "v1" cloudProviderBackoffMode: the factor to multiply cloudProviderBackoffDuration by for each retry iteration. Defaults to 1.5 when using cloudProviderBackoffMode "v1". |
| cloudProviderRateLimit | no | Use the Azure cloudprovider rate limiter to reduce the rate of calls to Azure APIs. Defaults to true. |
| cloudProviderRateLimitBucket | no | The size of the overflow read request queue when cloudProviderRateLimit is enabled. Defaults to the calculation "100 * number of agent pools in cluster configuration. |
| cloudProviderRateLimitQPS | no | QPS for Azure cloudprovider read request rate limiter enforcement. Defaults to a minimum factor of 0.1 (i.e., 10%) of the cloudProviderRateLimitBucket (e.g., given a cloudProviderRateLimitBucket of 100, cloudProviderRateLimitQPS defaults to 10), or 3, whichever is greater. |
| cloudProviderRateLimitBucketWrite | no | The size of the overflow write request queue when cloudProviderRateLimit is enabled. Follows the same defaults calculation as cloudProviderRateLimitBucket. |
| cloudProviderRateLimitQPSWrite | no | QPS for Azure cloudprovider write request rate limiter enforcement. Follows the same defaults calculation as cloudProviderRateLimitQPS. |
| cloudProviderDisableOutboundSNAT | no | For clusters w/ Standard LB only: enforces the disabling of outbound NAT for that load balancing rule. See here for more details. Defaults to false. |
addons is an interface to define user-configurable Kubernetes componentry. It is a child property of kubernetesConfig. Below is a list of currently available addons:
| Name of addon | Enabled by default? | How many pods | Description |
|---|---|---|---|
| tiller | false | 1 | Delivers the Helm server-side component: tiller. See https://github.com/kubernetes/helm for more info |
| kubernetes-dashboard | true | 1 | Delivers the Kubernetes dashboard component. See https://github.com/kubernetes/dashboard for more info |
| rescheduler | false | 1 | Delivers the Kubernetes rescheduler component |
| cluster-autoscaler | false | 1 | Delivers the Kubernetes cluster autoscaler component. See https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/azure for more info; only supported for VMSS clusters on the first agent pool. |
| nvidia-device-plugin | true if using a Kubernetes cluster (v1.10+) with an N-series agent pool | 1 | Delivers the Kubernetes NVIDIA device plugin component. See https://github.com/NVIDIA/k8s-device-plugin for more info |
| container-monitoring | false | 1 | Delivers the Kubernetes container monitoring component |
| blobfuse-flexvolume | true | as many as linux agent nodes | Access virtual filesystem backed by the Azure Blob storage |
| smb-flexvolume | false | as many as linux agent nodes | Access SMB server by using CIFS/SMB protocol |
| keyvault-flexvolume | true | as many as linux agent nodes | Access secrets, keys, and certs in Azure Key Vault from pods |
| aad-pod-identity | false | 1 + 1 on each linux agent nodes | Assign Azure Active Directory Identities to Kubernetes applications |
| scheduled-maintenance | false | 1 + 1 on each linux agent nodes | Cordon and drain node during planned/unplanned azure maintenance |
| azuredisk-csi-driver | true if using a Kubernetes cluster (v1.13+) with useCloudControllerManager enabled |
1 + 1 on each linux agent nodes | Allows Kubernetes to use Azure Disk volume |
| azurefile-csi-driver | true if using a Kubernetes cluster (v1.13+) with useCloudControllerManager enabled |
1 + 1 on each linux agent nodes | Allows Kubernetes to use Azure File volume |
| azure-policy | false | 2 | Open Policy Agent Gatekeeper with Azure Policy integration |
To give a bit more info on the addons property: We've tried to expose the basic bits of data that allow useful configuration of these cluster features. Here are some example usage patterns that will unpack what addons provide:
To enable an addon (using "tiller" as an example):
"kubernetesConfig": {
"addons": [
{
"name": "tiller",
"enabled" : true
}
]
}As you can see above, addons is an array child property of kubernetesConfig. Each addon that you want to add custom configuration to would be represented as an object item in the array. For example, to disable both tiller and dashboard:
"kubernetesConfig": {
"addons": [
{
"name": "tiller",
"enabled" : false
},
{
"name": "kubernetes-dashboard",
"enabled" : false
}
]
}More usefully, let's add some custom configuration to the above addons:
"kubernetesConfig": {
"addons": [
{
"name": "tiller",
"enabled": true,
"containers": [
{
"name": "tiller",
"image": "myDockerHubUser/tiller:v3.0.0-alpha",
"cpuRequests": "1",
"memoryRequests": "1024Mi",
"cpuLimits": "1",
"memoryLimits": "1024Mi"
}
]
},
{
"name": "kubernetes-dashboard",
"enabled": true,
"containers": [
{
"name": "kubernetes-dashboard",
"cpuRequests": "50m",
"memoryRequests": "512Mi",
"cpuLimits": "50m",
"memoryLimits": "512Mi"
}
]
},
{
"name": "cluster-autoscaler",
"enabled": true,
"containers": [
{
"name": "cluster-autoscaler",
"cpuRequests": "100m",
"memoryRequests": "300Mi",
"cpuLimits": "100m",
"memoryLimits": "300Mi"
}
],
"config": {
"max-nodes": "5",
"min-nodes": "1",
"scan-interval": "10s"
}
}
]
}Above you see custom configuration for both tiller and kubernetes-dashboard. Both include specific resource limit values across the following dimensions:
- cpuRequests
- memoryRequests
- cpuLimits
- memoryLimits
See https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ for more on Kubernetes resource limits.
Additionally above, we specified a custom docker image for tiller, let's say we want to build a cluster and test an alpha version of tiller in it. Important note! customizing the image is not sticky across upgrade/scale, to ensure that aks-engine always delivers a version-curated, known-working addon when moving a cluster to a new version. Considering all that, providing a custom image reference for an addon configuration should be considered for testing/development, but not for a production cluster. If you'd like to entirely customize one of the addons available, including across scale/upgrade operations, you may include in an addon's spec a base64-encoded string of a Kubernetes yaml manifest. E.g.,
"kubernetesConfig": {
"addons": [
{
"name": "kube-proxy-daemonset",
"enabled": true,
"data": "IGFwaVZlcnNpb246IHYxCmtpbmQ6IFNlcnZpY2UKbWV0YWRhdGE6CiAgbmFtZToga3ViZS1kbnMKICBuYW1lc3BhY2U6IGt1YmUtc3lzdGVtCiAgbGFiZWxzOgogICAgazhzLWFwcDoga3ViZS1kbnMKICAgIGt1YmVybmV0ZXMuaW8vY2x1c3Rlci1zZXJ2aWNlOiAidHJ1ZSIKICAgIGFkZG9ubWFuYWdlci5rdWJlcm5ldGVzLmlvL21vZGU6IFJlY29uY2lsZQogICAga3ViZXJuZXRlcy5pby9uYW1lOiAiS3ViZUROUyIKc3BlYzoKICBzZWxlY3RvcjoKICAgIGs4cy1hcHA6IGt1YmUtZG5zCiAgY2x1c3RlcklQOiA8a3ViZUROU1NlcnZpY2VJUD4KICBwb3J0czoKICAtIG5hbWU6IGRucwogICAgcG9ydDogNTMKICAgIHByb3RvY29sOiBVRFAKICAtIG5hbWU6IGRucy10Y3AKICAgIHBvcnQ6IDUzCiAgICBwcm90b2NvbDogVENQCi0tLQphcGlWZXJzaW9uOiB2MQpraW5kOiBTZXJ2aWNlQWNjb3VudAptZXRhZGF0YToKICBuYW1lOiBrdWJlLWRucwogIG5hbWVzcGFjZToga3ViZS1zeXN0ZW0KICBsYWJlbHM6CiAgICBrdWJlcm5ldGVzLmlvL2NsdXN0ZXItc2VydmljZTogInRydWUiCiAgICBhZGRvbm1hbmFnZXIua3ViZXJuZXRlcy5pby9tb2RlOiBSZWNvbmNpbGUKLS0tCmFwaVZlcnNpb246IHYxCmtpbmQ6IENvbmZpZ01hcAptZXRhZGF0YToKICBuYW1lOiBrdWJlLWRucwogIG5hbWVzcGFjZToga3ViZS1zeXN0ZW0KICBsYWJlbHM6CiAgICBhZGRvbm1hbmFnZXIua3ViZXJuZXRlcy5pby9tb2RlOiBFbnN1cmVFeGlzdHMKLS0tCmFwaVZlcnNpb246IGV4dGVuc2lvbnMvdjFiZXRhMQpraW5kOiBEZXBsb3ltZW50Cm1ldGFkYXRhOgogIG5hbWU6IGt1YmUtZG5zCiAgbmFtZXNwYWNlOiBrdWJlLXN5c3RlbQogIGxhYmVsczoKICAgIGs4cy1hcHA6IGt1YmUtZG5zCiAgICBrdWJlcm5ldGVzLmlvL2NsdXN0ZXItc2VydmljZTogInRydWUiCiAgICBhZGRvbm1hbmFnZXIua3ViZXJuZXRlcy5pby9tb2RlOiBSZWNvbmNpbGUKc3BlYzoKICAjIHJlcGxpY2FzOiBub3Qgc3BlY2lmaWVkIGhlcmU6CiAgIyAxLiBJbiBvcmRlciB0byBtYWtlIEFkZG9uIE1hbmFnZXIgZG8gbm90IHJlY29uY2lsZSB0aGlzIHJlcGxpY2FzIHBhcmFtZXRlci4KICAjIDIuIERlZmF1bHQgaXMgMS4KICAjIDMuIFdpbGwgYmUgdHVuZWQgaW4gcmVhbCB0aW1lIGlmIEROUyBob3Jpem9udGFsIGF1dG8tc2NhbGluZyBpcyB0dXJuZWQgb24uCiAgc3RyYXRlZ3k6CiAgICByb2xsaW5nVXBkYXRlOgogICAgICBtYXhTdXJnZTogMTAlCiAgICAgIG1heFVuYXZhaWxhYmxlOiAwCiAgc2VsZWN0b3I6CiAgICBtYXRjaExhYmVsczoKICAgICAgazhzLWFwcDoga3ViZS1kbnMKICB0ZW1wbGF0ZToKICAgIG1ldGFkYXRhOgogICAgICBsYWJlbHM6CiAgICAgICAgazhzLWFwcDoga3ViZS1kbnMKICAgICAgYW5ub3RhdGlvbnM6CiAgICAgICAgc2NoZWR1bGVyLmFscGhhLmt1YmVybmV0ZXMuaW8vY3JpdGljYWwtcG9kOiAnJwogICAgICAgIHNlY2NvbXAuc2VjdXJpdHkuYWxwaGEua3ViZXJuZXRlcy5pby9wb2Q6ICdkb2NrZXIvZGVmYXVsdCcKICAgIHNwZWM6CiAgICAgIHByaW9yaXR5Q2xhc3NOYW1lOiBzeXN0ZW0tY2x1c3Rlci1jcml0aWNhbAogICAgICB0b2xlcmF0aW9uczoKICAgICAgLSBrZXk6ICJDcml0aWNhbEFkZG9uc09ubHkiCiAgICAgICAgb3BlcmF0b3I6ICJFeGlzdHMiCiAgICAgIHZvbHVtZXM6CiAgICAgIC0gbmFtZToga3ViZS1kbnMtY29uZmlnCiAgICAgICAgY29uZmlnTWFwOgogICAgICAgICAgbmFtZToga3ViZS1kbnMKICAgICAgICAgIG9wdGlvbmFsOiB0cnVlCiAgICAgIGNvbnRhaW5lcnM6CiAgICAgIC0gbmFtZToga3ViZWRucwogICAgICAgIGltYWdlOiA8a3ViZXJuZXRlc0t1YmVETlNTcGVjPgogICAgICAgIHJlc291cmNlczoKICAgICAgICAgICMgVE9ETzogU2V0IG1lbW9yeSBsaW1pdHMgd2hlbiB3ZSd2ZSBwcm9maWxlZCB0aGUgY29udGFpbmVyIGZvciBsYXJnZQogICAgICAgICAgIyBjbHVzdGVycywgdGhlbiBzZXQgcmVxdWVzdCA9IGxpbWl0IHRvIGtlZXAgdGhpcyBjb250YWluZXIgaW4KICAgICAgICAgICMgZ3VhcmFudGVlZCBjbGFzcy4gQ3VycmVudGx5LCB0aGlzIGNvbnRhaW5lciBmYWxscyBpbnRvIHRoZQogICAgICAgICAgIyAiYnVyc3RhYmxlIiBjYXRlZ29yeSBzbyB0aGUga3ViZWxldCBkb2Vzbid0IGJhY2tvZmYgZnJvbSByZXN0YXJ0aW5nIGl0LgogICAgICAgICAgbGltaXRzOgogICAgICAgICAgICBtZW1vcnk6IDE3ME1pCiAgICAgICAgICByZXF1ZXN0czoKICAgICAgICAgICAgY3B1OiAxMDBtCiAgICAgICAgICAgIG1lbW9yeTogNzBNaQogICAgICAgIGxpdmVuZXNzUHJvYmU6CiAgICAgICAgICBodHRwR2V0OgogICAgICAgICAgICBwYXRoOiAvaGVhbHRoY2hlY2sva3ViZWRucwogICAgICAgICAgICBwb3J0OiAxMDA1NAogICAgICAgICAgICBzY2hlbWU6IEhUVFAKICAgICAgICAgIGluaXRpYWxEZWxheVNlY29uZHM6IDYwCiAgICAgICAgICB0aW1lb3V0U2Vjb25kczogNQogICAgICAgICAgc3VjY2Vzc1RocmVzaG9sZDogMQogICAgICAgICAgZmFpbHVyZVRocmVzaG9sZDogNQogICAgICAgIHJlYWRpbmVzc1Byb2JlOgogICAgICAgICAgaHR0cEdldDoKICAgICAgICAgICAgcGF0aDogL3JlYWRpbmVzcwogICAgICAgICAgICBwb3J0OiA4MDgxCiAgICAgICAgICAgIHNjaGVtZTogSFRUUAogICAgICAgICAgIyB3ZSBwb2xsIG9uIHBvZCBzdGFydHVwIGZvciB0aGUgS3ViZXJuZXRlcyBtYXN0ZXIgc2VydmljZSBhbmQKICAgICAgICAgICMgb25seSBzZXR1cCB0aGUgL3JlYWRpbmVzcyBIVFRQIHNlcnZlciBvbmNlIHRoYXQncyBhdmFpbGFibGUuCiAgICAgICAgICBpbml0aWFsRGVsYXlTZWNvbmRzOiAzCiAgICAgICAgICB0aW1lb3V0U2Vjb25kczogNQogICAgICAgIGFyZ3M6CiAgICAgICAgLSAtLWRvbWFpbj08a3ViZXJuZXRlc0t1YmVsZXRDbHVzdGVyRG9tYWluPi4KICAgICAgICAtIC0tZG5zLXBvcnQ9MTAwNTMKICAgICAgICAtIC0tY29uZmlnLWRpcj0va3ViZS1kbnMtY29uZmlnCiAgICAgICAgLSAtLXY9MgogICAgICAgIGVudjoKICAgICAgICAtIG5hbWU6IFBST01FVEhFVVNfUE9SVAogICAgICAgICAgdmFsdWU6ICIxMDA1NSIKICAgICAgICBwb3J0czoKICAgICAgICAtIGNvbnRhaW5lclBvcnQ6IDEwMDUzCiAgICAgICAgICBuYW1lOiBkbnMtbG9jYWwKICAgICAgICAgIHByb3RvY29sOiBVRFAKICAgICAgICAtIGNvbnRhaW5lclBvcnQ6IDEwMDUzCiAgICAgICAgICBuYW1lOiBkbnMtdGNwLWxvY2FsCiAgICAgICAgICBwcm90b2NvbDogVENQCiAgICAgICAgLSBjb250YWluZXJQb3J0OiAxMDA1NQogICAgICAgICAgbmFtZTogbWV0cmljcwogICAgICAgICAgcHJvdG9jb2w6IFRDUAogICAgICAgIHZvbHVtZU1vdW50czoKICAgICAgICAtIG5hbWU6IGt1YmUtZG5zLWNvbmZpZwogICAgICAgICAgbW91bnRQYXRoOiAva3ViZS1kbnMtY29uZmlnCiAgICAgIC0gbmFtZTogZG5zbWFzcQogICAgICAgIGltYWdlOiA8a3ViZXJuZXRlc0ROU01hc3FTcGVjPgogICAgICAgIGxpdmVuZXNzUHJvYmU6CiAgICAgICAgICBodHRwR2V0OgogICAgICAgICAgICBwYXRoOiAvaGVhbHRoY2hlY2svZG5zbWFzcQogICAgICAgICAgICBwb3J0OiAxMDA1NAogICAgICAgICAgICBzY2hlbWU6IEhUVFAKICAgICAgICAgIGluaXRpYWxEZWxheVNlY29uZHM6IDYwCiAgICAgICAgICB0aW1lb3V0U2Vjb25kczogNQogICAgICAgICAgc3VjY2Vzc1RocmVzaG9sZDogMQogICAgICAgICAgZmFpbHVyZVRocmVzaG9sZDogNQogICAgICAgIGFyZ3M6CiAgICAgICAgLSAtdj0yCiAgICAgICAgLSAtbG9ndG9zdGRlcnIKICAgICAgICAtIC1jb25maWdEaXI9L2V0Yy9rOHMvZG5zL2Ruc21hc3EtbmFubnkKICAgICAgICAtIC1yZXN0YXJ0RG5zbWFzcT10cnVlCiAgICAgICAgLSAtLQogICAgICAgIC0gLWsKICAgICAgICAtIC0tY2FjaGUtc2l6ZT0xMDAwCiAgICAgICAgLSAtLW5vLW5lZ2NhY2hlCiAgICAgICAgLSAtLWxvZy1mYWNpbGl0eT0tCiAgICAgICAgLSAtLXNlcnZlcj0vY2x1c3Rlci5sb2NhbC8xMjcuMC4wLjEjMTAwNTMKICAgICAgICAtIC0tc2VydmVyPS9pbi1hZGRyLmFycGEvMTI3LjAuMC4xIzEwMDUzCiAgICAgICAgLSAtLXNlcnZlcj0vaXA2LmFycGEvMTI3LjAuMC4xIzEwMDUzCiAgICAgICAgcG9ydHM6CiAgICAgICAgLSBjb250YWluZXJQb3J0OiA1MwogICAgICAgICAgbmFtZTogZG5zCiAgICAgICAgICBwcm90b2NvbDogVURQCiAgICAgICAgLSBjb250YWluZXJQb3J0OiA1MwogICAgICAgICAgbmFtZTogZG5zLXRjcAogICAgICAgICAgcHJvdG9jb2w6IFRDUAogICAgICAgICMgc2VlOiBodHRwczovL2dpdGh1Yi5jb20va3ViZXJuZXRlcy9rdWJlcm5ldGVzL2lzc3Vlcy8yOTA1NSBmb3IgZGV0YWlscwogICAgICAgIHJlc291cmNlczoKICAgICAgICAgIHJlcXVlc3RzOgogICAgICAgICAgICBjcHU6IDE1MG0KICAgICAgICAgICAgbWVtb3J5OiAyME1pCiAgICAgICAgdm9sdW1lTW91bnRzOgogICAgICAgIC0gbmFtZToga3ViZS1kbnMtY29uZmlnCiAgICAgICAgICBtb3VudFBhdGg6IC9ldGMvazhzL2Rucy9kbnNtYXNxLW5hbm55CiAgICAgIC0gbmFtZTogc2lkZWNhcgogICAgICAgIGltYWdlOiA8a3ViZXJuZXRlc0ROU1NpZGVjYXJTcGVjPgogICAgICAgIGxpdmVuZXNzUHJvYmU6CiAgICAgICAgICBodHRwR2V0OgogICAgICAgICAgICBwYXRoOiAvbWV0cmljcwogICAgICAgICAgICBwb3J0OiAxMDA1NAogICAgICAgICAgICBzY2hlbWU6IEhUVFAKICAgICAgICAgIGluaXRpYWxEZWxheVNlY29uZHM6IDYwCiAgICAgICAgICB0aW1lb3V0U2Vjb25kczogNQogICAgICAgICAgc3VjY2Vzc1RocmVzaG9sZDogMQogICAgICAgICAgZmFpbHVyZVRocmVzaG9sZDogNQogICAgICAgIGFyZ3M6CiAgICAgICAgLSAtLXY9MgogICAgICAgIC0gLS1sb2d0b3N0ZGVycgogICAgICAgIC0gLS1wcm9iZT1rdWJlZG5zLDEyNy4wLjAuMToxMDA1MyxrdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLjxrdWJlcm5ldGVzS3ViZWxldENsdXN0ZXJEb21haW4+LDUsU1JWCiAgICAgICAgLSAtLXByb2JlPWRuc21hc3EsMTI3LjAuMC4xOjUzLGt1YmVybmV0ZXMuZGVmYXVsdC5zdmMuPGt1YmVybmV0ZXNLdWJlbGV0Q2x1c3RlckRvbWFpbj4sNSxTUlYKICAgICAgICBwb3J0czoKICAgICAgICAtIGNvbnRhaW5lclBvcnQ6IDEwMDU0CiAgICAgICAgICBuYW1lOiBtZXRyaWNzCiAgICAgICAgICBwcm90b2NvbDogVENQCiAgICAgICAgcmVzb3VyY2VzOgogICAgICAgICAgcmVxdWVzdHM6CiAgICAgICAgICAgIG1lbW9yeTogMjBNaQogICAgICAgICAgICBjcHU6IDEwbQogICAgICBkbnNQb2xpY3k6IERlZmF1bHQgICMgRG9uJ3QgdXNlIGNsdXN0ZXIgRE5TLgogICAgICBzZXJ2aWNlQWNjb3VudE5hbWU6IGt1YmUtZG5zCiAgICAgIG5vZGVTZWxlY3RvcjoKICAgICAgICBiZXRhLmt1YmVybmV0ZXMuaW8vb3M6IGxpbnV4"
}
]
}The reason for the unsightly base64-encoded input type is to optimize delivery payload, and to squash a human-maintainable yaml file representation into something that can be tightly pasted into a JSON string value without the arguably more unsightly carriage returns / whitespace that would be delivered with a literal copy/paste of a Kubernetes manifest.
kubeletConfig declares runtime configuration for the kubelet running on all master and agent nodes. It is a generic key/value object, and a child property of kubernetesConfig. An example custom kubelet config:
"kubernetesConfig": {
"kubeletConfig": {
"--eviction-hard": "memory.available<250Mi,nodefs.available<20%,nodefs.inodesFree<10%"
}
}
See here for a reference of supported kubelet options.
Below is a list of kubelet options that aks-engine will configure by default:
| kubelet option | default value |
|---|---|
| "--cadvisor-port" | "0" |
| "--cloud-config" | "/etc/kubernetes/azure.json" |
| "--cloud-provider" | "azure" |
| "--cluster-domain" | "cluster.local" |
| "--event-qps" | "0" |
| "--pod-infra-container-image" | "pause-amd64:version" |
| "--network-plugin" | "cni" |
| "--max-pods" | "30", or "110" if using kubenet --network-plugin (i.e., "networkPlugin": "kubenet") |
| "--eviction-hard" | "memory.available<750Mi,nodefs.available<10%,nodefs.inodesFree<5%" |
| "--node-status-update-frequency" | "10s" |
| "--image-gc-high-threshold" | "85" |
| "--image-gc-low-threshold" | "80" |
| "--non-masquerade-cidr" | "0.0.0.0/0" (unless ip-masq-agent is disabled, in which case this is set to the value of kubernetesConfig.ClusterSubnet ) |
| "--azure-container-registry-config" | "/etc/kubernetes/azure.json" |
| "--pod-max-pids" | "-1" (need to activate the feature in --feature-gates=SupportPodPidsLimit=true) |
| "--image-pull-progress-deadline" | "30m" |
| "--feature-gates" | No default (can be a comma-separated list). On agent nodes Accelerators=true will be applied in the --feature-gates option for k8s versions before 1.11.0 |
| "--enforce-node-allocatable" | "pods" |
| "--streaming-connection-idle-timeout" | "4h" |
| "--rotate-certificates" | "true" (this default is set for clusters >= 1.11.9 ) |
| "--tls-cipher-suites" | "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256" |
| "--authentication-token-webhook" | "true" (this default is set for clusters >= 1.16.0 ) |
| "--read-only-port" | "0" (this default is set for clusters >= 1.16.0 ) |
Below is a list of kubelet options that are not currently user-configurable, either because a higher order configuration vector is available that enforces kubelet configuration, or because a static configuration is required to build a functional cluster:
| kubelet option | default value |
|---|---|
| "--address" | "0.0.0.0" |
| "--allow-privileged" | "true" |
| "--pod-manifest-path" | "/etc/kubernetes/manifests" |
| "--node-labels" | (based on Azure node metadata) |
| "--cgroups-per-qos" | "true" |
| "--kubeconfig" | "/var/lib/kubelet/kubeconfig" |
| "--register-node" (master nodes only) | "true" |
| "--register-with-taints" (master nodes only) | "node-role.kubernetes.io/master=true:NoSchedule" |
| "--keep-terminated-pod-volumes" | "false" |
controllerManagerConfig declares runtime configuration for the kube-controller-manager daemon running on all master nodes. Like kubeletConfig it is a generic key/value object, and a child property of kubernetesConfig. An example custom controller-manager config:
"kubernetesConfig": {
"controllerManagerConfig": {
"--node-monitor-grace-period": "40s",
"--pod-eviction-timeout": "5m0s",
"--route-reconciliation-period": "10s"
"--terminated-pod-gc-threshold": "5000"
}
}See here for a reference of supported controller-manager options.
Below is a list of controller-manager options that aks-engine will configure by default:
| controller-manager option | default value |
|---|---|
| "--node-monitor-grace-period" | "40s" |
| "--pod-eviction-timeout" | "5m0s" |
| "--route-reconciliation-period" | "10s" |
| "--terminated-pod-gc-threshold" | "5000" |
| "--feature-gates" | No default (can be a comma-separated list) |
Below is a list of controller-manager options that are not currently user-configurable, either because a higher order configuration vector is available that enforces controller-manager configuration, or because a static configuration is required to build a functional cluster:
| controller-manager option | default value |
|---|---|
| "--kubeconfig" | "/var/lib/kubelet/kubeconfig" |
| "--allocate-node-cidrs" | "false" |
| "--cluster-cidr" | uses clusterSubnet value |
| "--cluster-name" | auto-generated using api model properties |
| "--root-ca-file" | "/etc/kubernetes/certs/ca.crt" |
| "--cluster-signing-cert-file" | "/etc/kubernetes/certs/ca.crt" |
| "--cluster-signing-key-file" | "/etc/kubernetes/certs/ca.key" |
| "--service-account-private-key-file" | "/etc/kubernetes/certs/apiserver.key" |
| "--leader-elect" | "true" |
| "--v" | "2" |
| "--profiling" | "false" |
| "--use-service-account-credentials" | "false" ("true" if kubernetesConfig.enableRbac is true) |
cloudControllerManagerConfig declares runtime configuration for the cloud-controller-manager daemon running on all master nodes in a Cloud Controller Manager configuration. Like kubeletConfig it is a generic key/value object, and a child property of kubernetesConfig. An example custom cloud-controller-manager config:
"kubernetesConfig": {
"cloudControllerManagerConfig": {
"--route-reconciliation-period": "1m"
}
}See here for a reference of supported controller-manager options.
Below is a list of cloud-controller-manager options that aks-engine will configure by default:
| controller-manager option | default value |
|---|---|
| "--route-reconciliation-period" | "10s" |
Below is a list of cloud-controller-manager options that are not currently user-configurable, either because a higher order configuration vector is available that enforces controller-manager configuration, or because a static configuration is required to build a functional cluster:
| controller-manager option | default value |
|---|---|
| "--kubeconfig" | "/var/lib/kubelet/kubeconfig" |
| "--allocate-node-cidrs" | "false" |
| "--cluster-cidr" | uses clusterSubnet value |
| "--cluster-name" | auto-generated using api model properties |
| "--cloud-provider" | "azure" |
| "--cloud-config" | "/etc/kubernetes/azure.json" |
| "--leader-elect" | "true" |
| "--v" | "2" |
apiServerConfig declares runtime configuration for the kube-apiserver daemon running on all master nodes. Like kubeletConfig and controllerManagerConfig it is a generic key/value object, and a child property of kubernetesConfig. An example custom apiserver config:
"kubernetesConfig": {
"apiServerConfig": {
"--request-timeout": "30s"
}
}Or perhaps you want to customize/override the set of admission-control flags passed to the API Server by default, you can omit the options you don't want and specify only the ones you need as follows:
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorRelease": "1.8",
"kubernetesConfig": {
"apiServerConfig": {
"--admission-control": "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,AlwaysPullImages"
}
}
}See here for a reference of supported apiserver options.
Below is a list of apiserver options that aks-engine will configure by default:
| apiserver option | default value |
|---|---|
| "--admission-control" | "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota" (Kubernetes versions prior to 1.9.0) |
"--enable-admission-plugins"* |
"NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,ExtendedResourceToleration" (Kubernetes versions 1.9.0 and later) |
| "--authorization-mode" | "Node", "RBAC" (the latter if enabledRbac is true) |
| "--audit-log-maxage" | "30" |
| "--audit-log-maxbackup" | "10" |
| "--audit-log-maxsize" | "100" |
| "--feature-gates" | No default (can be a comma-separated list) |
| "--oidc-username-claim" | "oid" (if has AADProfile) |
| "--oidc-groups-claim" | "groups" (if has AADProfile) |
| "--oidc-client-id" | calculated value that represents OID client ID (if has AADProfile) |
| "--oidc-issuer-url" | calculated value that represents OID issuer URL (if has AADProfile) |
* In Kubernetes versions 1.10.0 and later the --admission-control flag is deprecated and --enable-admission-plugins is used instead.
Below is a list of apiserver options that are not currently user-configurable, either because a higher order configuration vector is available that enforces apiserver configuration, or because a static configuration is required to build a functional cluster:
| apiserver option | default value |
|---|---|
| "--bind-address" | "0.0.0.0" |
| "--advertise-address" | calculated value that represents listening URI for API server |
| "--allow-privileged" | "true" |
| "--anonymous-auth" | "false |
| "--audit-log-path" | "/var/log/apiserver/audit.log" |
| "--insecure-port" | "8080" |
| "--secure-port" | "443" |
| "--service-account-lookup" | "true" |
| "--etcd-cafile" | "/etc/kubernetes/certs/ca.crt" |
| "--etcd-certfile" | "/etc/kubernetes/certs/etcdclient.crt" |
| "--etcd-keyfile" | "/etc/kubernetes/certs/etcdclient.key" |
| "--etcd-servers" | calculated value that represents etcd servers |
| "--profiling" | "false" |
| "--repair-malformed-updates" | "false" (deprecated in v1.14) |
| "--tls-cert-file" | "/etc/kubernetes/certs/apiserver.crt" |
| "--tls-private-key-file" | "/etc/kubernetes/certs/apiserver.key" |
| "--client-ca-file" | "/etc/kubernetes/certs/ca.crt" |
| "--service-account-key-file" | "/etc/kubernetes/certs/apiserver.key" |
| "--kubelet-client-certificate" | "/etc/kubernetes/certs/client.crt" |
| "--kubelet-client-key" | "/etc/kubernetes/certs/client.key" |
| "--service-cluster-ip-range" | see serviceCIDR |
| "--storage-backend" | calculated value that represents etcd version |
| "--v" | "4" |
| "--experimental-encryption-provider-config" | "/etc/kubernetes/encryption-config.yaml" (if enableDataEncryptionAtRest is true) |
| "--experimental-encryption-provider-config" | "/etc/kubernetes/encryption-config.yaml" (if enableEncryptionWithExternalKms is true) |
| "--requestheader-client-ca-file" | "/etc/kubernetes/certs/proxy-ca.crt" (if enableAggregatedAPIs is true) |
| "--proxy-client-cert-file" | "/etc/kubernetes/certs/proxy.crt" (if enableAggregatedAPIs is true) |
| "--proxy-client-key-file" | "/etc/kubernetes/certs/proxy.key" (if enableAggregatedAPIs is true) |
| "--requestheader-allowed-names" | "" (if enableAggregatedAPIs is true) |
| "--requestheader-extra-headers-prefix" | "X-Remote-Extra-" (if enableAggregatedAPIs is true) |
| "--requestheader-group-headers" | "X-Remote-Group" (if enableAggregatedAPIs is true) |
| "--requestheader-username-headers" | "X-Remote-User" (if enableAggregatedAPIs is true) |
| "--cloud-provider" | "azure" (unless useCloudControllerManager is true) |
| "--cloud-config" | "/etc/kubernetes/azure.json" (unless useCloudControllerManager is true) |
schedulerConfig declares runtime configuration for the kube-scheduler daemon running on all master nodes. Like kubeletConfig, controllerManagerConfig, and apiServerConfig it is a generic key/value object, and a child property of kubernetesConfig. An example custom apiserver config:
"kubernetesConfig": {
"schedulerConfig": {
"--v": "2"
}
}See here for a reference of supported kube-scheduler options.
Below is a list of scheduler options that aks-engine will configure by default:
| kube-scheduler option | default value |
|---|---|
| "--v" | "2" |
| "--feature-gates" | No default (can be a comma-separated list) |
Below is a list of kube-scheduler options that are not currently user-configurable, either because a higher order configuration vector is available that enforces kube-scheduler configuration, or because a static configuration is required to build a functional cluster:
| kube-scheduler option | default value |
|---|---|
| "--kubeconfig" | "/var/lib/kubelet/kubeconfig" |
| "--leader-elect" | "true" |
| "--profiling" | "false" |
We consider kubeletConfig, controllerManagerConfig, apiServerConfig, and schedulerConfig to be generic conveniences that add power/flexibility to cluster deployments. Their usage comes with no operational guarantees! They are manual tuning features that enable low-level configuration of a kubernetes cluster.
Custom YAML specifications can be configured for kube-scheduler, kube-controller-manager, cloud-controller-manager and kube-apiserver in addition to the addons described above. You will need to pass in a base64-encoded string of the kubernetes manifest YAML file to KubernetesComponentConfig["data"] . For example, to pass a custom kube-scheduler config, do the following:
"kubernetesConfig": {
"schedulerConfig": {
"data" : "<base64-encoded string of your k8s manifest YAML>"
}
}NOTE: Custom YAML for addons is an experimental feature. Since
Addons.Dataallows you to provide your own scripts, you are responsible for any undesirable consequences of their errors or failures. Use at your own risk.
privateCluster defines a cluster without public addresses assigned. It is a child property of kubernetesConfig.
| Name | Required | Description |
|---|---|---|
| enabled | no | Enable Private Cluster (boolean - default == false) |
| jumpboxProfile | no | Configure and auto-provision a jumpbox to access your private cluster. jumpboxProfile is ignored if enabled is false. See jumpboxProfile below |
jumpboxProfile describes the settings for a jumpbox deployed via aks-engine to access a private cluster. It is a child property of privateCluster.
| Name | Required | Description |
|---|---|---|
| name | yes | This is the unique name for the jumpbox VM. Some resources deployed with the jumpbox are derived from this name |
| vmSize | yes | Describes a valid Azure VM Sizes |
| publicKey | yes | The public SSH key used for authenticating access to the jumpbox. Here are instructions for generating a public/private key pair |
| osDiskSizeGB | no | Describes the OS Disk Size in GB. Defaults to 30 |
| storageProfile | no | Specifies the storage profile to use. Valid values are ManagedDisks or StorageAccount. Defaults to ManagedDisks |
| username | no | Describes the admin username to be used on the jumpbox. Defaults to azureuser |
masterProfile describes the settings for master configuration.
| Name | Required | Description |
|---|---|---|
| count | yes | Masters have count value of 1, 3, or 5 masters |
| dnsPrefix | yes | The dns prefix for the master FQDN. The master FQDN is used for SSH or commandline access. This must be a unique name. (bring your own VNET examples) |
| subjectAltNames | no | An array of fully qualified domain names using which a user can reach API server. These domains are added as Subject Alternative Names to the generated API server certificate. NOTE: These domains will not be automatically provisioned. |
| firstConsecutiveStaticIP | only required when vnetSubnetId specified and when MasterProfile is not VirtualMachineScaleSets |
The IP address of the first master. IP Addresses will be assigned consecutively to additional master nodes. When MasterProfile is using VirtualMachineScaleSets, this value will be determined by an offset from the first IP in the vnetCidr. For example, if vnetCidr is 10.239.0.0/16, then firstConsecutiveStaticIP will be 10.239.0.4 |
| vmsize | yes | Describes a valid Azure VM Sizes. These are restricted to machines with at least 2 cores and 100GB of disk space |
| storageProfile | no | Specifies the storage profile to use. Valid values are ManagedDisks or StorageAccount. Defaults to ManagedDisks |
| osDiskSizeGB | no | Describes the OS Disk Size in GB |
| vnetSubnetId | only required when using custom VNET | Specifies the Id of an alternate VNET subnet. The subnet id must specify a valid VNET ID owned by the same subscription. (bring your own VNET examples). When MasterProfile is set to VirtualMachineScaleSets, this value should be the subnetId of the master subnet. When MasterProfile is set to AvailabilitySet, this value should be the subnetId shared by both master and agent nodes. |
| extensions | no | This is an array of extensions. This indicates that the extension be run on a single master. The name in the extensions array must exactly match the extension name in the extensionProfiles |
| vnetCidr | no | Specifies the VNET cidr when using a custom VNET (bring your own VNET examples). This VNET cidr should include both the master and the agent subnets. |
| imageReference.name | no | The name of the Linux OS image. Needs to be used in conjunction with resourceGroup, below. (For information on setting this for Windows nodes see WindowsProfile) |
| imageReference.resourceGroup | no | Resource group that contains the Linux OS image. Needs to be used in conjunction with name, above |
| imageReference.subscriptionId | no | ID of subscription containing the Linux OS image. Applies only to Shared Image Galleries. All of name, resourceGroup, subscription, gallery, image name, and version must be specified for this scenario. |
| imageReference.gallery | no | Name of Shared Image Gallery containing the Linux OS image. Applies only to Shared Image Galleries. All of name, resourceGroup, subscription, gallery, image name, and version must be specified for this scenario. |
| imageReference.version | no | Version containing the Linux OS image. Applies only to Shared Image Galleries. All of name, resourceGroup, subscription, gallery, image name, and version must be specified for this scenario. |
| distro | no | Specifies the masters' Linux distribution. Currently supported values are: ubuntu, ubuntu-18.04, aks-ubuntu-16.04 (previously aks), aks-ubuntu-18.04, and coreos (CoreOS support is currently experimental - Example of CoreOS Master with CoreOS Agents). For Azure Public Cloud, Azure US Government Cloud and Azure China Cloud, defaults to aks-ubuntu-16.04. For other Sovereign Clouds, the default is ubuntu-16.04 (There is a known issue with ubuntu-18.04 + Azure CNI). aks-ubuntu-16.04 is a custom image based on ubuntu-16.04 that comes with pre-installed software necessary for Kubernetes deployments. |
| customFiles | no | The custom files to be provisioned to the master nodes. Defined as an array of json objects with each defined as "source":"absolute-local-path", "dest":"absolute-path-on-masternodes".See examples |
| availabilityProfile | no | Supported values are AvailabilitySet (default) and VirtualMachineScaleSets (still under development: upgrade not supported; requires Kubernetes clusters version 1.10+ and agent pool availabilityProfile must also be VirtualMachineScaleSets). When MasterProfile is using VirtualMachineScaleSets, to SSH into a master node, you need to use ssh -p 50001 instead of port 22. |
| agentVnetSubnetId | only required when using custom VNET and when MasterProfile is using VirtualMachineScaleSets |
Specifies the Id of an alternate VNET subnet for all the agent pool nodes. The subnet id must specify a valid VNET ID owned by the same subscription. (bring your own VNET examples). When MasterProfile is using VirtualMachineScaleSets, this value should be the subnetId of the subnet for all agent pool nodes. |
| availabilityZones | no | To protect your cluster from datacenter-level failures, you can enable the Availability Zones feature for your cluster by configuring "availabilityZones" for the master profile and all of the agentPool profiles in the cluster definition. Check out Availability Zones README for more details. |
| cosmosEtcd | no | True: uses cosmos etcd endpoint instead of installing etcd on masters |
| auditDEnabled | no | Enable auditd enforcement at the OS layer for each node VM. This configuration is only valid on an agent pool with an Ubuntu-backed distro, i.e., the default "aks-ubuntu-16.04" distro, or the "aks-ubuntu-18.04", "ubuntu", "ubuntu-18.04", or "acc-16.04" distro values. Defaults to false |
| customVMTags | no | Specifies a list of custom tags to be added to the master VMs or Scale Sets. Each tag is a key/value pair (ie: "myTagKey": "myTagValue"). |
A cluster can have 0 to 12 agent pool profiles. Agent Pool Profiles are used for creating agents with different capabilities such as VMSizes, VMSS or Availability Set, Public/Private access, user-defined OS Images, attached storage disks, attached managed disks, or Windows.
| Name | Required | Description |
|---|---|---|
| availabilityProfile | no | Supported values are VirtualMachineScaleSets (default, except for Kubernetes clusters before version 1.10) and AvailabilitySet. |
| count | yes | Describes the node count |
| availabilityZones | no | To protect your cluster from datacenter-level failures, you can enable the Availability Zones feature for your cluster by configuring "availabilityZones" for the master profile and all of the agentPool profiles in the cluster definition. Check out Availability Zones README for more details. |
| singlePlacementGroup | no | Supported values are true (default) and false. A value of true: A VMSS with a single placement group and has a range of 0-100 VMs. A value of false: A VMSS with multiple placement groups and has a range of 0-1,000 VMs. For more information, check out virtual machine scale sets placement groups. This configuration is only valid on an agent pool with an "availabilityProfile" value of "VirtualMachineScaleSets" |
| scaleSetPriority | no | Supported values are Regular (default) and Low. This configuration is only valid on an agent pool with an "availabilityProfile" value of "VirtualMachineScaleSets". Enables the usage of Low-priority VMs on Scale Sets. |
| scaleSetEvictionPolicy | no | Supported values are Delete (default) and Deallocate. This configuration is only valid on an agent pool with an "availabilityProfile" value of "VirtualMachineScaleSets" and a "scaleSetPriority" value of "Low". |
| diskSizesGB | no | Describes an array of up to 4 attached disk sizes. Valid disk size values are between 1 and 1024 |
| dnsPrefix | Required if agents are to be exposed publically with a load balancer | The dns prefix that forms the FQDN to access the loadbalancer for this agent pool. This must be a unique name among all agent pools. Not supported for Kubernetes clusters |
| name | yes | This is the unique name for the agent pool profile. The resources of the agent pool profile are derived from this name |
| ports | only required if needed for exposing services publically | Describes an array of ports need for exposing publically. A tcp probe is configured for each port and only opens to an agent node if the agent node is listening on that port. A maximum of 150 ports may be specified. Not supported for Kubernetes clusters |
| storageProfile | no | Specifies the storage profile to use. Valid values are ManagedDisks, StorageAccount, or Ephemeral. Defaults to ManagedDisks. Ephemeral is an experimental feature - please read more on the feature status page |
| vmsize | yes | Describes a valid Azure VM Sizes. These are restricted to machines with at least 2 cores |
| osDiskSizeGB | no | Describes the OS Disk Size in GB |
| vnetSubnetId | no | Specifies the Id of an alternate VNET subnet. The subnet id must specify a valid VNET ID owned by the same subscription. (bring your own VNET examples) |
| imageReference.name | no | The name of a a Linux OS image. Needs to be used in conjunction with resourceGroup, below |
| imageReference.resourceGroup | no | Resource group that contains the Linux OS image. Needs to be used in conjunction with name, above |
| osType | no | Specifies the agent pool's Operating System. Supported values are Windows and Linux. Defaults to Linux |
| distro | no | Specifies the masters' Linux distribution. Currently supported values are: ubuntu, ubuntu-18.04, aks-ubuntu-16.04 (previously aks), aks-ubuntu-18.04, and coreos (CoreOS support is currently experimental - Example of CoreOS Master with CoreOS Agents). For Azure Public Cloud, Azure US Government Cloud and Azure China Cloud, defaults to aks-ubuntu-16.04. For Sovereign Clouds, the default is ubuntu-16.04 (There is a known issue with ubuntu-18.04 + Azure CNI). aks-ubuntu-16.04 is a custom image based on ubuntu-16.04 that comes with pre-installed software necessary for Kubernetes deployments. |
| acceleratedNetworkingEnabled | no | Use Azure Accelerated Networking feature for Linux agents (You must select a VM SKU that supports Accelerated Networking). Defaults to true if the VM SKU selected supports Accelerated Networking |
| acceleratedNetworkingEnabledWindows | no | Use Azure Accelerated Networking feature for Windows agents (You must select a VM SKU that supports Accelerated Networking). Defaults to false |
| vmssOverProvisioningEnabled | no | Use Overprovisioning with VMSS. This configuration is only valid on an agent pool with an "availabilityProfile" value of "VirtualMachineScaleSets". Defaults to false |
| enableVMSSNodePublicIP | no | Enable creation of public IP on VMSS nodes. This configuration is only valid on an agent pool with an "availabilityProfile" value of "VirtualMachineScaleSets". Defaults to false |
| LoadBalancerBackendAddressPoolIDs | no | Enables automatic placement of the agent pool nodes into existing load balancer's backend address pools. Each element value of this string array is the corresponding load balancer backend address pool's Azure Resource Manager(ARM) resource ID. By default this property is not included in the api model, which is equivalent to an empty string array. |
| auditDEnabled | no | Enable auditd enforcement at the OS layer for each node VM. This configuration is only valid on an agent pool with an Ubuntu-backed distro, i.e., the default "aks-ubuntu-16.04" distro, or the "aks-ubuntu-18.04", "ubuntu", "ubuntu-18.04", or "acc-16.04" distro values. Defaults to false |
| customVMTags | no | Specifies a list of custom tags to be added to the agent VMs or Scale Sets. Each tag is a key/value pair (ie: "myTagKey": "myTagValue"). |
linuxProfile provides the linux configuration for each linux node in the cluster
| Name | Required | Description |
|---|---|---|
| adminUsername | yes | Describes the username to be used on all linux clusters |
| ssh.publicKeys[].keyData | yes | The public SSH key used for authenticating access to all Linux nodes in the cluster |
| secrets | no | Specifies an array of key vaults to pull secrets from and what secrets to pull from each |
| customSearchDomain.name | no | describes the search domain to be used on all linux clusters |
| customSearchDomain.realmUser | no | describes the realm user with permissions to update dns registries on Windows Server DNS |
| customSearchDomain.realmPassword | no | describes the realm user password to update dns registries on Windows Server DNS |
| customNodesDNS.dnsServer | no | describes the IP address of the DNS Server |
Here are instructions for generating a public/private key pair for ssh.publicKeys.keyData.
At least one SSH key is required, but multiple are supported when deploying Kubernetes.
Here's a minimal example using just one key:
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": "ssh-rsa AAAA...w=="
}
]
}
},
And an example using two keys.
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": "ssh-rsa AAAA...w=="
},
{
"keyData": "ssh-rsa AAAA...w=="
}
]
}
},secrets details which certificates to install on the masters and nodes in the cluster.
A cluster can have a list of key vaults to install certs from.
On linux boxes the certs are saved on under the directory "/var/lib/waagent/". 2 files are saved per certificate:
{thumbprint}.crt: this is the full cert chain saved in PEM format{thumbprint}.prv: this is the private key saved in PEM format
| Name | Required | Description |
|---|---|---|
| sourceVault.id | yes | The azure resource manager id of the key vault to pull secrets from |
| vaultCertificates.certificateUrl | yes | Keyvault URL to this cert including the version |
format for sourceVault.id, can be obtained in cli, or found in the portal: /subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.KeyVault/vaults/{keyvaultname}
format for vaultCertificates.certificateUrl, can be obtained in cli, or found in the portal:
https://{keyvaultname}.vault.azure.net:443/secrets/{secretName}/{version}
windowsProfile provides configuration specific to Windows nodes in the cluster
| Name | Required | Description |
|---|---|---|
| adminUsername | yes | Username for the Windows adminstrator account created on each Windows node |
| adminPassword | yes | Password for the Windows adminstrator account created on each Windows node |
| windowsPublisher | no | Publisher used to find Windows VM to deploy from marketplace. Default: microsoft-aks |
| windowsOffer | no | Offer used to find Windows VM to deploy from marketplace. Default: aks-windows |
| windowsSku | no | SKU usedto find Windows VM to deploy from marketplace. Default: 2019-datacenter-core-smalldisk |
| imageVersion | no | Specific image version to deploy from marketplace. Default: 17763.737.190923. This default is incremented to include the latest Windows patches after being validated by the AKS Engine team. |
| windowsImageSourceURL | no | Path to an existing Azure storage blob with a sysprepped VHD. This is used to test pre-release or customized VHD files that you have uploaded to Azure. If provided, the above 4 parameters are ignored. |
| imageReference.name | no | Name of an Image. |
| imageReference.resourceGroup | no | Resource group that contains the Image. |
| imageReference.subscriptionId | no | ID of subscription containing a Shared Image Gallery. |
| imageReference.gallery | no | Name of a Shared Image Gallery. |
| imageReference.version | no | Version of an Image from a Shared Image Gallery. |
| sshEnabled | no | If set to true, OpenSSH will be installed on windows nodes to allow for ssh remoting. Only for Windows version 1809/2019 or later . The same SSH authorized public key(s) will be added from linuxProfile.ssh.publicKeys |
You can configure the image used for all Windows nodes one of the following ways:
The AKS Engine team produces images that are optimized for and validated with aks-engine. The latest version of these images are used as the default images for Windows nodes.
These images are published to the Azure Marketplace under the microsoft-aks publisher and aks-windows offer.
Release notes for these images can be found under releases/vhd-notes/aks-windows.
Aks-engine also supports running 'vanilla' Windows Server images published by Microsoft. These can be used by advanced users if a release other than Winders Server 2019 is needed.
If you want to choose a specific Windows image, but automatically use the latest - set windowsPublisher, windowsOffer, and windowsSku. If you need a specific version, then add imageVersion too.
You can find all available images with az vm image list --all --publisher MicrosoftWindowsServer --offer WindowsServer --output table, and the contents of these images are described in the knowledge base article Windows Server release on Azure Marketplace update history.
If you want to use a specific image then windowsPublisher, windowsOffer, windowsSku, and imageVersion must all be set:
"windowsProfile": {
"adminUsername": "...",
"adminPassword": "...",
"windowsPublisher": "MicrosoftWindowsServer",
"windowsOffer": "WindowsServer",
"windowsSku": "2019-Datacenter-Core-with-Containers-smalldisk",
"imageVersion": "2019.0.20181107"
},Listed in order of precedence based on what is specified in the api model:
To use an image uploaded to an Azure storage account (or any other accessible location) specify windowsImageSourceURL.
Note: URLs containing SAS tokens are not allowed by Azure!
"windowsProfile": {
"adminUsername": "...",
"adminPassword": "...",
"windowsImageSourceURL": "https://images.blob.core.windows.net/vhds/custom_windows_image.vhd",
},To use an Image from a Shared Image Gallery specify imageReference.name, imageReference.resourceGroup, imageReference.subscriptionId, imageReference.galllery, and imageReference.version.
"windowsProfile": {
"adminUsername": "...",
"adminPassword": "...",
"imageReference": {
"name": "custom-image",
"resourceGroup": "windows-images",
"subscriptionId": "00000000-0000-0000-0000-000000000000",
"gallery": "image-gallery",
"version": "0.1.0"
}
},To use a pre-existing Azure Image specify imageReference.name and imageReference.resourceGroup.
"windowsProfile": {
"adminUsername": "...",
"adminPassword": "...",
"imageReference": {
"name": "custom-image",
"resourceGroup": "windows-images"
}
},servicePrincipalProfile describes an Azure Service credentials to be used by the cluster for self-configuration. See service principal for more details on creation.
| Name | Required | Description |
|---|---|---|
| clientId | yes, for Kubernetes clusters | describes the Azure client id. It is recommended to use a separate client ID per cluster |
| secret | yes, for Kubernetes clusters | describes the Azure client secret. It is recommended to use a separate client secret per client id |
| objectId | optional, for Kubernetes clusters | describes the Azure service principal object id. It is required if enableEncryptionWithExternalKms is true |
| keyvaultSecretRef.vaultId | no, for Kubernetes clusters | describes the vault id of the keyvault to retrieve the service principal secret from. See below for format. |
| keyvaultSecretRef.secretName | no, for Kubernetes clusters | describes the name of the service principal secret in keyvault |
| keyvaultSecretRef.version | no, for Kubernetes clusters | describes the version of the secret to use |
format for keyvaultSecretRef.vaultId, can be obtained in cli, or found in the portal:
/subscriptions/<SUB_ID>/resourceGroups/<RG_NAME>/providers/Microsoft.KeyVault/vaults/<KV_NAME>. See keyvault params for an example.
Here are the cluster definitions for apiVersion "2016-03-30". This matches the api version of the Azure Kubernetes Engine.
| Name | Required | Description |
|---|---|---|
| apiVersion | yes | The version of the template. For "2016-03-30" the value is "2016-03-30" |
orchestratorProfile describes the orchestrator settings.
| Name | Required | Description |
|---|---|---|
| orchestratorType | yes | Specifies the orchestrator type for the cluster |
Here are the valid values for the orchestrator types:
DCOS- this represents the DC/OS orchestrator.Swarm- this represents the Swarm orchestrator.Kubernetes- this represents the Kubernetes orchestrator.Swarm Mode- this represents the Swarm Mode orchestrator.
masterProfile describes the settings for master configuration.
| Name | Required | Description |
|---|---|---|
| count | yes | Masters have count value of 1, 3, or 5 masters |
| dnsPrefix | yes | The dns prefix for the masters FQDN. The master FQDN is used for SSH or commandline access. This must be a unique name. (bring your own VNET examples) |
For apiVersion "2016-03-30", a cluster may have only 1 agent pool profiles.
| Name | Required | Description |
|---|---|---|
| count | yes | Describes the node count |
| dnsPrefix | required if agents are to be exposed publically with a load balancer | The dns prefix that forms the FQDN to access the loadbalancer for this agent pool. This must be a unique name among all agent pools |
| name | yes | The unique name for the agent pool profile. The resources of the agent pool profile are derived from this name |
| vmsize | yes | Describes a valid Azure VM Sizes. These are restricted to machines with at least 2 cores |
linuxProfile provides the linux configuration for each linux node in the cluster
| Name | Required | Description |
|---|---|---|
| adminUsername | yes | Describes the username to be used on all linux clusters |
| ssh.publicKeys[0].keyData | yes | The public SSH key used for authenticating access to all Linux nodes in the cluster |
Here are instructions for generating a public/private key pair for ssh.publicKeys[0].keyData.
aadProfile provides Azure Active Directory integration configuration for the cluster, currently only available for Kubernetes orchestrator.
| Name | Required | Description |
|---|---|---|
| clientAppID | yes | Describes the client AAD application ID |
| serverAppID | yes | Describes the server AAD application ID |
| adminGroupID | no | Describes the AAD Group Object ID that will be assigned the cluster-admin RBAC role |
| tenantID | no | Describes the AAD tenant ID to use for authentication. If not specified, will use the tenant of the deployment subscription |
A cluster can have 0 - N extensions in extension profiles. Extension profiles allow a user to easily add pre-packaged functionality into a cluster. An example would be configuring a monitoring solution on your cluster. You can think of extensions like a marketplace for acs clusters.
| Name | Required | Description |
|---|---|---|
| name | yes | The name of the extension. This has to exactly match the name of a folder under the extensions folder |
| version | yes | The version of the extension. This has to exactly match the name of the folder under the extension name folder |
| extensionParameters | optional | Extension parameters may be required by extensions. The format of the parameters is also extension dependant |
| rootURL | optional | URL to the root location of extensions. The rootURL must have an extensions child folder that follows the extensions convention. The rootURL is mainly used for testing purposes |
You can find more information, as well as a list of extensions on the extensions documentation.