Deploying Hybrid Kubernetes cluster (Windows and Linux) on Azure

Posted on April 3, 2018

Windows and Linux agent pools working together on Azure. We will deploy a Kubernetes cluster with 2 nodes on a Windows agent pool and 2 nodes on a Linux Pool and Kubernetes v1.9.

Optional, a KeyVault to store Service Principals and RBAC support with AAD.

Requirements

  • Bash. Windows Users the easiest way is to install Ubuntu bash for Windows 10 or your bash compatible terminal on Mac OS X or Linux (zsh, tsh)
  • Azure CLI 2.0 you can install it here, or use the Azure Cloud Shell in your browser (I am currently using azure-cli (2.0.30))
  • (Optional) jq for parsing data from json responses on bash.
  • Azure subscription where you are the owner or have a Service Principal with at least Contributor role to the subscription.
  • SSH keys in your local machine
  • VSCode for file edition.

Setting up the environment

  1. Download the latest version of the acs-engine in your computer. At the moment of writing this article, I am using v0.14.6 from their list of releases. If you are using Ubuntu Bash, you can type in the following commands from your terminal:

     mkdir hybrid-k8s
 cd hybrid-k8s
 curl -L https://github.com/Azure/acs-engine/releases/download/v0.14.6/acs-engine-v0.14.6-linux-amd64.tar.gz | tar zx
 cp acs-engine-v0.14.6-linux-amd64/acs-engine .
 chmod +x acs-engine
 rm -rf acs-engine-v0.14.6-linux-amd64/

    Just if you would like to check, the MD5 sum of the binary is the following: 8de3cb98f93238287f8c41c3d3c0dc1f.

  2. Make sure you have the latest version (az --version) of the Azure CLI, I have in this moment azure-cli (2.0.30). And make sure you are logged in (az login) to the right subscription (az account list -o table).

  3. Generate a Service Principal for your deployment, you can use my getAzureServicePrincipal.sh file to obtain it. You can run the following commands in your bash terminal to get one:

     curl -O https://gist.githubusercontent.com/brusMX/bab2224dc1b0c26d3aef4799cb97c045/raw/bf05884b5aeca9ae0c455af3ce0e695ec372cccc/getAzureServicePrincipal.sh
 chmod +x getAzureServicePrincipal.sh
 ./getAzureServicePrincipal.sh

    You will be using AZURE_CLIENT_ID and AZURE_CLIENT_SECRET in the next step, so keep them close. Also, remember that if you create a Service Principal for a subscription that was not the default one, you will have to run the following command to change into the desired subscription:

     az account set -s <<CHOSEN SUBSCRIPTION ID>>

  4. Create a .env file up environment variables that will define most of the configurations needed for your cluster. Just create on your hybrid-k8s folder a file named .env and edit it with VSCode:

     export LOC=southcentralus
 export RG_NAME=hybridk8srg001
 export DNS_NAME=hybrid-cluster-001
 # Get the Subscription ID you will get it from the previous command
 export AZURE_SUB_ID=XXXXXXX-XXXX
 # Copy the Service Principal you obtained from the ./getAzureServicePrincipal.sh script here
 export AZURE_CLIENT_ID=XXXXXXXX-XXXXXX-XXXX-XXXX-XXXXXXXXX
 export AZURE_CLIENT_NAME=http://azure-cli-XXXXXXXXXXX
 export AZURE_CLIENT_SECRET=XXXXXXXX-XXXXXX-XXXX-XXXX-XXXXXXXXX
 export AZURE_TENANT_ID=XXXXXXXX-XXXXXX-XXXX-XXXX-XXXXXXXXX
 # Usernames and pass for the vms, make sure to update them
 export VM_USER=azureuser1
 export WIN_VM_PASS=W!n*Us3r01

    And source it in your terminal:

     source .env

  5. Create your resource group where everything will be deployed (if something goes bad, you can always delete the whole RG and start again)

     az group create --name $RG_NAME --location $LOC

    If you are getting weird errors about the RG_NAME on windows, just install dos2unix apt-get install dos2unix and convert the special chars with

     dos2unix .env

  6. You need an SSH Key without a passphrase, you are able to see if you have one already with the following command: cat $HOME/.ssh/id_rsa.pub. If you don’t have one, you can easily create one ssh key by runnin the following command:

    NOTE: This command will ask you to override a previous SSH key, in this case is better to create one with a custom name right next to it. Just provide a new name when prompted

     ssh-keygen -t rsa -b 4096 -N ""

    You can now obtain it with this command:

     cat $HOME/.ssh/id_rsa.pub

  7. Create the json file that will be used to deploy the cluster. If you have all the right environment variables set up, you can copy paste the following command on your bash terminal and press enter. This will auto populate the file with your variables:

     cat <<EOT > hybrid-cluster.json
 {
     "apiVersion": "vlabs",
     "properties": {
         "orchestratorProfile": {
             "orchestratorType": "Kubernetes",
             "orchestratorRelease": "1.9"
         },
         "masterProfile": {
             "count": 3,
             "dnsPrefix": "`echo $DNS_NAME`",
             "vmSize": "Standard_DS2_v2"
         },
         "agentPoolProfiles": [
             {
                 "name": "linuxpool1",
                 "count": 2,
                 "vmSize": "Standard_DS2_v2",
                 "storageProfile" : "ManagedDisks",
                 "availabilityProfile": "AvailabilitySet"
             },
             {
                 "name": "windowspool1",
                 "count": 2,
                 "vmSize": "Standard_D2_v3",
                 "availabilityProfile": "AvailabilitySet",
                 "osType": "Windows"
             }
         ],
         "windowsProfile": {
             "adminUsername": "`echo $VM_USER`",
             "adminPassword": "`echo $WIN_VM_PASS`"
         },
         "linuxProfile": {
             "adminUsername": "`echo $VM_USER`",
             "ssh": {
                 "publicKeys": [
                     {
                         "keyData": "`cat $HOME/.ssh/id_rsa.pub`"
                     }
                 ]
             }
         },
         "servicePrincipalProfile": {
             "clientId": "`echo $AZURE_CLIENT_ID`",
             "secret": "`echo $AZURE_CLIENT_SECRET`"
         }
     }
 }

    Finally type in CAPS EOT and press Enter on your keyboard to close and save the file.

  8. Cat the file cat hybrid-cluster.json and make sure that the variables dnsPrefix, adminUsername, keyData, clientId and secret were populated. If not, you can edit .env and re-run the command above and it will replace the hybrid-cluster.json file with the new variables. Alternatively, you can open the file with VSCode and edit it with the right info: code hybrid-cluster.json.

    1. Make sure to use an unique name for dnsPrefix.
    2. Paste the content of your public SSH key into keydata, Remember, you need a key without a passphrase.
    3. Put the value of AZURE_CLIENT_ID from your service principal on clientId.
    4. Update the value of secret with AZURE_CLIENT_SECRET from that same Service Principal.

Deploying the cluster

Now that we have all our files in order, we can proceed to use ACS-Engine to create the artifacts of our cluster and deploy them.

  1. Create artifacts with ACS-Engine, you will see a folder called _output after you run the following command:

     ./acs-engine generate hybrid-cluster.json

  2. Deploy your cluster to Azure with the ACS Engine binary. This will ask you to login again in your subscription

     ./acs-engine deploy _output/$DNS_NAME/apimodel.json -l $LOC --subscription-id $AZURE_SUB_ID -g $RG_NAME

    You will see something like the following:

     WARN[0001] To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code B2PEZ5NUM to authenticate.
 INFO[0168] Starting ARM Deployment (hybridk8rg01-2035783392). This will take some time...

    Alternatively, you can deploy it using the Azure CLI:

     az group deployment create --resource-group="${RG_NAME}" --template-file="_output/${DNS_NAME}/azuredeploy.json" --name="${DNS_NAME}" --parameters @_output/${DNS_NAME}/azuredeploy.parameters.json

Connect to your cluster and test it out

After the cluster has been succesfully deployed we can obtain our kubeconfig file and start interacting with it.

  1. Obtain .kubeconfig file

         scp -o StrictHostKeyChecking=no $VM_USER@$DNS_NAME.$LOC.cloudapp.azure.com:/home/$VM_USER/.kube/config config-$DNS_NAME

  2. Paste this file in your the. For this case I will create an extra file so you don’t have to worry about your previous k8s config files.

     mkdir ~/.kube
 mv config-$DNS_NAME ~/.kube
 export  KUBECONFIG=~/.kube/config-$DNS_NAME

  3. If you don’t have kubectl installed. You can run from your bash with Azure Cli the following command:

     sudo az aks install-cli

    You should see an output like this:

     Downloading client to /usr/local/bin/kubectl from https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/linux/amd64/kubectl

 Please ensure that /usr/local/bin is in your search PATH, so the `kubectl` command can be found.

  4. Confirm that your cluster is up. You can start running the following commands and make sure that everything is working the way it should be:

     kubectl cluster-info

 Kubernetes master is running at https://hybridk8s004.southcentralus.cloudapp.azure.com
 Heapster is running at https://hybridk8s004.southcentralus.cloudapp.azure.com/api/v1/namespaces/kube-system/services/heapster/proxy
 KubeDNS is running at https://hybridk8s004.southcentralus.cloudapp.azure.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
 kubernetes-dashboard is running at https://hybridk8s004.southcentralus.cloudapp.azure.com/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy
 Metrics-server is running at https://hybridk8s004.southcentralus.cloudapp.azure.com/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
 tiller-deploy is running at https://hybridk8s004.southcentralus.cloudapp.azure.com/api/v1/namespaces/kube-system/services/tiller-deploy:tiller/proxy

     kubectl get nodes -o wide

 NAME                        STATUS    ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE                    KERNEL-VERSION   CONTAINER-RUNTIME
 18466k8s9010                Ready     <none>    9m        v1.9.6    <none>        Windows Server Datacenter   10.0.16299.309
                             docker://17.6.2
 18466k8s9011                Ready     <none>    9m        v1.9.6    <none>    Windows Server Datacenter   10.0.16299.309
                             docker://17.6.2
 k8s-linuxpool1-18466771-0   Ready     agent     15m       v1.9.6    <none>    Debian GNU/Linux 9 (stretch)   4.13.0-1011-azure   docker://1.13.1
 k8s-master-18466771-0       Ready     master    15m       v1.9.6    <none>    Debian GNU/Linux 9 (stretch)   4.13.0-1011-azure   docker://1.13.1

  5. Test the windows containers. Create a windows-service.yaml with the following content:

     apiVersion: v1
 kind: Service
 metadata:
     name: win-webserver
     labels:
         app: win-webserver
 spec:
     ports:
         # the port that this service should serve on
     - port: 80
       targetPort: 80
     selector:
         app: win-webserver
     type: LoadBalancer
 ---
 apiVersion: extensions/v1beta1
 kind: Deployment
 metadata:
     labels:
         app: win-webserver
     name: win-webserver
 spec:
     replicas: 1
     template:
         metadata:
             labels:
                 app: win-webserver
             name: win-webserver
         spec:
             containers:
             - name: windowswebserver
               image: microsoft/windowsservercore:1709
               command:
               - powershell.exe
               - -command
               - "<#code used from https://gist.github.com/wagnerandrade/5424431#> ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('> {0}' -f $$requestUrl) ;  ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$header='<html><body><H1>Windows Container Web Server</H1>' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='<p>IP {0} callerCount {1} ' -f $$_,$$callerCounts.Item($$_) } ;$$footer='</body></html>' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('< {0}' -f $$responseStatus)  } ; "
             nodeSelector:
                 beta.kubernetes.io/os: windows

    And deploy it with:

     kubectl apply -f windows-service.yaml

 service "win-webserver" created
 deployment.extensions "win-webserver" created

    In a few minutes you will see the service up and running and the containers created:

     kubectl get all

 NAME                             READY     STATUS              RESTARTS   AGE
 win-webserver-6f5c5c676d-nhvd9   0/1       ContainerCreating   0          1m

 NAME            TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
 kubernetes      ClusterIP      10.0.0.1      <none>        443/TCP        39m
 win-webserver   LoadBalancer   10.0.121.17   <pending>     80:32326/TCP   1m

 NAME            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
 win-webserver   1         1         1            0           1m

 NAME                       DESIRED   CURRENT   READY     AGE
 win-webserver-6f5c5c676d   1         1         0         1m

 NAME            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
 win-webserver   1         1         1            0           1m

 NAME                       DESIRED   CURRENT   READY     AGE
 win-webserver-6f5c5c676d   1         1         0         1m

    It might take up to 10 minutes to get an external IP. But when you have it, you can use it in your browser and you will see something like this:

     <h1>Windows Container Web Server</h1>

 <p>IP 10.240.0.5 callerCount 1</p>

  6. On the other hand, you can deploy a simple linux nginx service with the following command:

     kubectl run my-nginx --image=nginx --replicas=2 --port=80
 kubectl expose deployment my-nginx --port=80 --type=LoadBalancer

    Wait for the IP to come back with kubectl get svc and browse into it. You should see the welcome page from NGINX running on Linux

References