Deploying Talend ESB using Azure Container Service and Kubernetes

You’ve probably heard that Kubernetes on the Microsoft Azure Container Service (ACS) is now generally available. This is great news for the developer community. With a production-ready ACS for K8S, you’ll now be able to use it in real applications. In this tutorial, we’ll show you how to create an ACS and deploy the Talend Open Source enterprise service bus (ESB) application.

Before You Get Started

You’ll first need to fulfill a few prerequisites. First, create a client_id and client_secret_id. Refer to Microsoft’s instructions on creating an Active Directory application and service principal for information on creating your ids. You will also need RSA public and private keys, and you will need to add the private part of the key to the ssh client. Next, obtain azure CLI 2.0 and kubectl. Finally, you must have docker on your workstation.

Deploying

First, log into http://portal.azure.com, and click + in the upper right. Enter “Azure Container Service”, click “Azure Container Service”, and click “Create”.

Select “Orchestrator — Kubernetes”. Enter the Resource group and Location (for example: group1, East US), and click “ok”.

Complete the required fields, using your own SSH public key. Enter client_id and client_secret_id as the values for Service principal client ID and Service principal client secret. Enter 1 as the value for Master count. (Note that one master is enough for demonstration purposes, but is not enough for production use.) Click “ok”.

Set the Agent count to 2, select the size of VM, and select the OS (2, Standard DS2, Linux). Review the summary, and then click “ok”.

Your deployment will now start.

Validating the Deployment

To validate the deployment, go to the Dashboard and select “Resource groups” from the left menu. Then, click on “group1”.

Ensure that you have a couple of different types of resources as shown in the following example screen.

Connecting to the Management Console

Open the command line client with SSH support, type az login, and follow the instructions.

Type az account list, and find your subscription id. (This is the id you used to create the “rgroup1” resource group.)

Enter the following information, replacing the content in {} with your unique information:

az account set — subscription={your-subscription-id}

az acs kubernetes get-credentials — resource-group={your-resource-group} — name={name-of-container-service} — ssh-key-file={path-to-your-private-key}

(You can find the name of the container service under the resource group in the portal. You must specify the password of the key.)

You should now have a “config” file in the /home/user/.kube folder.

Ensure you have a connection to the cluster by typing kubectl get nodes. You should see a display like the following:

NAME STATUS AGE
 k8s-agent-e2fcdf77–0 Ready 49m
 k8s-agent-e2fcdf77–1 Ready 50m
 k8s-master-e2fcdf77–0 Ready,SchedulingDisabled 50m

Next, run kubectl proxy, and return to the dashboard. You can now start using your cluster.

Want a stress-free K8S cluster management experience? Download our demo, Kublr-in-a-Box.

Building the Docker Image

Before you can begin building your docker image, you’ll need to download the Talend ESB and you will also need an account on docker hub. Next, you must modify the build docker image file, Dockerfile, to ensure you have the correct paths.

In the following example, Runtime_ESBSE_light.zip is an archive which contains Talend ESB.

FROM openjdk:7-jdk
LABEL maintainer aocheretnoy@gmail.com
ENV TALEND_DOWNLOAD_URL=https://sourceforge.net/projects/talendesb/files/Talend%20Open%20Studio%20for%20ESB/5.6.2/TOS_ESB-20150508_1414-V5.6.2.zip/download
ENV TALEND_LOCAL_STORAGE=talend-release
ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64
ENV TALEND_DIR=/opt/talend
# Update ubuntu and install the required packages
RUN apt-get update && apt-get install -y make ant
# Download Talend
#
# Talend official packages are hosted on Sourceforge and it is slow to download in general
#RUN wget — no-check-certificate -O $TALEND_DIR.zip $TALEND_DOWNLOAD_URL && unzip $TALEND_DIR.zip -d $TALEND_DIR
# Copy Talend from local storage
#
# This is here until a fast server is available to download Talend packages
COPY $TALEND_LOCAL_STORAGE/Runtime_ESBSE_light.zip /opt
RUN mkdir $TALEND_DIR && unzip /opt/Runtime_ESBSE_light.zip -d /opt/talend; \
   rm /opt/Runtime_ESBSE_light.zip
VOLUME [“/opt/talend/Runtime_ESBSE_light/container/deploy”]
EXPOSE 8040 9001 1099 44444 8101 8000 8001 8888 61616 2181 1527 8082 8088 8090
CMD [“/opt/talend/Runtime_ESBSE_light/container/bin/trun”]

Enter “docker build -t tesb .”, where tesb is the name of a new image. Then, enter “docker tag tesb arkadiy/tesb:latest”, where arkadiy is the name of a public docker repository on docker hub. Replace arkadiy with the name of your own public docker repository.

To complete the build, enter “docker login” and then enter “docker push arkadiy/tesb:latest”.

Deploying an Application

To deploy an application, take a moment first to ensure you’ve completed the steps in “Deploying”, “Validating the Deployment”, “Connecting to the Management Console”, and “Building the Docker Image”.

Then, go to the dashboard and select “Deployment” in the left menu. Click “+Create” in the upper right, and select the “Upload a YAML or JSON file” option.

Modify tesb-deployment-acs.yaml to use your image name. Replace “image: arkadiy/tesb:0.2” with your image name from “Building the Docker Image”.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: tesb
spec:
 replicas: 2
 template:
  metadata:
   labels:
    name: tesb
  spec:
   containers:
   - name: tesb
     image: arkadiy/tesb:0.2
     ports:
     - containerPort: 8040
     - containerPort: 9001
     - containerPort: 1099
     - containerPort: 44444
     - containerPort: 8000
     - containerPort: 8001
     - containerPort: 8101
     - containerPort: 8888
     - containerPort: 61616
     - containerPort: 2181
     - containerPort: 1527
     - containerPort: 8082
     - containerPort: 8088
     - containerPort: 8090
     volumeMounts:
     - name: deploy
       mountPath: /opt/talend/Runtime_ESBSE_light/container/deploy
     volumes:
     - name: deploy
       hostPath:
        path: /home/azureuser/deploy

Select tesb-deployment-acs.yaml, and click “Upload”. Next, go to “Pods” in the left menu. Wait until “tesb-….” is ready. This should take approximately 5 minutes because the image is 2GB.

Go to “Services”, click “create”, and upload the two services.

First service: tesb-service-acs.yaml

apiVersion: v1
kind: Service
metadata:
 name: tesb
spec:
 type: LoadBalancer
 ports:
   - name: api
     port: 8088
     targetPort: 8088
   - name: apiint
     port: 8090
     targetPort: 8090
 selector:
  name: tesb

Second service: tesb-service-acs-nodeport.yaml

apiVersion: v1
kind: Service
metadata:
 name: tesb-int
 labels:
   name: tesb-int
spec:
 type: NodePort
 ports:
  - name: http
    port: 8040
    targetPort: 8040
  - name: https
    port: 9001
    targetPort: 9001
  - name: jmx
    port: 1099
    targetPort: 1099
  - name: jmxrmi
    port: 44444
    targetPort: 44444
  - name: jobserver
    port: 8000
    targetPort: 8000
  - name: jobserverfile
    port: 8001
    targetPort: 8001
  - name: karafssh
    port: 8101
    targetPort: 8101
  - name: jobservermonitor
    port: 8888
    targetPort: 8888
  - name: activemq
    port: 61616
    targetPort: 61616
  - name: zookeeper
    port: 2181
    targetPort: 2181
  - name: esbsam
    port: 1527
    targetPort: 1527
  - name: esbinstaller
    port: 8082
    targetPort: 8082
  selector:
    name: tesb

Go to “Services” and wait until service tesb displays an ip address. This will take a few minutes as load balancer and rules are created.

Deploying the Integration Scenario

To deploy the integration scenario, ensure you have completed the steps in the previous sections and that you have a Talend OpenESB compatible application (download here).

TalendESB is based on the apache karaf OSGI container, so you only need to copy archives to “deploy” deploy. Refer to tesb-deployment-acs.yaml for the path. In this example, we use /home/azureuser/deploy as the mount path for the container.

To test your integration scenario, use Postman to ensure the application works.

That’s it! You’ve created an ACS and deployed the Talend Open Source enterprise service bus (ESB) application.

Share your thoughts and questions in the comments section below.

Need a user-friendly tool to set up and manage your K8S cluster? Check out Kublr-in-a-Box. To learn more, visit kublr.com.