Effectively manage IoT Production with Docker MQTT Server
by Ponlakshmi
As of 2024, millions of IoT Applications are running on the cloud. More than 50% of the deployments leverage cloud based deployments envisioning large scale connectivity and scalability. MQTT Broker is not the only component of an IoT Deployment. We do have a lot of additional components like Queues, data storage, data aggregation engine, AI/ML, monitoring tools, and more as needed for the Solution. Managing such a mix of applications talking to each other for effective data collection, processing, and visualization needs a better and seamless orchestration. Docker is one such great tool which is available FREE and open source.
Docker compose for MQTT Broker
The core component of the IoT Deployment is the MQTT Server. There are two ways of using the same.
- Using it as a Central MQTT Broker - It will be used only for the data exchange. The IoT Application will be deployed at the edge more like the other clients, and the Host application subscribes to all the topics and will exchange data more like the other clients do.
- Using it with the separate IoT Data Storage - The MQTT application will be integrated into the MQTT Broker or will be behind the MQTT Broker communication more from a data perspective and Queues between the Application and the MQTT Broker.
For both these ways of running the creation of the docker is much similar except for a few configurations for the data storage or connection to the backend application.
This section helps you set up a docker MQTT Server environment with a data storage on a relational database. We have taken a ubuntu based deployment as an example to deploy the MQTT Broker. However this can be done with the other operating system as well.
Prerequisites
- Ubuntu
- Docker
- Mysql - if you are planning to store data into the database. The instructions will run MySQL on the physical vm itself. However it can be deployed inside the docker as required or Amazon RDS kind of data stores can be used.
Setting up the MySQL
We recommend MySQL to be run on a physical machine than a section for the data persistence purpose. Follow the below instructions to set up the MySQL. You have to install the MySQL Server first.
$ sudo apt-get update
$ sudo apt-get install -y mysql-server-5.7
Modify the MySQL configuration. By default the mysql runs at 127.0.0.1 and allows only the local application to connect to it. You can change it as 0.0.0.0 for application running on sections and other machines to access the MySQL.
$ gedit /etc/mysql/mysql.conf.d/mysqld.cnf
bind-address = 0.0.0.0
Restart the MySQL for the changes to take effect.
$ sudo service mysql restart
Providing data access: The root user of MySQL has access only form the local machine to read and write data into the databases. We need to allow the root user to access data from the section. Connect to the MySQL console from the localhost and then execute the following queries.
$ mysql -u root -p
Enter password: <provide your password here>
mysql> GRANT ALL PRIVILEGES ON *.* TO ‘root’@’%’ IDENTIFIED BY ‘root’
WITH GRANT OPTION;
mysql> FLUSH PRIVILEGES
Your MySQL server is all set to connect from any section that runs on your Ubuntu server.
Creating Docker MQTT Broker
Docker needs to be installed in your ubuntu to create the docker image of the MQTT Broker and also run the dockerized MQTT Broker. To install the Docker, run the following command.
$ apt-get install docker.io
Download Free MQTT Broker for Ubuntu and Extract it from your /home/ubuntu folder. If you are extracting the archive from some other folder, make sure you use the right reference folder
ubuntu@
Modify the configuration of the MQTT Broker for MySQL connections
$ cd /home/ubuntu/Bevywise/CrystalMQ
$ gedit conf/datastore.conf
DB_SERVER = MYSQL
DBHOST=<mysql host ip>
MYSQL_USER= <your mysql user>
MYSQL_PASSWORD = <mysql db password>
Create the docker file
$ gedit Dockerfile
FROM ubuntu
RUN apt-get update && apt-get install -y \
libexpat1 \
libsqlite3-0 \
wget \
unzip
RUN rm -rf /var/lib/apt/lists/*
ADD ./CrystalMQ /CrystalMQ
RUN chmod +x /CrystalMQ/bin/runbroker.sh
RUN chmod +x /CrystalMQ/bin/install_mysql_connector.sh
WORKDIR /CrystalMQ/bin
RUN ./install_mysql_connector.sh
EXPOSE 1883 8080 8883 10443 8081
CMD ["./runbroker.sh"]
Build the docker image
# docker build -t mqttroute .
Verify & Run Docker MQTT Broker
# docker images
root@ubuntu # docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
mqtt latest 45f7ee9a1ec7 7 seconds ago 81.15 MB
ubuntu latest 7feff7652c69 4 weeks ago 81.15 MB
Start using the docker command. The following command will run the MQTT Broker as a daemon process. If you are planning to run MQTT Broker in a secure mode, the document on deploying a secure mqtt server will help you create a secure instance and then package it with the docker.
$ docker run -d --name "enter-container-name" --restart=always -p "8080:8080" "enter-docker-image-name"
Note: If you've used a different port instead of 8080, just replace the port number in the command above.
Below is a downloadable Docker Image file of our MQTT Broker, which provides all the necessary resources, allowing you to easily set up and run Docker through a straightforward process.
Note: The Downloadable Docker Image setup only supports the SQLite database, and the data will not be stored; you can only view the real-time data.
After downloading, run the following command to decompress the file and prepare it for extraction:
bzip2 -d route.tar.bz2
Then run the following command to load the necessary resources into the Docker system.
docker load -i "file.tar"
Next, you can proceed by running Docker with the following command:
$ docker run -d --name "enter-container-name" --restart=always -p "8080:8080" "enter-docker-image-name"
Note: If you've used a different port instead of 8080, just replace the port number in the command above.
Data Storage & Best practices in Docker environment
MQTT Broker by default supports a number of database storage like MySQL, MSSQL, PYSQL, SQLITE, Elastic Search. Any one of these IoT Data storage can be enabled in the data storage configuration file itself. In addition to these storage, we can customise additional storage requirement using the Python MQTT Broker custom extenstions to transform data to the format as needed before storage. These custom extensions can also be used for connecting these data to the queues or enterprise applications.
The data storage or queue or any other application that receives or sends back data to the MQTT Broker should be placed out of the docker image. If your other component is a queue or application that does not store data, you can run that as well on a docker. If you are connecting to a database or a big data engine, then it has to be run on a physical or virtual server with persistent storage. It is recommended to use the storage as a service like Amazon RDS for better and assured storage of data.
Running MQTT Server ( IoT Platform) Components on Docker & Kubernetes
Solution provider IoT Platform is a multi-tenant framework and it is based on the micro services based architecture. Each component of the framework can be run on a docker container and deployed on the Physical or VM servers on the cloud or on premise. the dockerized components of the MQTT Server can be deployed on the kubernetes. The deployment in this case will be done by us. The complete monitoring scrips for the components including the functional and the application monitoring will be deployed and can be monitored by your Dev-ops team over the web UI console.
How Docker helps MQTT Broker deployments?
Containerization helps running the application in isolation independent of the Operating system of the physical or VM Server making the MQTT Server a predefined Virtual environment giving a higher level of consistency. As the dockerized VMs consumes less memory, the overall application will be highly scalable and portable. The combination of docker and kubernetes will make the deployment span across multiple VMs keep the control over a single orchestration for easy management.
Must Read Other Related Posts
Docker compose for MQTT
Scaling up your MQTT Deployment
Let us Collaborate
