Understand MQTT Protocol Packet & Build Reliable Applications

MQTT Protocol (Message Queuing Telemetry Transport) is a TCP-based messaging protocol specifically designed for transferring telemetry data from edge devices to a central server. It follows a Publish/Subscribe model, enabling MQTT clients to efficiently send and receive data through a central MQTT broker. Since its invention in 1999, MQTT has become the most widely adopted protocol for IoT implementations due to its lightweight nature and reliability. On the other hand, MQTT-SN (Sensor Network) extends the protocol for edge or peripheral devices, allowing communication between devices and gateways over low-power connectivity mediums like Bluetooth. Despite its potential, MQTT-SN hasn’t gained widespread adoption like the core MQTT protocol. While many MQTT implementations are readily available in the market, having a deeper understanding of the underlying concepts is crucial. This knowledge empowers you to leverage these implementations effectively, enabling you to build high-performance, fault-tolerant IoT applications.

MQTT Protocol Packet Types

The publish-subscribe interaction between MQTT devices and the MQTT server is facilitated through a set of predefined messages. For simplicity, both outgoing and incoming messages follow the same standard. At a high level, MQTT protocol packets can be categorized into two types for better understanding

• Data Packets - The messages that carry the actual data between the entities.
• Control Packets - The messages that help control the connections and the message delivery as per MQTT QoS Levels.

Beyond the MQTT Protocol packets, there are a few other specifications that takes care of how to handle sessions, retaining the last message and the disconnections which will be covered as well in this article as it is very important to understand that for the MQTT Implementation.

MQTT Protocol Data Packets

Data Packets help transfer the actual data between the MQTT Broker and the MQTT Client. Publish and Subscribe are the main data packets where the MQTT client sends its intentions to listen to a topic and the server publishes the data based on the requirement. We want to add the Connect packet as well to this list as a set of data is being sent as part of it.

• Publish - MQTT Publish packet contains the Topic to which the data needs to be published, the actual data, and a few control parameters like the request for message retention and Quality of Service.
• Subscribe - MQTT Subscribe is one of the important messages sent from the Client to the server indicating the list of the MQTT Topic filters, the client is interested in listening to. The Packet contains a Fixed Header, Variable Headers & a Payload.
• Connect - MQTT Connect Packet is more of an MQTT control packet, but holds data about the LastWill and Retain. In addition, it sends a Unique Identifier, MQTT Authentication details, and Clean session flag to say whether the session needs to be persisted or not. This packet is sent from the Client to the cloud hosted MQTT Server.

MQTT Packet Format structure for all the above three messages contains Fixed Header, Variable Header, and Payload.

MQTT Protocol Control Packets

Control packets are crucial for ensuring the success of data transfer in MQTT Protocol. A clear understanding of control packets and MQTT Quality of Service (QoS) levels is essential for effective implementation. While most control packets are managed at the core implementation layer depending on the MQTT client or server used, tasks like storing delivery states, handling acknowledgments, and resending messages need to be managed at the application level.

• CONNACK – This packet acknowledges the connection request and contains only the Fixed Header in the MQTT 3. MQTT 5 has options to send a reason code for the rejection of the connection. CONNACK sent on successful connection includes disclosure of the server capabilities like availability of retain, Wild card subscriptions, shared subscriptions, subscription identifiers and maximum QoS Supported, maximum packet size, and server keep alive for the client to efficiently communicate.
• SUBSCRIBE - MQTT Subscribe is one of the important messages sent from the Client to the server indicating the list of the MQTT Topic filters, the client is interested in listening to. The Packet contains a Fixed Header, a Variable Header & a Payload.
• SUBACK – This packet is sent in response to the subscription message from the MQTT Client for the MQTT Topic Filter(s). This packet contains Fixed & Variable Headers and Payload.
• UNSUBSCRIBE – This packet is sent from the client to the server to instruct the server to stop sending further messages for a particular topic. The message contains a Fixed Header, Variable Header and Payload.
• UNSUBACK – On receipt of the Unsubscribe for a topic filter from a client, the server should remove that topic filter from the list of topic filters it has stored as subscriptions for that client. The unsuback contains Fixed Header, Variable Header, and Payload.
• DISCONNECT – Once the MQTT Platform receives this request from the MQTT Device, it should disconnect the client and close the connection. The clean session parameter set during the connection will determine whether the client’s future subscription messages are stored or discarded. The WILL Message should be removed. The packet contains only the Fixed Header. In MQTT 5, the server can also send DISCONNECT to a client if that client does not conform to the protocol.
• PUBACK – This packet can be sent by the client or server as an acknowledgment to the PUBLISH packet it has received at QoS Level 1. If the PUBACK is not received, the sender should resend the message with the DUP flag set while resending. The sender can remove the message from the queue after receiving this acknowledgment. The packet contains a Fixed Header & Variable Header.
• PUBREC, PUBREL & PUBCOMP - MQTT QoS level 2 ensures exactly one delivery of MQTT PUBLISH packet. These control packets are exchanged in a pre-defined series to ensure this QoS Level 2 delivery. This series of packet flows is followed by both the client and the server. When the receiver receives a PUBLISHed packet, it sends a PUBREC as an acknowledgment. The sender of the PUBLISHed packet must now send a PUBREL packet in response to PUBREC. The receiver sends PUBCOMP in response to PUBREC. All these control packets contain Fixed & Variable Header.
• PINGREQ, PINGRESP - These packets are used to keep the connection alive and work based on the MQTT Keep Alive set during the Connect Request. It is the responsibility of the MQTT Client to send the PingREQ packet at an interval of keep alive time. The Standard MQTT Server should send back the PingRESP packet as the acknowledgment. The server will disconnect the client connection if the PingReq is not received. In general, a wait time of 2x the keep alive time is considered before disconnection. These packets have only a Fixed Header.

MQTT Protocol Packet Format & Sizes

MQTT Packets can be broadly divided into three parts namely Fixed Header, Variable Header, and payload. If you are starting your implementation over a standard MQTT Clients, then this information may not be much needed. However, for understanding how to use the right APIs to use the client, you can have this hands-on. The fixed header field comprises the control flags; the variable header contains the packet length; the payload will be equal to the length set in the variable header. The minimum size of a packet length field is 1 byte, which is for messages less than 127 bytes.

The minimum packet size is 2 bytes, which has only the control field and a single packet length field, while the maximum packet size is 256 MB. Smaller packets less than 127 bytes have a 1-byte packet length field. The Packets greater than 127 and less than 16383 use 2 bytes, and so on. 7-bits that are used with the 8th bit is the continuation bit.

Control Field

The 8-bit control field is the principal byte of the 2-byte fixed header. It is divided into two 4 bit fields and contains all protocol commands and responses. The first four most important bits are the command or message type field, while the remaining 4 bits are used as control flags.

Control Flags

While there are 16 possible flags, a limited set of bits are still used. The Publish message makes the most use of these flags.

• The duplicate flag is used when a message is republished for QoS 1 or 2.
• The QoS flags are used when publishing to indicate a QoS level of 0,1,2.
• The MQTT retain message flag is also used when publishing a message.

Remaining Length

The Remaining Length is the number of bytes left in the current packet, including variable header and payload data. The Remaining length does exclude the bytes used to encode the Remaining Length. The Remaining Length is encoded using a variable-length encoding scheme that uses one byte for values up to 127. The least seven important bits of each byte encode the data, and the most important bit is used to indicate there are following bytes in the representation. Thus, each byte encodes 128 values and one “continuation bit”. The greatest number of bytes in the remaining length field is four.

Variable Header

Some types of MQTT control packets have a variable header component. Variable-length header fields are not always present in MQTT messages. Some MQTT message types or commands require this field to carry additional control information. It remains between the Fixed Header and the Payload. The Variable Header will vary depending on the packet type.

MQTT Packet Format Example: Connect Message Structure

As an example, let’s take a look at the details of the MQTT Client CONNECT message.

After establishing the TCP level connection with an MQTT Platform, the first packet must be a CONNECT packet. The CONNECT packet only needs to be sent once over the network connection. The second CONNECT packet sent by the MQTT client is ignored and disconnected.

Fixed Header

Variable Header

The variable header sits next to the Fixed header and has four major parts as part of the header. They are the protocol name bytes, protocol level, connect flags, and keep-alive. Each of these parts has a predefined byte size and the order in which they are present.

Protocol name

These bytes store the Protocol name. The protocol name is stored as MQTT in 4 bytes. Any other characters in any of these fields will lead to disconnection of the client from the server.

Protocol Level

This defines the protocol version. The protocol level for MQTT broker is 4 for the MQTT 3.1/3.1.1 and 5 for the MQTT 5 version. Sending other values will lead to disconnection. The MQTT Broker can also disconnect the client based on the version it supports.

Connect flags

This is a single byte field and represents the presence or absence of data in the payload for MQTT Authentication (username and password), Will message. The other fields like Will retain and clean session will be either true or false and the QoS field specifies the QoS level to be followed for WILL Message.

MQTT Keep Alive

MQTT Keep alive is a 2 byte packet, and the maximum time you can set for the keep alive is 65535 seconds (18 Hours). However, it is better to keep it around 30 seconds or a minute based on your requirements. The MQTT client should send control packet (PINGREQ) at an interval that does not exceed the value of the defined Keep Alive. The Keep Alive value depends on the duration of transmission of the control packet by the client.

MQTT Protocol Payload Format

The payload contains the MQTT Client credentials (username, and password). The client ID must be unique for each client. The broker responds with a unique client ID for each client if the client does not specify a client ID. If you send a client ID in an empty field, the client will be rejected. The fields must be filled in the following order: Client ID, Will Topic, Will message, username, and password.

All the fields in the Connect packet should be in the string format. After establishing the TCP level connection, the client should send the MQTT Connect within a reasonable time. If the connect packet is not received, the MQTT Broker will disconnect the client. After successful validation, the broker will either respond with a CONNACK packet with zero payload or with the disconnection if there are some errors in connection. After successful connection, the broker will add the new client to the connected list, and the session will be maintained and monitored for the data packets as well as the keep alive.

If you are building from scratch, follow the Complete MQTT Client Developer Guide and the specifications to understand and implement it better.

MQTT Implementation - related reading

For a very successful MQTT Application, Do read additional article beyond the MQTT Protocol.

• MQTT Security Essentials
• Secure MQTT Broker communication with TLS Certificates
• Running MQTT Broker inside a Docker Environment