MQTT 5 is pretty new, a lot of things changed.We will walk through the most important changes of MQTT 5 in this blog.
I guess most of you are already familiar with MQTT but for those who are not quite sure what MQTT is, what it’s about and what are the main principles let’s start with a quick refresher this is also important to understand some of the changes in MQTT 5.
MQTT is an iot messaging protocol. It gained massive attraction in the last few years.It’s mostly used for device to cloud communication and cloud to device communication sometimes device to device communication directly.It has a lot of features which are good for mobile network use cases.If you have devices out in the field like cars or like gateways or physical hardware which needs to connect via mobile networks to the backend services.
MQTT has 3 different quality of services levels which you can define on the application level.You can send messages with fire and forget.you can also make sure that message arrives at least once or exactly once on the backend.
MQTT has nice features which are unique to the protocol is retained messages which essentially allows you to save messages on your MQTT message broker.
Persistent offline sessions
MQTT has a feature which allows the client to come back and the broker remembers the client and also sends out messages the client did not get at the time.In general, from an application perspective, you can program with MQTT like you never lost the connection.
MQTT is a binary protocol with a minimal overhead so it’s really tiny it saves a lot of bandwidth.
MQTT Use Case
MQTT is a very good protocol for constraint devices.If you don’t have too much computing power or a memory then MQTT is a very good protocol choice.This is typically on physical hardware at a few megabytes of memory.
Typical use cases for MQTT is push communication.We have reliable communication of unreliable networks this is mainly mobile networks.
Low Bandwidth and High Latency
It also stays extremely on the backend. Some MQTT brokers allow scaling to more than 10 billions of concurrently connected devices.Typically especially if you’re in a mobile network you offer a very low bandwidth in the high latency and MQTT make sure that you really get the best here so it doesn’t waste any bandwidth and the best at low latency.
MQTT uses the publish/subscribe mechanism.we have an MQTT broker which is in middle.MQTT broker is responsible for distributing data across clients.
We have some producing clients like here temperature sensor.It publishes data to MQTT broker and the broker distributes the data to devices which could be in this case a laptop or a mobile device it could be a car it is really anything which can connect to the internet.
This device works by a subscription mechanism.The laptop or the mobile devices subscribes to the MQTT broker it says hey “I’m interested in a particular data set” and the broker make sure that only the data which the client are interested in forward to their clients. You have a decoupling here and what is very important to know about MQTT you have a standing TCP connection here so all of the devices which talk to the MQTT broker are connected all the time so this is different to other protocols like HTTP which typically close the connection after doing stuff.
MQTT 5 Overview
MQTT 5 is the successor of MQTT 3.1.1.It is not backward compatible with the old specification.Officially released in January 2018. It also had a lot of clarifications from the MQTT 3.1.1 specification so to make sure that implementers of the specification get everything right.
MQTT 5 Goals
The goals of MQTT 5 is enhancements scalability and improved error reporting.Error reporting wasn’t most wished features by users because MQTT 3.1.1 has some blind spots when you come to error handling.MQTT 5 did a lot of work here and also formalized common patterns like request response.
One of the most wished features is the extensibility of the protocol because they(MQTT 3)weren’t headers like you know from HTTP. MQTT 3.1.1 wasn’t that flexible this changed.Performance improvement for small clients they’re also a big part.
MQTT is very easy to program on the client side but it’s important to notice that implementing MQTT broker is not as easy as it sounds.
MQTT 5 has some enhancement for the scalability.MQTT free brokers scale up to 10 millions of devices which is already a lot but also we expect that MQTT 5 allows us to scale even beyond the magic 10 million by concurrent connections.
Before digging into the specific features let us talk about the foundational changes of the MQTT protocol
First foundational change is Negative acknowledgments.I already mentioned that the error reporting in MQTT 3.1.1 wasn’t optimum.There was no way for the broker to tell “hey you’re doing something wrong”.So what was defined here was their negative acknowledgments.A broker can actually notify the client that something went wrong and also what went wrong. This is for many use cases this is very important especially for production use cases where it’s it’s hard to debug what happened.The client can react when something weird happens but also the client can send acknowledgments to the broker if something bad happens.
Another foundational change is returned codes for unsupported features when a client connects to an MQTT broker if the broker does not allow all MQTT features or doesn’t implement all MQTT features which is pretty common atypical like cloud platforms like AWS.It’s possible to tell the client, ok this feature is not available or this is not available for you.If you haven’t a permission control and you do not allow clients let’s say to subscribe or publish or something like this.
MQTT 5 can restrict the retain message feature so we can turn it off if we want for a specific client.
we can define a maximum quality of service level for the client.
We can restrict the use of wildcard subscriptions.
we can also restrict subscription identifiers share subscriptions.
A client can be notified what is maximum message size to broker support.What deserves a keepalive value is which is also very important because it also changes with MQTT 5.
The broker can tell the client how often it should send ping messages or heartbeat messages in order to recognize it for clients is offline.
Another notable change is MQTT 5 not allowed to send retry for the quality of service 1&2 messages. This is something which may come surprisingly for some people because We noticed that many projects rely on retries which are not allowed with MQTT5. So this is a common pitfall when upgrading to MQTT 5.
Passwords are now allowed to send without having usernames which may be interested in sending tokens.Clients are now allowed to send disconnect packages traditionally. The broker has no way to disconnect the client gracefully with MQTT 3. The client can connect and when it decides okay I want to disconnect now in a graceful then the client sent a disconnect packet to a broker but now it’s also allowed for the broker to say that this connect packet it back to the client and tell the reason why it was disconnected this is always something new and is used heavily for the negative acknowledgments.
New Features in MQTT 5
Let’s talk about some of the features in MQTT 5.We cannot dig into all features because there are more than 25 features available which take a lot of time but what I want to highlight some of the features more detail.
Session & Message Expiry
So one of them from interesting features of MQTT 5 is session and message expiry.
MQTT 3.1.1 allows two kinds of session. We have a clean session which ends when a client disconnects so the broker does not need to remember anything about the client and we have a persistent session which is a session, the broker saves and persists when a client is offline and come back. the client can just resume the session. So essentially what we get here is we have a state on the broker side.
The problem here is if we have some clients which connect to the broker and disconnect and never come back. The broker can have a very hard time because it needs to persist to data until the client reconnects.When a client never reconnects essentially we get a problem here and most brokers like mosquito allow to set that time to live for a session on an administrative level. So the broker can clean up after this time.
In MQTT 5 this feature went back to the specification and now all brokers must implement a cleanup because this is needed on the broker’s side otherwise it would be a problem of denial of services attacks. So the client can say okay when I’m connecting to this broker I want the session expiry interval in let’s say 100 seconds or 10 minutes or one day and then the broker is allowed to delete the session after this time.
The problem here is let’s assume we have a car which often for one week and when it comes back doesn’t really need to get all messages.Perhaps some messages need some messages not and now sending client can send publish expiry to a published message and say to the broker “if time to live message is over do not send out the message anymore”.This allows the broker to clean up messages especially if it’s queue of messages and also it saves a lot of bandwidth.
Return code for all acknowledgments is essential.
So now the broker and the client can say sorry we have a problem here and this is the problem there are reasons defined which are humanly readable. Which the broker and the client can send out if they want but they aren’t they don’t need to do this and also send a return code. These return codes are specified most of them are error codes and all client and broker must have the ability to react to this code to the returns code. So if you are a client and the broker disconnects because you were let’s say idle for long then you have the possibility on the client side to adjust the interval for sending heartbeats.
Payload format indication & Content Type
What you get here is a content type this is similar to mime types like this is a JPEG picture or this is a text.This can be sent as a meta information.You can also indicate what kind of content do we have it is also possible for up messages.We get two new headers in MQTT content type which is the mime type and you have a payload format indicator this is more interesting for debugging purposes.
Request and Response
It is possible to send hints like you want to request-response.What you can do here is you can send metadata like request response information. So a publisher can say that can send a message and it can also send a metadata.Request response information can be used that the receiver of the message can send the answer to the topic.
Share subscriptions are a very interesting concept for scaling out backend subscribers.
Let’s assume we have a publisher which sends a lot of messages.We have a high-frequency topic.The problem which could arise is that the backend clients cannot process data this fast. Because it says it writes to the database which is slow at the moment, what to do how can we scale this? with MQTT 3 you cannot scale this without using shear subscriptions.
MQTT 5 has a logical or mutual shared topic a shared subscription.The client can decide ok I want to share my subscription with others clients and then the broker can send one message to the one client and one message to the other client.If you have a stream of let’s 1,000 messages/second on a topic and you have a share subscription with two clients then each of these clients get 500 messages/seconds and now if you scale out three clients then each of these backend instances get 330 messages/second and so on.This is a way how you can elastically scale up the backend subscribers or client up and down for topics which have a lot of traffic.
MQTT 5 has headers and it’s possible to use user-defined properties which is you can just like with HTTP. Add any data you want to your MQTT packet. It can modify the publisher and user properties.Let’s say you have a publisher which has some application specific identifiers which you want to parse in the backend without looking into the whole payload of the packet contrast.the backend applications can just get out there the header without decoding the whole message.You can add unlimited user properties which is a bit controversial.
Topic Alias: The client can choose to shrink topics.If you have a very long topic and repeatedly published.They save a lot of bandwidth because they can just use an alias.
MQTT 5 has Flow Control so our client can decide how many messages you can actually receive.
It has maximum message size indication and an authentication flow.
We also get will delay. You can tell the broker, please wait 5 seconds before sending out the last New Testament message.
The broker can tell the client what keep alive. It expects the program also overwrite client identifier.
Unfortunately, I did not find any MQTT5 broker yet expect the eclipse paho test broker.