I've learned a lot about MQTT while setting up Home Assistant, and Mosquitto is a pretty important part of my Home Assistant installation. I use it to control all of my Zigbee devices via "zigbee2mqtt" (buttons, switch modules, etc.), and also a smart kettle running Tasmota [1]. I also added a little patch to Tasmota [2] so that my kettle responds with `418 I'm a teapot` [3]
Cool stuff and inspiring. Have a website or GitHub about your home automation projects? I checked out your blog, and the raspberry pi microwave was pretty sweet.
MQTT is becoming the defacto interface for home automation. As an example, Home Assistant used to have plugins for zwave and zigbee but they are now replaced by an MQTT broker that sits between Home Assistant and the program driving the wireless dongle.
Since I'm already running Mosquitto to interface with zwave devices, I decided to also use it to monitor my home lab. Servers push sensor statuses into Mosquitto and Home Assistant is used for the UI and alerting. That was a fun week-end project.
> Home Assistant used to have plugins for zwave and zigbee but they are now replaced by an MQTT broker that sits between Home Assistant and the program driving the wireless dongle
And now, they've moved on from that to having straight websocket communication between those two programs, eliminating MQTT altogether.
I use MQTT for a lot of stuff but it adds a layer of delay between devices that is particularly noticeable with flipping a light switch in the Home Assistant web ui.
I observed this as well and was just starting to dabble in MQTT with my ESP projects.
I have yet to find a good resource for outlining autodiscovery. Most of the projects I have come across just hard code some pub paths in the homeassistant namespace.
>MQTT is becoming the defacto interface for home automation.
The problem I have with MQTT is also a feature. The payload format isn’t defined, there is no request response structure, encryption of payload requires custom encapsulation, etc. You are on your own for everything.
This is made better in MQTT v5, but AWS still doesn’t support that.
If I was starting over today, I really like OSCORE and CoAP derivatives for their completeness.
This should have been my first comment, but there you go.
Hello, I'm Roger, the Mosquitto project lead. Thanks for the kind comments, I'm happy to answer questions!
I work for cedalo.com developing Mosquitto and other MQTT projects. We offer Mosquitto based support, please get in touch if that's of interest to you.
I just wanted to thank you for it. Over the years it has had a big impact on all types of projects, from industrial heavy ones to raspberry-based tinkering experiments :D
I've recently transitioned my home "IOT" network (and servers) to a Wireguard-only network. It's wonderful. All the routing accessibility through firewalls or with dynamic IPs just disappeared. All services are exposed only on the wireguard interface, meaning security and connectivity are solved (to the 80/20 level) with just Wireguard.
That kind of removed the need for a static broker entirely since now every node has a static IP on my wireguard network and are accessible no matter what network they are actually on. Pinging my mobile phone from an air quality sensor was an eye-opening experience.
Did you ever notice any issues with significant increased softirqs?
I started going down a similar path but a node that is acting as a gateway/bridge for UDP traffic routing to an outbound wireguard tunnel is now spending a constant ~20% CPU time on NET_RX softirqs with 10Mbps and 5~30Kpps. Almost no NET_TX.
It's specifically the combination of UDP-over-wg that seems to cause this. It's otherwise handling 1Gbps and way more packets just fine.
I'v hoped to find in your profile maybe more info/blog, but instead: > Group supervisor at NASA Jet Propulsion Laboratory working in autonomous vehicles and sensing for space exploration.
Oh well, a little out of scope and I see you blog about other things. Anyway - you mean I don't need static IP to form a network of distant IoT devices? This would be some really useful property.
I'll just pile on my own accolades. We use MQTT to connect handhelds with in the field ag irrigation controllers. There's probably about 10,000 valves on farms and other ag projects throughout, that are ultimately scheduled and monitored through MQTT. We've even been able to build a full request/event protocol on top of it.
The protocol is simple enough, that when I wasn't happy with not many full solutions existing in iOS, I just rolled our own in a day or two.
Currently adding an Elixir REST bridge to the whole thing. It's been pretty cool all around.
I started home automation in my late 40-ies, as an amateur developer.
Despite having casually developed for 25 years at that point, I found it amazing how much I learned quickly when setting up Home Assistant and pyscript (and obviously mqtt).
This drove me to use a bus (mqtt for instance) for non-home automation code just because it is so useful.
To friends who wonder how to help their kids get interested in coding (which is not easy at the beginning because converting farenheit to celsius gets boring after some time), I tell them to buy a few zigbee devices and have them automate stuff in pyscript. I am happy to see that some are on the way to graduate in CS :)
I've never used Mosquitto, but I have done a fair amount with Verne.mq [1], and I have to say that MQTT is downright pleasant to use in a lot of cases. I've not done a ton with it in "real world" situations, but I have used it for multiple hackathons, and I'm always impressed how little of a headache it is to build a decent "live" application with almost no effort.
If your frontend web project calls for any kind of messaging, I definitely recommend looking into trying MQTT before you jump straight into WebSockets. There's a good chance MQTT does what you need, scales better, can communicate over WebSockets, and will make your life easier .
For someone that has worked with RabbitMQ, is there a guide to the differences and appropriate usage of MQTT brokers with some real use cases explained?
You are aware RabbitMQ support MQTT[0] and MQTT over WebSockets[1]?
As for comparing it to RabbitMQ you should compare the AMQP protocol vs MQTT protocol, it seems MQTT with its lightweight design "won" IoT market over and should be rather compared against protocols like CoAP.
As for guides I have always found resources rather scarce and mostly learned by adding features to mqtt libraries or writing my own prototype games using it.
The node-red [0] project is a great complement to MQTT use-cases. [1]
"Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click."
Node-RED is one of those polarizing apps. People either love it or hate it with a passion.
I am in the latter camp. I mean, it does what it says on the tin, but I could never understand how laboriously "wiring" up a bunch of boxes and filling in parameters is any easier than whipping up a few lines of Python.
I'm a big fan of NATS. It a small binary which can be scaled as much as you want and it replaces Kalfka, lightweight MQTT brokers, etc and has clients for most languages.
Anyone have experience scaling with more than 256 devices?
I use MQTTS at work and for my own projects. It is a very thin protocol (low code footprint), especially combined with Wi-Fi where those stacks typically eat up most of your MCU's flash if you're using a single-chip solution with TLS. I've used it on a max of 256 devices on a single BSSID, but I would be curious to know how it scales.
EDIT: Specifically, when all devices are subbed to a noisy channel. In some cases we had to stagger pub with random jitter to prevent both Wi-Fi and edge processing issues.
Perhaps someone here that's familiar with MQTT (and variants) can shed light on how to handle this case: an IoT device that periodically comes to life, queries the broker for the last known state of topic 'xyz', then sends a message regarding another topic 'abc' and goes back to sleep. i.e. it's not awake to receive updates to the 'xyz' topic, but needs to know the last update when it awakes.
With mqtt you may do that with retained messages.
The emitter of a message mark the message as to be retained.
The server will retain it.
Now when your mqtt client will subscribe tout the topic, the server will send the last retained message for this topic.
If you need a highly available broker you can consider https://github.com/streamnative/mop which adds a MQTT protocol handler to Apache Pulsar.
I recently contributed auth support and TLS support also just landed.
There is perhaps a few non-standards compliant things that need to be worked out but it's in fairly good shape.
I really like Bevywise MQTTRoute[1] as it is good one for home automation solutions.
It allows us to create our dashboard[2] specific to our use case with pre-built widgets. The easy & flexible integration with home assistant[3] and the mobile app[4] permits us to automate our home devices.
MQTT is awesome. In deepstream.io we bridged the node MQTT protocol to deepstream which allows us to mix and match protocols. MQQT for IoT devices (and to be honest any device considering the large amount of libraries) and the deepstream JS client for node/browsers.
This pretty much allowed clustering out of the box + automatically saves messages with quality assurance set in an actual database. Plus a bit more authentication and permission tools.
Deepstream itself is under maintenance right now, but I find the concept of mixing protocols together (similar to what ably.io provides) to hit a sweet spot for development across multiple different environments/devices
I've been wanting a GCM/FCM alternative, could mqtt be used for notifications like that? I understand its usually used for IOT devices and stuff so I assume its suitable for not using loads of power
I'm developing a mqtt client/broker and the protocol is all but simple to implement.
I didn't found any implementation, mosquitto included, that was really lossless.
+1 on paho-mqtt for python and mosquitto. All of my fun projects such as XBee3 sensors and HA + Z-Wave go through MQTT. It has been rock solid and a joy to develop against.
[+] [-] nathan_f77|4 years ago|reply
[1] https://tasmota.github.io/docs
[2] https://github.com/ndbroadbent/Tasmota/commit/bbcf57faffbf5b...
[3] https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/418
[+] [-] seidoger|4 years ago|reply
[+] [-] matttrotter|4 years ago|reply
[+] [-] nicolaslem|4 years ago|reply
Since I'm already running Mosquitto to interface with zwave devices, I decided to also use it to monitor my home lab. Servers push sensor statuses into Mosquitto and Home Assistant is used for the UI and alerting. That was a fun week-end project.
[+] [-] parineum|4 years ago|reply
And now, they've moved on from that to having straight websocket communication between those two programs, eliminating MQTT altogether.
I use MQTT for a lot of stuff but it adds a layer of delay between devices that is particularly noticeable with flipping a light switch in the Home Assistant web ui.
[+] [-] oynqr|4 years ago|reply
[+] [-] cptskippy|4 years ago|reply
I have yet to find a good resource for outlining autodiscovery. Most of the projects I have come across just hard code some pub paths in the homeassistant namespace.
[+] [-] SV_BubbleTime|4 years ago|reply
The problem I have with MQTT is also a feature. The payload format isn’t defined, there is no request response structure, encryption of payload requires custom encapsulation, etc. You are on your own for everything.
This is made better in MQTT v5, but AWS still doesn’t support that.
If I was starting over today, I really like OSCORE and CoAP derivatives for their completeness.
[+] [-] ralight|4 years ago|reply
Hello, I'm Roger, the Mosquitto project lead. Thanks for the kind comments, I'm happy to answer questions!
I work for cedalo.com developing Mosquitto and other MQTT projects. We offer Mosquitto based support, please get in touch if that's of interest to you.
[+] [-] ciberado|4 years ago|reply
[+] [-] zorr|4 years ago|reply
[+] [-] jvanderbot|4 years ago|reply
That kind of removed the need for a static broker entirely since now every node has a static IP on my wireguard network and are accessible no matter what network they are actually on. Pinging my mobile phone from an air quality sensor was an eye-opening experience.
[+] [-] 3np|4 years ago|reply
I started going down a similar path but a node that is acting as a gateway/bridge for UDP traffic routing to an outbound wireguard tunnel is now spending a constant ~20% CPU time on NET_RX softirqs with 10Mbps and 5~30Kpps. Almost no NET_TX.
It's specifically the combination of UDP-over-wg that seems to cause this. It's otherwise handling 1Gbps and way more packets just fine.
(I'm aware MQTT is TCP but hey, worth a shot)
[+] [-] kobalsky|4 years ago|reply
[+] [-] jve|4 years ago|reply
I'v hoped to find in your profile maybe more info/blog, but instead: > Group supervisor at NASA Jet Propulsion Laboratory working in autonomous vehicles and sensing for space exploration.
Oh well, a little out of scope and I see you blog about other things. Anyway - you mean I don't need static IP to form a network of distant IoT devices? This would be some really useful property.
[+] [-] joshspankit|4 years ago|reply
[+] [-] travisgriggs|4 years ago|reply
The protocol is simple enough, that when I wasn't happy with not many full solutions existing in iOS, I just rolled our own in a day or two.
Currently adding an Elixir REST bridge to the whole thing. It's been pretty cool all around.
[+] [-] BrandoElFollito|4 years ago|reply
Despite having casually developed for 25 years at that point, I found it amazing how much I learned quickly when setting up Home Assistant and pyscript (and obviously mqtt).
This drove me to use a bus (mqtt for instance) for non-home automation code just because it is so useful.
To friends who wonder how to help their kids get interested in coding (which is not easy at the beginning because converting farenheit to celsius gets boring after some time), I tell them to buy a few zigbee devices and have them automate stuff in pyscript. I am happy to see that some are on the way to graduate in CS :)
[+] [-] tombert|4 years ago|reply
If your frontend web project calls for any kind of messaging, I definitely recommend looking into trying MQTT before you jump straight into WebSockets. There's a good chance MQTT does what you need, scales better, can communicate over WebSockets, and will make your life easier .
[1] https://vernemq.com/
[+] [-] jvanderbot|4 years ago|reply
[+] [-] zerkten|4 years ago|reply
[+] [-] oneweekwonder|4 years ago|reply
As for comparing it to RabbitMQ you should compare the AMQP protocol vs MQTT protocol, it seems MQTT with its lightweight design "won" IoT market over and should be rather compared against protocols like CoAP.
As for guides I have always found resources rather scarce and mostly learned by adding features to mqtt libraries or writing my own prototype games using it.
[0]: https://www.rabbitmq.com/mqtt.html [1]: https://www.rabbitmq.com/web-mqtt.html
[+] [-] rhodin|4 years ago|reply
[0] https://www.cloudmqtt.com/plans.html
[+] [-] boilerupnc|4 years ago|reply
"Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click."
[0]: https://nodered.org/
[1]: https://cookbook.nodered.org/mqtt/connect-to-broker
[+] [-] deepspace|4 years ago|reply
I am in the latter camp. I mean, it does what it says on the tin, but I could never understand how laboriously "wiring" up a bunch of boxes and filling in parameters is any easier than whipping up a few lines of Python.
[+] [-] IceWreck|4 years ago|reply
[+] [-] SavantIdiot|4 years ago|reply
I use MQTTS at work and for my own projects. It is a very thin protocol (low code footprint), especially combined with Wi-Fi where those stacks typically eat up most of your MCU's flash if you're using a single-chip solution with TLS. I've used it on a max of 256 devices on a single BSSID, but I would be curious to know how it scales.
EDIT: Specifically, when all devices are subbed to a noisy channel. In some cases we had to stagger pub with random jitter to prevent both Wi-Fi and edge processing issues.
[+] [-] canada_dry|4 years ago|reply
[+] [-] Kuinox|4 years ago|reply
[+] [-] jpgvm|4 years ago|reply
I recently contributed auth support and TLS support also just landed. There is perhaps a few non-standards compliant things that need to be worked out but it's in fairly good shape.
[+] [-] Hema0909|4 years ago|reply
[1] https://www.bevywise.com/mqtt-broker/
[2] https://www.bevywise.com/mqtt-dashboard/
[3] https://www.bevywise.com/blog/mqtt-route-integration-with-ho...
[4] https://play.google.com/store/apps/details?id=com.Bevywise.m...
[+] [-] zwieback|4 years ago|reply
[+] [-] mleo|4 years ago|reply
I use it with Webhooks deployed to Internet VM that pass the data to broker installed on VM.
Another broker in home network connects to Internet broker and relays messages to local clients.
I don’t have to open any ports into my home network for webhooks to work with other services.
[+] [-] yasserf|4 years ago|reply
This pretty much allowed clustering out of the box + automatically saves messages with quality assurance set in an actual database. Plus a bit more authentication and permission tools.
https://deepstream.io/blog/20190909-release-deepstream-4.2/
Deepstream itself is under maintenance right now, but I find the concept of mixing protocols together (similar to what ably.io provides) to hit a sweet spot for development across multiple different environments/devices
[+] [-] ajfally|4 years ago|reply
[+] [-] Kuinox|4 years ago|reply
[+] [-] luma|4 years ago|reply
https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-qual...
[+] [-] RedShift1|4 years ago|reply
[+] [-] eska|4 years ago|reply
[+] [-] idatum|4 years ago|reply
[+] [-] kitd|4 years ago|reply
[+] [-] xnyan|4 years ago|reply
[+] [-] nicolaslem|4 years ago|reply