Add support for external eventing (MQTT, webhooks, etc.)

[skjdghsdjgsdj]

When we use Baby Buddy, we interact with it a couple ways: the web interface and the API through various integrations (Android app, custom devices, etc.). What would be nice is support for webhooks in the API.

Use case: we log a feeding one of three ways:

• Directly in the web interface
• Via the Android app
• Using a custom-built ESP32 device that uses the REST API

But, regardless of how we log the feeding, we'd like to get a webhook callback to an arbitrary HTTP(S) URL. That would let us set up a timer (outside of Baby Buddy) that notifies us after a certain time interval like two hours "hey, it's been a while, a feeding time is coming up shortly."

That's a specific use case around feeding, but a generic publish/subscribe mechanism would be nice for things like feedings, changes, etc. Another random example: a webhook around changes could let us separately keep track of likely number of diapers used, and then we get notified "hey, you've logged n changes which probably means n diapers and you might be running low."

I searched the docs and existing issues and didn't see this implemented or requested yet.

[cdubz]

I’m surprised there isn’t a more mature Django package for this. I’m open to this but it will take some research and consideration.

[skjdghsdjgsdj]

A potential idea, and shoot it down as you feel: MQTT support?

For example, for a simplistic initial use case, define an MQTT broker in a config file. Then if any of the REST endpoints are hit, publish the HTTP verb, endpoint, and its payload to the broker. I have a Mosquitto instance available, and if Baby Buddy was just dumb--in a good way--and republishes the REST requests it gets to the MQTT broker, then another service can subscribe to the broker and take over from there.

This way, the solution is loosely coupled and not require Baby Buddy to actually send its own outbound HTTP messages to arbitrary endpoints, and MQTT is by its nature a very light protocol.

The MQTT payload could be as simple as a JSON struct like:

{
    "endpoint": "/api/changes",
    "verb": "POST",
    "payload": "(original payload)"
}

Logically, only non-idempotent would need to emit events. I can't think of a scenario where a GET would need to be relayed, but ¯\_(ツ)_/¯ .

Of course there could be information exposure, so after a basic implementation, maybe the MQTT relaying could be locked down to specific endpoints, users, etc., but it would be a nice starting point.

The de facto standard library for MQTT in Python seems to be https://pypi.org/project/paho-mqtt/ which I've used before and was pretty straightforward, at least as a subscriber

[cdubz]

Webhooks I have at least a passing familiarity with but MQTT is entirely new to me. This looks and sounds interesting but I'd probably have to lean on someone else contributing actual code here. As with webhooks -- I'm open to it, just can't say it's something I'll have the time to jump on. Though MQTT has piqued my curiosity for sure 😄

[skjdghsdjgsdj]

I may mess with it and see what’s possible. Doubtless whatever I contribute would need to be cleaned up by someone, but hey, that’s how this open source thing goes. In the meantime I’m going to donate because this software has been amazing for us and definitely helped improve our sanity taking care of our new baby!

[cdubz]

Appreciate the support! And happy to help review code and move the idea forward. Will be a good way to learn about MQTT myself, hah.

[skjdghsdjgsdj]

I made an initial proof-of-concept of this. Keep in mind I'm a hobbyist with Python and this is the first time I've touched Django, so I'm probably violating all kinds of best design principles and this isn't ready for an actual PR. But I welcome your comments.

tl;dr: I made DiaperChange extend a new intermediate BabyBuddyBaseModel class which is abstract and itself extends models.Model. I overrode save() to call the parent to persist the data to the database first so IDs get populated, and then if there's an MQTT configuration in production.py (example config added to production.example.py), then Baby Buddy will publish the model's data as a JSON payload.

skjdghsdjgsdj@cf46815

Screenshot showing it ending up in MQTT:

I'm using Mosquitto as an MQTT broker (server).

[cdubz]

@skjdghsdjgsdj this looks like a pretty interesting and straightforward start! Will you open a PR for the branch? You can set it as draft while we work through it.

For testing, it would also be helpful if you can give me some very basic instructions on how to set up the broker for testing.

Thanks for working on this!

[skjdghsdjgsdj]

I'll figure out the PR bit shortly. In the meantime, the high level approach you need to do is as follows. MQTT is a publish/subscribe mechanism. Clients publish messages to the MQTT "broker" (just a server) with a given topic and payload. Other clients can subscribe to that topic to listen for messages that got published. In this example, Baby Buddy publishes a message to the babybuddy topic with a JSON payload that contains the Django model's data.

• Install an MQTT broker like Mosquitto, like this tutorial shows for Linux. You don't need to do the TLS steps in that tutorial. On macOS, Mosquitto is available through Homebrew and I imagine the basic management commands are the same but with different config file paths.
• Add a block to production.py that looks like the one I added in my copy of production.example.py. You'll need to change the following, but port, topic, and client_id should all be okay just for testing:
  • hostname to something other than localhost if you're running Mosquitto on a different server than Baby Buddy
  • username and password, obviously, to what you defined when you set up Mosquitto
• Assuming you're using Linux or macOS, open a separate terminal running mosquitto_sub and listen to the topic babybuddy. On macOS, you can also use MQTTX which provides a more friendly UI for interacting with an MQTT broker.
• Launch Baby Buddy and try to save a diaper change. Hopefully, once the change gets saved, you should immediately see an MQTT message come through in the window running mosquitto_sub or in MQTTX.

My error handling is pretty bad so if it doesn't work, it'll be interesting at least.

My development environment involves running Baby Buddy in PyCharm with its built-in server. My MQTT broker is running on a separate Debian Linux server. Its /etc/mosquitto/mosquitto.conf looks like this:

pid_file /run/mosquitto/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d
password_file /etc/mosquitto/pwfile

listener 1883 192.168.1.193

This adds an optional dependency to paho-mqtt. I don't know how to set that up in requirements.txt or otherwise.

For future reference, AWS IoT Core offers an MQTT broker as well for people who don't want to self-host one.

[skjdghsdjgsdj]

PR: #894

[skjdghsdjgsdj]

As noted in #926: if this implementation included retained messages for most recent state (feedings, changes, etc.), then clients interested in real-time events via MQTT could receive the retained messages too instead of needing to first contact the REST API to get current state.