MQTT configuration for automatic polling messages (persistent)

[tnn85]

Hi,

I am trying to understand the concept of ebusd in regards to how data/messages can periodically be acquired. I already read the available documentation (Wiki) on Github. However, at some point the information given is not enough for me to fully understand how I could achieve my goal of monitoring certain messages / fields available for my heating system over time.

What I want to do is this:

1. Periodically acquire data from the Vaillant heating control / burner unit, mixer control via eBUS.
2. Make this data available via MQTT (in ebusd?)
3. Use Influx Telegraf + InfluxDB to store this data in a timeseries database
4. Create Dashboards with Grafana to visualize the data in a graph

Lets forget about 3)+4) because that is my problem to implement.

I have difficulties to understand how I am supposed to configure ebusd to read+poll and then send the data I am interested in every -for example- 5s via MQTT and how to make this configuration persistent in case of a power loss / restart (ebusd runs on a dedicated Raspberry Pi).

I have already achieved to read data (ebus messages) from the bus and publish them on MQTT.
However, before most messages are polled regularly I have to execute a "ebusctl read -pX -c CIRCUIT MESSAGENAME" (where X = 1..9) command on the local machine for each message I want to be polled
OR
send a specific "/get" topic suffix as well as "X?" as the value of an MQTT topic to the host ebusd is running on.
I just can't do this with my the backend (telegraf + influxdb) because it does not publish MQTT itself. I can only subscribe to topics.

AFAIK I can also download all necessary CSV message definition files for my system (Vaillant ecovit exclusiv VKK 226/4 + calorMATIC 470 VRC 470 + Mixer VR61) and change the first column to r1..r9 for every message I want to be read (&polled) automatically without further ado.

In this case I would have to modify the original message definition source files (CSV and INC) from and redirect the ebusd argument "--configpath=PATH" to the local path where I have saved the downloaded message definition files from https://github.com/john30/ebusd-configuration/tree/master/ebusd-2.1.x

Is this the way to go?

If this is the way:
I just didn't want to mess around in the original message definition files and would have preferred to use latest updated message definitions from https://cfg.ebusd.eu/ .
I thought maybe there is another way to configure ebusd to poll certain selected eBUS messages.

[ghhausi]

You may want to give the "--mqttinit=/etc/ebusd/mqtt-hassio.cfg" (and "--mqttretain") a chance ... or adapt the ".cfg" to your needs .... all the ebus readout parameters are going into mqtt .

[tnn85]

Thank you for your answer. In fact I have already taken a look at the MQTT integration configurations before. I just didn't know it would suit my needs because MQTT integration configurations are even harder to understand because the missing documentation and the comments in the file are a little confusing to me:

As it seems in the MQTT integration config the filter filter-seen = X is probably what I am looking for but it has 2 purposes:

1. If (filter-seen = 1): Messages that have not appeared on the bus because they are either not sent by other masters (QQ != ebusd) and are not broadcast messages and have no poll priority (r1-r9) set in the message definition files (= all messages when https://cfg.ebusd.eu/ is the selected source for message definition files) will be filtered. So basically no difference to if no MQTT integration is used.
2. If (filter-seen > 1): For all messages listed in the message definition files (which were chosen after the initial scan) the poll priority is set to 2 to 9 (depends on which value above 1 is set here ).

I set filter-seen = 2. Those messages are claimed to be polled with at least the poll priority of 2 (according to the information given in the comments of those MQTT integration config files).

What I have read in the other discussions/documentation poll priority 2 means those messages are only polled every second poll cycle?
The poll cycle interval can be set by the ebusd option --pollinterval=SEC in seconds, right? Default pollinteral is 5 seconds?.

I have set --pollinterval=1 (1 second) already. However, even if i only filter one message (filter-name = Flow1Sensor) with the MQTT integration configuration this message is sent with a frequency of approx. 0.1 msg/s (=10 seconds between subsequent messages) at most. 0.5 msg/s is expected for poll prio 2 -> 1 message every 2 seconds).
If i don't filter any messages with other filters at all the poll interval or lets better say the MQTT publish frequency is approx. 15 min for each message (in MQTT Explorer).
I assume this is because of the heavy load for a bus with only 2400 Baud rate ? Or what is the limiting factor here?
If I use the message definition modify method and set the messages I want to poll with r1 - r9 I have similar behavior.

Why is the poll priority value 1 not possible with automatic polling in the integration configuration? Wouldn't it be better to use a switch to enable polling for all messages and use a filter for messages that are not sent by ebusd? Maybe I have not understood the concept completely.

It also seems that the MQTT integration config doesn't stop broadcast or messages sent by other ebus masters (which appear on the bus without having a poll priority set for them) from being published as MQTT topics if those messages are filtered by filter-name = MESSAGE_NAME. Just like if I wouldn't use the MQTT integration config at all. I thought I could filter those messages with the config as well.

[VivLloyd]

Some clarification:
pollinterval [seconds] = frequency between each message read cycle in seconds. The default is once read/poll interval every 5 seconds.
priority [0,1-9] = message poll priority. Where 0 = don't poll. Messages with priority 1 are polled in each poll cycle, priority 2 messages only in every second poll cycle and so on.

Example (assuming pollinterval of 5 seconds):
If you had 100 messages each with a poll priority of 1, each message would be polled once every 500s.
If you had 100 messages each with a poll priority of 2, each message would be polled once every 1000s.

In summary, a message polling frequency is a function of the pollinterval, poll priority and total number of messages to be polled i.e. you cannot configure a fixed polling frequency for an individual message, but you can calculate it.

[toerni]

@tnn85
Did you eventually figure this out, since I am struggling with the same problem.
I want some specific sensors to be polled frequently and others should not be polled at all, since they just clutter the traffic

[VivLloyd]

You have a number of options.

1. Modify the configuration files by deleting all entries except the ones that you want. However, this means that you won't be able to perform ad-hoc reads/writes of other values as the messages won't be defined. Based on what remains you can tune the poll rate to meet your needs.
2. Modify the configuration files by setting the poll interval of each unrequired message to zero to disable polling. This way you can still perform ad-hoc reads/writes. Based on the an what remains enabled, you can tune the poll rate to meet your needs.
3. Leave configuration files unmodified and disable polling completely and reimplement polling in Node-RED. This gives you complete control of polling rates and is much simpler to modify/maintain. This is what I have done.

[tnn85]

@toerni
Not really. I think right now i am using these configuration options:
EBUSD_OPTS="--scanconfig -d /dev/ttyUSB0 -p 8888 --httpport=8889 --htmlpath=/var/ebusd/html --pollinterval=1 --accesslevel=* --configpath /etc/ebusd --mqttport=1883 --mqttjson=short

I am only interessted in a few messages (~20) anyway. I am not sure right now because I am not into the whole configuration of ebusd anymore (it was 1 year ago when i setup my configuration and it took a lot of time to understand everything).
I think i modified the configuration files for my needs (set the polling interval for single messages).
I didn't use the configuration for home assistant because it made it more difficult for the integration with InfluxDB / Telegraf.

Sorry I can't help you more.

[toerni]

@tnn85 tnx for the reply 🙏

[dietersiebs]

Modify the configuration files by setting the poll interval of each unrequired message to zero to disable polling. This way you can still
perform ad-hoc reads/writes. Based on the an what remains enabled, you can tune the poll rate to meet your needs.

How can I do this? With r0,,Name... it will nevertheless be polled.

I want to hold all findings in the csv files to see lesser not "decrypted" messages at the bus. But I want only some data...

[nightrope]

Thanks for your quick reply.
When I copy/paste, I'm getting an error in the import field:

SyntaxError: JSON.parse: expected ',' or '}' after property value in object at line 17 column 226 of the JSON data
= flow.get("messageTable)

Are you able to import it successfully from your post?

[nightrope]

Maybe pasting it using the code function in the reply box may help

[VivLloyd]

flows.json

Try this instead.

[VivLloyd]

Import as a file.

[nightrope]

great, thanks!

[Dobie007]

Hi, First of all big thanks to john30 for his work!
My solution to improve the poll rate is the following: (I use the mqtt explorer to see what messages are published and how often.)

To reduce the overhead:
Copy the mqtt-hassio.cfg and change some things using visual studio (its easier to see what is active):
filter-name = status|temp|humidity|count|power|runtime|hours|starts|mode|curve|^load$|^party$|sensor|speed
In this line i removed "timer" and some other texts and added "speed" because i don't use the timers and wanted the "FanSpeed" which was missing.
Next i changed this line:
filter-non-name = timer|Hc3|Hc2|Z2|Z3
so that i don't get any entities starting with timer and none from Heating circuit 2,3 and Zone 2,3 (not used in my setup).
Store the cfg file under a different name and change the reference in the ebus config file.
I have not understood much yet but these two helped to reduce the number of polled messages quite a lot.
I assume you can filter with more intelligence to refine things even further.

To me it is unclear what is meant with "circuit" and what with "fields". Those have separate filter settings that i haven't made use of yet..

[Christoph1111]

If you enable http support in ebusd config, you will get with host:8080/data a nice json response which you can watch e.g. in Firefox:

circuits are 700 and bai in this picture, rest is self explaining. I let Excel grab the json and convert the unixtime number to a readable date format.

[martin8310]

Hello together, How can I check if ebusd has found and correctly loaded the file /etc/ebusd/mqtt-hassio.cfg?" regards Martin

[Dobie007]

Use mqtt explorer and check what is available on your mqtt server. If it works you have lots of sensors and values visible.