4.0.0

README.md

@@ -1,76 +1,134 @@
-# DSMR MQTT
-MQTT client for Dutch Smart Meter (DSMR) - "Slimme Meter". Written in Python 3.x
-
-Connect Smart Meter via a P1 USB cable to e.g. Raspberry Pi.
-
-Application will continuously read DSMR meter and parse telegrams.
-Parsed telegrams are send to MQTT broker.
-
-Includes Home Assistant MQTT Auto Discovery.
-
-In `dsmr50.py`, specify:
-* Which messages to be parsed
-* MQTT topics and tags
-* MQTT broadcast frequency
-* Auto discovery for Home Assistant
-
-A typical MQTT message broadcasted
-{"V1":226.0,"V2":232.0,"V3":229.0,"database":"dsmr","el_consumed":24488767.0,"el_returned":21190375.0,"p_consumed":1130.0,"p_generated":0.0,"serial":"33363137","timestamp":1642275125}
-
-A virtual DSMR parameter is implemented (el_consumed and el_returned, which is sum of tarif1 and tarif2 (nacht/low en day/normal tariff)) - as some have a dual tarif meter, while energy company administratively considers this as a mono tarif meter.
-
-## Preparation
-Test if P1 adapter is functional and providing dsmr data by running in a bash shell:
-* `tail -f /dev/ttyUSB0`
-
-## Usage:
-* Copy `systemd/dsmr-mqtt.service` to `/etc/systemd/system`
-* Adapt path in `dsmr-mqtt.service` to your install location (default: `/opt/iot/dsmr`)
-* Copy `config.rename.py` to `config.py` and adapt for your configuration (minimal: mqtt ip, username, password)
-* `sudo systemctl enable dsmr-mqtt`
-* `sudo systemctl start dsmr-mqtt`
-
-Use
-* http://mqtt-explorer.com/
-to test &  inspect MQTT messages
-
-A `test/dsmr.raw` simulation file is provided.
-Set `PRODUCTION = False` in `config.py` to use the simulation file. No P1/serial connection is required.
-
-## Requirements
-Install following python3 libraries
-* paho-mqtt
-* pyserial
-
-Tested under Linux; there is no reason why it does not work under Windows.
-Tested with DSMR v5.0 meter. For other DSMR versions, `dsmr50.py` needs to be adapted.
-For all SMR specs, see [netbeheer](https://www.netbeheernederland.nl/dossiers/slimme-meter-15/documenten)
-
-## Licence
-GPL v3
-
-## Versions
-3.0.0
-* Change HA Auto Discovery. Solve warning:
-Discovered entities with a name that starts with the device name
-This stops working in version 2024.2.0. Please address before upgrading.
-Credits: ricko1
-
-2.0.0 - 2.0.1
-* Updated mqtt library
-* Removed need for INFLUXDB label
-* Added telegraf-dsmr.conf
-* Added example (dsmr50_Stromnetz_Graz_Austria.py) for Austria dsmr (Stromnetz Graz, by karlkashofer)
-
-1.0.13
-* Add zero/non-zero check on data (as sometimes eg gas and power consumed values in influxdb became zero)
-
-1.0.12
-* Fix exit code (SUCCESS vs FAILURE)
-
-1.0.2 - 1.0.4:
-* Potential bug fix in parser
-* Add MQTT last will/testament
-
-1.0.0:
-* Initial release
+# DSMR MQTT
+MQTT client for Belgian and Dutch Smart Meter (DSMR) - "Slimme Meter". Written in Python 3.x
+
+Connect Smart Meter via a P1 USB cable to e.g. Raspberry Pi.
+
+Application will continuously read DSMR meter and parse telegrams.
+Parsed telegrams are send to MQTT broker.
+
+Includes Home Assistant MQTT Auto Discovery.
+
+In `dsmr50.py`, specify:
+* Which messages to be parsed
+* MQTT topics and tags
+* Auto discovery for Home Assistant
+
+A typical MQTT message broadcast looks like:
+```json
+{
+"V1":226.0,
+"V2":232.0,
+"V3":229.0,
+"database":"dsmr",
+"el_consumed":24488767.0,
+"el_returned":21190375.0,
+"p_consumed":1130.0,
+"p_generated":0.0,
+"serial":"33363137",
+"timestamp":1642275125
+}
+```
+
+A virtual DSMR parameter is implemented (el_consumed and el_returned, which is sum of tarif1 and tarif2 (night/low en day/normal tariff)) - as some have a dual tarif meter, while energy company administratively considers this as a mono tarif meter.
+
+## Requirements
+Install following python3 libraries
+* paho-mqtt
+* pyserial
+* persist-queue
+
+## Test the P1 adapter & USB connection:
+Test if P1 adapter is functional and is providing dsmr data by running in a bash shell:
+* `tail -f /dev/ttyUSB0`
+* optionally first execute `sudo chmod o+rw /dev/ttyUSB0`
+* Results should be similar to:
+   ```
+  tail -f /dev/ttyUSB0
+  
+  /Ene5\T210-D ESMR5.0
+
+  1-3:0.2.8(50)
+  0-0:1.0.0(240204171638W)
+  0-0:96.1.1(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)
+  1-0:1.8.1(024790.294*kWh)
+  1-0:1.8.2(011140.310*kWh)
+  1-0:2.8.1(010474.138*kWh)
+  1-0:2.8.2(025578.620*kWh)
+  .....
+  .....
+  ```
+
+## Installation & Configuration
+* Install Python3 packages per recommendation for your distro
+* Install from github and configure:
+  * mkdir & cd to your install location 
+  * git clone https://github.com/hansij66/dsmr2mqtt.git
+  * cd dsmr2mqtt/ 
+* In `systemd/dsmr-mqtt.service`:
+  * Adapt ExecStart under [Service] to ExecStart=/<your location>/dsmr-mqtt.py (default: `/opt/iot/dsmr`)
+* sudo cp -p systemd/dsmr-mqtt.service /etc/systemd/system
+* Copy `config.rename.py` to `config.py` and adapt for your configuration (minimal: mqtt ip, username, password)
+  * Edit the MQTT configuration and know that the MQTT_TOPIC_PREFIX = "dsmr" will show these messages as topic dsmr/. Configuration will be shown as homeassistant/sensor/dsmr/
+
+* `sudo systemctl enable dsmr-mqtt`
+* `sudo systemctl start dsmr-mqtt`
+* And check if it is running properly
+  * sudo systemctl status dsmr-mqtt
+    ```python
+    user@server:/opt/dsmr2mqtt $ sudo systemctl status dsmr-mqtt
+    dsmr-mqtt.service - P1 dsmr smartmeter
+     Loaded: loaded (/etc/systemd/system/dsmr-mqtt.service; enabled; vendor preset: enabled)
+     Active: active (running) since Tue 2023-09-26 22:21:06 CEST; 18h ago
+     Main PID: 1026 (dsmr-mqtt.py)
+      Tasks: 6 (limit: 1595)
+        CPU: 56.349s
+     CGroup: /system.slice/dsmr-mqtt.service
+             └─1026 /usr/bin/python3 /opt/dsmr2mqtt/dsmr-mqtt.py
+             ```
+Use
+* http://mqtt-explorer.com/
+to test &  inspect MQTT messages
+
+A `test/dsmr.raw` simulation file is provided.
+Set `PRODUCTION = False` in `config.py` to use the simulation file. No P1/serial connection is required.
+
+Tested under Debian/Raspbian.
+Tested with DSMR v5.0 meter in Netherlands and Belgium. For other DSMR versions, `dsmr50.py` needs to be adapted.
+For all SMR specs, see [netbeheer](https://www.netbeheernederland.nl/dossiers/slimme-meter-15/documenten)
+For Belgium/Fluvius, open your P1 port through [Fluvius portal](https://mijn.fluvius.be/): 
+
+For encrypted P1/dsmr, there is a [fork](https://github.com/wehrmannit/dsmr2mqtt) available.
+
+## Licence
+GPL v3
+
+## Versions
+4.0.0
+* Simplified design
+* Patches from javl
+* Some Patches from smartathome
+
+3.0.0
+* Change HA Auto Discovery. Solve warning:
+Discovered entities with a name that starts with the device name
+This stops working in version 2024.2.0. Please address before upgrading.
+Credits: ricko1
+
+2.0.0 - 2.0.1
+* Updated mqtt library
+* Removed need for INFLUXDB label
+* Added telegraf-dsmr.conf
+* Added example (dsmr50_Stromnetz_Graz_Austria.py) for Austria dsmr (Stromnetz Graz, by karlkashofer)
+
+1.0.13
+* Add zero/non-zero check on data (as sometimes eg gas and power consumed values in influxdb became zero)
+
+1.0.12
+* Fix exit code (SUCCESS vs FAILURE)
+
+1.0.2 - 1.0.4:
+* Potential bug fix in parser
+* Add MQTT last will/testament
+
+1.0.0:
+* Initial release

config.rename.py

@@ -1,59 +1,70 @@
-"""
-  Rename to config.py
-
-  Configure:
-  - MQTT client
-  - Home Assistant Discovery
-  - USB P1 serial port
-  - Debug level
-
-  Configure the DSMR messages in dsmr50.py
-
-"""
-
-# [ LOGLEVELS ]
-# DEBUG, INFO, WARNING, ERROR, CRITICAL
-loglevel = "INFO"
-
-# [ PRODUCTION ]
-# True if run in production
-# False when running in simulation
-PRODUCTION = True
-
-# File below is used when PRODUCTION is set to False
-# Simulation file can be created in bash/Linux:
-# tail -f /dev/ttyUSB0 > dsmr.raw (wait 10-15sec and hit ctrl-C)
-# (assuming that P1 USB is connected as ttyUSB0)
-# Add string "EOF" (without quotes) as last line
-SIMULATORFILE = "test/dsmr.raw"
-
-# [ MQTT Parameters ]
-# Using local dns names was not always reliable with PAHO
-MQTT_BROKER = "192.168.1.1"
-MQTT_PORT = 1883
-MQTT_CLIENT_UNIQ = 'mqtt-dsmr'
-MQTT_QOS = 1
-MQTT_USERNAME = "username"
-MQTT_PASSWORD = "secret"
-
-if PRODUCTION:
-  MQTT_TOPIC_PREFIX = "dsmr"
-else:
-  MQTT_TOPIC_PREFIX = "test_dsmr"
-  MQTT_CLIENT_UNIQ = 'mqtt-dsmr-test'
-
-# [ Home Assistant ]
-HA_DISCOVERY = True
-
-# Default is False, removes the auto config message when this program exits
-HA_DELETECONFIG = False
-
-# Discovery messages per hour
-# At start-up, always a discovery message is send
-# Default is 12 ==> 1 message every 5 minutes. If the MQTT broker is restarted
-# it can take up to 5 minutes before the dsmr device re-appears in HA
-HA_INTERVAL = 12
-
-# [ P1 USB serial ]
-ser_port = "/dev/ttyUSB0"
-ser_baudrate = 115200
+"""
+  Rename to config.py
+
+  Configure:
+  - MQTT client
+  - Home Assistant Discovery
+  - USB P1 serial port
+  - Debug level
+
+  Configure the DSMR messages in dsmr50.py
+
+"""
+
+# [ LOGLEVELS ]
+# DEBUG, INFO, WARNING, ERROR, CRITICAL
+loglevel = "INFO"
+
+# [ PRODUCTION ]
+# True if run in production
+# False when running in simulation
+PRODUCTION = True
+
+# File below is used when PRODUCTION is set to False
+# Simulation file can be created in bash/Linux:
+# tail -f /dev/ttyUSB0 > dsmr.raw (wait 10-15sec and hit ctrl-C)
+# (assuming that P1 USB is connected as ttyUSB0)
+# Add string "EOF" (without quotes) as last line
+SIMULATORFILE = "test/dsmr.raw"
+
+# [ MQTT Parameters ]
+# Using local dns names is not always reliable with PAHO
+MQTT_BROKER = "192.168.1.1"
+MQTT_PORT = 1883
+MQTT_CLIENT_UNIQ_ID = 'mqtt-dsmr'
+MQTT_QOS = 1
+MQTT_USERNAME = "username"
+MQTT_PASSWORD = "***********"
+
+# MAX number of MQTT messages per hour. Assumption is that incoming messages are evenly spread in time
+# EXAMPLE 1: 1 per hour, 12: every 5min, 60: every 1min, 720; every 5sec, 3600: every 1sec
+# Actual rate will never be higher than P1 dsmr messag rate
+# MQTT_MAXRATE = [1..3600]
+# MQTT_MAXRATE = 720
+MQTT_MAXRATE = 60
+
+if PRODUCTION:
+  MQTT_TOPIC_PREFIX = "dsmr"
+  MQTT_CLIENT_UNIQ = MQTT_CLIENT_UNIQ_ID
+  HA_ID = ""
+else:
+  MQTT_TOPIC_PREFIX = "test_dsmr"
+  MQTT_CLIENT_UNIQ = 'mqtt-dsmr-test'
+  HA_ID = "TEST"
+
+# [ Home Assistant ]
+HA_DISCOVERY = True
+
+# Default is False, removes the auto config message when this program exits
+HA_DELETECONFIG = True
+
+# Discovery messages per hour
+# At start-up, always a discovery message is send
+# Default is 12 ==> 1 message every 5 minutes. If the MQTT broker is restarted
+# it can take up to 5 minutes before the dsmr device re-appears in HA
+HA_DISCOVERY_RATE = 12
+
+# [ P1 USB serial ]
+# ser_port = "/dev/ttyUSB0"
+ser_port = "/dev/tty-dsmr"
+ser_baudrate = 115200

dsmr-mqtt.py

@@ -29,7 +29,7 @@
 
 """
 
-__version__ = "3.0.1"
+__version__ = "4.0.0"
 __author__  = "Hans IJntema"
 __license__ = "GPLv3"
 
@@ -137,17 +137,23 @@ def exit_gracefully(signal, stackframe):
 def main():
   logger.debug(">>")
 
+  logger.info(f"P1 serial port = {cfg.ser_port}")
+  logger.info(f"MQTT Max Rate = {cfg.MQTT_MAXRATE}")
+
   # Set last will/testament
   t_mqtt.will_set(cfg.MQTT_TOPIC_PREFIX + "/status", payload="offline", qos=cfg.MQTT_QOS, retain=True)
 
   # Start all threads
   t_mqtt.start()
+  # Introduce a small delay before starting the parsing, otherwise initial messages cannot be published
+  time.sleep(1)
   t_parse.start()
   t_discovery.start()
   t_serial.start()
 
   # Set status to online
   t_mqtt.set_status(cfg.MQTT_TOPIC_PREFIX + "/status", "online", retain=True)
+  logger.debug(f'Meter status set to online')
   t_mqtt.do_publish(cfg.MQTT_TOPIC_PREFIX + "/sw-version", f"main={__version__}; mqtt={mqtt.__version__}", retain=True)
 
   # block till t_serial stops receiving telegrams/exits
@@ -157,6 +163,7 @@ def main():
 
   # Set status to offline
   t_mqtt.set_status(cfg.MQTT_TOPIC_PREFIX + "/status", "offline", retain=True)
+  logger.debug(f'Meter status set to offline')
 
   # Todo check if MQTT queue is empty before setting stopper
   # Use a simple delay of 1sec before closing mqtt