Going Modular

One year monolithic

Last summer, I hinted that I was about to switch from the monolithic programme with all its threads to a constellation of separate processes.

The main reason was that the monolithic application needs a restart for any change in the configuration and that a crash/exception on one thread breaks everything.

True, having several separate processes impose to find a way to start them all in the first place and something else to look after them. It also uses quite a bit of memory (because of Python overhead, a tiny process is almost as memory hungry as a small one). Last but not least, the interprocess communication can be problematic.

Enters the Raspberry Pi 2

Fortunately, the Raspberry Pi Foundation released the Raspberry Pi 2 which has now a quad-core CPU, 1GB of memory (and even the Raspberry Pi 3, more recently, but I doubt it would make much difference here). At ~ 3MB of memory per process, there is plenty of available RAM! And also, starting a Python process is now almost immediate compared to the 5-10 seconds needed on the 1-B+ model.

Communication : MQTT

Thinking about it, there is not much need of communication between processes. In the majority of cases, it is all bout sending the data to the display interfaces and to a database (for the sensor part).

Note that so far, I have very little automation (besides a couple of sockets).

Anyway, the ubiquitous MQTT can solve all the communication problems... These days, it seems that there isn't a similar hub project around which isn't using MQTT either at the core or at least for plug-ins communication.

I have already detailed the way I format the topic and the payload of the messages.

Every process is now using a bootstrap library which manages the daemonisation, the MQTT communication, and logs. There are 2 types of messages : the data and metadata (starting, heartbeat, ...).

Current state

Currently the model used is the following:

MQTT Hub

Notes :

  • All measurements have the MQTT retain option activated to keep the last value available to a reconnecting process
  • Pushover is a notification system for mobiles (and desktop)
  • I am currently using 2 storage systems: RRD and a timeseries database (test in progress)
  • 'display_bikes' and 'display_transport' call external webservices and/or doing web scraping of pages. Resulting data is only displayed but never stored.