Last summer, I hinted that I was about to switch from the monolithic programme with all its threads to a constellation of separate processes.
The main reason was that the monolithic application needs a restart for any change in the configuration and that a crash/exception on one thread breaks everything.
True, having several separate processes impose to find a way to start them all in the first place and something else to look after them. It also uses quite a bit of memory (because of Python overhead, a tiny process is almost as memory hungry as a small one). Last but not least, the interprocess communication can be problematic.
Enters the Raspberry Pi 2
Fortunately, the Raspberry Pi Foundation released the Raspberry Pi 2 which has now a quad-core CPU, 1GB of memory (and even the Raspberry Pi 3, more recently, but I doubt it would make much difference here). At ~ 3MB of memory per process, there is plenty of available RAM! And also, starting a Python process is now almost immediate compared to the 5-10 seconds needed on the 1-B+ model.
Communication : MQTT
Thinking about it, there is not much need of communication between processes. In the majority of cases, it is all bout sending the data to the display interfaces and to a database (for the sensor part).
Note that so far, I have very little automation (besides a couple of sockets).
Anyway, the ubiquitous MQTT can solve all the communication problems... These days, it seems that there isn't a similar hub project around which isn't using MQTT either at the core or at least for plug-ins communication.
I have already detailed the way I format the topic and the payload of the messages.
Every process is now using a bootstrap library which manages the daemonisation, the MQTT communication, and logs. There are 2 types of messages : the data and metadata (starting, heartbeat, ...).
Current state
Currently the model used is the following:
Notes :
All measurements have the MQTT retain option activated to keep the last value available to a reconnecting process
Pushover is a notification system for mobiles (and desktop)
I am currently using 2 storage systems: RRD and a timeseries database (test in progress)
'display_bikes' and 'display_transport' call external webservices and/or doing web scraping of pages. Resulting data is only displayed but never stored.
Just last week, I was mentioning the existence of Minibian which is basically a Raspbian Jessie but with a minimum set of pre-installed packages.
Raspbian Jessie... Light
And a couple of days ago, the "Raspbian Jessie Light" was anounced! (I don't know if it is linked to the Raspberry Pi Zero release, but one can say that they were focused on reducing things this week!)
I haven't tried this new distribution yet but it can be downloaded on the official download page and more info about it is available on the forum.
For almost a year, I ran my Raspberry Pies with Arch Linux. The Raspberry needs a special armv6 or armv7 version but it is overall well supported.
The main avantages of Arch Linux are a very recent version of all the packages, a relatively small image footprint and a very optimised compilation of packages. For example it boots within seconds compared to sometimes almost a minute with Raspbian.
On the downside, it is important to keep the OS updated all the time because it is a rolling upgrade system.
But a few month ago, I wanted to add new package and I had to catch-up with the versions of the libraries. In theory, no problem with the command:
pacman-Syupackage_name
The upgrade went well without warning, I rebooted... and the system never ever came back...
Total crash! A first, impossible to boot. After a while, I managed to go the root prompt but hardly further. Knowing that I normally used my Raspberries headless, having to plug keyboard and screen really adds to the pain.
I figured out it would be easier to copy the data and to restart afresh.
Yet, a couple of weeks ago, same scenario... AGAIN!
Raspbian
This time I gave up! In the meantime, Raspbian Jessie was out. Trouble is, it is now a pinnacle of bloatware (at least to use as a pure server).
Anyway, while I was at it, I made the switch to a new OS on a new Raspberry Pi 2. I got stung by the new Device Trees Overlay system when I had to change the GPIO 1-Wire pin but at least I ended-up with a stable OS.
Minibian
The other day, I discovered a new-ish distribution called Minibian which is basically a minimal version of Raspbian.
It works very well indeed and was exactly what I was looking for. There are a couple of differences between Minibian and its huge brother but nothing bad. Video memory split is for example already set to the minimum and you connect straight to "root".
Transfering the image to the SD Card is business as usual:
When a Raspberry Pi -- which doesn't have a hardware clock -- boots, the time can be set thanks to NTP. It usually works well enough to be totally transparent.
Trouble usually starts if there is a power cut. By the time the Raspberry is up and running, the modem-router is usually still booting/synchronising with the ISP and no clock is set. Programmes start using the Epoch, i.e. 1st of January 1970!
The best hardware workaround is certainly to use an external clock like "real" computers. Maxim's DS1307 has been a reference for years but is both 5V and not reputed for its accuracy.
Raspberry Pi's I²C being in 3.3V, I tried to use a cheap RTC Clock using a DS3231 also from Maxim.
There are loads of webpages explaining how to setup Linux to use this hardware clock and if at first everything worked fine, I then started to notice some problems.
Symptoms
With Raspbian, the main problem was that after a while, I couldn't SSH to the server anymore. The Raspberry being headless, it wasn't easy to guess what was going on but it turned out this behaviour was the result of the disappearance of the root directory.
And this was itself caused by an I/O error on the SD Card. It seems that the 2 main reasons for this to happen are:
Problem with the power supply
Faulty (or incompatible) SD Card
The power supply was bought from RS with the first Raspberry and is rated 1.5A. Swapping with the second one (from Farnell this time) rated 2A didn't change anything. I then decided to use the other SD Card with a brand new Arch Linux on it.
At first, no failure but then 2 problems also showed up:
Loads of log lines related to mmcblk or
An extremely slow Raspberry Pi with an non-explicable high load (~ 6)
Solution?
I was a bit lost but I found a forum message which gave me a very important clue:
Second: disconnected the Dallas RTC DS3231, as it was pulling the 3.3v line too low, along with the DHT22 sensor.
Another test
Since I use a Level Shifter (3.3V/5V) on the I²C bus, it was very easy to move the DS3231 IC to the 5V side and try this theory. Unfortunately, this didn't make any difference.
Since I removed the module, the Raspberry Pi is no longer slow and load is now ~0.05.
Software workaround
With Arch Linux, I now force the application to start only when the network setup has completed (this includes proper NTP initialisation) by adding in the [Unit] section the following lines:
