Freesteel Blog » Do not do data logging

Do not do data logging

Tuesday, November 18th, 2014 at 5:54 pm Written by:

I have experienced much joy from this hardware hacking. I must have spent a couple hundred pounds on components. The bits arrive in little plastic trays like very expensive chocolate sweeties. There’s always a thrill when you first wire them up and they actually work perfectly. Not only that, you can have fun with them the next day and the day after that, because they have not turned into poop.

I have a few surpluses by now. I got a realtime clock which is 5V, and a microSD card reader which is 3V3; the Jeenodes run on 3.3V and the normal arduinos are 5V, so I can’t easily use either as the controller for datalogger. Some of the more idiot-proof breakout boards have converters on them, so they are safe for either voltage. Adrian has warned me to prepare for the coming of the 1.8V standard everywhere soon. I bought a combined ArduLog-RTC Data Logger, which for the moment is not playing ball.

Meanwhile, I’ve made a rule for the data logging of sensor data. Don’t do it. It’s not an end in itself. Too often people take on projects to collect sensor data and upload it to the internet (it’s Tuesday, so the site must be called Xively) with the idea that anyone else in the world could download it and [rolls eyes] “Do whatever they want with it.”

“Like what?”

“Whatever they want!”

If you can’t think of a single interesting application for your data, why do you think anyone else in the world will be able to? And even if there was anyone in the world who could do something with it, they’re probably the sort of person who’d have their own data which is guaranteed to be lot more interesting to them than yours. There’s a reason we don’t have a CCTV channel of someone else’s back door at night on cable TV.

I’ve formulated a stronger principle:

The value of sensor data is inversely proportional to the product of the time that has ellapsed since it was collected and the distance you are from the subject of the data.

Let’s take a simple case.

Patrick made CoffeeBot by putting the coffee machine onto an electronic weighing scale connected to an Arduino with an ethernet shield that talks to the internet.

He writes:

The optimum condition for using the CoffeeMonitor [a second arduino device that shows an arrow to the level of the coffee] would be out-of-sight of it, but still in the same building.

Clearly, the data about the level of the coffee in the pot at some moment 30 days ago is of no value.

Similarly, the data about the level of the coffee right now is of no value to someone in Manchester.

The blurb concludes:

While collection of the level of the coffee in the machine is an interesting exercise and it’s nice to know if there’s coffee available before getting up, there are some more practical uses once the data has been collected. Averaging the number of cups consumed during each day might give guidance on when to stop making coffee in the afternoon, and how many cups to make in the last pot. Tracking total cups made would mean that it’s possible to determine how regularly coffee beans need to be purchased so as to keep them available, but still fresh.

All very worthy, if over-optimized suggestions. So, how come when I get to the hackspace before everyone else in the morning and put the coffee on, the people who normally do it have to come in and ask me if I’ve put the coffee on just to make sure that they haven’t just poured themselves a cup that’s been baking on the hotplate overnight?

It seems that nobody has got round to writing the simple code that works out if the coffee has been on there for more than six hours and lights up a poop-coloured LED.

Data logging should be therefore be banned as a distraction.

My god, is there really that much noise in the coffee scale signal?

Data streaming, however, is essential for the development of algorithms that can turn raw sensor data into actually useful information.

So I deleted my sophisticated hour-by-hour file single day directory CSV data logging system whose files could be loaded into excel and badly graphed to make you go “Aw, that’s interesting”, and replaced it with a brutally simple sequential hex dump from the sensors to the SD card that is intended only to be parsed and replayed back into the system, either from the SD card or down the serial port — once I have worked out how to do either.

This will make it possible to systematically develop, debug and test how I am going to handle noise and detect signals relative to the various scatterbrained projects I have already engaged myself in.

a different patrick's desk

These are:

FridgePhoton (aka Fridgahedron) – Detect the parameters of the cooling and warming cycle of your fridge, and see how it varies throughout the day and night, and what are the effects on it from the outside temperature, the fullness of the fridge, and the cleaning of detritus from the rubbery seal that’s letting the cold air leak out, the marked effect of which you should be informed of in less than an hour so that you know that you earned that beer in the fridge that you’re going to have. My scientific hypothesis is that a full fridge will have a longer cycle because of the thermal mass. The market for this product is as an upgrade of the Fridgeezoo. Additionally, I contend that you’re not going to get retrofitted smart demand response fridges until the plug-in device that turns it on and off (probably by responding to signals transmitted over the mains A/C frequency) knows exactly what’s going on in the fridge so it doesn’t switch it off at the worst possible time and melts your freezer. It would help to have this intelligent toy in the fridge first, which is then able to communicate with and over-ride the dumb wall socket switching unit so it cuts the power only when it doesn’t matter to you but reduces a spike on the grid in an arbitrage deal. Smart arduino-based fridge controllers that are able to learn the thermal overshoot degree have been developed for the purpose of brewing beer.

Methane detector – This is a prototype for CO2 sensing which can control ventilation in Andrew’s air-tight house. I had hoped that I could track the transfer of air around my not-at-all air-tight house, but early experiments show that you have to hold it quite close to the gas cooker to get any reading. I do have a more sensitive analogue-to-digital converter than the arduino analog pin, so I am not done yet. I will not be satisfied until I have measured a positive concentration in at least one fart.

Total light switch duration – A device you can stash above the strip lights in Becka’s office building that logs their on-off periods over a week and prove to the facilities manager which ones have their motion sensors buggered because they’re always on even when no one is there.

PALvario – Following the discovery that gliding variometers (which detect the rate of ascent or descent) can be made with a very accurate altimeter chip (one of which I now have), I decided I would like to have a Peace-At-Last vario that does not bleep at you all the time while you are flying. My haptic vario is going to be connected to a vibration motor against my skin, so I can feel the information as to whether I am going up or down, leaving me to enjoy the sensation of the clear air rushing past my head.

Wing nerves – This is the daddy. A hang-glider is the ultimate in wearable tech. Birds have an advantage because they can feel the changing pressure on their living wings as they soar through the air. Can I install air pressure sensors along the top surface of my wings which will be able to detect changes in its flight characteristics at high speed or close to stall? Most electronic barometers are connected to the I2C bus, which has a range of about a metre. We need an oscilloscope to check if we have the right shielded cabling to make it go longer. There’s the issue of finding a way to attach it onto the surface of the wing, probably taping a wire back over the trailing edge and along to a box on the keel. The barometric device is said to be sensitive to sunlight. My first flight will merely be data logging to see if anything can be found in the data. Relating the data stream to a events on the flight (turn left, turn right, go slow) will require synchronizing it with the video.

The flight video, of course, is the ultimate in data logging, and I will continue to do this, so the title of the post is all wrong. I am going through rapid iterations and changing my mind in ways that would not be possible if I was instructing someone else to do the work for me. It would drive them nuts. It must be done myself.

Imagine in a larger organization where the inertia is fully formalized, and engineers (who are never allowed to hold any authority) have to tediously argue the case to a middle manager or other responsible adult to release the funds and employee time to work on a new idea. Then on day one it doesn’t look like it’s going to work so well, and you’re forced to either flog this dead horse for the duration, or attempt to undo all that work you just did with the middle managers building up the idea and persuade him that you want to do something else.

There’s no way you’re going to be bothered. They need to get this inactive talking shop idea clearing process out of the picture and instruct engineers to do different things all the time. The manager should say, Do whatever you want, but I’m going to come in once a week, and I want to see you doing something different each time I see you. Not the same project day after day. That’s the only way you are going to absorb the breadth of experience to start making stuff that puts together something new.


Meanwhile, last night’s home fridge trace: horizontal lines are degrees centigrade from zero at the bottom, vertical lines every hour, and the red spikes are the fridge light when I put the device inside in the evening and took it out in the morning. It’s working to a four hour cycle, and there’s a mysterious two slope step during the cooling section that lasts half an hour. There’s a large freezer compartment on the bottom for keeping a year’s supply of Becka’s breakfast blackcurrants.

I was going to spend some time running it through my peak detection algorithms, but now I’ve just wasted the whole afternoon on a blog.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <blockquote cite=""> <code> <em> <strong>