## Freesteel Blog » 2013 » June

Thursday, June 13th, 2013 at 8:13 pm - Machining

## Thinning sensor time sequences geometrically

Apparently the following terribly obvious idea isn’t terribly obvious, and everyone who has been collecting time series data from, say, temperature sensors has simply been stuffing them into their flavour-of-the-month database as, basically, rows of the form (time, value, sensor_id) in copious quantities with not much idea about what to do with it.

If you’ve spent your life generating machine tool paths and thinning out the points so that the controllers don’t get overloaded, the first thing you say is: “Hang on, why don’t you just fit some straight line segments to all this data?”

Our good friends at Open Energy Monitor immediately began running with the idea as soon as I mentioned it — and show every sign of comprehending the depth of the science behind the practice which I have unfortunately not had time to investigate in our machine tooling case.

Here is a section from a temperature time sequence that contains 10090 values collected over seven days:

Here’s what it looks like when you thin it to the tolerance of 0.25 degrees which results in 210 points — a compression factor of 50 times:

The important thing about these algorithms is to model the lines, not the points, because that’s what you will be integrating with. This means a record in the database is of the form:

`(time0, value0, time1, value1, tolerance, sensor_id)`

Now you might think this is unnecessary redundancy because each time and value is listed twice in the database, when you should be able to get away with simply listing the sample points once, but it’s actually the linear segments that define the curve. Databases are very poor at joining rows against sequentially subsequent rows to recover these records, so don’t try to do it that way. The tolerance value refers to how well the line segment from (time0, value0) to (time1, value1) fits the data.

The algorithm I use to do this thinning is pretty simple and recursive on a stack:

```segs = [ (t0, v0, t1, v1), (t1, v1, t2, v2), ... ]  # input
thinsegs = [ ]                                      # output

cstack = [ (0, len(segs)-1) ]
while cstack:
i0, i1 = cstack.pop()
di, dtol = GreatestDeviation(segs, i0, i1)

if dtol > maxtolerance:
cstack.append((di+1, i1))
cstack.append((i0, di))
else:
thinsegs.append((segs[i0][0], segs[i0][1], segs[i1][2], segs[i1][3], dtol))
```

All we do is fit a line to each sequence of segments, find the element of greatest deviation (on the (t1, v1) point), and split at that place if it exceeds the pre-set tolerance.

Doing it this way in a batch is quite a bit easier than attempting to thinning ongoing at load-time, but it is simple-minded because on an evenly curved path it will tend split to power of 2, which means it’s on average around 25% worse than optimal. And sometimes it’s possible to fit lines to the sequence whose endpoints are not on the sequence, but where the lines fit better. I’ve never done this; it’s hard, doesn’t gain much and may have disadvantages. But it is an important part of the science. One thing I do like about this algorithm is it preserves corners well (although there are some exceptions). I also think that the tolerance should be variable, so it can be tighter where the slope is long and shallow because the low frequency undulations on a long stable sequence could contain important data.

A notable observation of this curve is that it looks it contains exponential decay segments going up and down as the person in the room switched on and off a fan heater to keep warm. The decay curves going down are much clearer, and I wonder if they are all tending towards the ambient outside temperature at a rate given by the insulation rating of the room (very poor).

So we tried fitting exponential decay curves to this data, made possible with the addition of a further parameter in the segment record, called ex.

```    v = v0 + f (e-(t - t0) ex - 1)

where:
f = (v1 - v0) / (e-(t1 - t0) ex - 1.0)
```

This reduced the segment count to 161, which is promising. I can’t plot the curved segments using the annotated timeline, so they show up as straight lines below. However, I can plot the exponent values which show that the decay rate is pretty steady across the intervals whenever the heater is switched off:

So that’s the fun and games we can play with one sensor alone. Imagine what we can do when we have 100 sensors all in the same house and can relate the time displacements of the temperature changes between one sensor and the next.

Thursday, June 13th, 2013 at 8:03 pm - Kayak Dive

## A kayak diving, then a bit of underground and not much in the air

Half way through June and it feels like I’ve already blown this summer. Got to get busy.

On Saturday the 1st of June in the afternoon, I did a kayak dive on the SS Editor (wrecked in 1887) on tide rip rock at Penrhyn Mawr where the sea kayaks like to play. A bunch of them came through while I was thinking about diving, and I should have filmed them for context. Otherwise, it’s a more watchable vid than most and could be popular with those paddlers curious about what’s on the sea bed.

For me, this dive was a collector’s item. I hadn’t expected to kayak dive such an extreme place, but the psychological boat cover from our club diving the Missouri nearby gave me the impetus to go for it. Odd to note, my quivering fear of these actions seems to have vanished and I just got on with it. I don’t know why.

On Sunday 2nd of June a cave diver friend came out to Anglesey for a dive, having spent all Friday night in Large Pot on a trip where they grabbed 400m of new cave passage without surveying it. This was extremely rude and thoughtless, but we forgave him. Sea conditions were ideal, and we headed over to the wreck of the Hermine. I couldn’t recharge the gopro, so there’s no movie of the dive in which every living sea creature was encountered in its proper place, from the conger in its hole, to the crayfish on a ledge as though in an aquarium. Becka played surface cover in a boring sea kayak. We took a second dive out to 20m from the Hermine (having juggled diving bottles between us) hoping for a drift dive, but there was no current. There were, however, scallops. And that was the weekend, where I got 5 dives and Becka only got 2, but our visitor got 2 as well so it wasn’t completely wasted.

Becka spent the whole of the week agonizing over the extensions in Large Pot and couldn’t wait to get me underground at the weekend on Saturday 8 June when we would survey all the passages the first explorers had trampled through, but not drawn up. Her plan was that we would go down on two consecutive days because there was too much to survey in one trip. I knew this was implausible because (a) the length of the new discoveries was grossly over-estimated, and (b) I become very grumpy when I do more than 6 hours of caving per week — particularly in a very grubby wetsuit that needs to be patched after every trip. The mud in one of the crawls was exactly like Philadelphia cream cheeze — soft and very sticky. You won’t forget it. Hopefully the passage of lots of visitors will eventually smear it out of the way so it becomes a dim distant bad memory.

There’s not a lot of point taking a gopro underground in big passage, even with the brightest light I have. The Olympus camera that would have worked is terminally broken again after being taken on its second trip. This video shows us putting down conservation tape into the fresh passage so that cavers can keep to one well trampled path and avoid trashing 100% of the cave floor with their muddy boots.

On Sunday we stomped around the hillside looking for evidence of our discoveries on the surface (eg where this huge tunnel would have intersected with the valley) but drew a blank. Then stomped up Whernside looking for a place I could potentially fly my glider sometime when there is an east wind and I have energy for an insane carry up.

Finally, on Monday, I did actually fly my glider with almost disastrous style. I’m not pleased with it, so here is a link to the unlisted YouTube video. Fasten your seatbelts. It rapidly turns sheep shaped.

Tuesday, June 4th, 2013 at 6:48 am - Kayak Dive

## Anglesey Diving Again

Stunning weather. Astonishing visibility. We had the guts to go beyond the scary Penryn Mawr into the sheltered coves beyond and dive the wreck of the Kyle Firth just beyond.

This wreck is right about here:

View Larger Map
More vids to follow.