Freesteel Blog » Flightlogger

Wednesday, April 19th, 2017 at 11:41 pm - - Flightlogger

I’m not going to learn too much from myself, so I’ve got the raw data from Tim’s flight as measured by his Kobo/bluefly IGC logger.


As you can see, he ranged a good deal further than I did from Mam Tor, reaching an altitude of 1500m, and he reported scoring glide angles of 10 to 1 on his glider much-fancier-than-mine-of-which-I-am-now-jealous.

Let’s see if we can corroborate this from his data with the power of pandas and matplotlib.

pIGCtim = fd.LoadIGC("Tim-1492470000.igc")
pQ = pIGCtim.resample("1S").interpolate(method="time")  # fill uneven records

dstep = 30 # rolling windowed measurements in seconds
vario = (pQ.alt.diff(dstep)/dstep)  # m/s averaged over 30seconds
groundvelocity = numpy.sqrt(pQ.x.diff(dstep)**2 + pQ.y.diff(dstep)**2)/dstep

Here is vario:

Here is ground velocity:

The glide slope (normally called the glide-angle, even though we don’t quote it as an angle) is the groundvelocity divided by the descent rate. When the descent rate is close to zero, you get stupid numbers, so we need to throw those out and only quote for the parts where there is reasonably direct gliding flight.


Thursday, April 6th, 2017 at 11:22 am - - Flightlogger

On my last sorry flight I was carrying a total of four GPS units. One of them is a pile of poo. It’s the bluefly one I programmed to read at the frequency of 10 times a second, so greedy was I for precision and data.


Why do they have that option if it’s a complete waste of my time? All that work learning how to remotely step up its baudrate so that it could transmit data at this higher rate.

Here’s how the error comparisons line up, including the minus 3second correction on the 6030 vario’s timestamps. The “u” value is the number of milliseconds from midnight according to the GPS satellite timestamp.

gnames = [ "gps6030", "gpsxcsoar", "gpsTopShelf", "gpsBlueFly" ]
print("samples, gpsA, gpsB, stddev")
for i in range(len(g)):
    for j in range(i+1, len(g)):
        qA, qB = g[i], g[j]
        dt = 3000 if i == 0 else 0 # observed timeslip on 6030 unit
        dx = qA.x - numpy.interp(qA.u-dt, qB.u, qB.x)
        dy = qA.y - numpy.interp(qA.u-dt, qB.u, qB.y)
        k = (dx**2 + dy**2)[tf0:tf1]
        print(len(k), gnames[i], gnames[j], math.sqrt(k.mean()))
samples gpsA gpsB stddev
400 gps6030 gpsxcsoar 6.086
400 gps6030 gpsTopShelf 7.914
400 gps6030 gpsBlueFly 17.040
550 gpsxcsoar gpsTopShelf 7.845
550 gpsxcsoar gpsBlueFly 19.095
4120 gpsTopShelf gpsBlueFly 17.451

Now I have to program the BlueFly to sample at 200ms like the spare top-shelf GPS does, and see if that fixes it.

Meantime, has is that the accelerometer doing with this not-crap GPS?

Friday, March 24th, 2017 at 12:19 pm - - Flightlogger

We move on to the temperature sensor work, and the controversial concept that the temperature of the rising air in a thermal is hotter than the non-upwardly-mobile surrounding environmental atmosphere.

I say it’s controversial because the meaning of “the temperature” in relation to a mobile and turbulent airmass whose structure spans hundreds of metres in the vertical dimension and thousands of Pascals in relative pressure is undefined. Remember that the adiabatic temperature differential is about 0.7degrees per 100m change in altitude.

We do, however, have a single point temperature sensor on a mobile glider which is also progressing up and down the air column (depending on the wind currents and skill of the pilot). The location of the glider with the single temperature sensor is imperfectly known (due to bad GPS (see previous post) and inexplicable barometric behavior (see next post)), and the sensor itself has a thermal mass which means its readings have a delay half-life of about 9 seconds in flowing air (see this other post).

I have taken the precaution of jettisoning my slow, accurate and low resolution dallas temperature sensor for two humidity/temperature sensor combos and an infrared thermometer that has quite a good ambient temperature sensor within its metal can.

These sensors tracked one another pretty well, except for one small problem when I spotted a series of spikes in one and then the other humidity/temperature sensor.


What is going on?


Monday, March 20th, 2017 at 1:45 pm - - Flightlogger 2 Comments »

Last week I finally had my first flight of the year with my newly build flight data logger. I can’t believe the number of issues it’s already thrown up.

At least I may be making quick enough progress to get past the issues (rather than being swamped by them) using this exceptionally efficient Jupyter/Pandas technology.

For example, my code for parsing and loading the IGC file is 15 lines long.

The code for loading in my flight logger data into a timeseries is just as brief, if you consider each data type individually (there are more than 13 of them from humidity sensors to an orientation meter).

The GPS time series from my flight logger (at 10Hz) can be crudely converted it to XYs in metres, like so:

# pQ is the GPS position pandas.DataFrame
earthrad = 6378137
lng0, lat0 = pQ.iloc[0].lng, pq.iloc[0].lat
nyfac = 2*math.pi*earthrad/360
exfac = nyfac*math.cos(math.radians(lat0))
pQ["x"] = (pQ.lng - lng0)*exfac
pQ["y"] = ( - lat0)*nyfac
plt.plot(pQ.x, pQ.y)


Note the suspicious sharp turn near (-1000, -400). Here’s another sharp turn somewhere else in the sequence covering a 1 minute 5 second period using time slicing technology:

t0, t1 = Timestamp("2017-03-09 15:42:55"), Timestamp("2017-03-09 15:44:00")
q = fd.pQ[t0:t1]
plt.plot(q.x, q.y)


The dot is at the start point, time=t0.


Friday, March 3rd, 2017 at 11:04 am - - Flightlogger, Hang-glide, Uncategorized

To be clear, I haven’t got mathematical proofs here (I don’t have the time), but the experimental evidence is quick to get.

Take the differential barometer sensor (used to measure airspeed) of the hang-glider flight logger. The Arduino code which updates the reading every 200ms looks like this:

long lastpx4timestamp; 
void Flylogger::FetchPX4pitot()
    long mstamp = millis(); 
    if (mstamp >= lastpx4timestamp + 200) {
        sdlogger->logpitot(px4timestampset, px4pitot-rawpressure, px4pitot->rawtemp); 
        lastpx4timestamp = mstamp; 

Why did I choose 200 milliseconds? It sounded like a good number to read it at. This is a quick way to program it to be a regular reading.

A better way is to actually synchronize it with the clock divided rather than simply add 200ms to the next time, like so:

int mstampdivider = 20; 
int prevmstampdivided = 0; 
void loop()
    long mstampdivided = millis()/mstampdivider; 
    if (mstampdivided != prevmstampdivided) {
        prevmstampdivided = mstampdivided; 
        P(micros());  P(" ");  P(singlereading());  P("\n"); 

Now that code reads at 20ms rather than 200ms, but it prints a load of output which I can cut and paste into a file and read into pandas, like so:

rows = [ (int(s[0]), int(s[1]))  for s in (ln.split()  for ln in open("../logfiles/dmprapidtest.txt").readlines())  if len(s) == 2]
k = pandas.DataFrame.from_records(rows, columns=["t", "d"])

And then we can plot the autocorrelation (the covariance) with itself shifted in time, like so:

d = k.d   # just the measurement Series
dm = d.mean()
ss = [((d - dm)*(d.shift(i) - dm)).mean()  for i in range(400)]


Let’s zoom in on the first 50 covariances:

Wednesday, March 1st, 2017 at 7:55 pm - - Flightlogger

Lately I have become interested in noisy sensors; all sensors are noisy, so this is an applicable subject.

The interest comes from my attempts to read up on Kalman filters where the variance of the sensor measurements is a crucial input into calculation of the weight to apply to that measurement as well as the subsequent variance of the output.

I was wanting to do something more scientific with this noisy CO2 sensor, other than simply filtering it and saying to myself: “That looks like a nice clean line”.


At the same time, a minor brush with the concepts of statistical seasonal adjustment lead eventually to the autoregressive model and a way to test the quality of the noise — eg whether it is pure noise or the result of noise that has been filtered by a smoothing process.

For example, if you take pure white noise function X[*] with a standard deviation of 1 and pass it through the usual exponential decay filter:

Y[n] = Y[n-1]*f + X[n]*(1-f)

the standard deviation of Y[*] is


and the covariance between Y[*] and Y[*-1] (formed by averaging the product of the sequence with an offset of itself) is f times this value (whereas the covariance between the purely random sequence X[*] and X[*-1] is obviously zero).

In the absence of attendance at a university lecture course taught by a professor who has structured a procession of scientific concepts into a coherent order in which to learn over the course of a semester, I am happy to follow this journey where it takes me.

My glider data logger has two wind sensors now which, as with most sensor problems, disagree unpredictably. Airspeed is a tricky thing, due to local turbulence and large scale vortices which exist owing to the existence of a levitating wing.

One of the sensors is a pitot tube connected to a MS4525DO differential pressure device purporting to measure to 14 bits of precision as often as every 0.5milliseconds.

I’ve set up the pitot to read as fast as possible, about every millisecond, like so:

delayMicroseconds(500); // needs a delay of 500microseconds or the reading is marked stale

Wire.requestFrom(0x28, 4);
uint16_t pmsb =; // this reading takes 400microseconds whatever happens
uint16_t stat = (pmsb & 0xC0) >> 6; // 0 good, 2 stale, 3 fault
uint16_t plsb =;
uint16_t rawpressure = ((pmsb & 0x3F) << 8) | (plsb);

uint16_t tmsb =;
uint16_t tlsb =;
uint16_t rawtemperature = (tmsb << 3) | (tlsb >> 5);

Now immediately you’re going to have a problem because the measurements aren’t continuous; the values are quantized.



Friday, December 30th, 2016 at 9:44 pm - - Flightlogger 1 Comment »

Not been taking time to blog much. I have been lost in a vast pool of non-working projects, and trying to learn scipy and pandas and jupyter, which is an effective replacement for twistcodewiki, which itself was derived from Scraperwiki. The in-browser interactive programming environment Jupyter is everything that Scraperwiki was supposed to have become (eg see data journalism example). But that didn’t happen because every other software engineer I knew assured me in No Uncertain Terms that nobody was ever going suffer the sheer excruciating pain of coding in the browser — when real programmers all want to use vi and git.

But I digress to a world where practically all of my coding is in the browser and I’m using tools like matplotlib at a cycle time of around 10 seconds, which is about a quarter of the time it takes to lose any image file you have generated somewhere on your goddamn disk!

Today I put my new hang-glider data logger (which is held together with lots of glue) into the floor of the elevator and recorded the accelerometer in Z sampling at 100Hz.


With my new powers of filtering

b, a = scipy.signal.butter(3, 0.005, 'low')
pZ["fz"] = scipy.signal.lfilter(b, a,

we can get this:


The art of using sensors is to use a set of them in order to access the underlying reality.

Here is the raw barometer reading as the elevator goes up and down several floors.

Once again it’s a good idea to filter it, only this time we set the frequency of that butterworth filter to 0.01 instead of 0.005 because the barometer readings are at 50Hz, and we need the time delay from the filters to be comparable.


Because it’s nice and smooth after filtering, we can differentiate it by differencing it, using the pF[“df”] = pF.f.diff()*50/10. The times 50 is because it’s at 50Hz, and the divide by 10 is to convert from barometric readings to metres (approximately). The scale is metres per second, and metres per second squared. The elevator travels at about 1.5m/s. The section on the right with the green line lifted for 12 seconds was traveling down about 3 floors at once, so the air pressure was constantly rising.


Acceleration is the differential of the velocity, so setting ddf = df.diff()*50 gets us the following plot:

What we now see are just horrendous oscillations in the barometer. Every time you differentiate you magnify the noise.

But we can reduce the lowpass frequency of this filter by a factor of 5, so it’s 0.001 for the accelerometer and 0.002 for the barometer, and get a curve that agrees (where all the scaling seems to work out without any fudging).


The problem now is you can’t see the elevator moving between the different floors at all. What are those barometric oscillations from? Was I breathing too hard?

Let’s plot the barometer differences against the filtered barometer differences.

There’s a little too much noise, so filter the barometer difference by 0.1 instead and plot it against the 0.01 filter.


No, I can’t see those oscillations in the underlying data either.

To my eyes they only start to spike up with a filter low pass frequency of 0.03.


So I’m pretty stumped. But now I’ve got my eye on it, I can see a slight waviness at about this frequency in the original barometric reading in the third picture from the top, so it’s not an illusion or a creation of this filter.

If I hadn’t just glued my entire unit together I might have separated just this device out to see if it’s caused by yet another oscillation in the circuitry, like the kind I’d seen on a temperature sensor.

Maybe I’ll have to rerun this elevator experiment with all the other sensors turned off as much as possible to see if I can get the the bottom of this mysterious 4 second oscillation in the barometer.

Then there’s the temperature decay curves to sort out (from the use of a hairdryer this morning), which are not as good as I had hoped.

I am pleased to have finally implemented an AB Butterworth filter on my arduino.

It’s not a big deal. The code is roughly as follows, and you can get the a[] and b[] arrays from SciPy by executing the function scipy.signal.butter(3, freq, ‘low’):

class ABFilter
    float* a, *b, *xybuff; 
    int n, xybuffpos; 

ABFilter::ABFilter(const float* la, const float* lb) :
    a(la), b(lb), n(len(la)), xybuffpos(0) // len(a) == len(b)
{  xybuff = (float*)malloc(2*n*sizeof(float));  }

float ABFilter::filt(float x)
    xybuff[xybuffpos] = x; 
    int j = xybuffpos; 
    float y = 0; 
    for (int i = 0; i < n; i++) {
        y += xybuff[j]*b[i]; 
        if (i != 0)
            y -= xybuff[j+n]*a[i]; 
        if (j == 0)
            j = n; 
        j -= 1; 
    if (a[0] != 1)
        y /= a[0]; 
    xybuff[xybuffpos+n] = y; 
    xybuffpos += 1;
    if (xybuffpos == n)
        xybuffpos = 0; 
    return y; 

Saturday, September 24th, 2016 at 1:51 pm - - Flightlogger

I’ve been flying around with my data logger for two years now and I have only this week developed a theory of how to process the temperature data. I might have got to it sooner had I known enough physics or bothered to test the actual response time of my Dallas temperature sensor, which was so abysmal I had to start doing something about it. To be fair, it did take me over a year to even shelter it from the sunlight.

Part 1: The responsiveness of the temperature sensor is 0.09

The apparatus involved running a vacuum cleaner against the sensor housing to create a fast airflow while drawing the air either from the room or down a tube containing a bag of ice.


This was the graph it made, holding the intake near the ice and away from the ice for about 40seconds at a time

The curves you see are exponential decay curves, as the temperature sensor cools to the air cold temperature, then warms to the room air temperature, quickly at first, and then slowly as it converges.


Thursday, July 28th, 2016 at 11:25 am - - Flightlogger, Hang-glide 1 Comment »

I don’t know which direction to go now, so I did a quick bit of video editing of a flight in Italy. I got plans to write this story up in more detail if I can get over the writers block that’s making it difficult even to complete my logbook.

I am losing control over direction. I should be running the machine tool on something, to keep it oiled. But then there is the GroundWindow app that I’m converting to work in Yorkshire with the cave data we have there. And also analyzing my flight logger data, which I seem to put hundreds of lines of code into a week, but continues to get nowhere. It’s strange.

The WebGL situation with GroundWindow is diving into a real rabbit hole. I have long known I needed to learn how to code GPUs, but could never find a way in. GPU technology, as I have observed, makes much of the software engineering I have done over the years entirely redundant.

Last week I spotted that the genius who made the early webgl water demo I’ve been hawking around to prove my point finally written a blogpost about one small aspect of the demo.

In it he referred to another blog about the GPU graphics pipeline (written in 13 parts), that I am trying to work through. I’m getting about 50% of it at most.

The question then comes down to whether there are any meaningful new machining algorithms that can be built based on this technology, using OpenCL or whatever, because that’s one of the few things I am uniquely qualified to research better than most — even if I can’t persuade someone with a budget to pay for it.

That’s just normal stupidity and mis-allocation of capital by those in control of developmental budgets.

For example, let’s take the massive public engineering program called Trident just authorized by Parliament this month to build four nuclear powered submarines to provide that essential round-the-clock service of threatening indiscriminate annihilation to targets unknown at a few minutes notice.

Now some MPs believe that investing in high technology is good merely for the sake of it, like Toby Perkins, who said in the debate:

The most depressing exchange was with representatives of the GMB union in Barrow [where the submarines are built], when… [the MP] for Islington South and Finsbury suggested that they might like to make wind turbines instead. They politely but firmly informed her that they were involved in designing and producing one of the most complex pieces of technology on the face of the earth, and that wind turbines had already been invented.

Setting aside the fact that nuclear submarines have already been invented too, the difference is that wind turbines produce electricity, which has value. Military nuclear submarines, on the other hand, have no return on investment. They are not put for up sale as part of the international arms market to foreign dictators (and you won’t get away with selling used ones to Canada again). The subs are not applicable to a modern resource war, like the invasion of Iraq where the plan was to win the wealth back by stealing their oil, because the victims don’t have navies. And there is no program for technology transfer, given that the nuclear power industry has been entirely outsourced to France on a strategic level

In fact all the engineering being budgeted for this program is wasted and will be subtracted from the engineering brains available nationally, just when we need them most and the availability of immigrant engineers is going to be choked off.

Nuclear war, in terms of the way the politicians handle it, is worse than low-grade Science Fiction. So at this time I picked up the 1964 Heinlein post-apocalyptic novel Farnham’s Freehold, where an all-out nuclear war blasted the Goldwater republican right-wing Americans (with the same mind-set as the author) two thousand years into the future from their private fall-out shelter. Here’s one of the characters in the future civilization looking back at the recorded history trying to interpret the events:

The war itself he didn’t find hard to believe. He had experienced only a worm’s-eye view of the first hours, but what the scrolls related matched the possibilities: a missile-and-bomb holocaust that escalated in its first minutes into ‘brilliant first strike’ and ‘massive retaliation’ and smeared cities from Peking to Chicago, Toronto to Smolensk; fire storms that had done ten times the damage the bombs did; nerve gas and other poisons that had picked up where the fire left off; plagues that were incubating when the shocked survivors where picking themselves up and beginning to hope–plagues that were going strong when the fallout was no longer deadly.

Yes, he could believe that. The bright boys had made it possible, and the dull boys they worked for had not only never managed to make the possibility unlikely but had never really believed it when the bright boys delivered what the dull boys ordered.

Not, he reminded himself, that he had believed in ‘Better red than dead’–or believe in it now. The aggression had been one-sided as hell–and he did not regret a megaton of the ‘massive retaliation’. [Chapter 14 p190]

Two things: Being ‘red’ is actually a temporary phenomenon (unlike radioactive and dead). Just ask the East Germans.

Secondly, the Cold War was stoked and prolonged by the dull boys in America, from their endless lies about the missile gap, to their intrusive U2 surveillance flights across Soviet airspace that utterly wrecked the four powers peace summit that had been scheduled to de-escalate the Cold War in 1960.

Ironically, those U2 flights were collecting intelligence that proved there was no missile gap whatsoever, yet the President and Presidential candidates continued to lie about Soviet capabilities to paint their political opponents as “weak on defense” in the forthcoming election.

It’s the old game of elites clinging to power by scaring the bejeezus out of the public, and then offering dangerous answers that don’t work, and successfully displacing consideration of the real problems at hand that require solutions they don’t want anything to do with.

The problem with our thinking is that future exists only in the human mind, and we are not carefully discriminating between the challenges ahead that are entirely within the various states of mind, such as the threat of war and the causes and consequences of property distribution and financial debt– and challenges out there in the physical world that are not going respond to any of our puny beliefs, like climate change and the polluting energy systems in the modern world.

In a sane world the Committee on Climate Change would get the £200billion engineering budget to start building the stuff we need now, like tidal barrages and CCS, and the nuclear warriors would instead sit in smoky committee rooms writing strategic reports on paper and getting sent off to international conferences to sign treaties– in other words do the sorts of things that would solve those problems completely.

That’s the way round it should be. But it’s like we think we’re looking through a window on the future, and instead it’s just a mirror reflecting the past behind us. And this would be fine, if it weren’t for the point of reality that time does not in fact run backwards.

Friday, May 27th, 2016 at 12:07 pm - - Flightlogger

The datalogger is fitted to the right hand upright of the control frame. According to my shoddy measurements the upright is 176cm long and the basbar is 138cm long. There are two wires from the corners of the control frame to a point on the keel 151cm forward, each of length 206cm. And there are another two wires to a point on the keel 80cm backward, each of length 205cm.

This seems to add up to a vector of (69.0, 42.06344442205491, -156.2492943010685) from the top of the control frame to one corner of the base bar, which is the alignment of the datalogger and its BNO055 orientation sensor strapped to the upright.

I definitely don’t have time for a complete blog record of this, and a lot of it isn’t working, even though I’ve spent no end of time hacking on things, but if I plot the logger vectors that go backwards and perpendicular to the upright on the gps track, I get this picture:


For scale, the red lines are 11m long and the plot rate is 3x on the 100Hz samples, so there are 33 of those lines per 1second of motion.

It’s reasonably smooth and consistent, unlike the accelerometer values, which I think represent a lot of vibrations in the system, if the creaking wicker chair-like noise on the gopro soundtrack is anything to go by. It’s about right, if you think of the airspeed being in line with the keel of the glider (which it is probably not, because of the yawing and side-slipping in turns).

I nearly bought a sonic anemometer second hand off ebay last week, but missed it. I’ve got an alert watch on a new one and have designs on fitting one to the top of the kingpost where it is out of the way and least likely to get damaged by flying into a hedge or the ground.

Regarding the kinematics of the system, every part of the flight is the same as everything else, so I can strip out the simpler bits to see if there is a relationship. In particular, this is a plot that claims to be 10 second long sections of the flight where the bank angle is less than 3degrees.


This doesn’t fit too well, and I can explain it that the calibration value on the device was at 1 out of 3 for most of the flight, in spite of starting off on the ground at 3/3, so it’s telling me it got it wrong.

Anyway, it’s straight enough to move on to this complex plot of some of those flight sections:


Time goes from left to right of about 10 to 20 seconds. The following traces have been displaced close to the zero horizontal white line so you can see their correlations, if any. Green is pitch angle, Yellow is smoothed acceleration in forward direction (approx because it’s not aligned with gravity yet), Red is airspeed, and White is differentiation of windspeed.

What I would hope for is a delayed correlation between pitch (green) and airspeed (red). If you push the nose up and hold it there, the glider will slow down, and then fly at a slower speed. If you pull the bar in and hold it there, the glider will speed up and then fly at a constant higher speed. But of course you’re moving the bar all the time. But you can almost convince yourself of the response; the green line goes down, and the windspeed then starts to rise, and vice versa.

This should be built this into a kinematic model of control and response. It would actually also align with the barometric reading, where the sink rate goes up with a higher speed, so it’s quite 2 dimensional in the vertical plane, with energy equations.

The other correlation should be between the forward acceleration and the derivative of the airspeed. Both the airspeed signals and the acceleration signals are quite full of noise, so they needed to be smoothed (I think I’ve done this incompetently here, but that’s for another fortnight of coding).

Here you would expect a direct correlation in still air, because if the accelerometer says I am accelerating at 1metre per second per second, then over the course of 2 seconds my absolute velocity will increase by 2 metres per second. And, if the air is still, then my measured air speed should also increase by 2 metres per second.

So these two ought to be scaled the same. Except the smoothing flattens out all the humps and troughs, so you really can’t tell what the absolute value of anything is from this anymore.

Some of the bumps seem to line up, but this is not adequate for a kinematic model.

The numbers get hairier and hairier as I try to approach something useful. And that useful thing would be to separate the absolute motion into that which is due to my control and the flight characteristics of the glider, and that which is due to the underlying wind speed and direction.

Accordingly, I should be doing lots of boring flights in completely still air at the start of the day with a working logger where everything is calibrated, rather than having lots of fun in thermic air. This is a job to do in Austria where the hills are high and I should have hours and hours in the campsite not coding on other things.

This work just takes so much time and has hit so many technical failures already that I doubt it could be done by someone who is answerable to a boss.