Freesteel Blog » Whipping
For a number of years I have been familiar with the observation that the sophistication of, in particular, time series data analysis is adversely impacted by the use of the Excel spreadsheet program. More recently I have discovered exactly how it is an irreparably deficient application and I am convinced that its use should be abolished from all non-small business accounting applications (ie everything except what it was originally intended for).
Hitherto I did not attach much importance to this view, owing to the fact that it is considered an anti-Microsoft bias as well as a “lost cause” because “everyone uses it”. However, on learning the existence of a large body of signal processing theory which is all but inaccessible to users of Excel due to its fundamental nature, I submit my observations for consideration below.
My first remark is that if data scientists don’t know about the benefits and substantial applications of multi-source data combinations, Kalman filters and seasonal adjustments reliant on the autoregressive-moving-average model, they are missing an important part of their job and are deferring the implementations of these concepts to mere “estimation by eye” from the graphs.
My second remark is that when external software exists that can be used to, say, calculate and compensate for the seasonal adjustment, it generally requires the data to be submitted in a time series format, and this requires a great deal of preparation of the spreadsheet data. Thus the appearance of being able to open up and immediately (and supposedly do) work with the spreadsheet data within seconds is deceptive, because there is now a longer route for the data to move it back out in a form to be processed and re-imported back into the spreadsheet for familiar viewing.
Let us consider a couple of time-series data sets. For example, the monthly national GDP and the employment statistics, or imagine one minute intervals of temperature and electricity use in a building.
What elements of the data are required to perform any useful processes on it, beyond simply the production of visual graphs?
For time series data (which a great proportion of data can be described as being), the existence of a reliable datetime value is paramount. Excel may in theory have a datetime cell type, but it is not visibly distinguishable from an unstructured string type with its ambiguous American and English date orderings. As such, it cannot be consistently used because improper use does generate an unignorable error (eg anything in column A must be in this datetime form or you can’t save the file).
Furthermore, just the datetime is not enough, because there are time series intervals (for example, monthly or quarterly data) and these cannot always be approximated by a single time point. By convention quarterly intervals can either be represented by the end of the quarter (eg sales results) or the start of the quarter (eg sales targets) but both need to be boxed into the same entity in any subsequent regression model.
Finally, when you have different data series from different data sources they usually work to different sample rates, so you cannot represent them adequately as a single row per time sample. This would apply to the power use for the heating system which is provided every minute, when the average outdoor temperature is recorded daily.
Accordingly, the primary dimension of the data points, the datetimes, are problematical. But what of the data point values, the measured quantities? If they are each recorded into a single spreadsheet cell we will invariably be lacking an associated standard deviation/confidence interval for them. The standard deviation is an crucial input to the Kalman filter for the determination of the weight applied to each individual measure.
Take the example of the monthly political polling data prior to an election. These are often supplied by different agencies and almost always come with a confidence interval that depends on the size of sample so we know to take less notice of a poll which defies the steady trend when it has a wide margin of error. But then if there are more polls with the same wide margin of error that are also in line with that new trend, the best guess of the trend will be pulled in the new direction as much as it would have been by one very good accurate poll with a narrow margin of error. This balancing of the estimations from the aggregation of the location and accuracy of the measures is optimized by the Kalman filter, and should not be done by eye from the charts themselves merely because it can’t easily be applied in Excel and we’re too lazy to convert our working platform to something where it could have been easily applied.
And this brings me to the final point about Excel, which apparently can do anything because it can run programmed macros. Really? Who can honestly think, if they have every stopped to consider it, that it is a good idea to co-mingle software with data? You might as well nail your vinyl record onto the record player and then parcel-tape it into a heavy cardboard box to prevent interchangeability.
The co-mingling of data and code with no reliable means of disconnection leads to dangerous and ridiculous practices, such as copying the data into and out of the spreadsheet by hand just to access the service of the macros.
Come on folks. If you’re going to call yourself data scientists, you cannot rely on a single tool that prevents you from applying the last fifty years of data science know-how optimally and productively — and then rely on its inadequacy as an excuse to not challenge yourself to learn and master the amazing mathematical technology that would have been at your disposal had you not chosen ignorance over education.
We have got to get beyond the pointless issue of formatting data into “machine readable form” for the mere purpose of making graphs for visual titillation, and get to grips with actual intelligence and effective control theory.
There is, for example, nothing smart about having to control a car with a human looking out through a window for white lines on the tarmac and the traffic lights on the verge in order to move the steering wheel and pedals in response. Smart is getting the data to feed-back directly into a computer that controls these motions optimally while you sit back in awe having merely specified the destination. But if someone out there building the tech has dared to embed a copy of Excel within the process chain between the sensor acquisition and the motor control actuators, then we are doomed.
Thanks to the famous Go to considered harmful letter of 1968. There was a heated debate about it, and 30 years later it was unconscionable that programming languages could even be conceived of to include a goto statement. Kids these days probably don’t even know what one is.
But just think about all that programming wasted and how much further on we could have been without the inclusion of that single statement which caused so much unnecessary expense and buggy code throughout the years, and then imagine how much damage is being caused by inappropriate use of this inadequate data non-analysis tool up to now and for the next 20 years before it too finally gets buried in the ash-can of history and people don’t even remember that we ever used it in the first place.
This has been the moment of truth.
Well, it happened. Donald Trump got elected. And into the hard vacuum of his political philosophy, he’s sucked in all kinds of plague carrying rats.
Rats like Myron Ebell.
People ask me if I am going to restart my proxy blog for Ebell, which I wrote from 2004 to 2011, called The Myron Ebell Climate chronicling his part in the suicide of the human species.
But there’s no point.
Not only is it too late, but everyone else is covering this walking cancer now.
Where were you folks when it mattered? Back in 2004 I said we should run a proxy blog for each one of these think-tank bastards and form a shadow network for this corporate funded disinformation infrastructure. It was required to correct the reputations of these monsters who were constantly popping up on the TV and in the newspapers carrying out lies and damaging all of society. It would have been easy. They’re slippery pricks, but one could systematically keep their record alive to make sure some of it sticks.
Friday, June 17th, 2016 at 12:03 pm - Whipping
I was quickly trapped in a dark room full of suits as the Government man painstakingly powerpointed his way through the process by which they were going to hand over several millions of pounds to the sorts of respectable businessmen who ultimately don’t give a damn if the technology can be made to work productively or not, because the underlying purpose of their job titles is to extract financial profits for themselves as the primary objective.
That is the very definition of a respectable businessman these days. It’s not ever defined as someone with the character to manage and inspire the teams of technologists who would have the capability to deliver the technology, or a preparedness to use finance in the service of these ends rather than as an excuse to impose inefficient and ineffective research and development methods onto any program they control.
Back in the old days when the Government wanted some new technology developed, like nuclear weapons, space travel or machine tools, people sought out and spoke to the engineers who were ultimately going to have to do the job, and then designed programs around the necessity of organizing these engineers to get it done as efficiently as possible.
If their one idea is that people like me have to be subserviently employed under a contract to deliver closed source software for the know-nothing suits they’ve deliberately empowered to curate this technology by virtue of proving them the grants, then it’s going to be a disappointment.
I walked out to the main hall where FW de Klerk was being introduced prior to his speech.
Tuesday, May 17th, 2016 at 4:22 pm - Whipping
I met my first Leave Europe supporter yesterday, but didn’t have enough time to quiz them. But they referred me to Boris Johnson’s speech in Manchester as an explanation.
I listened to it all.
Boris banged on about those nasty EU regulations infiltrating every part of the nation:
“They can’t tell us what sort of trains we can run, can they?
“Oh yes they can!
“Oh yes they can!
“The EU Commission told us that by 10 Nov 2018 we must create a rail freight corridor to Glasgow and Felixstowe, which means that Network Rail can be legally obliged to accept rail freight trains in place of passenger trains.
“Of course our excellent transport minister spotted this insanity, with the west coast main line full to capacity. If we had more freight trains, fewer passenger trains, there would me more overcrowding and higher fares. So he wrote a fierce letter to the Commission complaining that they were circumventing requirements, bending the rules, and of competence creep aka sticking their nose into something that wasn’t their business. But the Commission told him to ‘go and jump in a lake.’
“So we took them to the European Court of Justice, and what did the Court of Justice say? They told us ‘allez vous plonger dans un lac.’ They ruled on that occasion as they have done in 80 per cent of the cases in which Britain has been involved – they rules against us.”
Now these are the days when the world is interconnected, and we have a Channel Tunnel, and roads that are utterly chockablock with a hundred thousand fat diesel trucks that can freely drive onto the motorways in enormous numbers.
In the meantime it’s practically impossible to arrange for a zero-carbon electric train to haul heavy steel products direct from the steel mills on the Clyde to the construction sites in Barcelona because of the amazing national railway bureaucracies along the way.
It takes years of painstaking systematic work to identify and address these problems, like those articulated in a Select Committee report from 2005:
Many of our witnesses told us that getting rail freight through France was very difficult. The significance of this problem is greater because France’s geographical position means that international rail freight to and from Great Britain and the Iberian Peninsula travels through France. We were therefore encouraged by Mr Hilbrecht’s confirmation that Europorte 2, a subsidiary of Eurotunnel, had received a licence and a safety certificate to operate in France…
The French two-part tariff system was also said to be a particular barrier to open access and fair competition within France. Mr Hilbrecht was happy “to say that we have achieved agreement with France . . . . They agree that it (the two-part tariff system) should be changed”. Unfortunately the French government claim that because of the public service contracts with regions they cannot do so before 1 January 2006. This two-part tariff system needs to be abolished. We hope that the Commission will ensure that the French government abolish it as soon as possible.
The last problem that we identified is the least tangible, but is nevertheless an important challenge facing the rail freight industry. The evidence we received led us to believe that the rail industry in general, and in particular the rail infrastructure managers, have inadequate incentives to win new traffic. We recognise that, for political reasons, rail passengers are given priority over the movement of rail freight. This appears to have resulted in an institutional framework within the rail industry in which there is little incentive to increase and improve rail freight.
Whatever the cause of this lack of commercialism and competitive performance, it has to be overcome if the rail freight industry is to revive and achieve its potential.
But Boris doesn’t give a toss about this if he can make people laugh at his stupid jokes.
He’ll say all EU directives and ECJ rulings are about Brussels bureaucrats meddling in everything with their crappy regulations, when these ones are about unpicking the thicket of stultifying regulations and prohibitive monopoly rules that make it impossible run trains across national borders.
You have to pass laws to repeal laws, and fight hard to get rid of bureaucracy.
Anti-democratic? Never mind that each one of these directives is examined, amended and passed by a European Parliament that gets elected every five years where every vote counts equally, so that when the Green Party gets 6.7% of the votes, it gets 6.7% of the seats — unlike in the utterly screwed up system in the UK that gives disproportionate representative power to tosswits like Boris to tell us like it isn’t.
Yes, I mentioned that steel industry, the jewel in the crown of Britain’s awe-inspiring industrial revolution of the 19th century. You remember how a private corporation in one of our former British empire colonies bought it in its entirety for small change ten years ago and then shut the whole thing down last week? How humiliating is that?
So much for the Commonwealth Dream, eh? I don’t see Britain owning any comparable assets in foreign lands, other than a few seedy tax havens.
This should have been a seminal moment in our nation’s self-image.
But it isn’t because our political feelings have become pathologically detached from reality.
Apparently the EU tried to save the steel industry by putting tariffs on Chinese impots, but this effort was blocked by the UK government because our policy is to toady up to everything the Chinese want no matter what the cost.
That’s because the country that built the first commercial nuclear power station in the world in 1956 needs that Chinese finance to pay French engineers to consider building a new one — at a cost not quite as high as the International Space Station.
It’s not going to fly. And it doesn’t include the cost of taking it down, let alone in time for the sea levels to rise and wash the resulting radioactive sludge inland to Bristol and all along the North Devon coast with the tides.
Isn’t it curious the highest profile proponents of the Britain to Leave Europe are also climate change denialists? Once you have one delusion of supremacy, it’s easy to get more.
The UK is not a typical country in Europe. If we were more typical I would have more patience with those who suggest that we could leave and our lives would improve. But in many ways we’re a poorly performing affluent country. This poor performance has little to do with the EU, and a lot to do with us, and our legacy of having had an Empire. From the Suez crisis right through to the Panama papers, there’s a series of embarrassments that have occurred and, in a way, this referendum is just another one of those embarrassments.
Some people have a fantasy (enjoyed by the majority of the Brexit group, particularly the Cabinet ministers) that if we were to leave we would become ‘Great’ again. We could become the richest country in the world again, and our EU membership is why we are not ‘Great’.
And, so, because a lot of sensible people are sick and tired of arguing with these total idiots, we feel like letting them have what they desire, and getting us out of Europe. And then the Europeans are not going to help us with driving our trains, cars, people, money, goods or anything else across our the borders and into their lands. Things will pretty quickly stop working and go south.
And these politicians who sold us this knackered bill of goods will absolutely own it. We’re going to have to finally learn the hard way how Not-Great we are. It’ll be good in the long run — if there was a long run — for us to have a more realistic perspective about who we are. After all, losing their big wars seems to have done a power of good to the standard of living of the people in Germany and Japan.
In the end none of this matters one little bit. We are arguing about these petty matters when we should be freaked out by the fact that the near-term human extinction has become inevitable.
At some point everyone will learn that we are nothing more than ten billion monkeys farting around on one lonely planet with made-up borders. We’ve used our big brains just enough to fuck things up royally, but we just cannot be arsed to turn it up another notch to avert disaster, can we?
If voting changed anything, they’d make it illegal — Emma Goldman.
So, if you want people to vote, they have to believe that it can change something.
The Labour Party is undergoing a sudden and spectacular revolution with hundreds of thousands of people signing up on the belief their vote will make a difference when they elect Jeremy Corbyn. No one saw this coming.
Just one month ago the former leader Tony Blair said that anyone who supported Corbyn should get a heart transplant.
Funnily enough, Blair only became party leader (and, by default, Prime Minister) because John Smith had a heart attack and died. Blair was then stupid enough to believe that he was there because of his awesomely crappy policies that caused so many people to quit the Labour Party he had to fund his 2005 election by selling seats in the House of Lords.
Voting in Scotland in a referendum was going to make a difference, and the turn-out there was massive.
But in the wider country there continues to be a problem with General Election where necessary change is not coming about and people are getting screwed.
Young people don’t vote because they know it doesn’t make a difference. The system is too skewed. The old people in the rural constituencies reliably root for the Tories and provide their base. The Tories return the favour by redistributing the wealth from the youth to their elders on a massive scale through rising house prices, tuition fees (after this older generation got educated for free), historically low wages, a rising retirement age, a declining pension (which doesn’t effect the current generation of pensioners), expensive public transport while car driving becomes cheaper, cuts in inheritance tax (how old are the “kids” when they actually get the money?), and huge bank bailouts to protect the savings of those with hundreds of thousands of pounds on deposit.
Thursday, April 30th, 2015 at 10:13 am - Whipping
I’ve not been doing as much as I should regarding this General Election. A few leaflet rounds, one canvassing session. After attempting (and failing) to contribute code to the Election Leaflet website, I’ve been handed the job of reading through hundreds of election leaflets each morning to look for anything interesting, which I report by entering it into a google excel spreadsheet. Urgh. But it’s my duty. Takes hours, and I’m going crazy with it.
Top issues are: NHS more funds, HS2 abolished, Green belt protected, increasing recycling, cutting carbon use, and opposing those ineffective flickering noisy windfarms that clutter up the countryside when we need more flood defences that aren’t going to work due to rising sea levels, you dumb-dumbs.
Basically, this election should be cancelled for lack of interest. I’ve driven from one end of the country to the other, from Land’s End to Liverpool, then to Newcastle and back to Liverpool, and there are approximately zero election posters of any kind (plus or minus less than 5) in gardens, on walls and billboards. Even the news media is bored to the extent that it barely makes it into the first half of the news hour each day. There is nothing to say.
Now I’m going camping in a field in Southeast Wales to get humiliated and intimidated at a HG competition for the next few days so I’ll miss whatever comes about internetwise. Be back on Wednesday night in time for the 5am leaflet drop on election day and the count (unless I can avoid it). The real fact is that it’s only the votes that count on the day. Nothing else matters.
I’ve washed up on the annual Easter university diving trip, though my heart’s not in it. There’s a long period of stable weather forecasted, which should mean the silt will have time to settle out of the water ready for when the novices to get good enough to come out to more exciting locations.
snakelocks anemone encrusted wreckage in Sennen Cove
It’s a bit of a rehash: I’ve done them all before in previous years in better conditions, with Becka by kayak back in 2010. I’m too tired at the end of the day to do any of the hacking I’d hoped for, so I’m marking time. Maybe I should go to the pub more often and not try to make best use of my time all the time.
Curiously, that last time in Cornwall (but one) also coincided with a General Election campaign, and I remember a big Conservative Party poster in a farmer’s field at the end of the lane. There isn’t one there this year. Either the land-owner is not so keen on Cameron this time, or he can’t be bothered, or he’s sold up to a new owner, or who knows? It’s another metric that could have been noted and cross-correlated over the years if we really had the data. For the life of me, I don’t know why these posters never became a substrate for some time-limited concentrated geocaching game. Geocaching happens on a lot sillier things, and this could have been like tracking down sightings of rare wild animals.
Fish approach between the boulders and kelp
Watching them discuss stuff I realize I’m totally lost in the last century in terms of the technology. It’s a full time job just keeping up. (And in the large software company I briefly worked for, nobody seemed to be employed to keep up, so they didn’t.) Nowadays I don’t know much more than the difference between JPEGs and PNGs.
We are using the RabbitMQ messaging system, our queue server is run by CloudAMPQ (Big Bunny instance, dedicated server)…
Our worker servers also live behind an ELB but don’t have auto-scaling enabled; we manually manage the amount of instances based on the size of our queues, we can check using the RabbitMQ management console…
All of our MySQL queries are handled by the Doctrine ORM and written using the Doctrine QueryBuilder. These doctrine queries are also cached in Redis as SQL…
Our application is based on Symfony 2.6.* standard edition.
For Redis we use the SncRedisBundle. For RabbitMQ interactions we are using the RabbitMqBundle.
We’re using the DoctrineMigrationsBundle for database migrations and the data-fixtures and AliceBundle for database fixtures.
Our CI tool Jenkins runs all of our tests and triggers a new capistrano deployment if they pass.
Is it me, or does it feel like I’m in the world of The Hitchhiker’s Guide to the Galaxy reading about how to build a Globular Cluster Information Hyperdrive?
And this, all in the name of electing Members of Parliament, an institution whose daily procedures were already antiquated back in the Victorian era.
Once the process of governance starts getting anywhere near state of the art web technology, it’s going to be awesome.
Or it will be a whole lot worse. You never know.
As the human debacle around the science of climate change has proved, this tech is equally good at spreading knowledge and intelligence or ignorance and stupidity. It’s our choice as to what we want from it.
Monday, March 23rd, 2015 at 6:08 pm - Whipping
I’m doing a lot of politics stuff running up to the general election, but I don’t feel like blogging about it much.
Actually, I’m not doing that much. What’s happening is that all the work I did ten years ago is finally being put into use today by other people, resulting in an event such as this fine rant from the LibDem MP in Bristol West:
To claim, as the website Public Whip does, that I ‘voted very strongly for selling England’s state owned forests’ is misleading in the extreme. I have never voted in favour of selling forest land – I voted against two poorly worded and hyperbolic motions submitted by the Labour Party.
I never believed I’d see the day where MPs would have to answer for the things they voted for in Parliament. Anyway, there’s that and electionleaflets.org and Francis’s Candidate CVs project gathering steam. What a difference five years of internet technology advancement and greater generational awareness can make.
Meanwhile, I was in Bristol for a few days helping a friend with some DIY, because I want to put this kind of laminate floor down in our kitchen on top of some real insulation:
Then I did some painting before getting relieved of my duties for spreading paint all up the paint brush handle.
I spent the night on the Blorenge, then flew at lunch time completely alone for two hours until I suddenly got dumped down in the bottom landing field in Abergaveny. Nobody took any notice of my death spiral down to the ground; just kept walking their dogs. I am still working hard to process the data into something meaningful — if this is possible.
An invite went out for a surf on the Dee Bore on Saturday morning. I thought it might be special, being the day after a partial solar eclipse, but it was a damp squib and most of us lost the wave within a few hundred metres.
Monday, November 10th, 2014 at 10:39 am - Whipping
It was an expensive London and Cambridge weekend for me and Becka (£99.20 return train ticket each), but the chance to get home on Sunday night directly from the middle of London to the middle of Liverpool in under three hours without needing to be awake beat the plan of car shuttling onto a local train via some backstreet parking spot in St Albans to avoid driving to the centre of London.
You win some, you lose some.
I got a motion for electionleaflets.org to be done properly accepted by the members at the UnlockDemocracy AGM on Saturday. This has the potential to get some professionalism on the situation in time for the next election.
The purpose of the project was to learn how to use PDF.js, which Francis told me about the day before.
I thought I had a good chance with it (being as it is completely practical and could be implemented by the Public Accounts Committee right away), but it did not even get an honourable mention. That honour went to Richard whose Parliamentary Bill analyser disclosed how many goats would need to be skinned to print out the Act, among other things. For more details, see my blogpost from six years ago: The vellum has got to go.
We met Rob for dinner who had a brain machine on the bookshelf, which Becka was very taken with. I can tell you that someone will be learning how to solder in the next couple of weeks, because that is the only way they are going to get one of their own.
International governance is a serious issue with major consequences. It doesn’t help that the British public enjoy electing clowns from UKIP and the BNP to these positions where they don’t even pretend to do any work. Politics is not a sports show, like the football league. This stuff matters. And it is logical.
This morning there were two politicians debating on the radio news program: LibDem MP Martin Horwood and UKIP Leader Nigel Farage squabbling about immigration, and tripping over themselves to praise the Transatlantic Trade and Investment Partnership treaty being secretly negotiated between the EU and the US. Horwood asserted, without any evidence, that this deal would “eventually help to create millions of British jobs”, while Farage said that if we were out of Europe we could sign this deal much faster than allowing Europe to do it.
Farage argues for Britain to get out of Europe on the basis that it gives us greater national sovereignty and control over our own laws — something which is flatly undermined by these trade deals whose sole purpose is to establish the supremacy of corporate rights over people’s rights, and where there will be no Parliament to over-see its operation.