Over the years I’ve owned quite a few computers. An endless series of PC compatibles that mostly differed from each other in CPU speed or memory capacity. But before the PC era computers were a bit different in that they differed vastly from each other. The advent of the PC homogenized personal computing to a large extent, this is now due to the proliferation of mobile platforms and various tinkerer devices (Raspi, Arduino, etc) changing again but for the longest time it seemed as though x86 was what computing was all about.

The very first computer that I owned outright was a TRS-80 pocket computer. It was actually manufactured by Sharp and marketed by Radio Shack in the USA and Tandy in Europe. I’d befriended a number of Tandy store managers in and around Amsterdam, they let me play with the bigger machines so when I’d finally delivered enough papers and repaired enough TVs to buy a computer it was a fairly natural decision to get a TRS-80 of some kind and at 549,- guilders it was just within reach. The paper route was pretty steady income, the TVs mostly hit and miss (scan the garbage on wednesdays and fridays for likely candidates for fixing up or parts, then sell the TVs through classified ads for 25 guilders a piece). But eventually I got there and my first computer was mine. I still have it and it still works!

A cassette interface was another 150 guilders or so so I ended up writing out all my first programs into a copybook. 2K of RAM isn’t a whole lot so most programs were under a hundred lines or so.

The next step up was a pretty big one. From the pocket computer I went on to a ‘dragon 32’. The dragon was a clone of the TRS 80 color computer, it had an interesting mix of hardware and software, with the software doing the job where other computers of that time would have used a piece of hardware. For instance, the sound interface and the cassette tape interface were entirely in software. This taught me a lot about timing, phase locked loops in software and serializing memory to tape and back again. I fixed a friends computer (burned out RAM chip) and we realized that the chips were 64 Kbit x 1, and there were 8 of them. So our ‘dragon 32’ was really a dragon 64! The 6809 chip that powered it could address 64K, the memory was laid out with 32K of RAM in the low part, then 16K of ROM and then another 16K of ROM + peripherals in the final portion. The controller could bank switch each 16K block between ROM and between RAM, an undocumented feature. We found out how to move the BASIC rom to RAM and had endless fun documenting and modifying the BASIC interpreter using 6809 assembly.

Because I couldn’t afford a disk drive I wrote a disk simulator that would seek to a specific spot on the tape and then hit the ‘stop’ button on the cassette drive, then switch to record or play mode depending on whether I wanted to read or write a block. Auto-reverse cassette recorder doubled the storage space available :) I’m pretty sure Sony never foresaw that putting electrical push-buttons on their cassette recorders instead of the usual piano keys would allow some crazy dutch kid to use them as random access storage.

The dragon could interface to a TV or, if you fiddled around a bit with it to a monitor. I scavenged a ‘real’ monitor from a studio TV camera that I found in a dumpster, an old tube driven beast with a super sharp 8” black and white display. It kept losing sync as it warmed up but it had a ton of thumbwheels to adjust Hsync, Vsync and so on. Especially in winter this was lots of fun since the room I lived in didn’t have any heat. Effectively that old tube monitor was both my screen and my heater :)

My friend Henri and me had the same machine, we’d write assembly programs and then we’d call each other and transfer our code in ‘hex’ by typing it in line by line on the other end with a checksum to make sure we had not made any mistakes. Human modems!

Then came the KIM-1. In many ways this was a huge step down from the dragon. The KIM-1 was a very barebones computer, basically nothing but a board with a couple of 7 segment displays and a keypad that wouldn’t look bad on a calculator. The KIM-1 had only a few hundred bytes of memory and a 6502 CPU. This was interesting because it allowed me to learn 6502 assembler. Tape interface was through an FSK chip. What’s funny is that this was such a restrictive environment that you ended up having to be extremely creative in order to do anything useful at all.

Ever tried to play ‘snake’ on a couple of 7 segment displays?

Next up, one of my big loves, the BBC micro model ‘B’. The ‘beeb’ as it was known at the time was an absolutely amazing machine. Excellent hardware build quality, a switching power supply, several rom slots which you could use to install software available at boot time (exmon, word and a ton of others). What really set the BBC apart from the rest was the language that came with it, Acorn Soft structured basic. A basic which - unlike other basics at the time - allowed you to define functions and procedures by name rather than by line number. This opened up the world of structured programming for me and I distinctly remember that resounding click when I got it. Suddenly I could write programs several times larger without getting bogged down. Names are so much easier to remember than line numbers, and it made it much less of a chore to renumber a programs lines. (remember the renum command and how you had to re-set your whole internal symbol table about what lived where? No? lucky you!).

Mass storage ended up being two 160K teac 5 1/4” floppy drives, and the internal memory was expanded using a thingy called a ‘solidisk’ to 128K, using a technique called bankswitching. Every one of the 16 rom slots could be paged in for read or write into a 16K segment of the memory map.

After the BBC micro I moved on to the Atari ST. The ST was a serious step up from the 8 bit era. The Motorola 68000 that powered it could address a lot more memory. The initial run of the ST (the 260) had 256K of ram, then the 520 came and after that the 1040 with a full meg of RAM. That was an amazing amount of memory for that time and it seemed as though I would never write a program that would fill it all. (this didn’t take long, Jenny Yips laws of computing have held true over the years: eventually computer programs will fill every cycle, every byte of RAM and every bit of secondary storage that you’ve got). The ST came with something else as well, it had a hard drive interface and you could use this to connect a small selection of 5 1/4” hard disks to it. I eventually got a 20M hard disk and bought a C compiler. This bootstrapped my career as a free lance programmer and I proceded to write computer games profesionally for a studio in Leiderdorp (Aackosoft, long since bankrupt, two titles that I wrote that shipped were ‘FlightDeck’ and ‘Indy 500’.).

After the ST there was a brief period where I used an Acorn Risc Machine, but since it was a pre-production demo unit and it took forever for the machine to pick up more steam in the marketplace I ended up abandoning it. This was a real pity because it was an extremely powerful computer for the day, the chip that powered it (the ARM) is no used in countless mobile platforms and has survived where lots of others have not.

Next up, the PC era. I worked for a bank for a bit and got my first introduction to PCs there. The first PCs seemed quite underpowered compared to the computers that I’d been using and for the longest time I didn’t get what all the fuss was about. Even though the ST only had a megabyte of RAM the CPU was already using larger registers than the CPU in the PC and it could address that megabyte without resorting to bank switching or other tricks. But PCs got better with every new generation of CPU and by the time the 286 came out I got an el-cheapo mother board from some taiwanese manufacturer and a ‘hercules’ graphics board. A bit more scrounging got me another drive (a Seagate 40 MB unit) and this eventually gave me enough experience to start offering my services as a PC programmer as well. This was important because economically speaking the ST was doomed forever to be a ‘game computer’ (though you’d be surprised to see what we did with it, including a full CAD-CAM program directly driving lathes and mills!), and the PC was the choice of just about every business.

Programming the PC (after the ST with its linear memory space) really felt like a step back in many ways. But over time it got better, and after I discovered the QnX operating system it really came into its own. I wrote tons of software for the PC platform either for Windows or for QnX. Cad systems (for instance, for classical sail designs), process control, a whole suite of software around the idea of digitizing pictures, printing them on credit cards and distributing these (you probably have a descendant of that system in your pocket right now), editors, database applications, a good chunk of a message switch for telex messages used to broker shipping capacity and so on, and finally my own 32 bit QnX clone called ‘Unite!’, a multi-tasking multi-user operating system with built in networking and primitive clustering capabilities.

Around the time the world wide web launched I used SGI machines for several years. IRIX, SGI’s version of Unix was a really nice implementation and was quite fast. Hardware accelerated graphics meant that you could do reasonably complex 3D scenes with acceptable frame rates. It also doubled as my first web server, hooked up to a nearby provider using what was probably the longest run of coax on the internet at that time ;) The Indy as it was called was this funny little pizza box with a MIPS cpu running at 200 MHz, 32M of ram and internal 3.5” hard drive. It was quite fast and the operating system felt extremely well integrated with the desk top environment. To this day, whatever machine I’ve got I try to re-create that feeling as much as possible.

SGI eventually bit off more than they could chew with the acquisition of CRAY, the hardware was way overpriced compared to the PC offerings and eventually the company died and I was back to PCs.

In spite of the PCs (especially from the 386 onwards) dominance in the market, their utility and eventual performance and the fact that they powered a huge part of my professional career there was something special about those funny little 8 bit machines.

Apart from the operating system exercise (which, together with games programming taught me more about computers than anything else) I have over the years come to feel more and more out of touch with the machine. Those 8 bitters you could actually know, you’d be aware of each and every memory location and what was going on there, you could know those computers exhaustively, until there was literally nothing left that you did not know.

Nowadays when I boot my trusty Ubuntu machine (24G of ram, 48T of attached storage, 2.6GHz clock, one of those 8 bitters would fit in a very small corner of that machine) I feel strongly that it is impossible for me to fully ‘grok’ that computer. I have to rely on trust rather than knowledge that everything is the way it should be and when I program I know I write on top of countless layers of code written by others rather than directly to the machine. API gluing rather than actual coding. Of course the effect of tying together a few APIs with some glue code will outmatch anything you could have achieved in a lifetime writing on an 8 bitter but still, there is some aspect that is missing there.

I don’t actually know the computer that I’m using for my day to day work any longer. Those oldies they had personality, they were quirky and strange, each computer completely different from the others in many ways, and knowing them intimately gave you a tremendous edge. In that sense they were more personable, more individual.

A man named Jim Lawless just mailed me, we seem to have more than just a title in common, read his post as well if you feel like it and spot the 7 differences :) : Computers I have known.

Over the last couple of weeks I’ve been working off-and-on on a little side project. The idea is that there are lots of people that are not in a position to visit a university or that don’t have a public library with ‘access’ at their disposal but that do need to be able to read scientific papers. Ever since the advent of the world-wide-web the funding for public libraries has slowly but surely dried up. There are fewer of them and they offer ever fewer facilities, including more limited or non-existent access to scientific journals. And that is a pity because libraries offer something the web does not, access to a wealth of copyrighted materials for a very small fee.

Scientific research costs lots of money. So much that most if not all research can trace its funding back to public money. Every paper cites the papers that it builds upon, and even if a piece of research is not funded directly with public money you won’t have to trace back very far through the citations before you hit papers that were funded with public money. So, effectively, *ALL* research that is conducted is funded directly or indirectly through public money.

Historically, publishing anything was a costly affair. Copyrights on scientific papers submitted to journals were routinely assigned to those journals by the scientists that wrote the articles. The copyrights didn’t matter much to them as long as their work made it out into the world, to get to an audience larger than the scientists themselves could reach. Quite a few of those journals even had such copyright assignment as a requirement. At that time nobody could foresee that the world would change to the point where this mechanism no longer made any sense at all. Lots of people are aware of this problem, but nobody is in a position to do much about it. The reason why is that the lobbying power and the legal armies that are at the disposal of the few publishing companies that find themselves in the luxurious position of holding hostage the last 70 years or so of the results of scientific research are so formidable that they serve as an effective deterrent against correcting this historic mistake.

It’s a beautiful business to be in: publish research that you took no part in, claim the copyrights to the results of that research, publish the research in a very expensive journal, publish reprints at exorbitant fees and finally, when a more efficient distribution method appears get rid of all the costly components of the business but keep the prices the same. According to one person I spoke to who is knowledgeable about the publishing field the profit margins dwarf even those of the publication of pornography.

In our current technological age storing and distributing 50 to 75 T of data (estimated amount of storage required to store all the papers produced historically) is an affair that even a private individual can afford. Passing a paper from a central service to a person that wishes to read that paper no longer takes a printed volume and a physical distribution network. Such a service can operate not on several 10’s of Euros/Dollars per reprint of a document (which was, even when it was needed excessive), but it can operate on very small fractions of pennies. And yet, the publishing industry that seeks rent on the content that (they claim!) has been assigned to them routinely charge excessive amounts of money for access to this information. Many words have been written about this subject, but in the end the publishers have pockets deep enough and lobbying power sufficient to protect their business interests against the very people that fund the research.

If you are living in a country that is poor, if you are a private individual (whose taxes have likely been used in part to fund the very research you might be interested in) and if you have a desire to enhance your understanding of how some aspect of the universe works, or if you want to help lift your country out of poverty by studying hard chances are that you will not be able to do so without paying exorbitant amounts of money to parties that did not contribute to the original research in a meaningful way.

A Small Detour

In the 1980’s, when computers were a bit more scarce and a lot less powerful than they are today, and when a half decent development environment was even more scarce I finally found my home on the Atari ST, an MC68K computer that was more suited to writing games on than software of a more productive nature. It came with your option of several not-so-hot and fairly buggy programming languages, mostly limited to BASIC and assembler. And then a little known company called The Mark Williams Company came out with ‘Mark Williams C’, a very good C compiler and a standard library implementation to go with it. I was hooked instantly, this was what I’d been looking for, a computer with a very powerful CPU and a more ambitious amount of memory than what my 8 bitters could provide, an environment to learn more about programming (I’d been reading Kernighan and Ritchie’s introduction to the C programming language back-and-forth but had no access to an environment where I could actually write programs and try them, so ‘desk checking’ was my lot until that point).

Mark Williams C really changed my life, it literally launched my career as a free-lance programmer. It came with a *fantastic* reference manual, that I kept with me until it fell apart from use and even my attempts at taping it together were no longer enough to keep it serviceable. Long after all my Atari machines had all died from old age and hard use that manual came with me, to Poland where I lived for a while, and then back to the Netherlands.

The Mark Williams Company derived its curious name from William Mark Swartz, the father of the man that ran the company, Robert Swartz.

End Detour

Late in 2010, A young man named Aaron Swartz, the son of the above mentioned Robert Swartz worked very hard to achieve one of his goals, to ensure that everybody would have access to scientific papers, no matter whether they were affluent or whether they were affiliated with a university or not. He went about it in an unfortunate way which got him into trouble with the law. Aaron had already achieved several noteworthy victories for someone as young as he was but the combined might of the United States federal justice system and MIT (who insisted on labeling him a felon and jailing him) proved too much for him to deal with and he took his own life on the 11th of January 2013. Volumes have been written about Aaron’s saga, I won’t go into the details but if you feel like reading more about it I recommend this slate article.

What really bothers me that someone so young, so nice and accomplished and so idealistic would be pressured to such an extent for trying to do something that was good by any definition of the word that I’m familiar with.

Most of the work I do is born from a love for my profession, this work was born in anger and frustration at a system that causes the best and brightest lights to be extinguished.

So, here is ThePaperBay.com, a way for those that seek scientific papers to make contact with those that have access. It’s my salute to the man that tried to give us access to all the products of science and a very small token of gratitude to his dad.

We should all wish for sons to be so proud of.

This article is in response to an earlier one comparing Haskell and C, which made the claim that Haskell beats out C when it comes to speed. On a Hacker News thread I made the bold claim that the C code is rife with problems and that it is tempting to fix it.

I’ve earned a good part of my life’s income to date taking other people’s C code, cleaning it up, making it work and then making it work better. In one sense the original program that this article is about is perfect, it takes the prescribed input, processes it and produces the desired output using the test set provided. That’s really all you can ask for in any program, under normal circumstances. Provided of course that the program produces its results in an acceptable time. What acceptable is is up to the user of such a program, in this particular case the writer of that article (linked at the end) used acceptable to mean ‘slower than Haskell’ and as a long time observer of the language wars this irks me.

For one, I think if you compare two languages side by side I think you should at least be as proficient in the one language as you are in the other, and you should be proficient in both. Second, you should if you are not proficient in one of the two limit yourself to highlighting the strong points of your favorite language rather than talking down the other language that you don’t know as much about. Third, a comparison on speed between Haskell and C is almost meaningless, speed is not Haskells strongest point and it would be folly to bench it against C for that particular reason.

The monetary gain from using a higher level language than C is made because the programmer spends less time writing/debugging/maintaining the software. As compilers get better and languages are better capable of using raw speed of the hardware without investing a lot of programmer time this trade-off will be made more frequently to the disadvantage of languages lower on the totem pole such as C.

I have no clue about the specific strengths or weaknesses of Haskell. I hear a lot about it but there are so many languages and there is only so much time. My own language repertoire is probably outdated, limited and not ‘hip’ by any stretch of the imagination but I think I know my old tools well, a bit like an elderly carpenter, please keep your fingers away from the chisels.

C may be old, but beating C for raw performance is quite hard (if it is possible at all, even FORTRAN still has its uses) and there are applications where the sheer number of computers working on a single problem outweighs the programmer cost advantage offered by an easy to use language (I’ve never heard anybody describe Haskell as ‘easy’, but it may very well be easier than C, I just do not know) by a considerable margin. The example chosen is quite apt because a lot of the problems in the field of computational genomics are exactly such problems. Other advantages from efficient code may translate into hard real time guarantees, improved battery life, lower charges from your service provider or a lower power bill. And since the environmental impact of computing is more and more on the radar optimization may find a new lease on life. That said, any optimization that you don’t know for sure that you need is likely to be premature and it is important to be aware of the cost of optimization and to make the appropriate decision when to employ it.

So this is an attempt to set the record straight and will give a level comparison between the ‘old’ version of the code as supplied by the original author and a re-written piece of code. I’ve purposefully not looked at other C entries in the language shoot-out section to keep things fair and to show what you can do to fix this code. A from scratch implementation would go for the end result in a much more direct way.

The time taken by the version as posted on http://paulspontifications.blogspot.nl/2013/01/when-haskell-is-faster-than-c.html on my machine (i7 920, 2.66GHz) when compiled with -O3 (a C compiler optimization directive) to process a 250M ‘fasta’ format file is about 7.4 seconds (average of 10 runs), so we’ll be using that as a baseline.

As we go the original program will be transformed into one that hopefully reads a bit easier and performs better as well.

The first re-factor will be a simple cleanup.

After turning on some compiler warnings (-Wall -Wextra) the first thing that jumps out is an unused variable ‘title’ in the ‘main’ function (happens to everybody without warnings on) and the fact that characters (signed entities) are used as array indices. Changing the base type of the arrays and pointers to ‘unsigned character’ removes these warnings and takes care of a potential problem if the program would ever be fed bytes > 0x7f, in which case it would start to refer to memory before the lookup table. You really don’t want to do that. C’s signed character type has tripped up many programmers, especially when used as an index. Main really has a type ‘int’ and you should declare it as such, and add a proper return value.

As usual when dealing with programs that are undocumented other than a few brief comments here and there part of the fun in doing things like this is figuring out how it does what it does. This one is fairly simple but still, the principle holds, I like puzzles.

C does not require you to set static global arrays to ‘0’, so the for loop in the main function can go, you also don’t need to declare empty blocks so the {} after the while can go as well.

First thing to hit is the loop that writes the output. This is a character loop that is a bit inefficient the way it is written. The fasta format uses 60 characters to a line and we can simply write out lines of 60 bytes in one go, adjusting the size for the last line to whatever is left. New runtime of the program is 5.2 seconds.

I don’t really like the way the feof / fgets bit at the beginning of the process function looks so I’ve changed it a bit using the return value of the fgets function as a guide whether we’re at the end of the input or not. A bit more fiddling to the output loop to make it look nicer (obviously no speed gain, that’s just cosmetics). What is not cosmetics is that the title string is messed up if it is longer than 132 characters, the remaining characters would be dropped. This is a problem because the FASTA format does not allow lines > 120 characters but if we receive input like this we have the tough choice of either keeping the input and performing the job or throwing an error. Silently discarding the excessive input seems wrong to me. If this were a contract job I’d aks for guidance at this point what the company policy is in situations like these ;) Some more looking at that loop and you see that you can simply drop the title variable altogether and simply use the loop that scans for the ‘\n’ character to process the whole title. End of file handling of this program (and bad input) are pretty weak, but those things are not specified so we’ll just leave them the way they are. For robust production code this would have to be substantially improved on.

Next up, the read loop. The read loop is also character based, just a minor tweak and instead of reading characters we’re reading lines using fgets. New runtime: 1.2 seconds. C has a nasty problem with looking ahead in the input stream, we need to know if the first character of the next line is a greater than sign, if it is then that’s the sign that we are looking at the beginning of a new sequence. This is done rather un-elegantly using a getc / ungetc pair, which effectively allows you to peek one character ahead.

That 1024 bytes buffer is a bit anemic, this is 2013, we’re processing 250M of data and we’re going to pull it all in anyway. Let’s use 256K buffers, we’ll still use exactly the same amount of memory + on average half an empty 256K buffer. That’s a very small price to pay. Runtime now 0.97 seconds.

So far we’ve been quite nice with our optimizations, no tricks, no gimmicks. The speed of the program is not fantastic but quite respectable.

But suppose your boss is at your desk breathing down your neck. It’s fast but really not yet fast enough, what can you do to make this code faster? ‘Use as much memory as you want’ he said when leaving…

Optimization is a funny game, the more you optimize the code the uglier it gets. Programs that have been optimized are - if you don’t stop where we’ve gotten so far - a lot harder to maintain and a lot harder to debug as well. So now we’re entering the twilight zone of optimizations, things that you really don’t want to do unless you’ve got an extremely good reason for doing them, and things that might subtly alter the way you can use your program, so you need to be vigilant against that. Since there is no clear specification here we have a lot of leeway but in a real world setting there would be all kinds of constraints.

A quick check of how fast the program runs when you disable ‘reverse_complement’ shows that this takes approximately 20% of the runtime. This means that even if we’d reduce that portion to ‘0’ there would still be 80% of the time used for other bits of work. Another quick check, cat’ing the file to /dev/null shows that this takes about 50 ms. Clearly raw IO speed is not what is holding us up.

That leaves the two loops that we’ve already optimized a bit as the main offenders.

One of the problems with those loops is that (unlike reverse_complement) we can’t do the work in parallel, so the usual ‘get out of jail free’ card does not help us here. If we were CPU bound for the larger part of the runtime the effect of multi-threading would be much more pronounced, but in this case we can never win more than 20% that way. Reading from an input stream is an inherently serial operation and writing to an output stream is too. When you’re optimizing there are a bunch of trade-offs that you can make, the most common one is space versus speed. If memory is significantly larger than the largest possible input sequence you could read all of the sequence into one giant chunk of memory. that would mean a single read. The interesting possibility that approach offers (versus the block oriented approach used by the original program) is that the linefeeds would remain where they are. Simply skipping them would be enough. (this skip would require two tests in the inner loop of the reverse_complement routine, which would incur a penalty, but that penalty is not as large as the one imposed by the way the linefeeds are processed now). A single read and a single write could then do the whole job.

A very quick check, increase the block length to something > than the input file length shows about 5% improvement, but we’re still going through those loops. I like the idea of a very fast scan, let’s see what we can do with that.

So, this is the first major departure from the original code, We start off with allocating buffers again, but this time we read huge chunks of data until all data has been read, and then we move all the data into one single buffer which we can then process in place.

To make reverse_complement a bit more universal we pass it two pointers to the data instead of the ‘block’ pointer.

Reading and writing the data now comes in at 0.22 seconds with 1M blocks. (reading from a file and writing to /dev/null).

Next up the processing. We leave all the data in place, we don’t mess with the linefeeds (we just skip them) and we stuff it all into one giant buffer. Runtime: 0.44 seconds. .22 of that comes for the reading/writing, and the other half is the actual processing.

Note that this change means that the program will need an end-of-file at some point in order to function, and that end-of-file should arrive before we run out of memory. This is a serious drawback and we should really do something about that, because now the program is no longer up to spec. It may be fast, but if it contains a bug like that the speed is not relevant. Make it correct first, then make it fast!

The problem is (like with many text protocols) that we have absolutely no idea how much data to read to get the next working set into memory. The only thing we have to go on here is that single ‘>’. Scanning for that character is expensive! It requires a full pass through the data just to figure out where the one sequence ends and the next one starts. And since this is crucial information we can’t really skip this step. The good news is, we had that scan anyway so whatever the reader gains in complexity is lost by the processing step (which will now only process one sequence).

The final result is a runtime of about .75 seconds, quite acceptable and this would put it #1 on the shootout if submitted. There are stil two bits of low hanging fruit left to optimize this code further:

The first is very obvious. If we’re processing multiple sequences then (in the case of the original program) we can process blocks in parallel on multiple cores using threads. In the current incarnation of the program you’d apply that technique to a full sequence. As shown above this would yield at most a 20% improvement, my best guess is that with the input streams given the yield would be lower than that (about 7%, 1/3rd of 20% because there are 3 input sequences).

The second is a bit less obvious. The reader loop now reads in as much as it needs or a bit more. Once detected the bit more is saved for the next round of reading. Every time that scan takes place it re-scans from the beginning of the sequence found so far. That’s fast (because it is in memory), but still very wasteful (because a sequence is larger than the CPU cache it is in fact very wasteful), you only need to scan the new part after you’ve done the initial scan. So you could improve on this by adding another scan in the first portion of the reader loop just after it re-cycles the saved portion. For every iteration of the loop after that one you only scan the bit that you just read in.

Total memory consumption of this version is about twice as good as the best C version in the shootout, and elapsed time slightly better. Using the threading and scan tricks above memory consumption should be about the same and elapsed time should be significantly better.

In the words of Colin Wright: “You can’t make a computer do stuff faster, but you can make it do less work”, and that really is the essence of optimization.

The realloc is not as bad as it looks, since it is almost always the last allocated block growing it is nearly free.

C gives you a lot of leeway in how you want to express your code. Over the years I’ve settled on a style that is pretty defensive, this helps in avoiding some of the pitfalls in the C language. When you read the code below you might think ‘I can do this quicker using a pre-decrement’ or you might think ‘I can push these two separate statements onto a single line’. These things are all true, and in fact they might give you a small performance boost at times. But for me such optimizations are not worth it unless we’re talking about an inner loop where a substantial amount of time is spent. And even then I’d work hard to get rid of the number of times we iterate through the loop before resorting to such micro-optimizations trading readability for speed.

I hope you enjoyed this little exercise, I certainlyh did. The final code is below.

(afterword: Joachim Schipper points out that memchr could be used to speed up the inner loop in the scanner for the ‘>’ character see here, that’s important because it makes the inner loop both faster and more readable, and this is a good illustration of why you want code review).

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define BLOCKLEN (1*1024*1024)

#define TRUE 1
#define FALSE 0

/* Initialised by calls to set_complement. Holds the complement codes. */

static char complement[256];

void set_complement(int c1, int c2) {
    complement [c1] = c2;
    complement [c2] = c1;
    complement [c1 | 0x20] = c2;
    complement [c2 | 0x20] = c1;
}

/*
 *  this walks two pointers towards each other, every time one of the pointers
 *  hits a linefeed the linefeed is skipped and the loop restarted. Whenever neither
 *  pointer is on a linefeed (the most common case) the characters are complemented
 *  and swapped.
 */

void reverse_complement(unsigned char * ptr1, unsigned char * ptr2) {
    unsigned char c;

    while (ptr1 <= ptr2) {

        if (*ptr1 == 10) {
            ptr1++;
            continue;
        }

        if (*ptr2 == 10) {
            ptr2--;
            continue;
        }

        c = complement[*ptr1];
        *ptr1++ = complement [*ptr2];
        *ptr2-- = c;
    }
}

/*
 * Fasta files do not have any parser friendly features such as length fields. This means 
 * that if you want to know if a sequence is complete you have to scan for the next '>' 
 * character of the title following the current sequence. This is quite time consuming, 
 * and even if you find it you still have a problem because you'll have read *too much* 
 * data from the input stream. That excess is then saved in a separate buffer to be re-used 
 * at the beginning of the next round of reading. 
 * 
 * This loop is pretty slow, it takes up about 40% of the total runtime of the program. 
 * There is a clear way to improve it, which is to scan only the part that was just read 
 * in after the first pass through the loop in read_sequence. I'll leave that as an exercise 
 * for those that want to beat this version of the code, there is an easy #1 spot in the 
 * language shootout waiting for you if you add that feature ;) (and multithreading if
 * you want to keep that #1 spot!)
 */

int sequence_complete(unsigned char * p, int n, unsigned char ** saved, int * savedsize)

{
    // skip the leading '>' for the title of the current block, we want the *next* block

    p++;
    n--;

    // now search the rest of the data for a sequence start

    while (n) {
        if (*p == '>') {
            // we've found a title for the next sequence, figure how how much excess we have
            // and save the remainder

            *saved = malloc(n);

            memcpy(*saved, p, n);
            *savedsize = n;

            return TRUE;
        }

        p++;
        n--;
    }

    return FALSE;
}

// read in a sequence, if we read in more than one whole one save the excess

unsigned char * read_sequence(int * s, unsigned char ** saved, int * nsaved) {
    int size;
    unsigned char * p;
    int read;
    int n;

    if (*saved == NULL) {
        // first round or a really lucky break
        size = BLOCKLEN;
        p = malloc(BLOCKLEN);
        read = 0;
    }
    else {
        // round which starts from excess data read during a previous round
        size = *nsaved;
        p = *saved;
        *saved = NULL;
        read = *nsaved;
        size <<= 1; // fix for bug spotted by Robert G. Jakabosky
    }

    while (!feof(stdin) || read) {
        p = realloc(p, size);

        n = fread(p + read, 1, size - read, stdin);

        read += n;

        // if the sequence is now complete save the excess for the next
        // iteration and process this sequence. Like that we don't need
        // to read all the way to end-of-file before we can start processing
        // if we have read more than one whole sequence this still works, it
        // just means that we will re-visit this on the next round through
        // the reader to fetch another whole sequence.

        if (sequence_complete(p, read, saved, nsaved)) {

            read -= *nsaved;    // compensate for saved portion
            break;
        }

        // if we did not read any data we can return now

        if (n == 0) {
            break;
        }

        // every time we reach the end of the block we scale up

        if (read == size) {
            size <<= 1;
        }
    }

    *s = read;

    return p;
}

void process_sequence(unsigned char * p, int size) {

    unsigned char * start;

    // skip the block title
    start = (unsigned char*)strchr((char*)p, '\n') + 1;

    // figure out start and end of this sequence and do the complement

    reverse_complement(start, start + size - (start - p) - 1);
}

void setup() {

        set_complement ('A', 'T');
        set_complement ('C', 'G');
        set_complement ('M', 'K');
        set_complement ('R', 'Y');
        set_complement ('W', 'W');
        set_complement ('S', 'S');
        set_complement ('V', 'B');
        set_complement ('H', 'D');
        set_complement ('N', 'N');
}

/*
 * process sequences one-by-one until we're out of sequences and end-of-file 
 * has been reached on the input and the readahead buffer is empty
 */

void process() {

    unsigned char * readahead = NULL;
    int nreadahead;
    unsigned char * p;
    int s;

    while (readahead != NULL || !feof(stdin)) {
        p = read_sequence(&s, &readahead, &nreadahead);

        process_sequence(p, s);

        fwrite(p, 1, s, stdout);

        free(p);
    }
}

int main() {

    setup();

    process();

    return 0;
}

Finally, this code is still terrible! Yes, really. It is fragily because it does not deal well with error conditions such as broken input and many others besides. Because the original does not and the specification doesn’t give any requirements I left it that way but I would never ever consider a piece of code like this finished. At a minimum it would require checks on all system / library call return values, unit tests for every function and so on. Even a ‘rivial’ little program like this is not trivial if you plan on putting it into a real world situation where people will run their business based on your code.

One thing that struck me about the FASTA format is that there is absolutely no attempt to detect/correct errors in the data, for a file format that stores information where mutations are not without effect you’d hope they would at least put in some safe-guards to make sure that mutations are of the biological variety rather than the digital one (bitrot).

Another thing that I noticed when I was finished and looked through the language shootout entries was OCaml #3, about 3 times as slow as the C version here, but only 12.8M of ram in use, I have absolutely no idea how it does that!

the language shootout entry

skybrian notes about battery life/cloud charges

FASTA file format

Original HN thread

@ay: I left you a message

The 14th of August, 2003 was an interesting day in many ways. I’d just taken the family down to Toronto to spend a day there and pick up our newly minted passports at the Dutch embassy. We had some fun in the city seeing the sights, visited Niagara Falls and were on the way back on a sunny August thursday afternoon. What could possibly go wrong? We pulled into a gas station just outside of Barrie. We parked behind the person gassing up and waited our turn. The guy took his time, walked into the store to pay, got into his car and drove off. We moved up the line to the pump, I got out and put the nozzle into the fuel port but nothing happened. This normally means that the pump hasn’t been ‘cleared’, usually that takes 10 seconds or so (an annoying little bell starts ringing in the office over the blinking light indicating the pump that needs to be cleared). After a minute of waiting I went into the office. “Sorry bud, the power is out”.

Bummer! The car we were driving was our family car, one of those ford minivan affairs and it wasn’t exactly a paragon of fuel economy. 3.8 liter engine, still slow as could be and it drank like a fish. So when we pulled into the gas station the needle was at ‘empty’ and the fuel warning light was on. We were stuck for the duration.

I looked at Barrie from the highway down and noticed that there were lots of people in the streets, many more than you’d normally see in a place like that. Traffic lights were out. Billboards were dark. Little by little it dawned on me that this was more than just a trivial outage. I tried my cellphone, it still had a signal and called to the Island where we lived. We had a business there (a gas station, coincidentially) and I asked them if they had power. No, they didn’t. Oh oh. But the generator was up and running and as far as they were concerned it was business as usual. Maybe a bit more busy because some traffic from the highway pulled in on the island to gas up. I asked them to call the supplier (Wardlaw fuels in Sault Ste Marie, an awesome company to work with) to make sure they were going to be provisioned.

An hour and a half into the blackout a truck from OPG pulled into the station. The guy in it took all the ice his truck would carry, paid cash and drove off again. Now I was getting really worried. OPG is the “Ontario Power Generating Company”, they are the company that runs all the grid infrastructure, and more importantly they run the nuclear generators. If a guy with that much knowledge buys ice he must know something that the general public doesn’t know yet: that this outage is going to take a very long time, long enough that food will spoil. Nuclear power is funny in that it is excellent baseline power, but when you shut down a nuke it’s going to be down for a while. They go down quite quickly but they take forever to get back online.

I saw one guy walk around to the spot where the air hose for refilling tires was. A few minutes later the same guy walks by with the length of air hose coiled up and on his shoulder like it was the most normal thing in the world. Presumably he was going to use it to siphon fuel from one car into another. Let’s hope they were at least both his cars. Another guy made off with the fire extinguisher. I’ve heard it say that the veneer over our society is very thin and that it won’t take much to bring out the ugliness underneath but I had not expected to see that illustrated so clearly so quickly. Suddenly I didn’t feel safe and I did not want to wait for who knows how long next to an unfamiliar town in a vulnerable position like this with my wife and kid with me.

My mobile still worked, so I called the gas station on the Island again. The mood around us continued to detoriorate from resigned to menacing, people were getting angry with the operator of the station (just some kid that was clearly not the cause of the trouble). I asked the people at our gas station on the Island to please go to our house, get our other car (a honda civic hybrid, capable of doing 1200 km on a single tank if you were careful), load it up with fuel cannisters and drive down to where we were. They said they’d get on it, and that the station was being mobbed by people that wanted gas, some coming all the way from DesBarats and Sault Ste. Marie (a good 50 Km or 30 Miles away). They’d already gone through one full drop (50,000 liters) and another one was on the way. Our trusty diesel was doing just fine running the pumps and half of the freezers, they’d sold the food in so they could be switched off, the other half at a discount rather than to let it go bad.

We talked to some of the people stranded around us. One group was a bunch of motorcycle enthusiasts, guys that not only drive but build their own bikes. Lots of nice metalworking there so we talked for a bit about that. One girl that was stranded came from Toronto and was on the way to Muskoka, a popular place to spend the holidays. We waited… One of the bike guys went into town and scored a pizza. One of the pizza bakers had set up in the street and was selling pizzas for $5 as long as he had ingredients. We munched our pizza, talked some more and waited some more.

Barrie is about 500 km from the Island, and around 10 pm I saw our saviors roll into the station. Two kids with huge grins on their faces and music blaring loud from the open windows. I don’t think I was ever so glad to see my own car. The honda was a veritable bomb, it was filled with cannisters as many as they could cram in. We gassed up, then gassed the bikes of the bike guys (we swapped addresses, they invited us for a pig roast which we attended and they came to visit us on the island). The girl got enough gas to get back to Toronto, and we parcelled out the remaining fuel in the same way, just enough to get as many people home as possible. No charge, obviously (people were on the whole expecting to pay the gas station with a credit card or debit card). We made a lot of friends that day :), and quite a few enemies too (all those that we couldn’t help).

So, finally done we drove back to the island. It was the weirdest ride on Highway 17 North ever. Normally even at night there is light traffic there, this time not a single car. Just lots of wildlife, deer, moose and the occasional bear that had already taken over the highway so we went pretty slow. In none of the little villages on the road were there any lights. When we got back to the island it was 5:30 am or so, the gas station was still open, the pumps were still manned and going pretty much continuously. For 75 miles around there was no gas. The distribution point had huge tanks (millions of liters) that did a gravity dump into the delivery trucks, and then the truck would do a gravity dump into the underground storage tanks at the station. So none of that required power. And then the generator driven pumps would pull the fuel out of the underground tanks and deliver it to the customers just like normal.

The outage lasted 2 days and a bit. We went trough our monthly amount of fuel in that time and we made the Sault Star (the local newspaper) after the power was restored. (Good thing too, their presses were down just like everything else, we wouldn’t have been able to cope with the demand if there had been any publicity during the outage, word of mouth we could deal with, barely).

When we bought the station one of the first things I did was install that generator. I felt that what with the station being critical infrastructure for many of the islanders that we could not be dependent on Ontarios power grid. The island was fed by cables to a step-down transformer, and after that there was a ‘low voltage’ (not really that low, but low compared to long distance lines) distribution net on the island and every house had its own drop transformer. Outages were frequent, once every couple of months, and more frequently in the winter months. Whenever lightning struck it was 50/50 that the power would go out. I imagined a situation where the power was out for just a bit longer than normal and decided that we could not run that risk. My software background makes me abhor single points of failure and to run an infrastructure business without a back-up in place seemed unwise and ran against my nature (even if the partners in the bussines thought that it was a bit over the top they still helped installing the genny).

I didn’t have any idea that there would be such a massive power outage (there had not been one like that in decades). I also didn’t know that I was going to be one the main beneficiaries of having a back-up. I definitely didn’t plan to get our business in the newspaper this way and did not realize how many people would end up relying on our silly little out-of-the-way gas station with it’s puny 2x50K liter reservoirs for a critical resource (a typical highway station has a 1/4 of a million liters). But being prepared for the occasion sure didn’t hurt. What is really scary is that apparently none of the other (much larger) stations than ours had thought this scenario through. Critical infrastructure being down is annoying enough, but the knock-on effects are devastating.

Our house didn’t have any issues at all, we were ‘off the grid’ by that time so our business worked uninterrupted. One thing that I was very impressed with was how well the cellphone network dealt with the outage, for at least 24 hours all the base stations worked, after that point they slowly dropped out one by one as their batteries ran dry.

Infrastructure is invisible, as long as it works. And only when it fails do you realize just how much we are all dependent on it and how badly we are able to cope with such infrastructure being unavailable for any length of time. No power translates quickly into: no fuel, no water, no food and so on. If this had happened during the winter instead of a nice summer day the results would have been much more severe.

There is no moral to this story, but I’d like to add one: if you’re in the infrastructure business design as if lives depend on it. One day they may, even if you can’t foresee how that is possible. Even if you’re just running a lousy gas station on some barely accessible island.

For a while now there have been tantalizing hints that Elon Musk is at it again. His usual way of approaching a problem is to look at an industry, find some fatal flaw in it, fix that flaw and then leave everybody else in the dust, wondering why they didn’t do what he did. He’s one of the most interesting people on the planet at the moment and there isn’t a week that I do not come across some bit of information that re-inforces that view. I’m not a person that will readily admit to being a ‘fan’ of someone but let’s leave it at that I find it hard to not be infected by his devil may care attitude towards all things holy in industry and convention. Recently there has been a lot of buzz regarding the ‘hyperloop’.

The Hyperloop is the code name for his plan to disrupt the transportation industry, and maybe to hedge his bets on Tesla (which attempts to do the same in a less revolutionary way), and because of the lack of details speculation runs rampant about what it could be.

I’ve seen the most fantastic interpretations of the hyperloop name, including space bound filaments kept aloft somehow that people carrying projectiles are fired into (doesn’t anybody think of the children, or the grandmothers) and other science fiction interpretations.

This is my attempt at putting together what it could be taking into account some of the common elements in Tesla, SpaceX and paypal.

One thing Elon Musk doesn’t dabble in is science fiction, he seems to be stuck solidly in the science and engineering departments so I’ll go with ‘make it as simple as possible, but no simpler’ principle. Airborne tunnels don’t have that simplicity component so I’ll happily discard them, they’re almost impossible to construct and they are terribly fragile once they’re up (and taking them down for maintenance would suck).

There are a few facts known from things that Elon has said on the subject:

“Cheaper than high speed rail”

“ground based”

“it leaves when you arrive”

“no rails required”

“It will revolutionize the transportation industry”

“It is not an evacuated tunnel”

That last one really piqued my interest. At first sight says very little, but that depends on how you interpret the sentence and I suspect that Elon may be playing a little practical joke on us here, let’s not forget he knows a thing or two about computers.

You can parse that sentence two ways to get a completely different meaning, depending on where you place your ‘brackets’, but in a spoken sentence these are invisible. So they’re a neat spot for a verbal trick:

It is not (an evacuated tunnel)

and

it is a (not evacuated) tunnel

So, in other words he may be putting the truth out there for everybody to see: it is a tunnel that is explicitly not evacuated.

Which I read to mean that he is saying that it is pressurized. Pressure is a lot easier to maintain than a vacuum is, especially if you’re building long stretches of tubing. A really good vacuum needs a considerable support structure because the atmosphere will happily crush your apparatus. There is also a low limit of how effective you can be (1 atmosphere, to be precise is the upper limit of diference that you can achieve by definition). Then there is the problem that to brace against that atmospheric pressure you have to push outward, which would be annoying because that is exactly where you want you vehicles to run, which means that if you want to make a high speed transportation device using a tunnel based on a vacuum that you’re going to have to make the tunnel double walled, which will greatly increase the cost. So clearly a vacuum would have all kinds of drawbacks and is a non-starter. Then there is the problem of convincing the general public to enter a cart in a tunnel that is evacuated, no way to refresh the air while it is moving and so on. Vacuum: bad idea.

So here is my vision for what ‘hyperloop’ stands for:

If you reverse the picture and you imagine a tunnel that is pressurized then there is a much easier way to make the whole thing work, in fact it would explain one of the two parts of the word ‘hyperloop’, the fact that it apparently is a loop, which means that it is going to recycle some critical component. (the other, the ‘hyper’ bit I assume refers to the maximum speeds attainable by such a system).

And that re-cyclable component may very well be the moving air inside the tunnel. If the pressure is supplied from the outside then it could leak away causing vehicles to have to work harder than they were designed for and the transit speed would suffer, but there is an elegant solution for this and it involves seeing the system as a whole rather than as bits and pieces.

A packet switched network of carriages that can be inserted and removed at various locations that are propelled into a slot on the ‘loop’ and taken back out again when they reach their destinations. Energy would be expended when accelerating a carriage to the speed of the loop, after that has been done on a parallel track the carriage is shunted into the loop, and energy expenditure drops rapidly. Such shunt points would exist every few thousand meters along the length of the loop, comparable to on-ramps on highways. The whole thing uses the carriages themselves as the impellers that cause the air to move through the tunnel system, effectively everybody is drafting with everybody else. So the pressurized air is pressurized by the combined motive force of all the carriages. The more carriages in the loop the lower the energy budget per carriage. Carriages would be electrically propelled, mostly seal the tunnel walls and use maglev principles or something similar to reach their speed. One hyperloop could be the ramp up to the next by keeping ‘packets’ for the same destination together, shunting them out to an intermediary phase and speeding up again for the next level. Downshifting would be the same but in reverse.

So this system would actually be cheaper and more energy efficient as usage goes up, compared with ordinary roads where congestion and fuel consumption goes up with usage.

There would be a fair amount of redundancy in the system, since if a carriage would for some reason slow down or if it would be really busy they could be made to move in such a way that a rearward carriage would start to help push the one in front of it, effectively reducing aeronamic drag on the rearward carriage to near 0, using the air between the carriages as a buffer medium and shock absorber. Pumping the remaining air between two carraiges out would effectively lock them together for the duration of the trip. Instant train formation, with some maximum length based on the radius of the loop, and even if the carriages would not be locked together from an energy consumption point of view if they get close enough the effect is much the same.

Loops around a city would be joined to other loops around other cities by long stretches of tubing, with the occasional spot for egress in case of emergencies. The whole thing resembles the internet loosely, both are packet switched, where the electrical wiring needs a return path for the electrons the tube system needs a return path to keep the moving air in the system. If traffic is light dummy carriages can be inserted into the system to keep the air moving, and as traffic increases more and more of these can be circulated out. That way the system would work even when there is only very little payload.

No vacuum (with associated risks and technical issues), simple, reasonably cheap to set up and maintain with all the active components in the carriage and with good economies of scale and redundancy. It would still cost a bundle to install such a system but compared to todays roads it will probably come in cheaper, especially in the long run asphalt is terribly costly to maintain. No traffic jams, just a fixed maximum capacity that can be expanded by installing parallel tubes.

I’m not sure what the end-points of such a system should look like, I’m imagining something akin to what I saw in a mailroom where they used pneumatic tube mail. And effectively, that is what this is, a scaled up version of pneumatic tube mail with some elegant tricks thrown in to make it energy efficient.

So, that’s what I think the ‘hyperloop’ concept stands for. Time will tell if I was right or not, I wished I had some drawing skills so I could give you an artists impression of the words above.

Hello again, Hackernews.

I tried to post this on HN but the limit is 2000 characters.

Two weeks ago to the day I asked for your help in finding a lawyer for a HN’er that was about to be arraigned in front of a Delhi judge.

The original thread:

http://news.ycombinator.com/item?id=4739649

To say that the results were overwhelming would be an understatement but I don’t have more superlative words in my vocabulary.

We got offers from people for shelter, legal advice, money and all kinds of other help and most importantly, several pointers on how to quickly obtain a good lawyer.

A special mention for Shubham Goel who found us Rohit Kaliyar, advocate in Delhi.

Rohit met with his newfound customer on arrival in Delhi (our friend was snatched from her home in Pune in a bogus criminal case) so instead of facing a scared single young person the opposition faced a legal team that was briefed as good as time permitted and the result of that was that all efforts to keep our friend under some form of custody fell flat.

Right now she’s free, still there are people trying hard to ruin her life but at least for now the main threat is under control and a sense of normalcy has returned.

In the original thread there were a lot of requests for more details, we gambled on our combined reputations to keep these details from you but still ask for your help. That gamble paid off, and if at some point this story is to be told then we’ll leave that to the protagonist, rest assured that you won’t be bored if and when it ever comes to that. But for now the details of the background behind this will have to remain out of the public eye. Our friends’ peace of mind and safety depends on keeping her identity and circumstances as much of a secret as we can.

So, thank you Hacker News, for trusting us about the urgency of the request in spite of the lack of details and for pulling together when it really mattered. On behalf of our friend: “Thank you so much for being there for me and for helping to keep me safe.”.

We hope that there will not be any re-runs of this, and that our friend is now safe from further bogus prosecution but in India not much is sure in this respect, especially not when the opposition has significant clout with the police who will apparently not be deterred by such simple things as laws. Fortunately, with the help of a competent lawyer and a somewhat more solid judiciary it seems you can get a semblance of justice. But that does mean having to live a life on the run and in hiding, which isn’t much of a life at all. Working on that.

best regards to all & thank you again,

Jacques Mattheij (jacquesm) and Daniel Tenner (swombat)

After moving to ‘the Island’, a place I won’t make too obvious here to protect the innocent and the guilty alike we settled down and built a house. We got to know a lot of people and were quite happy to employ a few of the people there to help run our various businesses.

We had money in the bank and a very positive outlook on life, the breathtaking nature and some hard earned time-off had renewed the store of energy that I had come to rely on over the years. A couple of the local people that we got to know expressed their desire to run a company of their own, and it didn’t take long before an opportunity presented itself.

The Island had two stores, one a general store where you could buy just about everything but it was on the far side of the island. The other a gas station, right next to the only access point to the island by car (technically it wasn’t an island any more since the bridge had been put in but it definitely still felt like one). The owners of the gas station were getting quite advanced in age and working a company like that in Canada, especially during the harsh winters is pretty heavy. Operating a gas station is a funny business, you make almost nothing on the gas (1 to 2 cents per liter, and of that you have to operate and service the pumps, pay for inspections, deal with occasional theft and so on), but you make it all back in the store. If you run such a station wisely it can hold its own, stay afloat and provide employment for a fairly large number of people (2 people, 2 shifts, 7 days per week) and turn a profit. So with the old owners willing to sell (they’d tried selling it before but that had not gone through for some reason) and new people willing to step up there was a possible deal to be made.

We put together a package, bankrolled by us which would result in one family having a house, one new company subletting the large garage space next to the gas station and a group of individuals that would be employed and would be co-shareholders in the gas station. It looked like a pretty good plan on paper and with the blessing of the bookkeeper and the various authorities it was put in motion.

The first year things looked to be going quite well. The man that had stepped forward to manage the whole thing had many years of experience in retail and he did a good job of making the transition from the old owners to the new one. We kept the name and we re-established ourselves as a constant factor in the lives of the local population. The gas station was quite important to the locals, it had been around longer than many of the people and there was a large element of loyalty involved. Be nice, serve up a good product, improve the store bit by bit, what could possibly go wrong?

In the winter of that first year I got the first sign of trouble. We’d done pretty good over the summer but somehow there wasn’t enough money in the bank to cover the tax bill. I really didn’t understand what was happening, all the books seemed to be in order and yet, even after the best quarter we were down enough that extra money was required to plug the hole. After discussing the situation we borrowed the missing amount. The store manager back-pedalled on his original commitment, and wanted to be relieved. This was a major problem. Even before the whole thing had started I had made it more than clear that we were willing to bankroll the operation but that we did not have the time or the knowledge required to run a retail operation ourselves and that we were relying on everybody to do their part.

A solution was found, one of the other partners in the venture stepped forward and made a good case why she should be allowed to take the position. She had been one of the hardest workers over the summer and seemed absolutely reliable. So the management change was made and things were back on track.

For about another year things seemed to be going well. We survived the winter (which had always been touch and go, the energy bills were huge), spring arrived and we were set for a fantastic summer. That year things derailed spectacularly. At some point the fuel supplier called me on my personal number inquiring when the next payment would come in. Fuel is sold in ‘drops’, a typical drop is one truckload of a mix of diesel and regular, sometimes premium (if that’s still sold, we still had one pump serving premium). The original arrangment was that I had asked specifically not to be given a line of credit. In spite of that agreement the fuel supplier had delivered three drops without payment, about $150,000 worth of fuel! Their reasoning was that since we had such a good reputation that the money would be coming any moment, and in fact they were assured to that effect by the people running the station. The call came as a complete surprise to me, I didn’t have clue that we were in arrears on anything, let alone in excess of a hundred grand.

The decision to shut down the station and liquidate the assets was a tough one. But it had to be done, there was no way we were going to survive a hole this size. But that still left the question of how we had gotten into this situation in the first place. Little by little the truth dribbled out, I’ll likely never know all the details but it seemed that some people figured that since the cash isn’t counted on a daily basis, why not take that cash and visit the casino. Surely if you gamble a bit then you can make some money, pay back the difference and nobody will be the wiser. And then, if it doesn’t work the next time you go gambling, you just take a bit more. And so on. In a span of 3 months a business that was the 3rd largest employer on the island was destroyed by those that were supposed to be minding it.

I blame myself. Trust but verify should have been my motto, trust alone simply doesn’t cut it. I should have kept a much closer eye on things. Coming from a software background I had 0 insight in the retail business, the one thing that I thought would offset this was the fact that we had a retail expert on board and that the people we were dealing with were effectively destroying their own livelihood if they would do anything against the interest of the group. But apparently not everybody is smart enough to see even that far ahead. What a pity!

So, a lot of people lost their jobs, some their house and reputations. We lost a bunch of money, the station got sold and the amount it fetched was high enough to cover all the debts (it turned out that there were many more bills that had not been paid) with a little bit of money left over.

And a very expensive lesson was learned: If you invest, do it in a business that you understand, and keep a close eye on your investment otherwise you risk getting burned.

There are lots of bad things that you can insert into your system through various ways that all seem to be represented in physical form as white powder. Heroin, Cocaine, various other chemicals that people will consume in order to alter their state of consciousness. I hate drugs, with a passion. I’ve seen up close in many people during my years in Amsterdam what drugs can do to a healthy human body and it isn’t pretty. Having a dad that was a full blown alcoholic (brilliant when he was sober, mean tempered and agressive when he was not) probably helped a lot too.

So, long ago, when I was 12 or so I told myself that that would never ever happen to me. There was absolutely no way I was going to end up an addict to any of this stuff and become a living wreck. So instead of giving in to peer pressure and joining the club, I vowed to simply abstain from any drug at all. No Beer, Wine, Spirits, smokes, hash, coke or other dope for me. I wanted to be healthy and to live as long as possible while keeping my body in as good a shape as I could. Peer pressure by the way, especially to consume alcohol or to smoke pot is absolutely enormous when you’re a teenager. The amount of scorn heaped on me and the kind of treatment I got for not joining in any of this ought to be illegal. Kids can really be dumb when it comes to things like this and pressuring someone to join you in an act against your own interest especially when the pressure is from a group aimed at an individual is something that ought to be illegal. Fortunately I can take heat like that fairly well or I would have surely given in.

My plan was to simply not start any of this, that way I wouldn’t have to stop either. (one of my weaknesses is that if I like something I tend to do it a lot, by not finding out of I like something I thought I could elegantly side-step the problem).

Good plan! If only I had realized I was already addicted when I made that vow. The white powder seems to have gotten hold of me at an age early enough that I didn’t even realize how far in I was when I told myself ‘no drugs’.

Here is my vice: sugar.

It doesn’t seem as though it is very bad compared to the other ones in the list above. After all, you can buy it (legally, for a change) on every other street corner in convenient 2 pound bags. It’s really cheap too. Buy as much as you want, there is more if you need it. The DEA is not going to lift you from your bed for moving it from the distribution point into your house, it is transported by truck in plain sight instead of by submarine or ingested plastic balls because it is not a proscribed substance. Your kids will not be placed in foster homes if you are identified as a heavy user and everybody around you will be of the opinion that it is perfectly ok that you consume this chemical.

Sugar is poison, and much as I hate it, it seems that I can’t kick the habit. It really angers me to know that this substance seems to have a hold over me that I can’t break. Periodically I’ll try to get rid of it. I drink tea, lots of it, several liters per day. And I sweeten that tea with sugar. If I don’t drink my tea my head feels like mush. That’s why kicking the habit is so hard. It is as if there is something real and tangible that I get out of all this sugar, it raises the fog and makes it easier to think about stuff. If I don’t use sugar (but still drink the tea) then I simply get (or feel, hard to tell the difference) very low on energy. All I can do then is consume media, mostly reading and sit around. I’ve tried to write code while abstaining from sugar but it just doesn’t work. I’m not sure what the link is here, all I know is that without that chemical in my bloodstream for some reason I can’t get my head in first gear, let alone second or third.

Very frustrating. Extra frustrating because I know just how bad taking in these quantities of sugar is. Sugar has been linked to all kinds of disorders, especially when used in excess. None of it instantly fatal, more a kind of steady erosion of the system. A slow poisoning. When I observe myself aside from the dependency I can’t find any real signs of degeneration yet, but that’s something that probably every addict would say. And likely that’s just a matter of time. At 3 liters of tea per day (a fairly accurate estimate), 1.5 teaspoons per glass, 5 glasses per liter that’s 15 * 1.5 teaspoons per day, or 22.5 teaspoons. A teaspoon is 5 grams of sugar, so that makes for 112.5 grams, or a kilogram of sugar for every 9 days. That’s 40 Kg per year, I started this when I was 11 or so, figure 36 years since then so about 1.5 tonnes of sugar consumed since then!

Kicking this habit is something that I’ll need to do. Tomorrow, maybe. Right now I feel like having a nice cup of tea.

http://articles.mercola.com/sites/articles/archive/2010/04/20/sugar-dangers.aspx

Yesterday I wrote up a post that I had on my list of the last couple of years but never got around to. My all time favorite project, building a windmill and some of the stories behind it.

I made a promise to open source the design, and today I finally hope to make good on that promise. This comes with a word of warning though, I am flooded with work at the moment so I do not have the time to make this a picture perfect release. I am making a lot of assumptions about the skills of the people that would attempt an undertaking like this and if you are in doubt you probably should not even try this. In some ways releasing this scares the crap out of me because the possibility of someone getting hurt is > 0, even simply building this (without ever using it) is not without risk. I almost lost a bunch of fingers mounting the magnetic drum, so please take this as a warning (and I really like my fingers, and I’m really careful). A single ‘neo’ magnet is already dangerous, 18 of the mounted to a chunk of steel is a formidable thing. No pretty pictures today, just dry technical stuff.

Welcome to this terribly messy jumble of bits and pieces that I recovered from my subversion 
repository. It's a snapshot of the state of the project after the machine was finished. 

I wished I could say I was proud of the result but that's far from true.

In this archive you will find a number of directories with a fairly large number of files
that should in theory allow you to re-create some or all of the parts that went into
making the Big Blue Windmill that is on display at 

    http://jacquesmattheij.com/how-to-make-a-windmill

and some of the tools (specifically, the software to drive the plasmacutter/3D mill and to 
visualize the toolpaths).

I promised I'd open source the design and this is my attempt at doing just that. Here are some
of the caveats regarding this design:

 (1) Only *ONE* of these was ever built. That means there are likely design flaws, errors and
     omissions that will cause problems in the first few hundred copies made according to these
     (possibly incomplete or inaccurate) plans.  

 (2) I'm releasing this information as a private individual and accept absolutely no liability
     of any sort with respect to this release of information. If you build this machine even
     if it is exactly according to these plans then I do not warrant anything whatsoever. These
     plans are provided 'as is' without any warrants as to suitability, accuracy or even being
     fit for reproduction. If you, your pet, your children, spouse, innocent bystanders or any
     other person or good are harmed, lost, killed or damaged because you use these plans then
     *YOU* accept full responsibility for that. 

 (3) These plans are now 8 years old. 3 of those years the windmill was operational in Northern
     Ontario, Canada. The climate there is fairly harsh but there may be local factors where 
     you live that affect the design in a negative way. If you don't do your research before 
     you embark on building a machine like this then you will likely end up doing damage to 
     yourself or others. 

 (4) Windmills of this size are *NOT* toys. The amount of energy in a machine this size moving
     at full speed is downright scary, you could very easily do considerable damage to
     structures and people (including killing them) in a radius of 2 Km (1.3 miles) around the 
     location of the tower, possibly even further if you put it on a really tall tower. Keep 
     this in mind when you experiment with windmills, there is absolutely no fun at all in 
     turning a school bus into a giant shish-kebab by hurling an 8' windmill blade through it 
     at 400 miles per hour. 

     If you do not understand the engineering and manufacturing challenges of building
     something like this then please go and collect poststamps or find another hobby that is
     safe and within your area of expertise. I have a large amount of experience designing
     mechanical stuff and in fabrication and the complexities and challenges involved in 
     windmill design - as well as the enormous energies involved - have left me with nothing
     but respect for the wind and its ability to destroy things and to surprise you.

 (5) If this is your first windmill by all means stop now and build something smaller and 
     simple first, see 

            http://www.fieldlines.com/ 

     the people there have tons of experience making machines that are much simpler to 
     reproduce than this one. This machine is complex, far from perfect and requires a lot 
     of tooling and expertise to build and to operate safely. Please take this serious.

 (6) This project, the code and the design is licened under the:

     http://sam.zoy.org/wtfpl/

     Wtpfl license. 

Now, with that out of the way, and assuming you are still reading:

This package partially (there is some stuff missing, but if you can't figure that bit out
you really shouldn't be building this) describes the bits and pieces that are required to 
build a horizontal-axis 2.5 KW design power 16' wind turbine. 

Such a machine is suitable for the power requirements of a single home assuming there is 

  - enough wind

  - some kind of backup system (for instance, solar power or a diesel generator)

  - a battery pack 

  - an inverter

  - a rectifier to take the AC current from the windmill and turn it into DC

A windmill is part of a system, it is not a stand-alone source of usable power and you need 
to take this into account.

The windmill itself (professionals (and snobs) call them 'turbines' because they don't mill
your mustard seeds but here in .nl where I live we call water pumpers windmills and so on 
so forgive me if I stick to the term windmill, if you have trouble with that then please
imagine it milling the wind) consists of several structural elements that give the machine
a way to attach to the post that it is mounted on, a main shaft, a rotor, a stator made 
up from laminates, a governor, a set of blades and a tail. 

Lots and lots of pictures about the construction of this machine can be found at

http://pics.ww.com/v/jacques/renewables/windmill/ 

the directory tree that goes with this package contains:

cad-drawings         All of the .dxf files that I could find 
cutter-jobs          The job files ready for transmission to the homebrew plasmacutter/mill
programs             Software used to design the various parts in a parametric way

Here is a verbal description of all the parts/construction of the windmill, all the bolts are 
copiously covered with 'never seize' aka copper grease. You will likely balk at the price when 
you buy this stuff but you will balk a lot worse when you want to dissassemble the machine
three years down the line if you don't use it. I have not listed the 'small fry', nuts and
bolts, I assume you're smart enough to figure out that if a hole is threaded for 3/8" that
that's the bolt that goes in there. All the bolts used in the construction are regular grade
hex head inox or anodyzed. Bolts that are subject to vibration are held down with locktite 
blue. 

There is no handholding here and I'm afraid I can't offer you support if you decide to build
this machine. So be prepared to do some figuring out and fixing if you find something in this
description is wrong or if you think you have a better plan. Most of the sizes used are in 
*Imperial* measurements, I was building this machine in North America. With a little bit of 
work you can adapt those measurements to the metric system, I'd go for a drum outer diameter
of 300 mm and work back from there if I wanted to build this machine using metric components.

- support structure
  (1) Main housing, upright tube
      4" od, 1/2" wall technical tubing, 12" long, hole milled in the center for the slipring
      assembly (bronze riders on trespa making good contact with sliprings around the shaft
      that connects to the tower stub) milled out at a 15 degree angle at the top to accept (3)
      which will be welded in place. The bottom end of this tube is milled out to accept two
      bearings id 2", these bearings are locked in place using lockrings, which are placed
      in a groove turned into the wall of the tube.

  (2) The support shaft. 
      This shaft is hollow so the wiring for the 3 phases can pass through
      the center and be connected to the sliprings. Another bearing is seated at the bottom
      of this tube. The shaft itself will be stationary with respect to the tower, so the 
      machine rotates around it as it orients itself with respect to the wind. 

  (3) Main housing, horizontal tube
      4" od, 1/2" wall technical tubing, 18" long 2 bearing holes cut out at each end. Bearings
      are 2" id which will accept the main shaft (4), again secured with lockrings placed into
      a groove turned into the wall of the tube. 

      http://pics.ww.com/v/jacques/renewables/windmill/dscf0272.jpg.html 

      Shows these components before welding up, in the picture you can see that the shaft and
      the stator mount are already in place. 

 (3a) the statormount
      See the file spacerring.job

  (4) The main shaft is a 2" solid high tensile steel shaft, a (4a) 3/4 " steel disc with a 
      hole od 4", id (of course) 2" is welded on to the end sticking out the front onto which 
      the governor base plate will be bolted. The end of the shaft is threaded with 3/4" 
      thread to accept the central post along which the rider will move.

 (4a) Disc welded to the main shaft, 3/4" mild steel, 12 hole pattern to mate with magnet drum
      and governor base plate (13)

  (5) 4 bearings, 3" od, 2" id (I used SKF but this is not overly critical, I'd use a quality
      brand though because a bearing failure can have dramatic consequences).

  (6) Lockrings to secure the bearings in their spots

  (7) A stub that matches the tower inner diameter, with a hole large enough to accept the 
      shaft. Keep a bit of space here, secure the machine against lifting (yes, that can
      happen, don't let it happen to you) by bolting it down from three sides with bolts
      that are threaded trough the stub *into* the shaft. 

- The stator
  (8) 48 sheets of mild steel (preferably with good magnetic properties) cut into the shapes
      described in the files statorproto2_00.job through statorproto2_47.job These sheets are
      placed on top of each other after making sure they are nicely flat, then welded *ON THE
      INSIDE* where they ride against tube (3). This is to reduce eddy currents, if you weld
      them anywhere else you will be generating a lot of heat but very little power.

  (9) Copper coils #18 wire, 100 windings per coil in sets of 6, leave wires plenty long
      so you can route them comfortably to the electrical box.

 (10) slot paper to coat the inside of each coil slot so the coild don't ride against the metal

 (11) after placing the coils liberally coat the stator with thermally stable resin to fix 
      the coils into position 

 (12) mark the 'hot' end of each coil and mark the set number to make sure you know what to 
      connect to what when you're done. 

- The governor
 (13) Cut the governor base plate according to governorbase.dxf, I made the baseplate for my
      machine out of 3/8" mild steel, if I did it again I'd use 1/2". The reason why is that 
      the bearings for the blade shafts can bind under heavy load and this may interfere with 
      the governing action leading to overspeeding. 

      A 1/2" base is a lot more sturdy (it is also a bit heavier) and will prevent this 
      binding. Another way to improve on this part is to strengthen it by welding a piece 
      of angle iron left and right of the spots where the bearing support blocks will be 
      mounted.

      http://pics.ww.com/v/jacques/renewables/windmill/dscf0289.jpg.html

      shows the governor base plate

      The governor baseplate is mated to the alternator drum, the bolts go right through the
      alternator drum into (4a) mounted on the main shaft. 

  (14) The bearing support blocks (2 sets of 3)
      These blocks are made out of mild steel and are shaped to receive a bronze bearing with
      id 1". These bearings need lubrication and there is a little lube nipple mounted in 
      each of them with a channel that reaches the bearing (which has a hole mated up with the
      hole in the support block).

      In this image you can see the bearing support blocks on the right hand side

      http://pics.ww.com/v/jacques/renewables/windmill/dscf0286.jpg.html

      Originally I was going to use only three of them but later I changed the design and added
      the three short blocks at the back. The short blocks are mounted with two bolts, the 
      longer ones (which take the brunt of the force of the blades) are mounted with 4 bolts 
      each.

  (15) The bearings (2 sets of 3)

      These are bronze bearings, 1" inner diameter with a spiral groove for lubricant. I've 
      sized them to sit just inside the bearing blocks (14), they're a press fit inside the 
      blocks.

      At the pressure end (so at the rear) there is a small 1" id needle bearing that makes sure
      the blade shafts (17) can rotate freely even when the mill is spinning. If there wouldn't 
      be a bearing here then the back of the nut that holds the shaft under tension would rub 
      against the stationary bearing housing. On either side of the needle bearing there is a 
      ring id 1", and a similar arrangement is present on the blade side. The nut at the back 
      end of the shaft holds the whole arrangement under very light pressure. 

 (16) The weights (set of 3)
      The weights are made out of 1/4" steel sheet, according to weights.dxf and the large 
      shapes on linkage.dxf. They're stacked together as shown in this picture:

      http://pics.ww.com/v/jacques/renewables/windmill/dscf0288.jpg.html

      And then welded up with 'invisible' welds. A 1" nut is welded onto one side of each 
      weight.

 (17) The blade shafts, 1" stock shaft, welded onto (18), with a small ring welded onto the 
      end 

      closest to (18) so the blade mounts ride free from the bearing blocks.

      http://pics.ww.com/v/jacques/renewables/windmill/dscf0303.jpg.html

      The shaft is threaded over the length from the back of the main bearing block all the way
      to the end, this thread will thread into the nut on the weight and will be used to secure
      the blade at the rear bearing block. (yes, this means that the back bearing block is
      touching the thread, which is not ideal, on the positive side the movement is very subtle
      and there is not much force on that part, it mainly helps to stabilize the shaft and to
      prevent binding of the linkage).

 (18) The blade mounts (3x)
      these are made of 2"x1" bar stock, threaded with bolt holes to accept 4 bolts on each side
      to fix the blade mounting plates. The blade mounts are welded onto the blade shafts. This 
      is a *VERY* critical weld, take your time, make sure it is 100% true, check for 
      deformation after welding and correct if neccesary, anneal this weld to make sure there is 
      no remnant stress and avoid hardening. 

 (19) The Rider
      The rider is a small tube of mild steel with a 3/4" hole in the center which will ride 
      along the 3/4" threaded rod sticking out of the main shaft. It has three sets of 'ears' 
      welded on to it that are used to connect the linkage (with a bolt through each hole).  

 (20) The linkage (3 sets)
      The linkage connects the weights with the rider, the linkage transfers the force exerted 
      by the weights and causes the rider to be pulled into the direction of the governor 
      baseplate.

      The rider is pushed in the other direction by the spring that is coiled around the 3/4"
      threaded rod sticking out of the main shaft. The top of the threaded rod sticking out
      of the main shaft is stabilized by running threaded rods from the perimiter of the 
      governor base plate all the way to a small steel disc at the end of the rod. You can
      also use that to connect a nose cone to protect the governor mechanism from the 
      elements.

- The rotor

 (21) The Rotor Drum
      The rotor drum is 12" od tube that with 1/4" wall, 4" high. This holds the 18 magnets (23) 
      in place. 

 (22) The rotor drum spider
      The spider is the shape sitting at the top of the rotor drum, it connects the rotor drum 
      (21) to the disc at the end of the main shaft (4a). The spider has been drilled with 12 
      holes, 6 of those holes are recessed holes for bolts that will hold the rotor drum in 
      place when the governor is taken off. 

      The governor bolts pass right through the rotor drum spider into the holes in (4a).

 (23) The magnets
      18 2"x1"x0.5" neodymium magnets are placed at equal distances along the inside of the 
      drum, aligned in such a way that each of them has about 1/4" of magnet above and below 
      the stator pack. 

- The blades

 (24) The blade mounting plates (set of 6)
      Size pieces of 1/4" mild steel 8"x8" drilled with holes to mate with (18) and drilled 
      with a hole pattern to mate with the blades (25)

 (25) The blades, made according to the coordinates in 

      jam2.back.job
      jam2.front.job

      Registration of front-to-back is critical, make sure that the Z axis registers 
      perfect or there can be substantial variation in the shape of the blade (because it 
      is very thin even the slightest change will cause huge changes in contour).

      Feel free to play with blade.py to create your own job files for your own blades.
      You will need pygame, tkinter and the scientific python package, this stuff was done
      in the dark ages (2004), chances are that you'll need to tweak things a bit to get it 
      running because of changes to those packages. 

      Sharp cookies will have noticed the governor.dxf file in the cad-drawings directory, 
      note that that is not the design that was eventually used (the shafts were made longer 
      and the a second bearing block at the back was added for more rigidity). 

      In the 'plasmacutter' directory you will find a whole pile of code that can be used 
      to read .job and .dxf files, to control a plasmacutter if you should have one laying 
      around and to simulate a 3D gantry mill on the screen of a computer to save on 
      materials. 

 I'm not sure if any of that is of any use but if you should get it to work I'd like to 
 hear about it. My email is j@ww.com, twitter is @jmattheij

 You will be able to find the files that come with this article on:

 http://jacquesmattheij.com/windmill.tar.gz (300 MB, be kind to my server, if you don't 
 need it then don't download it!)

 If you find I made any mistakes above (which I surely did!) please leave the corrections 
 in the comments thread on this page, that way if someone decides to follow in your 
 footsteps they can save themselves some time and trouble. 

 Have fun tinkering!

Windmills have always fascinated me and for a long time I was dreaming of one day building my own. Where I live there are still machines from the 1600’s that are just as magnificent today as when they were first built. When you see one standing in a field rotating and converting passing air into usable energy they look like simplicity incarnate, but nothing could be further from the truth.

They’re complicated beasts even the most humble water pumper or a decorative item in someone’s garden. As soon as it is allowed to rotate freely driven by the wind there are a lot of things that you should take into account. And if you don’t one day you’ll wake up to find bits and pieces all over your yard, or worse.

The reason for this lies in the nature of the wind. That gentle breeze cooling you down on a summer morning and a gale 10 force flattening trees and ripping up buildings are only separated from each other by a single metric, the wind speed. The amount of energy that passes through a given area goes up as the cube of the speed(*). So a windmill that is spinning happily at its designed windspeed of 10 meters per second will have to survive in an environment where the amount of energy put into the system is a very large multiple of the energy at the designed (or ‘rated’) speed. That is a very difficult engineering challenge to overcome.

There are a number of strategies for mitigating this problem: shutting the windmill down (which requires a reliable mechanism to do so) above the rated speed or gradually reducing the fraction of energy that the windmill extracts from the wind.

Even the shutdown concept, which seems to be at first glance the safest approach is quite a challenge. Once a windmill is moving shutting it down is not something that is going to work unless you’ve really thought the problem through. Mechanical brakes have been used since the dawn of windmill construction, and they’ve caused many a windmill to burn to bits before the wind could rip them apart. Typically they work best when the windmill isn’t moving yet but the usual time when you find out that you want to shut down is when the machine is already working. Friction + a ton of energy = lots of heat. If the heat has nowhere to go the machine will at some point ignite the grease in the bearings or it will simply function as a very large scale fire bow. The end result is as predictable as it is sad.

So enterprising engineers figured out other ways to deal with this seemingly simple problem. Some involve designs that will move the windmill at an angle to the wind reducing the effectiveness of the blades. Figure the surprise on their faces when they realized that a windmill once it is spinning has two stable attitudes if you discount the tail, sideways into the wind where it will slow down and head on where it will speed up until it blows apart. So that trick does not work very well with windmills that can seek their own direction. Lots of methods to do this have been tried, many of them are usable to some extent (furling as this technique is called can also be done along other axis, for instance upwards). Others concentrate on the blades themselves, changing the angle of attack so the blades will not be as effective as they are below the rated speed. This latter method is known as pitch control and it is one of the most complicated but also one of the most effective and reliable ways of dealing with overspeeding.

The design parameters of my homebrew windmill are: 2.5 KW of output at a windspeed of 10 m/s, variable pitch blades, rotor diameter 5 meters (about 16’). The total weight of the machine is about 85 kilograms (or about 170 pounds). It has survived numerous storms and worked very well supplying our house with reliable power, far more reliable than the solar panels we had used exclusively up to the point the windmill was finished. The plan was to open-source the design and to make available a list of parts. I really should get around to that one of these days, the fact that I finally had the time to do this write up means that there is hope ;) If you bought a machine with those specs commercially it would have cost about $10,000, but that would not be a variable pitch one. This machine cost a (fairly) large multiple of that, not counting our time, tooling and so on, but it could be reproduced well under that $10,000 mark if you already had all the tools and the knowledge and you didn’t have to go through a prototyping stage. Prototyping is very expensive and time consuming.

Designing a machine that size was a lot harder than I ever thought it would be. What was intended to be a one summer project turned into a two year tour of technology including magnetic theory, power generation, mathematics, mechanical engineering, woodworking, metalworking, meteorology, CAD/CAM, computer programming, electronics and aerodynamic theory. If you feel like acquiring some real world skills, go build a windmill! I never ever realized how much knowledge goes into making one of these until I tried it for myself.

The most demanding bit in my spec was that the windmill should be super reliable and should not require maintenance other than a lube job once every year or so. The place where it was put up has some of the harshest climate conditions on the planet, winters with days of -40 celsius and summers of +35 and sometimes even higher. Taking the windmill down during the winter was absolutely not an option (too much snow and no grip for the tractor used to raise and lower it) so reliability was extremely important, being without power meant that our house would be inhabitable almost instantly (no backup power grid, we did have a stand-by diesel generator which was used quite a bit before the windmill was installed).

When starting out I thought this was going to be a one-man job. I did not intend to involve as many people as it eventually took and I’m really grateful to all the people that lent a hand (or two) at fulfilling my dream and that helped with their knowledge, skills and time. Without that it would have never ever worked.

Windmills consist of several major components: a structure that holds the generator, typically a tower of some sort, an alternator, an optional gearbox (the one described here is direct drive so there is no gearbox), a rotor plate + (optional) governor that holds the blade assembly and the blades themselves, and finally a tail.

To learn more about how windmills work I decided to build a 1/2 scale model first. It probably doesn’t seem like a big difference, 15 cm (about 6”) or 30 cm (about 12”) for the rotor drum that holds the magnets but the scale model you can pick up with one hand whereas the ‘real’ thing (the scale model is just as real, it is a functioning machine) takes two adults to lift it. The idea here was that all or most of the trouble that would be experienced in the large machine would be present in a smaller one, where mistakes would be cheaper and easier to fix.

The first problem we hit when building the scale model was that even though the machine was quite simple and small, the precision required to make the pieces was well beyond what I could do with the hand tools that I had available when I started this project. Bit by bit we assembled a pretty formidable array of metalworking tools. In the 80’s I’d co-founded a CAD/CAM start-up which had taught me a lot of metalworking skills and those came in pretty handy. We bought a lathe and a mill and used these to make the drum and various other componenents of the scale model. Of course I could have outsourced the work but I really wanted to do this myself. What use are skills if you can’t use them :)?

We hit a serious roadblock when we came to manufacturing the stator, the part of the alternator that holds the coils used to generate the electric current. The difficulty was that the design called for an ‘inside out’ stator. The reason why is that I’d observed a tendency in windmills run by members of the same forum that I was active on at the time tended to spit out magnets leading to catastrophic failure. By inverting the design, putting the rotor on the outside as a drum instead of on the inside the force on the magnets would simply push them harder against the drum they were placed in, instead of attempting to eject them from a spinning core where they were on the outside. At first sight that was a no-brainer, but making an inside-out stator was a lot harder than using any one of a 100 different models used in the industry where the coils are on the inside (almost any electric motor has that arrangement) of the laminations. So we ended up having to manufacture our own stator elements.

The stator consists of a stack of thin metal sheets, each insulated from the other to reduce an effect known as eddy currents. Eddy currents are bad for two reasons, the first is that they cause the windmill to be less efficient, the second is that they tend to warm up the stator and that can lead to failure of the whole machine if it gets out of control. Coil losses are bad enough to contend with.

Each stator sheet is a circle with some mounting holes on the inside, and then a bunch of ‘fingers’ radiating outward and little edges attached to the fingers to hold the coils. Imagine a dinner plate with capital ‘T’ letters tacked on to the edge radiating outward. Drawing that in the cad tool was really easy, but realizing the drawing in metal turned out to be very hard indeed.

At this point we took a 6 month detour in the wonderful world of sheetmetal fabrication. This was a lot harder than it seemed at first. Finally we settled on using a plasma cutter to cut the stator elements from standard sized sheets of metal (a special kind, which is low on carbon and has very good electromagnetic properties). Die cutting was impossible due to the cost of the dies, but if this were ever produced in volume that would be the way to go).

A rig was built to hold the plasma cutting head and a sheet of metal, the whole thing operated by stepper motors connected to a small computer that ran some homebrew controller software to translate the design (made using ‘qcad’) into metal pieces. Building that machine and getting it to work properly was a fairly big project all by itself, in many ways more complicated than building the windmill! One interesting sub-problem was that to make the plasma-cutter gantry we needed to have - you guessed it - a working plasmacutter. We solved this by hacking one out using the hand torch to bootstrap the machine, and then used that to make the real gantry.

Working thin sheets with a plasmacutter is very difficult. The first attempts failed horribly, the sheet would deform during the cutting causing the plasma cutter to come in contact with the sheet of metal (this is called a ‘strike’) which would short out the arc ruining the plasma tip (they’re consumables but they are not cheap) and this would cause terrible deformation. During cornering the cutting speed was too low which would cause the tops of the ‘T’s to melt off because too much heat would be injected. Thin sheet is a very hard material to work using torches. Little by little we overcame these difficulties, the cutting speed by coming up with a way to make the stepper motors work much more smoothly (instead of ramping down to ‘0’ at right angles we ended up using very small arc segments or to purposefully overshoot the desired spot at speed and then to hit the 90 degree turn by making a 3/4 circle going the other way, which kept the cutting head moving at a nearly constant velocity, and any slowdown would occur in the scrap portion of the sheet).

To make the increase in speed possible the gantry was redesigned to be much lighter than the original which allowed for faster acceleration and deceleration which in turn made the cuttings much more regular and consistent. Originally the gantry system weighed about 80 Kg (about 160 pounds), the redesigned version weighed in at only 18 Kg (36 pounds). I came up with a funny way of using the cutter, I don’t think it was intended that way but it worked wonderfully. Instead of cutting all the way through some parts of the machine we’d simply score them, running at a speed much higher than the one required to make a full cut. This then leaves you with a nice ‘fold here’ line, after which you just fold them over and weld the edges up. You end up with all kinds of very rigid and precise box like structures like this. This technique was used to make the motor casings and the Z-axis support structure.

Once we finally had the stator sheets ready they could be stacked and placed in the now nearly finished 1/2 scale alternator. During testing it produced about 500 Watts at 600 RPM, which was more than we had hoped for (the reason for this turned out to be that I had missed one elemment during the calculation of the the gap between the magnets and the rotor drum, I’d simply subtracted the magnet height from the rotor drum inner diameter but of course (well, in retrospect everything is obvious) the rotor isn’t flat but curved inward there, which caused the magnets to ride slightly higher and hence the airgap was smaller than it should have been resulting in an increase in output power).

With a working scale model of the alternator in our hands we began work on the larger version. The design was scaled up and we spent a few months on making all the parts for the real thing. All of the experience we gained making the scale model paid off and the 1:1 version of the alternator worked as designed the first time we tried it.

Here you can see the finished stator mounted on the base, the slant in the stator pack is to reduce ‘cogging’, the effect caused by the magnets lining up with the stator elements. If you don’t do anything about that the machine will vibrate terribly which will reduce the life-span and it will need a lot of wind to get going which means that you will be making substantially reduced power over a given amount of time. We used the big lathe as a test rig, the alternator fixed on the tool support and the lathe driving the rotor drum. We could easily stall out the lathe by loading up the alternator!

Next on the todo list was the rotor plate which would hold the blades and the bearings that allowed the blades to change pitch. The idea behind this rotor plate was a very simple one: each of the blades would be free to rotate around a pivot mounted on the rotor plate, the three blades would be kept synchronized by tying them all to a central ‘rider’ which would go up and down on a central shaft. Piece of cake ;). It took three tries to get this right, the pitch control mechanism worked by having a weight attached to the shaft for each blade, which was pushed out by the centrifugal force and a spring that pushed the rider back up. This gave a lof of adjustment options to change the dynamic response of the governor. Spring tension, starting angle, ending angle, pre-tension and an ending stop to limit the travel of the blades all controllable with two nuts riding up and down on the central shaft. In retrospect I would change one small detail here, I would add a hydraulic or air based shock absorber to reduce the speed with which the governor rider can move up and down the central shaft. That would have made the machine a bit more gentle in its responses to sudden wind speed changes. I never realized how quickly the wind can change direction or speed and the windmill would follow these changes almost instantaneously.

Even though this was time consuming and lots of solutions were tried and discarded this was by far the easiest portion of the whole prototyping process, the governor was tricky to figure out but relatively easy to manufacture. The weights in the picture as well as the base plate were cut out of 8 mm sheet steel (3/8”) using the plasma cutter. The weights are stacked segments that are then welded up and finished. This gives the same effect as a solid but is much easier and quicker to make than milling the same structure.

After the rotor plate and the governor mechanism were done we began work on the blades. Making three windmill blades that are identical in wood is hard. Wood, being a natural material is of variable density, and moisture will cause that density to change over time. Wood also doesn’t really like to keep its shape. After trying to make some blades by hand we settled on converting our trusty plasmacutter rig to become a 3D milling machine. The size was just about right so we could make a blade in two passes (one for the front, one for the back).

At this point I found that I had a friend in Texas who really liked the project and would mail me in the evening with the solutions for the problems we’d have the next day.

He kept on doing this and after a while I jokingly told him that if he really knows what we’ll be up against tomorrow, why not come over and give us a hand. And so he did! Enter Ron Brown, Texan, woodworker extraordinaire. Without Ron this whole project would have failed miserably at the ‘making the blades’ stage, the bit that we all thought would be the easiest part of the whole project.

Ron taught us a lot about wood, woodworking, making windmill blades in general and using a 3D gantry mill to make complicated objects with repeatable accuracy.

I wrote a small interactive python program to help with the design the 3D contours of the blade which then produced a file used by the program written earlier to control the gantry mill.

Trial runs in wood are expensive so I made a simulator of the 3D mill that showed on the computer screen what would have happened if we ran on the mill. This saved several forests. After we were confident we got the geometry right we prototyped a bunch of blades in foam and when we finally had ironed out the kinks and dealt with the challenges of ‘registration’ in a two sided piece (both sides of the blade are curves) we switched to wood. A very nice lot of knot free white pine was hand picked by Ron and used for the blanks that we made the final cut of the blades from.

A few months after Ron joined our little band of windmill making friends we had three identical to a very small fraction of a millimeter absolutely flawless blades sitting on the bench with trailing edges so thin that you could read the newspaper through them. We sat around for a day just looking at them :) These were then given a coat of fibreglass and using an ingenious system that Ron came up with that would make them rot-proof for many years were mated to the rotor plate. For the first time all the components were ready.

The machine was painted a spiffy blue, the blades white, sliprings and an electrical connection box were added and we set off to place it on the tower that we’d installed over the summer.

The final assembly took place in the rain, and by the time we were done it was too dark to even see the machine sitting on top of its 20 meter high (60’) tower so we took it back down and waited until the following morning. We also had to tweak the balance a hair once it started moving, you can get only so close with static balancing alone. For proper balance you have to observe the machine moving. The slightest error and it will show up as ever increasing oscillations and vibration in the mount. We ended up balancing the machine on the ground and there is not much else that is scarier up close than a windmill of this size running at a few hundred RPM. You really have to have good faith in your design and welding skills. I must have checked the mount 50 times before trying this. The rotor weighs in at a good 40 Kg and I’d hate to think what would happen if it struck something (or someone). There is a lot of energy in a system like that and the blade tips move at roughly 150 meters per second ((3.14 * 5 * 600) / 60) (550 Km/h or about 330 MPh) when it is going at its rull speed of 600 RPM! Nothing to sneeze at.

Running it for the first time and seeing it make power was - outside of my children being born - one of the high points of my life to date and even though from an economical point of view this project was an absolutely terrible loss the amount of satisfaction of seeing a dream like that come true is truly amazing. In the end the machine produced 2250 Watts at 10 m/s winds, so a hair under design power. Probably with some more tuning of the airgap we could have gotten spot on but 10% more did not seem worth re-doing a substantial portion of the work on the rotor drum. That would also mean running the coils close to their maximum temperature and it doesn’t hurt to have a bit of safety margin there.

Many thanks to:

Ron Brown

Tom Wilkinson

Dan Bartman

Matt Bartman

Wayne Abraham (+)

Mirka Jucha

Chris Stevens

Ed Arthurs

Johannes van den Dop

Laurence & Dina McKay

All the people at otherpower.com/fieldlines.com that helped with tips and tricks, especially ‘flux’, ‘kurt’, ‘donc’ and ‘marty’. Out of sight is not out of mind guys, you may be a continent away these days but you’re not forgotten.

For many more pictures: http://pics.ww.com/v/jacques/renewables/windmill/

(*) Edited after a comment on HN pointed out an error here. Thanks HN :)