Jacques Mattheij

technology, coding and business

Please Do Not Be a One Trick Pony

Paul Simon has a song called ‘One Trick Pony’ with a bit in it that goes like this:

He’s a one trick pony
One trick is all that horse can do
he does one trick only
It’s the principal source of his revenue

In the world of programming being a one-trick-pony is not an option. What it means is that when that one technology that you’re currently married to becomes obsolete you’ll be instantaneously out of a job or out of customers. Sure it pays off to be an expert in ‘x’ whilst ‘x’ is in fashion. But that is a limited time window.

Do yourself a favor and recognize that the tech world is moving much faster than it used to and that having only one blade in your arsenal is a surefire way to eventual obsolesence. It also helps to make you appreciate other points of view in debates about languages and their relative merits (not as important as you might think) and will make you a much better programmer. The single biggest boost to your programming career will come when you master a second technology as far removed as the one that you are currently using as you can stomach.

So if you’re a die hard python fan, learn yourself some Clojure, Erlang or maybe even prolog if you’re feeling brave. If JavaScript is your thing then move to the severside for a while in a language other than Javascript. If PHP is your first love then please do go and learn some Ruby or C#. If you know how to use just one framework and it looks as if you’re going to be doing this forever then forget about that and make a non-toy project in another framework. If you’ve only done client server stuff then try your hand at embedded, learn some assembly (yes, that’s still useful knowledge, even if in a more indirect way than in the past).

Lifetime careers no longer exist. I knew a bunch of people that were charging $200 per hour to do HTML in the early 90’s. Frontpage ate their lunch. Don’t become a dinosaur waiting for the impact of the comet in your safe little life be adaptive, and start adapting before you really have to.

Oh, and Happy 2015!

Saving a Project and a Company

I often get the question what it is exactly that I do for a living and usually I can’t really talk about it due to non-disclosure agreements and possible fall-out but in this case I got permission to do a write-up of my most recent job.

Recently I got called in on a project that had gone off the rails. The system (a relatively low traffic website with a few simple pages) was built using Angular and Symfony2 used postgres and redis for its back-end, centos as the operating system for the unix portion of the system, 8 HP blades with 128 G RAM each in two blade enclosures and a very large HP 3par storage array underneath all that as well as some exotic hardware related windows machines and one special purpose application server running a java app. At first glance the hardware should have been easily able to deal with the load. The system had a 1Gbps uplink, every blade ran multiple (VMware) VMs. After interviewing the (former) CTO I got the impression that he thought the system was working A-ok. Meanwhile users were leaving or screaming blue murder and the management was getting seriously worried about the situation being out of control and possibly beyond repair, an existential threat if there ever was one.

So for the last 5 weeks I’ve been working pretty much day and night on trying to save the project (and indirectly the company) and to get it to a stage where the users are happy again. I don’t mind doing these jobs, they take a lot of energy and they are pretty risky for me financially but in the end when - if - you can turn the thing around it is very satisfying. Usually I have a pretty good idea of what I’m getting into. This time, not so much!

The job ended up being team work, it was way too much for a single person and I’m very fortunate to have found at least one kindred spirit at the company as well as a network of friends who jumped to my aid at first call. Quite an amazing experience to see a team of such quality materialize out of thin air and go to work as if they had been working together for years. My ‘plan B’ was to do an extremely rapid re-write of the system but fortunately that wasn’t necessary.

It turned out that the people that had originally built the system (a company in a former east-block country) had done a reasonably good job when it came to building the major portions of the application. On the systems level there was lots that was either inefficient or flat-out wrong. The application software was organized more or less ok, but due to time pressure and mis-management of the relationship and the project as a whole the builders had gotten more and more sloppy towards the end of the project in an attempt to deliver something that clearly wasn’t ready by the agreed upon deadline. I don’t blame them for this, they were placed in a nearly impossible situation by their customer, the (dutch, not associated with the company that built the whole thing) project manager was dictating what hardware they were going to have to run their project on (I have a suspicion why but that’s another story) and how that was all set up and so the whole thing had to become the justification of all the expenses made up front. He also pushed them to release the project at a stage where they were pretty much begging him (in writing) for more time and had made it abundantly clear that they felt the system was not really ready for production. That letter ended up in a drawer somewhere. A single clueless person in a position of trust with non technical management, an outsourced project and a huge budget, what could possibly go wrong… And so the big red button was pushed and the system deployed to production and from there it went down very rapidly. By the time I got called in the situation had become very serious indeed.

Here is an enumeration of some of the major things that we found:

When I first started working on the project the system was comprised of a whopping 130 VMs, each of those seriously restricted in terms of CPU and memory available (typically: 3G RAM, 2 or 4 cores). I wished I was joking here, but I’m not, for every silly system function there was a VM, a backup VM and another two that were make-believe running in another DC (they weren’t, that second blade enclosure was sitting one rack over). Now that it’s all done the whole thing is running comfortably on a single server. Yes, that puts all your eggs in one basket. But that single server has an MTBF that is a (large) multiple of the system the way it was set up before and does not suffer from all the communications overhead and possible sources of trouble that are part and parcel of distributed systems. Virtualization when used properly is a very powerful tool. But you can also use it to burn up just about any CPU and memory budget without getting much performance (or even reliability) in return. Don’t forget that if you assign a VM to just about every process you are denying the guest OS the ability to prioritize and schedule, you’re entirely relying on the VM architecture (and hence on yourself) to divide resources fairly in a mostly static fashion and that setup doesn’t have the in-depth knowledge the guest OS does about the multiple processes it is scheduling. Never mind the fact that each and every one of those VMs will have to be maintained, kept in sync, tuned and secured. The overhead of keeping a VM spun up is roughly equivalent to keeping a physical server alive. So use virtualization, but use it with care, not with abandon and be aware of where and how virtualization will affect your budgets ($, cycles, mem). Use it for the the benefits it provides (high availability, isolation, snapshotting). Over time we got rid of most of the VMs, we’re left with a handful now, carefully selected with regards to functional requirements. This significantly reduced total system complexity and potential attack surface and made a lot of the problems they were experiencing tractable and eventually solvable.

Application level documentation was non-existent. For the first few days we were basically charting the system as we worked our way through it, to figure out what we had, which bits went where and how they communicated. Having some usable application level documentation would have been a real time saver here, but as is usual with jobs like these documentation is the thing everybody hates to do and pushes as far ahead as possible. It’s usually seen as some kind of punishment to have to write documentation. What I wouldn’t have given for a nice two level drawing and description of how the whole thing was supposed to work on day #1.

The system was scaled out prematurely. The traffic levels were incredibly low for a system this size (< 10K visitors daily) and still it wouldn’t perform. First you scale ‘up’ as far as you can, then you scale ‘out’. You don’t start with scaling out, especially not if you have < 100K visitors per day. At that level of traffic a well tuned server (possibly a slightly larger one) is what you need. Maybe at some point you’d have to off-load a chunk of the traffic to another server (static images for instance, or the database). And if you can no longer run your site comfortably on a machine with 64 cores and 256G of RAM or so (the largest still-affordable server that you can get quickly today) then the time has come to scale out. But you want to push that point forward as far as you can because the overhead and associated complexity of a clustered system compared to a single server will slow down your development, will make it much harder to debug and trouble-shoot and in general will eat up your most precious resource (the time the people on your project have) in a hurry. So keep an eye out for that point in time where you are going to have to scale out and try to plan for it but don’t do it before you have to. The huge cluster as outlined above should have been able to support more than one million users per day for the application the company runs, and yet it did not even manage to support 10K. You can blow your way through any budget, dollars, memory, cycles and storage if you’re careless.

There was no caching of any kind, anywhere. No opcode cache for the PHP code in Symfony, no partials caching, no full page cache for those pages that are the same for all non logged in users, no front-end caching using varnish or something to that effect. No caching of database queries that were repeated frequently and no caching of the results either. All this adds up to tremendous overhead. Caching is such a no-brainer that it should be the first action you take once the system is more or less feature complete and you feel you need more performance. The return is huge for the amount of time and effort invested. Adding some basic caching reduced the CPU requirements of the system considerably at a slight expense in memory. We found a weird combination of PHP version and opcode cache on the servers, PHP 5.5 with a disabled xcache. This is strange for several reasons, PHP 5.5 provides it’s own opcode cache but that one had not been compiled in. After enabling xcache it turned out the system load jumped up considerably without much of an explanation to go with that (it should have gone down!). Finally after gaining a few gray hairs and a suggestion by one of the people I worked with we threw out xcache and recompiled PHP to enable opcache support and then everything was fine. One more substantial jump in performance.

Sessions were stored on disk. The HP 3par is a formidable platform when used appropriately, but in the end it is still rotating media and there is a cost associated with that. Having a critical high-update resource on the other side of a wire doesn’t help, so we moved the sessions to RAM. Eventually these will probably be migrated to redis so they survive a reboot. Moving the sessions to RAM significantly reduced the time required per request.

The VMs were all ‘stock’, beyond some basic settings (notably max open files) they weren’t tuned at all for their workload. The database VM for instance had all of 3G worth of RAM, and was run with default settings + some half-assed replication thrown in. It was slow as molasses. Moving the DB to the larger VM (with far more RAM) and tuning it to match the workload significantly improved performance.

The database didn’t have indices beyond primary keys. Hard to believe that in this day and age there are people who call themselves DBA who will let a situation like that persist for more than a few minutes, but apparently it’s true. Tons of queries were logged dutifully by postgres as being too slow (> 1000 ms per query), typically because they were scanning tables with a few hundred thousand or even millions of records. Adding indices and paring down the databases to what was actually needed (one neat little table contained a record of every request ever made to the server and the associated response…) again made the system much faster than it had been up to that point.

The system experienced heavy loadspikes once every hour, and at odd moments during the night. These loadspikes would take the system from its - then - average load of 1.5 to 2 or so all the way to 100 and beyond, and caused the system to be unresponsive. This took some time to track down, eventually we found two main causes (with the aid of the man that had configured the VMware and storage subsystem, to rule out any malfunctioning there). The first is that the linux kernel elevator sort and the 3par get into a tug of war over who knows best what the drive geometry looks like. Setting the queue scheduler to ‘noop’ got rid of part of the problem, but the hourly loadspike remained. It turns out that postgres has an ‘auto vacuum’ setting that when it is enabled will cause the database to go on some introspective tour every hour which was the cause of the enormous periodical loads. Disabling auto vacuum and running it once nightly when the system is very quiet anyway solved that problem.

The system was logging copious information to disk, on every request large amounts of data would be written to logfiles that would get extremely large. Disabling these logs (they were there for debug purposes) sped up the system considerably and also closed a major security hole. In a bit of a hurry the builders had made the root of the web tree world writeable and the log files were stored there for easy access by the general public, wanna-be DDOSers, hackers and competitors. So disabling these logfiles killed two birds with one stone, it significantly reduced the amount of data written to disk for a performance gain, it also closed a major security hole.

What didn’t help either is that the hardware - in spite of being pretty pricey - broke down during all this. Kudos to VMware, I can confirm that their high availability solution can save your ass in situations like that, but still, it’s pretty annoying to have to deal with hardware failures on top of all the software issues. One of the blades failed, was fixed (faulty memory module) and then a few weeks later another blade failed (cause still unknown). Highly annoying, and for hardware this expensive I’d expect better. It probably is nothing but bad luck.

Besides all of the above there were numerous smaller fixes. So, now the load is at 0.6 or so when the system is busy. That’s still too high for my taste and I’m sure that it can be improved upon but it is more than fast enough now to keep the users happy and spending more time to make it faster still would be premature optimization. We fixed a ton of other issues besides that had direct impact on the user experience (front end stuff in Angular, and some back end PHP code) but since I’ve been mostly concentrating on system level stuff that’s what this post is about. The company is on a very long road to recover lost business now, it will take them a while to get there. But the arterial bleeding has been stopped, they’re doing an ok job of managing the project now with an almost entirely new local team in concert with the original supplier of the bespoke system. Emphasis will now be on testing methodology and incremental improvements, evolutionary rather than revolutionary and I’m sure they’ll do fine.

Instrumental in all this work was a system that we set up very early in the project that tracked interaction between users and the system in a fine grained manner using a large number of counters. This allowed us to make detailed analysis of the system under load in a non-intrusive way, and it also gave us a quick way to analyze the effect of a change (better or worse). In a sense this became a mini bookkeeping system that tracked the interactions in such a way that if it all worked the way it should that this would be reflected in certain relationships between the counters. A mismatch indicated either a lack of understanding in how the system worked or pointed to a bug (usually the latter…). Fixing the holes then incrementally improved the bookkeeping until we hit a margin of error for most of these counter that we were confident the system is working as intended. A few hold-outs remain but these have no easy fixes and will take longer to squash.

Lessons learned for the management of this particular company:

  • Trust but verify

It’s ok to trust people that you hand a chunk of your budget and responsibilities to but verify their credentials before you trust them and make sure they stay on track over time by checking up on them with regularity. Demand that you be shown what is being produced. Do not become isolated from your suppliers, especially not when working with freelancers. And if you’re not able to do the verification yourself get another party on board to do that for you who is not part of the execution. Do not put the executive and control functions in the same hands.

  • Don’t be sold a bill of goods

Know what you need. If you need an ‘enterprise level’ solution make sure your business is really ‘enterprise level’. It sounds great in to have a million dollars worth of hardware and software licenses but if you don’t actually need them it’s just money wasted.

  • Know your business

Every business has a number of key parameters that determine the health of that business. Know yours, codify them and make sure that every system that you run in house ties in with that in realtime or as close to realtime as you can afford (once per minute is fine). Monitor those kpis and if they are off act, immediately.

  • Be prepared to roll back

If the new system you accept on Monday causes a huge drop in turnover by Tuesday roll back, analyze the problem and fix it before you try to deploy again. A roll-back is a no brainer, if the drop is on the order of a few percent it may be worth to let the system continue to work but if you are operating at any kind of serious level of turnover a roll back is probably the most cost efficient solution. Having a turnover drop may be explained by some other cause but usually it simply is indicative of one or more problems with the new release.

  • Work incremental, release frequently

Try to stay away from big bang releases as if your company depends on it (it does). Releasing bit-by-bit whilst monitoring those kpis like a hawk is what will save your business from disaster, it will also make it much easier to troubleshoot any problems because the search space is so much smaller.

This was a pretty heavy job physically, there were days when I got home at 4:30 am and was back in the traces at 10:30 the next day. That probably doesn’t sound too crazy until you realize I live 200 km away from where the work was done, part of the time I spent in a holiday resort nearby just to save on the time and fuel wasted on traveling. I’ve been mostly sleeping for the last couple of days, recovery will be a while after this one, I’m definitely no longer in my 20’s when working this hard came easy. Even so, I am happy they contacted me to get their problems resolved and I’m proud to have worked with such an amazing team thrown together in such a short time.

Thanks guys! And a pre-emptive Happy and Healthy 2015 to everybody reading this.

Good Consultants vs Bad Consultants

Bad consultants make money off their customers, good consultants make money for their customers. That’s it. That’s the whole trick.

If you make money for your customers you can basically write your own ticket (within reason, but reason translates into ‘more than you can spend reasonably’). Bad consultants typically enter a job at some hourly rate, pull in their buddies (also at hourly rates), start ordering hardware with kick-backs (unknown to the customer) and leave right around delivery time (or are kicked out) because what they deliver is either not cost effective or simply does not work. And then usually someone more experienced gets called in to fix the mess.

This is bad for everybody in the consultancy business. It gives us all a bad name and it makes it look as if consultants are vampires that will suck their customers dry until there is no life left in them and then they move on to the next victim. But that’s really not what consultancy is all about, that’s a separate brach of consultancy called ‘scamsultancy’. I’ve seen enough of these cats in operation to know that it must be a lucrative business but I personally can’t stand it. Real consultants make money for their customers.

You can stop reading here if you don’t feel the above addresses you, but on the off chance that it does, keep reading:

If you don’t make money for your customer but you make money off your customer then you’re doing it totally, completely and utterly wrong. Your model is flawed, your reputation will suffer and in the end both you and your customer will be behind in the game. I’ve seen companies wrecked by this attitude and it is very sad whenever people lose their jobs because some joker spent the budget on unnecessary hardware, on losing IT strategies, on giving advice that cost the company money (directly or indirectly) or that derailed the company in such a way that their competitors could take away their customers or their entire market. Please stop doing this, for the sake of your employers, their employees and shareholders and ultimately yourself.

It is all a matter of speed. Making money off a customer without giving much value (or negative value) in return can seem like a good game to be in for so called IT specialists (or growth hackers) or any one of a hundred different names these people go by. It’s a huge temptation. Clueless customer, nearly unlimited budget, what could possibly go wrong?

But that’s short term thinking, you’ll make some money inititally and then:

  • you may end up getting sued

  • you may end up getting charged for fraud (if you take money from suppliers of hardware that you ordered that was not needed for the job, in general, avoid that situation entirely, never take money (‘kickbacks’) from suppliers when working in a consultancy role, that’s not where you should be making your money, you should be trying to save your customer money, which is the same as making money for them).

  • your reputation will be in the toilet after a few of these jobs (word gets around and the IT world is small enough that this will eventually result in you not being able to get another job)

  • your customers may go bankrupt, which hurts employees and shareholders (and customers!) of the companies you pluck

  • someone will end up having to clean your mess

  • you will (in the end) make much less money for yourself

Of course the company also has a role in all this, they hired you without doing due-diligence on whether or not you are capable and have a good reputation. But if you start seeing your customers as partners rather than as victims then you will be able to make much more money in the longer term. So if money is your motivation then do yourself a favor and align your customers interests with your own. Make them money and save them money, you’ll be happier, your reputation will steadily climb and you’ll be doing better financially.

Once you get on that train the problem becomes to not adjust your lifestyle too much and to realize that one of the reasons why you can charge that much is also because you will leave when the job is done. So no job security for you, just your skills and reputation, both of which you’ll have to protect zealously. Keeping your skills up can be hard work all by itself (and that’s not typically billable time either), your reputation is entirely up to you. The temptation to make money off your customer is the biggest one for screw-ups as far as I can see.

And when all that is done there is taxes and savings. So it’s not as if you get to save a million or more every year in most countries but if you’re smart about it and don’t waste your money on stuff you don’t need and a lifestyle then you’re going to look pretty good in a very few years of reasonably hard work (in so far as anything involving just a keyboard, a screen and your head is ‘hard work’).

If Growth by Itself Was Good Then Getting Cancer Would Be Good News

Disclaimer: the cancer analogy is in no way meant to cause mental grief to anybody, I’ve lost enough family members to it that I realise such an analogy may be a step too far for some, but I feel that in this case its use is justified. If you do feel offended by this in any way then my apologies.

One of my favorite VC bloggers wrote an article that I’m pretty sure was meant to be serious but that had me in stitches. Fred Wilson, famous in VC circles and a very smart guy wrote an article that contains a whole series of gems, the best one of which was this line:

“At some point you have to build a real business, generate real profits, sustain the company without the largess of investor’s capital, and start producing value the old fashioned way. ”

The reason why I thought this was supremely funny is that if VCs start to complain that their start-ups are not behaving like bootstrappers it’s like hearing an SUV driver complain about their mileage. Who bought the thing in the first place? Who gave this money to be used as fuel to a company without a profitable model in the first place? Who thought it was a good idea to buy a vehicle that requires so much fuel that using it to haul a product you sold from ‘a’ to ‘b’ means you lose all your margin on the fuel bill?

VCs tend to get their return from a relatively small number of investments out of a much larger portfolio. The earlier stage they invest, the larger that portfolio has to be. In order to maximize the returns the typical VC funded start-up will do everything to substitute capital where you’d normally expend time. All this to chase the one number that they are all interested in, which is growth.

That would be all find and dandy by me, if it wasn’t that growth per se is not a good thing. Organic growth is a good thing, it’s healthy, it gives you time to maintain your corporate culture and it makes sure your company does not get away from you, or ends up crashing spectacularly. In biology, uncontrolled growth is called cancer, and it’s arrival is not generally heralded as a good thing. Cancer needs fuel, and in order to get at it it will rob healthy tissue of the supply lines that it needs to fuel its rampant growth. Until the organism no longer knows how to deal with these ever increasing demands on its various systems and succumbs because critical organs no longer get what they need.

Rewind to the summer of 1998. Camarades.com had launched in March and was doing very well. We were growing fast but not too fast (though ordering servers timely was a tough balancing act), we had a relatively solid ad fueled model and we did not need much in terms of capital to keep this up. It wasn’t a huge business but it was doubling in size roughly every month from an income point of view and we managed to keep our payroll limited and our other expenses in check. Not a cent was spent on marketing, all our referals were organic, people simply liked the product.

Roughly at this time we were approached by an American based company called ‘spotlife’ for a potential technology buy-out. Now spotlife was from a tech point of view doing a poor job, but they had two things that we could not easily match: $30M in the bank from a combined investment from Philips and Logitech, and on top of that access to Silicon Valleys finest. The price-point of their product was set so low that it made me wonder if they had been to the ‘Minderbinder school of business’:

“We lose money on every transaction, but we’ll make it up in volume”

In the words of their CEO (some MBA guy): “You and I are on a collission course, and we have just thrown away the steering wheel, who do you think is going to flinch”.

Famous last words, we didn’t flinch and refused the offer. Spotlife went down in flames less than two years later, having burned through the larger part of that $30M as if it was toilet paper. Fancy offices, subsidized product, huge staff, over-hyped launches, underwhelming product, you name it they had it.

So, spotlife practically killed the market for us. And they were behaving just like the start-ups that Fred Wilson is lamenting about. They spent their money to do a classical ‘landgrab’, but they forgot that you also have to hold the territory that you grab and that at some point your VC capital will run out and then you had better be able to make ends meet.

And this is where I think quite a few start-ups get it wrong. They’re inspired by the likes of YouTube and Instagram and see nothing but growth.

But in the end, for founders rather than investors there are more ways than one to the finish line and besides the fact that such fast growing companies without a clear path to profitability can spoil an entire segment (once users have gotten the product for free from your soon-to-be-bust competitor I wish you good luck in trying to charge for it) is bad for everybody in the long run. For VCs (especially earlier stage) there is only one important factor: they need a few ‘out of the ballpark’ successes to make up for the the losses on the rest of the portfolio. So their goals and founders goals are rarely aligned in this respect.

These companies are in a race to be acquired or to find a model before their precious capital runs out are spoiling the soup. And woe to the founders if they miss that window, that ‘B’ or ‘C’ round could easily result in them losing the ability to call the shots in their own house.

I suspect - but have absolutely no proof - that if you would take the total sum of the valuation of all VC backed companies that came out of SV in the last decade or so and you divided that by the total number of founders that went in and did the same for the ‘bootstrappers’ that the bootstrappers on average did better. None of them (or at least, not many of them) will be billionaires. But they’re going to have to do exactly those things that Fred Wilson wants his companies to do: concentrate on revenues and profitability rather than on growth for growth’s sake. So they’ll be building profitable companies which will then be called (derisively) ‘life style’ companies rather than ‘real companies’. As if a company with say 25 employees and $10M turnover is a bad thing or a hobby. The Go-big-or-go-home attitude runs deep and I’m perfectly ok with people attempting to do this (see three roads to the top of the mountain) but I don’t understand why simply running a good old conservative business with a substantial chance of long term survival is a bad thing. Not everybody has to play the lottery.

The silver lining on the cloud is that once these burn-baby-burning competitors go under that you usually can pick up their assets for a song once they go bust (Aeron chairs at $0.05 on the dollar are a good deal). But once you’ve fine-tuned your business model and you are ready to roll out internationally and you really need that Venture Capital (which is supposed to be used as an accelerator, not as life-support) then you will find that your locust competitors have left the field barren. That segment is now poisoned and you’ll have to slog it out on your own unless you get very lucky.

Surviving the intermediate period while they still have capital and are spoiling your niche with freebies that you can’t match is going to be a long and hard exercise in frugality and teamwork. Hang on to your core team during that phase at any price, do not under any circumstances try to imitate them at their game, that will reduce your chances of survival even further!

VCs should be far far more critical about the companies that they invest in, that the path to break even is clear and that they are not going to invest in a company whose business model is broken but where the cracks are paved over with marketing and growth by burning investors money. Especially companies where investment is made in B, C or even later rounds should be looked at very carefully. A healthy company would survive and grow even without VC investment. If it can’t do that then it probably is better to not invest at all rather than to have to write angry blog posts later.

Whose Phone Is It Anyway?

When telephones were still solidly connected to the wall, either by being bolted on to it or by an umbilical (sometimes consisting of one or more phone extension cords) it was pretty clear who owned them, the phone company did. And bad things (such as disconnection) would happen to you if you connected something to the network that wasn’t authorized.

When computers were still unconnected to each other using networks it was pretty clear who owned them too, the company or person that paid for it did.

Life was simple and all was good.

Then one day in the 60’s, before I was even born the world changed in a remarkable way though it would take decades before the implications would become clear. The idea of networked computers first saw the light of day and by the early seventies the first real networks of computers were a fact.

Telephones were still mostly tethered to walls, apart from some very important people who had access to ‘mobile phone technology’. In practice this meant that they gave up the trunks of their cars to enormous radio transmitters and some pretty bulky interface components in the interior. Such was the price of being an important enough person that your thoughts had to be available to those that needed instructions or information.

For the most part life was still simple and all was good, the few people that had advance warning of a tremendous shift seemed to ignore it. There was a thing in the 60’s called ‘the mother of all demos’ and it successfully extrapolated about 30 years of the future, up to and including the world wide web.

In the late 80’s and early 90’s mobile phones underwent a revolutionary shrinking and suddenly instead of a select few a much larger number of people got access to them. These phones were still quite bulky (the size of a toaster, maybe a bit thinner) and weighed a lot but those disadvantages did not seem to slow their adaptation at all. Suddenly not only royalty, presidents and the CEOs of the very largest companies had the power of consultation and direction at every hour of the day, it was available to just about every business person and the wealthier private individuals.

Computers were still treating being networked as ‘optional’, though in offices networks were now commonplace. Phones and computers occupied very distinct networks (though for the ‘long haul’ computer networks often employed telephone lines altered specifically for that purpose, so called ‘dedicated lines’ which had their filters removed) and phone companies still viewed the gear they rented to you as their property and the contracts in place were pretty specific on that. You paid a monthly fee for the equipment. In some countries the rules had been relaxed a bit and third party gear was permitted but in most cases only after an expensive approval process. After all, the phone company did not want any weird/dangerous/unlicensed gear on its network and infrastructure, the telephone network was - and is - critical infrastructure and only trusted parties are supposed to be on it.

In ‘95 a revolution happened. Suddenly unconnected computers went from being useful and productive machinery to sad lonely devices that did not have access to the majority of the available information resources embodied in this new fangled thing called the world-wide-web.

Phones continued to shrink and mobile phones shrunk even faster. By 1998 the mobile phone revolution was well under way and even though the phone network was still very much holy there were some inroads into the phone companies territory. For one cable companies started to offer phone services (which has since been mirrored by traditional telcos offering TV channels). You were allowed to buy your equipment rather than rent it. Clever phone company, sell you a peripheral of their network at a huge mark-up and then make you pay for it again whenever you use it for its intended purpose. I think that’s called a ‘win-win’ ;). There was no more lower limit on owning a phone, everybody seemed to have one, including kids! In just 30 years the ability to communicate world wide at any given moment of the day had gone from being a privilege only afforded the wealthiest and the most powerful to children that were still learning how to spell properly, an incredible example of commoditization.

Computer owners continued to see their machines as theirs almost without exception, and the network card was clearly a peripheral with the browser just another application on the computer. The demarcations were very clear but the pattern of convergence was at least as clear and with it came a slow blurring of the lines.

By the mid ’00s we were well under way in revolutionary territory. Suddenly pocket computers with internet connections became wide-spread. And regular computers without an internet connection all but disappeared. Providers required the installation of their software on your computer in an attempt to establish a beach-head.

Phone companies selling phones from various vendors had long also established a beach-head on your phone, the so called ‘baseband processor’. It is a dedicated computer that talks to the phone company network, the software that it runs is not under the control of the owner of the phone and upgrades and or functionality are not transparent at all.

And so we arrive today. Where you think that you own the computer that you use and the phone in your pocket. You see the network as a peripheral of your computer and your phone, with your device as a node in a network operated by ‘peers’.

But that’s not the view of the phone company and your ISP. They see that phone in your pocket and that computer on your desk as a peripheral of their infrastructure.

In smartphones the line of demarcation runs somewhere in the middle of a circuit board on a channel between the baseband processor and the main processor. But memory of the mainprocessor and other peripherals may be readable/writeable/controllable by the baseband processor. You won’t know about it as long as they don’t tell you about it. That phone in your pocket is about as nice an analogy of a trojan horse as you could imagine, you’re lugging along a ton of software that runs on behalf of others and you can’t even inspect it.

Meanwhile, on the home network that demarcation line runs roughly between your router (unless you built it yourself) and the computers connected to that router. In some cases special software has been installed on your computer (which serves absolutely no other purpose than to control what you can do with your internet connection, monitoring and control).

This situation will persist as long as people accept full fledged computers running tons of software supplied by third parties with convenience rather than data ownership as the main focus. So don’t be surprised if the phone company has access to all your contacts, to each and every email you’ve ever sent via your phone, to each and every website that you have visited (even if it was with https), to the voice contents of all your calls (historically they had that last one already but with the technology at the time such massive datacollection would have been all but impossible).

So, whose phone is it anyway? You paid for it, you should own it. But the phone company and your ISP have a different view of the world than you do.

To them you’re nothing more than a money making peripheral, just like in the old days, when phones were still wired solidly to the wall.

My New Vehicle

After looking over many advertisements for all kinds of super nice looking vehicles I’ve decided to go with this baby:

It was custom built for me by my eldest son, he spent a ton of time on it and the result is really quite nice. I had a whole pile of weird requests so buying a bike off-the-rack was out (I did quite a bit of shopping around but I could not find anything with these specs).

The demands were:

  • really good all weather braking

  • fast

  • strong

  • light (or as light as possible)

  • street legal

  • comfy

The first he solved by getting some pretty weird brakes, a hybrid mechanical/hydraulical brake set. I never even realized such a thing existed! The idea here is to have very good stopping power regardless of whether or not the rims are wet without the downside of using disc brakes, which work well but are super maintenance intensive and tend to go out of whack at the slightest disturbance. Definitely impressed with these brakes, if you hit the front wheel too hard without thinking about it you’ll be flying, and in the rain they work very well indeed.

As for speed, I like to go places when I cycle, not stand in one place so I like the old fashioned racing bikes a lot. This decided the basic geometry of the bike, a regular racing bike set-up with curved handlebars. I’ve had it up to 44 Km/h and for an old guy like me that’s reasonably ok. (And let’s not factor in the two week recovery period afterwards ;) ).

The bike is strong, the frame is very stiff and it has 32 spoke wheels for lots and lots of kilometers without having to worry too much about things coming out of alignment due to potholes or other disturbances.

It’s pretty light for a bike with all the trimmings and this level of strength. The frame is recycled, carbon/aluminum, strong where you need it and light where it matters (bought on the local equivalent of ebay for a song), the rest was bought new from a variety of bike parts stores.

Being street-legal is a must, I know that you can get away with a lot (especially in NL), but I like to be seen (lights) heard (bell), to have fenders (because it always rains) and I like to be able to go out in the evenings. Besides it being good that it is street legal from a fines perspective it is also simply good to be able to see and be seen.

It’s comfy because it has a very nice saddle and cushy handlebar tape, it also works well with my posture.

I’m still going to add a pack carrier, and maybe a secondary one in front.

All in all, a super bike and I’m totally happy with it, when I held my newly born son in my hands what feels like only a very short while ago I never anticipated that one day he’d build me a bicycle. I always felt pretty good about letting him loose in the workshop when he was still small and it looks like that confidence and trust has paid off.

Thanks son, well done!

It Is Not About the Money, Silly! It Is All About the Time.

One of the oldest business sayings that stuck into my head was ‘Time is Money.’. It’s deceptive in its simplicity, it generates in the mind of the beholder of this simple 3 word adage that there is a direct equivalent between time and money, and that that equivalence runs both ways.

Ways in which you see this at work in every days life are people working (spending their time) in order to get some money. People spending their money to give them more time (for instance, by using some form of mechanised transportation to arrive at their destination sooner, bicycle, moped, car, jet etc) is another way in which it is illustrated.

And so we all seem to have blindly bought in to the fact that in order to be able to spend money we will have to expend our time and in order to be able to maximize the time we have available we have to spend more money. And thus the treadmill was born. And once you’re on that treadmill it gets very hard to get off it.

Before you know it you’re hip-deep in debt (credit cards, mortgages, car loans, student loans) and you need to work even harder (and so spend more of your precious time) in order to keep up with the now much higher speed of the treadmill.

But nobody is born into debt (at least, not in any functional society), so at some point in our lives we derail from the path where we are debt free onto a path where we are no longer debt free and take out a mortgage on our future health, well being and free time budget in order to be allowed to enter the treadmill and then we willingly assist in speeding it up.

When I look around me I see people making endless purchases of stuff they don’t actually need, brandnew cars, houses large enough to live in with two families (and then there’s only two people living there), parents that barely see their kids because of the pressure to earn money and a continuous stream of complaints about how it is absolutely impossible to make ends meet.

I don’t understand any of it. Maybe it is because I don’t allow any advertising at all into my life but the draw of all these gadgets and the need to show off wealth by buying expensive brand clothing and so on are things that I’m a stranger to. In a way I consider myself very lucky, I’ll buy what I (think) I want when I want it and because I can (easily) afford it. Not because there is some kind of peer pressure or a sense of having to keep up.

In my family there is an interesting set of divides between people with some money, people somewhere in the middle and people with no money at all. And in my circle of friends and their extended families there is a similar set of divides. Most of the complaining about the situation comes from the middle group, which really surprises me. The people that have no money are surprisingly good at managing the little bit that they have. Month after month they impress the hell out of me by making a couple of credits stretch well beyond what you’d normally think would be possible. Absolutely superb money management skills. The people that are somewhat wealthy have little reason to complain and none of them do.

But oh, that middle group. Two incomes, 160K euros / year jointly is not exceptional. That’s a very large amount of money. And yet, at the end of every year it’s all gone. They can’t get rid of their mortgages (someone explained to me in a very serious tone of voice that getting rid of your mortgage is stupid because it is a deductible, better to wait with paying it off when you sell the house in 25 years), they are supremely bad at figuring out what the cost of all this consumption driven borrowing is and how big a difference it would make if instead of buying stuff they’d be saving for a while (or at least, lowering their mortgage).

It’s almost as if common sense about finances is something that is extremely rare. People that are by their own admission barely able to make ends meet show off their new cars, smartphones, laptops, tablets and so on. And all bought on ‘cheap’ credit. But that credit really isn’t all that cheap (read the fine print on your credit card statement, if you have a 1.9% card make sure that it isn’t accidentally per month instead of annually).

Seriously. If you are making good money, you’re in debt and at the end of the year you are further in debt or the situation hasn’t improved then learn to control your spending. Saving is so much easier than earning, and it’s a habit that once built will pay you back for the rest of your life.

And once you’re out of debt, maybe have a bit of an investment strategy going you can safely spend the surplus on things that you can actually afford. But until you reach that milestone getting rid of debt should be priority #1 and the gadgets, the new car and other luxury spending should have to wait.

And then, when you are in that luxurious position you can maybe slow down that treadmill again and recover some of your precious time. That’s the only form of capital that you will NEVER have more of. So spend it wisely.

Choosing a Web Framework/Language Combo - Ruby on Rails

In case you’re tuning in late, the previous installment in this series is here. This is an absolutely ridiculously long blog post, feel free to either totally ignore it or to read only the top portion, it will likely contain all you need or want to know to keep up to date with what’s happening here. The rest is more of a detailed log of what I did to be able to get back into it quickly if I decide that ‘Ruby Is the One’.

A picture paints a thousand words I guess:

And yes, it works, you can actually learn Romanian with it, so I guess that’s ‘mission accomplished’ :) It’s also butt-ugly, works on mobile, tablets and so on (courtesy of the template, I did not do much work on that other than to resolve some conflicts).

The short version of what the last couple of weeks were like (resulting in that screenshot) is:

  • Install RubyOnRails

  • Dive headlong into rails

  • Run aground, decide to learn the Ruby language first

  • Learn Enough Ruby to make sense of rails

  • ported the PHP command line application to ruby

  • re-write all database interaction to use activerecord

  • move all the business logic into the data model

  • port the datamodel over to the demo rails application

  • drive to Amsterdam

  • re-install RoR on the machine here at the office

  • make a ‘user creation’ form

  • make a ‘login’ form

  • figure out how to log-in a user

  • install a template and chop it up

  • make the ‘available courses’ page

  • make the learning page

  • make the feedback from the server about the answer given work

  • make a top row that shows the user info + logout link

All in all it was more work than I thought it was and I ended up spending a frustrating amount of time just searching for stuff and being stuck on simple little details. The problem with rails is not the the information isn’t out there, the problem is that there is a ton of stuff out there that simply is no longer relevant but because everybody is playing the SEO game that data will forever pollute your ability to find out what is relevant and what is not. In the end it worked out though and I’m pretty happy with the result. This of course does not say anything yet about how it would be to deploy this particular solution, it only means I’ve managed to build a first version.

My first impression is that developing on RoR is (when you know what you’re doing, always helpful) fairly quick but the result is quite fragile, far more fragile than I expected. This is in part due to Ruby the language (which Rails can’t really do all that much about) which has some peculiar quirks and in part because of the way Rails is set up, it’s all ‘conventions’ and conventions translate into ‘things you should know and learn by heart’. This makes it a bit harder to get going. I’m used to reading a program much like I’d read a book, and that’s also how I normally write programs. In comparison, working with Ruby on Rails feels like someone tore the book up and scattered the pages in a hundred different places. Ruby is a funky little language that allows you to take all kinds of short-cuts, the Ruby code was on average about 25% smaller than the corresponding PHP code. And I don’t know Ruby and I know PHP like I know my back pocket so likely with some more knowledge I could save more code.

Another thing that strikes me is that Ruby is a language that almost asks for unintended side-effects to be present in your code. And that in turn is an excellent way to get security issues, if your ‘counterpart’ knows Rails (much) better than you do chances are that they’ll find something exploitable that you overlooked.

Even so, I’m fairly impressed with how concise the final solution is, how quick it got built In spite of all the hours of frustrating searching and the continuous feeling of going very slow and working ‘against’ the frame work only 16 days have passed since the last installment, when I didn’t know anything about Ruby yet, and 3 of those days were spent on the road driving from Bucharest to Amsterdam.

This code is so fresh and most likely contains tons of security holes and bugs that I’m not confident to put it out there, besides, this is only the evaluation of Ruby On Rails and the list of frameworks that remain is a long one.

The logfiles (you *really* should stop reading here)

Warning, this is an extremely long blog section detailing weeks of work to try to get up to speed with Ruby, then a couple more to work through Rails tutorials and then more still to get a minimally working version of the SRS application up and running. If you just want the ‘verdict’ you’ll have to wait until I have done this with all the platforms so I can compare them side-by-side, think of this as a progress report and a way to segment the various platforms into one-page-each. It is written the way it is so you get an idea of the work I did, in what order I did it and what I was thinking while I did it and what issues I encountered along the way and how I (hopefully) solved them. This is probably of limited use to you unless you are currently evaluating Ruby / Ruby On Rails.

I’m going to try to keep the external representation of the various implementations as close together as I can. So starting off with a ‘full fledged’ batteries included combo like RoR is a good way to find out what is missing in other offerings, and it will also get the first web based version of this working in (hopefully) short order. I don’t recall having ever written a line of Ruby before so consider me a total noob when it comes to this particular set.

While you’re reading this keep your eye on the ball and remember that I’m trying to learn some Romanian.

I start out where I imagine everybody else starting with Ruby On Rails starts out, the getting started guide and going step-by-step.

ruby -v ruby 1.9.3p484 (2013-11-22 revision 43786) [i686-linux]

sqlite3 –version 3.8.2 2013-12-06 14:53:30 27392118af4c38c5203a04b8013e1afdb1cebd0d

gem install rails

After a long pause and a whole bunch of disk grinding and fan noises:

    "ERROR:  Error installing rails:
    ERROR: Failed to build gem native extension.

    /usr/bin/ruby1.9.1 extconf.rb

/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require’: cannot load such file – mkmf (LoadError)

    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from extconf.rb:1:in `<main>'"

Some googling later it turns out that ruby-dev is also required:

apt-get install ruby-dev

Maybe someone should update that tutorial (and retry it periodically to see if it all still works as advertised). Little gotchas like this in the first few minutes can turn off newbies and will reduce adoption.

gem install rails

More disk grinding. It’s pretty hot here today and the gem installer uses a lot of CPU cycles according to the sounds coming from the machine. The machine I’m doing this all on is pretty ancient, all of 2G of ram and a regular drive. My ‘main’ machine sits back in the Netherlands under my desk there and would probably make short work of all this but I have this one to work with and I guess it is a lot better than chiseling 1’s and 0’s into stone tablets. After a good 15 minutes it is finally done. Rails is installed :)

rails -v Rails 4.1.1

Neat. So at least it looks like once you get past that ruby-dev hickup the rest works as advertised.

On with the tutorial!

rails new blog

After a couple of minutes this prompts for my root password, then after yet another couple of minutes tons of errors. Scrolling back I find that this is caused by not having sqlite3-dev installed (a header file is missing).

Easy to correct.

sudo apt-get install libsqlite3-dev

On error, retry:

rails new blog

This asks me if I want to overwrite a secrets file, I’ve yet to modify files here so I say ‘Y’.

A few minutes later the installation completes without further issues. At the end there is this line in the output:

“Warning: You’re using Rubygems 1.8.23 with Spring. Upgrade to at least Rubygems 2.1.0 and run gem pristine --all for better startup performance.”

Ok, sounds good. How do you upgrade Rubygems? More googling leads me to:

gem install rubygems-update

followed by:


Which seems to install RubyGems 2.2.2

The tutorial next advises to start the embedded rails server from within the blog directory:

rails server

Only that doesn’t work…

/usr/local/lib/site_ruby/1.9.1/rubygems/dependency.rb:298:in `to_specs’: Could not find ‘railties’ (>= 0) among 0 total gem(s) (Gem::LoadError)

    from /usr/local/lib/site_ruby/1.9.1/rubygems/dependency.rb:309:in `to_spec'
    from /usr/local/lib/site_ruby/1.9.1/rubygems/core_ext/kernel_gem.rb:53:in `gem'
    from /usr/local/bin/rails:22:in `<main>'

In a stackoverflow thread I found a hint that I should re-run ‘gem install rails’ as a regular user:

gem install rails

Rather than as root. So I tried that.

“You don’t have write permissions for the /usr/lib/ruby/gems/1.9.1 directory.”

This is a very bad idea:

chmod -R 777 /usr/lib/ruby/gems/*

So we won’t be doing that. Let’s try again as root:

sudo gem install rails

Try again:

cd blog rails server

/usr/lib/ruby/gems/1.9.1/gems/execjs-2.0.2/lib/execjs/runtimes.rb:51:in `autodetect’: Could not find a JavaScript runtime. See https://github.com/sstephenson/execjs for a list of available runtimes. (ExecJS::RuntimeUnavailable)

Hm. What does ruby or rails have to do with finding a javascript runtime?

More googling. In this stackoveflow thread I read that I have to enable a gem called ‘therubyracer’ in a file called ‘gemfile’. I load ‘gemfile’ into vi and indeed, there is a line like that. Remove the ‘#’ in front of the line (# marks are used as a comment indicator in ruby).

rails server

Could not find gem ‘therubyracer (>= 0) ruby’ in the gems available on this machine. Run bundle install to install missing gems.


bundle install

Tons of errors, the topmost of which suggests that I should install g++ first.

sudo apt-get install g++

bundle install

At this point we’re about 1.5 hours into the installation process. To save RAM I’ve killed firefox.

Lots of grinding and about 1/2 hour later:

“Your bundle is complete!”

rails server

=> Booting WEBrick => Rails 4.1.1 application starting in development on => Run rails server -h for more startup options => Notice: server is listening on all interfaces ( Consider using (–binding option) => Ctrl-C to shutdown server [2014-05-26 22:49:55] INFO WEBrick 1.3.1 [2014-05-26 22:49:55] INFO ruby 1.9.3 (2013-11-22) [i686-linux] [2014-05-26 22:49:55] INFO WEBrick::HTTPServer#start: pid=7154 port=3000

Started GET “/” for at 2014-05-26 22:50:07 +0300 Processing by Rails::WelcomeController#index as HTML Rendered /usr/lib/ruby/gems/1.9.1/gems/railties-4.1.1/lib/rails/templates/rails/welcome/index.html.erb (34.9ms) Completed 200 OK in 325ms (Views: 215.4ms | ActiveRecord: 0.0ms)

Yay! It appears we have a working RoR installation.

Excellent. It took a while but it looks like it is working.

The built in server is quite convenient when testing code on your local host.

Next the getting_started guide walks you through a bit of configuration and then you will be able to see your test page. After that you are prompted to create a ‘resource’, by making some changes to the config/routes.rb file.

The listed example shows that you should make it look like this:

Blog::Application.routes.draw do

resources :articles

root ‘welcome#index’ end

But that throws up all kinds of errors.

rake routes

rake aborted! NoMethodError: undefined method `application’ for Blog:Module

Changing ‘Blog’ back into ‘Rails’ seems to fix this for now (my understanding of the routing dsl is so minimal at this point that I have no clue what this all means, I will read the doc on the routing in a moment, maybe then I’ll have a better understanding).

Next we add a form and there is something weird about that, the tutorial tells you to change the first line of the form to:

<%= form_for :article, url: articles_path do |f| %>

But gives no explanation why the ‘url:’ has a trailing colon and the rest of the : characters in that form are at the beginning of the terms. It doesn’t explain much of anything anyway but this is a little strange.

The explanation underneath reads:

“The form needs to use a different URL in order to go somewhere else. This can be done quite simply with the :url option of form_for. Typically in Rails, the action that is used for new form submissions like this is called “create”, and so the form should be pointed to that action.”

Hopefully this will be explained in:


So time to do some reading.

So what is it? url: or :url? According to the routing docs it is indeed url: so the tutorial got that bit wrong in the text but right in the example.

So the magic works, the predicted ‘Template is missing’ error shows up.

Next we add a bit of code to allow us to see what the controller receives.

There is this sentence in the tutorial at this point:

“The render method here is taking a very simple hash with a key of text and value of params[:article].inspect.”


“render plain: params[:article].inspect”

And I really can’t make soup of this. What hash? What key of ‘text’?

I suppose the ‘key’ refers to the name of the field ‘text’, but that value then makes very little sense (after all, there are two fields, title and text), and that still does not explain the ‘hash’.

At the end of section 5.6 we’re told to insert the following bit of code to get around a ForbiddenAttributesError:

private def article_params

params.require(:article).permit(:title, :text)


But after I do this and refesh the page the error remains.


Says: you appear to be following a pre rails 4.0 tutorial with rails 4. You need to use strong params now.

But at the top the tutorial explicitly confirms that it should work with rails versions > 4.

I use the second line in the SO post answer:

# @comment = @post.comments.create!(params.require(:comment).permit(:comment_text,:link) 

Convert its format to what I think should be required to make it work for articles:

# @article = Article.create(params.require(:article).permit(:title,:text))

Comment out the original line that made a new article:

# @article = Article.new(params[:article])

And commented out the bit that section 5.6 said I should have added:

#  def article_params
#    params.require(:article).permit(:title, :text)
#  end

And try again. Now rails complains about not having a ‘new’ action.

Checking above, I did not change ‘create’ to ‘new’. That’s an easy fix.

And now it indeed gives me:

“The action ‘show’ could not be found for ArticlesController”

Whoever maintains that tutorial certainly hasn’t been paying attention lately, and without stackoverflow and google to help I would have either thrown my computer out the window at this point or de-installed rails. Maybe this is some kind of secret plot by the RoR overlords to test newbies for the joint start-up required qualities of persistance and ingenuity? Or maybe they just don’t like newbies and want them to go away. I don’t know which it is but I can’t imagine this tutorial is leaving the people following it in good spirits.

Anyway, on we go.

I add the following line to config/routes.rb:

article GET    /articles/:id(.:format)      articles#show

And hit refresh:

/home/jam/svn/src/srs/www/ror/blog/config/routes.rb:4: syntax error, unexpected ':', expecting keyword_end article GET /articles/:id(.:format) articles#show

Most helpful. I change the line according to the format listed in an example lower down in the routes file to read:

get '/articles/:id(.:format)' => 'articles#show'

With ‘rake routes’ I verify that the route is indeed present:

      Prefix Verb   URI Pattern                  Controller#Action
             GET    /articles/:id(.:format)      articles#show
    articles GET    /articles(.:format)          articles#index
             POST   /articles(.:format)          articles#create
 new_article GET    /articles/new(.:format)      articles#new
edit_article GET    /articles/:id/edit(.:format) articles#edit
     article GET    /articles/:id(.:format)      articles#show
             PATCH  /articles/:id(.:format)      articles#update
             PUT    /articles/:id(.:format)      articles#update
             DELETE /articles/:id(.:format)      articles#destroy
        root GET    /                            welcome#index

I add the action and the view, hit refresh and indeed, the ‘article’ appears.

So far so good. One of the problems undoubtedly is that I’m a total noob to both ‘ruby’ and ‘rails’, but I suspect I’m not the only person that finds this tutorial as their first point of contact. And I just can’t get over how frustrating this whole experience is. It’s like being a failed Harry Potter, you’re being told to use all these spells that are supposed to have some magic effect, only they don’t and throw back errors at you all the time implicating you’re the one doing something wrong. And then after consult it always turns out that the spells you were given were wrong. Really annoying. Anyway, if I wasn’t the persistent type I would have never gotten into programming in the first place so I’ll try real hard to stop bitching about this but it’s getting to me. I sure hope the quality of ‘ruby’ and ‘rails’ is at a higher level than this tutorial.

The main issue I have at this point is that given the fact that Ruby does a ton of things under the hood that appear to be black magic to the user you should at least build up the students confidence that the magic actually works. Right now I feel as if a single ‘.’ or ‘,’ misplaced that does not have a relevant stack-overflow message waiting for me (assuming I know what to search for) will leave me totally stranded. You’re asked to do all these things without a proper explanation of what is going on, it’s a huge exercise in faith.

I suspect that one of the problems here is that I dove head-first into RoR without learning some Ruby first. So let’s divert for a bit and get a feel for ‘Ruby’ the language without looking at the framework.


Some things to remember (when coming from some other language):

  • Expressions work like they do in most other languages
  • functions are defined using a ‘def’ … ‘end’ pair
  • if a function has no parameters you can omit the () both during the declaration and the subsequent call
  • even if a function has parameters, if the context is unambiguous then you can omit them
  • Within a string #{varname} expands varname to the contents of varname.
  • functions can be defined only once (so no erlang like tricks based on arity)
  • you can call a method of an object by using object.method
  • variable names are not prefixed by any special character
  • irb (interactive ruby) will only attempt to process the code when it thinks it is complete (this can become quite messy with multi-line function definitions containing errors)
  • you can use the respond_to? method to figure out what an object understands, this allows for code to reflect on what a class can do and adapt accordingly. For instance, you could try to see if you received a list or a simple variable by checking of the object you received understands “each”
  • attr_accessor :someattribute will generate a getter and a setter method
  • puts “Goodbye #{@names.join(“, “)}. Come back soon!” shows that you can do just about anything in between #{ }

Something weird happened during the tutorial: I defined a simple class according to what it said on the page there and then listed the instance methods:

irb(main):043:0> class Hey
irb(main):044:1> def initialize(name="World")
irb(main):045:2> @name=name
irb(main):046:2> end
irb(main):047:1> def say_hi
irb(main):048:2> puts "Hi #{@name}!"
irb(main):049:2> end
irb(main):050:1> def say_bye
irb(main):051:2>  puts "Bye #{@name}, come back again!"
irb(main):052:2> end 
irb(main):053:1> end
=> nil

=> [:say_hi, :say_bye, :h, :nil?, :===, :=~, :!~, :eql?, :hash, :<=>, :class, :singleton_class, :clone, :dup, :initialize_dup, :initialize_clone, :taint, :tainted?, :untaint, :untrust, :untrusted?, :trust, :freeze, :frozen?, :to_s, :inspect, :methods, :singleton_methods, :protected_methods, :private_methods, :public_methods, :instance_variables, :instance_variable_get, :instance_variable_set, :instance_variable_defined?, :instance_of?, :kind_of?, :is_a?, :tap, :send, :public_send, :respond_to?, :respond_to_missing?, :extend, :display, :method, :public_method, :define_singleton_method, :object_id, :to_enum, :enum_for, :==, :equal?, :!, :!=, :instance_eval, :instance_exec, :__send__, :__id__]

Spot the weird one? That ‘h’ in there is a definition I made before defining the class ‘Hey’, and yet it is a method in the class!

Let’s try that again, completely from scratch.

jam@homebox:~/svn/src/srs$ irb
irb(main):001:0> def h(name)
irb(main):002:1> puts "hello #{name}!"
irb(main):004:1* end
=> nil
irb(main):005:0> h("hey")
hello hey!
=> nil
irb(main):006:0> class Hey
irb(main):007:1> def initialize(name="World")
irb(main):008:2> @name=name
irb(main):009:2> end
irb(main):010:1> def say_hi
irb(main):011:2> puts "Hi #{@name}!"
irb(main):012:2> end
irb(main):013:1> def say_bye
irb(main):014:2> puts "Bye #{@name}, come back again!"
irb(main):015:2> end
irb(main):016:1> end
=> nil
irb(main):017:0> Hey.instance_methods
=> [:say_hi, :say_bye, :h, :nil?, :===, :=~, :!~, :eql?, :hash, :<=>, :class, :singleton_class, :clone, :dup, :initialize_dup, :initialize_clone, :taint, :tainted?, :untaint, :untrust, :untrusted?, :trust, :freeze, :frozen?, :to_s, :inspect, :methods, :singleton_methods, :protected_methods, :private_methods, :public_methods, :instance_variables, :instance_variable_get, :instance_variable_set, :instance_variable_defined?, :instance_of?, :kind_of?, :is_a?, :tap, :send, :public_send, :respond_to?, :respond_to_missing?, :extend, :display, :method, :public_method, :define_singleton_method, :object_id, :to_enum, :enum_for, :==, :equal?, :!, :!=, :instance_eval, :instance_exec, :__send__, :__id__]
irb(main):018:0> Hey.h "hello"
hello hello!
=> nil

Yup, that’s reproducible. So a class absorbs any methods defined separately just prior to that class?

Weird. I can also still call ‘h’ outside of the class:

irb(main):019:0> h("hello")
hello hello!
=> nil

irb(main):020:0> say_hi "jacques"
NoMethodError: undefined method `say_hi' for main:Object
    from (irb):20
    from /usr/bin/irb:12:in `<main>'

But the class methods are not accessible without an instance of the class. I don’t understand this, but I assume it is somehow intentional. Maybe someone more versed in Ruby can explain what is going on here.

=> [:say_hi, :say_bye]

Shows only what we defined in the body of the class.

The cliff notes to ‘Why’s poignant guid on ruby’, about as short a crash course on Ruby as you’ll get:

  • variables are all in lowercase
  • strings can be made with both single and double quotes
  • constants start with an upper case letter, so Constant is a constant
  • :symbol is a symbol
  • instance method invocation works like this: variable.method
  • class method invocation works like this: Classname::method
  • the normal argument passing looks like lots of other programming languages: variable.method(param1, param2)
  • ‘kernel methods’ are core language functions such as ‘print’ they don’t require an object to be attached to, the default object Kernel:: is always searched if you type a method name.
  • global variables have their name prefixed with a ‘$’
  • instance local variables start with an @ sign, their scope is the local object
  • class variables start with @@, their scope is the whole class
  • code blocks are { between }, or between do and end, these are equivalent
  • blocks can have 0 or more arguments between || characters just inside the block
  • ranges look like this: (1..3) or (‘x’..’z’)
  • arrays look like this: [1, 2, 4]
  • hashes http://www.ruby-doc.org/core-2.1.2/Hash.html – a ‘hash’ in ruby parlance is a dictionary that can only store definitions, they look like this: { “a” => “test”, “b” => “you know” } – you get the value stored at key ‘a’ back out of a hash like this: thehash[‘a’]
  • regexps look like this: /regexp/
  • Interesting: 5.==(6-1) returns ‘true’, so == is a method like every other
  • << concatenates
  • .reverse reverses a string

The guide lists “string”.each as a valid construct, but the interpreter complains when I try that. Apparently ‘each’ is no longer available, you now have to use ‘each_char’. I wonder why one would deprecate an existing behavior, breaking existing code and then to re-implement that behavior using a new name. It seems like a pointless exercise to me, nothing changes except that a bunch of people end up having to debug their code. And in an interpreted language such changes are extra nasty because you don’t get the benefit of a compile time check to see if your code is still at least correct at that level.

  • string search/replace is called gsub, it is a method of the string class so you use it as string.gsub(“old”,”new”)
  • affixing ! to a method name does the operation ‘in place’ instead of returning a new result, it doesn’t seem to work for all operators though, a.+!(3) is not the equivalent of a+=3 (which works fine all by itself…), so the ! is part of the method name, not some kind of modifier that makes any method destructive

  • File::read(“filename.txt”) will return a string with the contents of the file

  • File::open(“filename.txt”,’r’) do |f| block end executes the code in block with the filehandle of the open file in ‘f’
  • f << string where ‘f’ is a file descriptor will concatenate the string to the file contents (what about seek? does it write or only concatenates??)
  • very clever dir globbing example: Dir[‘*.txt’].each do |file| system “cat ” << file end
  • there are multiple ways to do the same thing. t = [:a => ‘e’] + [:a => ‘d’] is equivalent to t = [ {:a => ‘e’}, {:a => ‘d’} ]


“Yukihiro Matsumoto: Ruby inherited the Perl philosophy of having more than one way to do the same thing. I inherited that philosophy from Larry Wall, who is my hero actually. I want to make Ruby users free. I want to give them the freedom to choose. People are different. People choose different criteria. But if there is a better way among many alternatives, I want to encourage that way by making it comfortable. So that’s what I’ve tried to do. Maybe Python code is a bit more readable. Everyone can write the same style of Python code, so it can be easier to read, maybe. But the difference from one person to the next is so big, providing only one way is little help even if you’re using Python, I think. I’d rather provide many ways if it’s possible, but encourage or guide users to choose a better way if it’s possible. ”

  • t.sort_by { |e| e[:a] }.each do |t| print t end # sorts ‘t’ by key ‘a’ for each element e by retrieving the value of e[:a]. That’s pretty concise!
  • next functions like ‘continue’ in ‘C’
  • break like it does in most other languages (it breaks out of the innermost loop)
  • a switch statement looks like this in Ruby: case when value when value when value end (a bit strange how they chose to use confusing names for widely accepted constructs, it’s like exchanging ‘-’ and ‘+’ just because you can)
  • return value of a function is the value of the last expression of that function
  • (5..10) === 7 evaluates to ‘true’, because 7 is between the range start and end values

Slight detour through:

http://lambda-the-ultimate.org/node/934 http://www.tbray.org/ongoing/When/200x/2005/08/27/Ruby

Because of Ruby’s variable scoping rules.

Since the _why guide was written Ruby has evolved quite a bit, for instance, the block variable scope is now a lot more sane, the example on page 48 (fortunately!) no longer works as advertised. Variables declared outside of a block are no longer affected by what goes on inside of a block. See here:


  • classes are defined like this: class ClassName bodyoftheclass end
  • 5.next gives you the next number after 5
  • (5.5).floor.next first truncates the float to an int, then calls next
  • object.class returns the class for that object
  • b = a.class.new -> make another object ‘b’ of the same class as object ‘a’
  • you can reach the current object from within a class definition by using ‘self’
  • you can add new methods to existing classes, including kernel classes! (bad idea (tm))

For instance, this is how you can add a ‘square’ method to numbers

irb(main):152:0> class Fixnum
irb(main):153:1> def square
irb(main):154:2> self * self
irb(main):155:2> end
irb(main):156:1> end
  • respond_to?(“methodname”) applied to any class tells you if that class has a method ‘methodname’ (or should I say ‘knows how to respond to a message ‘messagename’)?
  • you can slice strings like this: “abcde”[3..4]
  • you can split strings on delimiters using “ab-cd-ef”.split(“-“)
  • new class creation: class ClassName < parentclass; end (why does it need that ‘;’ there?)
  • you can query the parent class with the superclass method: ClassName.superclass
  • modules govern namespaces
  • to define a class with all the entities in a module you use ‘extends’ as in ‘class X extends ModuleName; end’
  • a *argument to a function will be passed in as an array
  • attr_reader will create getters for the symbols passed as arguments, for instance attr_reader :test :test1 will create reader functions for variables with that same name
  • attr_accessor does both readers and writers, attr_writer of course only writers
  • class << ClassName is used to add class methods (methods that can be called on the class rather than on an instance of the class)
  • a mixin is a module ‘included’ into a class, it instantly endows the class with all the stuff in the module
  • reading and processing the contents of a web page: require ‘open-uri’ # Searching all found items containing the word `truck’. open( “http://preeventualist.org/lost/searchfound?q=truck” ) do |truck| puts truck.read end
  • similar, but line-by-line processing: require ‘open-uri’ open( “http://preeventualist.org/lost/searchfound?q=truck” ) do |truck|
     truck.each_line do |line|
     puts line if line['pickup']

    end end

I don’t particularly like the ‘if x’ conditional appended to that line, it feels a bit backwards to me. Imagine the instructions for disarming a bomb read ‘cut the red-green wire if the blue wire is still connected’. boom. Too bad, you should have parsed the instructions first before you started to execute them!

  • yielding is a way to let two blocks of code communicate, the ‘inner’ block passes information to the ‘outer’ block with every time yield is executed. Or should I say ‘inner blocks pull data from outer blocks whenever they need to’? It’s reminiscent of co-routines (which ruby also has: http://rubydoc.info/stdlib/core/1.9.2/Fiber), one routine the producer, the other the consumer.

  • ‘brief’ style versus verbose style (both are identical in function):

    open( “idea.txt” ) { |f| f.read }

    open( “idea.txt” ) do |f| f.read end

  • traits in a parent class allow for elegant subclassing without having to pass in tons of (unnamed) parameters in the initializer use them like this in the parent class ‘traits :traita, :traitb, :traitc’

  • @x ||= {} is a little trick to ensure that a variable exists before it is used. The ||= says ‘if this varible does not exist yet then make a new one of this type, but if it does exist then adding an empty list to it won’t change anything’. I’m not sure why that works, you’d say that the ||= should fail if x does not exist yet, because it’s syntactically equivalent to x = x || {}

  • I got a little lost around page 109, the metaprogramming bit. It looks like one program is writing another, and not only is it able to evaluate code that wasn’t present in the original program, it can be used to construct entirely new objects / classes and so on. This is powerful but also asking for trouble because it can become very hard to reason about such programs. It’s akin to macro expansion, but more powerful.

  • you can overload just about everything too, including the basic operators

This probably makes ruby absolutely great at writing DSLs but it will be a nightmare to debug because you’ll never know what you are looking at means without backtracking very precisely through how you got where you are. Imagine looking at some program and seeing ‘a = b * c’. In most languages it would be fairly obvious what this does, the product of b and c is stored in variable a. Maybe b and c are arrays or matrices but at a minimum you’d expect some kind of multiplication to be going on there. In Ruby this is not the case. ‘a = b * c’ could mean literally anything, the only thing you can distill from that bit of code is that b and c are somehow combined using the operator ’’ and that the result of that combination is stored in a. Context decides whether this is two variables being multiplied or two battle ships exchanging fire or two images being compared. For a lark, someone could have decided to turn * into the division operator… Powerful, but also very* dangerous, hard to debug and easy to have unintended side effects.

I should re-read the portion on meta programming when I understand Ruby a bit better.

  • string formatting in Ruby: “abc %s and %d” % [“jaja”, 3]
  • inside double quotes #{ } allows you to escape to run ruby code (security concerns here, this goes way beyond variable substitution?)
  • reminder: globals start with a $
  • you can generate exceptions using ‘raise ErrorNo, “Description”
  • you can recover from errors by using the rescue label
  • you can create an array of strings quickly by using %w{a list of words} this gets transformed into [‘a’, ‘list’, ‘of’, ‘words’]
  • %x{} -> execute in a shell, %Q{} -> acts like a regular double quoted string
  • if ‘a’ is a list and you do ‘a = b’ then b contains a reference to a, not a copy of the list so modifying b modifies a and vice-versa
  • you can make an actual copy by using the .clone method, and then there is ‘.dup’ to create copies of objects that have been locked http://ruby.about.com/od/advancedruby/a/deepcopy.htm
  • you append a value to an array using the << operator (contrary to what the stackoverflow article on the same question says!), so a << 5 appends the value 5 to a pre-existing array a.

If you run irb with

irb –readline -r irb/completion

You can type the name of an object, append a ‘.’ and then hit Tab, this will show you all the methods you could call.

For some reason the default ruby install does not come with the embedded documentation (ri).

Ok, that’s the end of the ‘Poignant Guide’, thanks Jonathan. (some joker now has _why’s domain so I won’t be linking there from here).

Porting the command line app

The PHP command line application is as good a place as any to start working with ruby. It’s not overly large and has all the business logic in it and needs to work with the database. Let’s port it to ruby to see what that feels like.

First we need to install the mysql client development libraries:

sudo apt-get install libmysqlclient-dev

Next, we need the mysql gem:

gem install mysql

Now ruby scripts should be able to access mysql databases.

A quickie check in irb if that’s true:

irb(main):001:0> require "mysql"
=> true
irb(main):002:0> con = Mysql.new('localhost', '', '', 'srs')
=> #<Mysql:0x8a48f14>
irb(main):003:0> rs = con.query('select * from users')
=> #<Mysql::Result:0x8a190c0>
irb(main):004:0> rs.each_hash { |h| puts h['nick']}
=> #<Mysql::Result:0x8a190c0>
irb(main):005:0> con.close
=> #<Mysql:0x8a48f14>

Looks like it works :)

About 20 minutes into porting the console application I hit a bit of a roadblock. The standard mysql library module does not allow you to fetch a result from a prepared statement. This results in terribly ugly code which causes a one-liner to become something that spans a couple of lines. In the PHP world I use my own wrapper library for stuff like this, but for ruby I decide to see what’s available out there (this also gives a bit of an idea about how the ruby eco system works wrt to contributed software). I find this: http://sequel.jeremyevans.net/

gem install sequel

installs it, and we’re off again. I’ll change the bits of code written up to that point to use the sequel gem, it looks pretty easy to use. I could use ‘activerecord’ as well I guess but why not give this lightweight module a shot and see how far it takes me.

hash = $dbcon.fetch(“select * from users where nick=’%s’” % nick).all[0][:pwhash]

that looks manageable. Still has an ugly * and .all[0] in there that’s technically not needed but that’s fixable:

$dbcon = Sequel.mysql("srs")

# fetches the first record of a set

def $dbcon.single (q)

# fetches the first value returned by a query

def $dbcon.simple (q)

This is a little snippet from the db initialization code. To ‘fix’ the Sequel class we add two new methods to the class, one that will simply return the first result from a set, another to return the first field from the first result from a set. This allows you to do:

hash = $dbcon.simple("select pwhash from users where nick='%s'" % nick)

That looks better. Of course modifying code from the outside like this is a bit (ok, a lot) hackish but I want to get a feeling for what Ruby can do and even if this is probably not idiomatic ruby the fact that you can extend libraries so easily is a powerful concept (and probably a very dangerous one, imagine if the vendor of ‘Sequel’ decides to add a ‘single’ or a ‘simple’ method to the base class with different meaning! So for anything but this ‘toy’ program this is probably a very bad idea.

Next up, the hash comparison. The passwords in the system are hashed using ‘bcrypt’, with a workfactor of ‘10’ and the blowfish algorithm. This results in hashes with the prefix $2y$10$ followed by the salt and the hash. Testing these passwords with the default Ruby ‘BCrypt’ library results in a non-match, even with the correct password. It turns out that the maintainer of the BCrypt gem made a decision to add ‘2x’ instead of ‘2y’ and that any ‘2y’ passwords will not match, even if the lib is perfectly capable of doing the verification.

The solution? Replace the ‘y’ with an ‘a’ and then the comparison will succeed for correct hashes. See: http://stackoverflow.com/questions/20980859/using-bcrypt-ruby-to-validate-hashed-passwords-using-version-2y and https://github.com/codahale/bcrypt-ruby/pull/91 lists an open pull request with a fix.

This little gotcha cost me about 2 hours to figure out, but for now the ‘replace-the-y-with-an-a’ workaround seems to do the trick.

Ruby does not have a ++ operator!! So no increments of integer variables without doing something like += 1.

No big deal, but I’m sort of used to this and it would come in handy when porting code. But using iterators most of the increments should disappear. Still, it wouldn’t cost anything to include this operator in ruby and it’s surprising that such a simple thing is missing.

Interesting: Porting the php code to ruby turned up a bit that I should have extracted into a function (the loop where the choice was made which course to study). The reason why I found that this should have been a function was that there simply was no elegant way to do it ‘in place’, but then once the function was made it looked a lot better!

Ruby’s inconsistent use of {} for blocks is driving me crazy. I never know when they are permitted, expected or obligatory!

loop {


Is fine but

if a {


Is not… Just check this out:

puts [1,2,3].map{ |k| k+1 }
=> nil
puts [1,2,3].map do |k| k+1; end
=> nil 

What is happening here is that the {} have different precedence than do .. end. Very confusing, even after multiple tries I can’t find a way to convert the curly braces example to a do .. end example.

Maybe something like this:

irb(main):029:0> [1,2,3].map do |k| puts k+1; k+1 end
=> [2, 3, 4]
irb(main):030:0> [1,2,3].map { |k| puts k+1; k+1 }

At least now they both behave identical.

A nice little ‘gem’, wirble, it adds syntax highlighting and auto completion:

sudo gem install wirble

see http://tuxdna.wordpress.com/2011/11/13/autocomplete-and-colorize-your-ruby-shell-irb-with-wirble/

From here on forward porting the command line version of the ‘learn’ script to ruby went fairly straightforward, a couple of minor glitches and a few functions that I had to find a work-around for but nothing that I would not have experienced going the other way (from Ruby to PHP).

I note that the lines on average are a bit shorter than in PHP, they look cleaner but there are definitely more of them, in part this is because Ruby is so terse at times that you feel the need to explain what’s going on. Code + comments = a constant…

Part of this lies in that PHP basically contains anything you would like to use as soon as you start the interpreter in terms of crypto, database, string transliteration, json and so on. With ruby you have to include that explicitly.

Of course there is a very good chance that I simply do not really know what I’m doing in Ruby just yet. Another thing I noticed is that because not everything is in the ‘base package’ you have to resolve external dependencies all the time, there is always ‘one more gem’ to install and it could be quite easy to either forget such a gem or to find that when you build up a machine that your ‘favorite gem’ is no longer being supported. For this reason I’ve always tried to rely as little as possible on external dependencies but this is the ‘ruby way’ I guess. PHP has this too (pecl) and I’ve stayed away from that for much the same reasons. But in PHP land you can get away with that, in Ruby land you need to include those external dependencies just to get basic stuff.

All in all porting the learn.php program took less than a day, I learned a bit more about how ruby works under the hood and I have a better grasp of the syntax. When I’m more proficient at Ruby I should probably re-visit this to see if I can do a better job of it, which hopefully will make the program more compact and/or easier to read.

Here are the cloc stats of both the ruby and the php version:

$ cloc learn.rb
http://cloc.sourceforge.net v 1.60  T=0.03 s (30.1 files/s, 12351.2 lines/s)
Language                     files          blank        comment           code
Ruby                             1            123             92            196
PHP                              1            105             57            193

That’s very close. In fact, if you discount the functions that were added for compatibility with php then ruby comes out slighly ahead.

To gain some more experience, I got rid of the sequel gem and re-did the whole thing using active record. This also lines me up a bit better for using Rails.

Extremely frustrating moment. Examples all over the place on how to use ActiveRecord with ruby. If I use one of these examples (the most trivial), for instance:

Most of those examples/documentation give absolutely no indication what version of activerecord / ruby / possibly rails they are for.

So this works:

>> u = User.where(:nick => 'jam').first
=> #<User id: 2, nick: "jam", pwhash: "$2y$10$Hpp9rAx9d/eWIFi.E.f7pu1YDV4qfzfAwGvMpBzF6wb...", score: 6055, email: "jacques@mattheij.com", newcardsperday: 20, cardspersession: 25, decay: 3, language: "en", initialinterval: 5>

And this does not:

>> u = User.find(:first)
ActiveRecord::RecordNotFound: Couldn't find User with 'id'=first

Even though that example is used in just about every tutorial out there.

Finally I found this guide which appears to be up-to-date and relevant:


Usercourse.joins(:user).where(user_id: 2)

Tbh, after fiddling around with activerecord for a day or so now I’m really wondering how ORMs improve over SQL, but that’s probably just the novelty and the frustration talking. For instance, if I have a database with users collection scores I could do this: “update users set score=score+10 where id=1” without having to worry about locking records or anything like that. A user logged in more than once on the system could hit the server multiple times and it would all work like clockwork. But using an ORM that o-so-simple problem suddenly gets a lot harder. Just using user.score += score; user.save is not going to cut it! After all, who is to say that in another thread the exact same thing isn’t happening, causing one of the updates to be lost! Anyway, that’s a contrived example but you can see that ORMs are not a silver bullet by any means. What they do though is make your code more compact, hopefully easier to read.

The activerecord version of the console application in Ruby clocks in at 164 lines of code, but a good part of that is the elimination the ‘grade_answer’ function and inlining it. The reason why that was feasible is that the active record code was so much more readable compared to hitting the database directly using SQL statements that the duplication of the branches in the grade_answer function and the switch statement jumped out at me. So I merged the two and eliminated the function, resulting in a slightly longer version of the process_answer function but all in all still a good chunk of code elided and a cleaner overall flow. Not that that matters in a toy project like this but in a larger project such savings can add up.

Language                     files          blank        comment           code
Ruby                             1            100             86            161

I should probably work a bit more on this code to move all the business logic to the model rather than to have it spread out all over the program. Especially the fetching of the next card to learn, grading and rescheduling are all bits that should live in the model.

Ok, back to our scheduled program of learning Romanian… Further reading to be done:


I like the concept of migrations, but for now I’ll stick with doing schema maintenance through ALTER and CREATE TABLE statements in mysql, I don’t like the idea of getting so tied into any one eco system that the price of switching becomes too high, it would make evaluating other language/framework combo’s harder skewing the choice towards ruby/rails before all the data is in.

For some reason active record never detected the relationship between the various tables that are present in the schema. Not sure what is going on there, it’s easy enough to fix using some magic in the model but this should have been handled automatically, otherwise what’s the point of defining the relationships in the schema to begin with? I also suspect that under the hood there is a lot more chatter between AR and the database going on than I’m currently aware of but since this is a prototype any kind of attention spent on that would be a serious case of premature optimization so we’ll let all of that rest for now.

Another thing that worries me is the start-up time of the ActiveRecord module. Initializing the model takes well over 1 second on this machine(!) and I’m the only person using it. Ok, it’s a slow box, but still, that seems ridiculously slow to me.

So, after a gigantic detour this is the page where we left off:


In section 1.2 I now finally understand what they meant with this snippet:

<%= link_to 'Patient Record', patient_path(@patient) %>

All it does is reverse the route, given a patient it creates the link to that specific patient.

notes for the routing guide:

  • ‘resources :photos’ generates a whole pile of default routes for standard crud operations on ‘photos’
  • “get ‘profile’, to: ‘users#show’” simple routes look like this
  • singular resources still map to plural controllers, so ‘resource :customer’ will map to the customers controller
  • you can nest controllers inside namespaces like this: namespace :xxx do resources: objects end, this makes /xxx/resources/ etc

This is extremely compact, but there is some risk here as well, you could be creating more routes than you’re aware of and with default controller actions that might cause trouble. Do controllers exhibit default actions?

Here is the spec for all the actions that we want our web application to perform:

  • *the usual account creation
  • *login
  • logout
  • *generating a list of available courses
  • *a single page with a larger description of the courses
  • *a way to start learning a course

  • *a page where the actual learning takes place

  • *a place where the answer to a question gets posted
  • a page where you can track your progress
  • a page where you can view your profile details, courses selected etc
  • a page where you can add cards

Only the actions marked with a ‘*’ are required for the evaluation.

Since it all starts with the routes, let’s lay those out first:

resources :users, :courses
post '/users/login', to: 'users#login'
get  '/learn/:filter', to: 'usercards#learn'
post '/learn/:usercardid' to: 'usercard#answer'

Ok, back in NL 2275 km of driving later…

Re-play the installation of Ruby and Rails on my machine here:

apt-get install ruby

Ah no, that installs 1.8

apt-get remove ruby apt-get install ruby1.9.3

(why don’t they make that the default?)

apt-get install sqlite3 apt-get install libsqlite3-dev gem install rubygems-update update_rubygems apt-get install g++ gem install execjs apt-get install libmysqlclient-dev gem install mysql

gem install therubyracer

for some reason therubyracer did not work for me this time around but installing nodejs seems to have done the trick. Weird dependency!

apt-get install nodejs

gem install rails gem install bcrypt

mkdir ~/test cd ~/test

rails new blog

rails server

=> Booting WEBrick

So even though that’s the second time I’m doing this it still required some fixes to the process. I’ll need to test this on a VM to make sure I really have all the requirements down now.

Fragile! Anyway, it works (again). On to more productive matters.

(verified that ruby learn.rb works again as well).

In preparation to making this into a rails website, I’m first going to move all the business logic into the model. Hopefully that will survive a transition into a web app intact.

How do you ensure that for a given ruby / ruby on rails application all the relevant gems have been installed?

Ah, that’s what ‘bundle install’ is for.

added the bootstrap gem

gem install twitter-bootstrap-rails

Update the gemfile, then run bundle install again

I installed a template from one of those template sites (http://html5up.net/verti).

Installing a template is a bit messy, you have to rip it apart and place the various pieces where rails expects them, and even then it can be quite a chore to work out why things are not doing what they’re supposed to. Rails has an ‘asset pipeline’, a piece of magic that tries to compile all your css and whatnot for speed but frankly it is more trouble than it is worth. In the end I stuffed all the css files in /css/ and did a bunch of @imports to include them in the layout. Not as nice and probably not as fast but for now it works, I can re-visit this later if I want to continue with rails.

So, now we have ‘basic navigation’ and we can add the links to the template. I need to figure out how to log in a user / log a user back out again.

What’s a real pain is that every time I find a tutorial or some piece of information on how to achieve ‘x’ in rails it is invariably out-dated, deprecated, no longer maintained or just plain wrong. For instance, a tutorial on how to create a user signup process in Rails contained a whole bunch of errors, bad advice (which could lead to security issues) and deprecated stuff. Fixing that took a long time.

You’d never guess it but that ‘comment’ at the end of the line is not a comment. This had me stumped for a long time.

validates :password, :password_confirmation => true #password_confirmation attr

So, Ruby -> Rails -> JavaScript (Unavoidable) -> Coffeescript? Wtf. So I have to learn yet another language to be able to use rails. This sucks. By the way, the information on how to do stuff is literally all over the place. Railscasts (what? I have to watch a video in order to do something with a framework? I really would simply like to see a solid example that does not involve me hunting across 12 different websites to pull all the pieces together, let alone some video that was likely out-dated on the day that it was made. Forgive me for being grumpy but if frameworks are supposed to make you more productive then I’m currently not experiencing that feeling).

Finally, after many hours of messing around and reading the answers to this stack overflow question solved it:


I still can’t really believe that that is the real solution but at least it seems to work. You’d think there would be a much more generic way of handling the response rather than to write a bunch of jquery to pull the validation errors out of the json response.

But ok, it works so let’s not complain. Well, it works if you have javascript enabled. I really dislike the way the webs is developing in this sense, javascript should not be a requirement per-se and the amount of cruft that gets added to a webpage in order to achieve something that could have just as easily been achieved by expanding the HTML standard (for instance, by adding a ‘validate’ attribute to fields with a regexp of permitted values, or even a ‘validate_src=’url” for an online validator in case you need to do backend stuff to determine if the contents of a field are valid. Anything but JavaScript!).

Another thing that would really help here is a reference application that you can simply download that shows you ‘how it’s done’ using rails version ‘x’ rather than that every developer has to replicate the effort to piece it all together. Maybe I should post the SRS app sources so people can refer to them.

Useful reading:


Little by little I cribbed together the knowledge of how things are done the ‘ruby on rails’ way, it’s incredible how fragmented the information is.

Moniker.com Is Being Murdered

For many years I was a ‘bulkregister’ customer, I landed there after Network Solutions went mad. Then Bulkregister was bought by enom.com and it went from good to bad to worse in a very short time. I called around to my colleagues to see where they were registering their domains and they pointed me to a small but scrappy upstart registrar called Moniker, the brainchild of one Monte Cahn. Monte was awesome, he worked like a demon and rightfully claimed never to have lost a customers domain.

In 2005 Moniker changed hands for the first time, Seevast Corp bought it and I thought this was going to be the end. I decided to hold out for a while to see how things would go and - surprise - in spite of how most corporate take-overs are done Moniker appeared to be in good hands and I continued to be a customer. On occasion - when the support people were not able to help - Monte would step in to protect the good name of the company that he’d created. All was good.

Then in 2008 Moniker changed hands again. This time to Oversee.net. Again, I bided my time and was rewarded with a company that even though technically a small subsidiary of a larger mothership operated with some independence, and continuity was one of their prime focal points. So I stayed. Occasionally I had issues, most of these were resolved in time and competently, where that wasn’t the case we found ways to deal with it, just another customer-supplier relationship, no longer the personal treatment from the old days but that was to be expected. As these things go, for a company that had changed hands twice it could have been a whole lot worse.

In 2012 KeyDrive SA acquired Moniker.com. And in February 2014 Bonnie Wittenburg (already CEO of KeyDRive) was named CEO of Moniker. Wittenburg has apparently not heard of the age old mantra of computing systems that have been around for a while: If it isn’t broken, don’t fix it. Full of fresh and vigourous energy the CEO declared in February that ‘they’ve gone back to the drawing board’. I wished they’d stayed there. Last week Moniker.com released it’s new and revamped interface, and it is - without exaggaration - an unmitigated disaster.

I’m a pretty loyal customer, but what’s happening at Moniker is at a scale that I’ve yet to see elsewhere. The number of problems, the severity of the issues and the incredible lack of openness with respect to what is going on has managed to completely trash the excellent reputation that the Moniker.com brand had for me in the space of a few days. I’ve been a customer there for almost a decade and I’m more than willing to excuse the occasional error but this is an extremely good example of how not to do a public roll-out of a revamped website. The words “On Monday, June 2nd, 2014, visit Moniker to experience the change for yourself.” in this moniker announcement turned out to be truly prophetic, I have unfortunately tried to visit moniker on June 2nd 2014 and ‘experienced the change for myself’. What was once one of the easiest and most reliable user interfaces used at a registrar has turned into a textbook example of how you alienate an existing customer base.

Let me give you a short listing of what I’ve been able to come across so far:

  • it’s all just change, no improvements

  • stuff has been moved around for no apparent reason, basic common sense design has been replaced by a ‘hunt and seek’ kind of user interface that is totally un-intuitive and does not in any way connect to the old interface, it’s a radical breach with how things were. Sometimes this can be a good thing but imagine Microsoft would re-arrange all the items in the menu system for ‘Word’ at random and would rename a couple for good measure.

  • it’s as if there was absolutely no testing whatsoever.

  • account history is gone (so, how do I do my bookkeeping?)

  • most commonly used functions (such as domain push) do not work

  • newly registered domains are not showing up in the interface (so how do I do my work?)

  • domains that are bought and paid for are stuck in ‘awaiting payment’ even though the invoices for those domains do show up

  • charges are made for domains that have been set to expire long ago (and even for some that had expired!)

  • domains are expiring that are set to auto-renew

  • support questions are going utterly unanswered

  • the support phone system is overloaded permanently, hours on hold without any help whatsoever

  • transfering domains out can only be done on a one-by-one basis

  • I wished I was the only person experiencing all this the comments here are telling

And on and on. Monte Cahn is probably getting quitely drunk somewhere, they’re murdering his baby and it seems there is not much that can be done about it. What a total mess. Imagine, you’re being given the reins over one of the most respected registrars and within 6 months of that you take a legacy like that and totally destroy it. There has to be a prize for something like that.

If they keep this up for a while they’ll be the proud owner of a reputation worse than GoDaddy’s, which I would not have thought possible.

Wittenburg should step down or be fired and someone competent brought back in or KeyDrive will find that they have paid the most money ever for just a single domain because there will be no customers left. As soon as I figure out how to (they definitely did not make this easier) I’ll be taking my business elsewhere. Gandi looks pretty good. I’m only about 1/1000th of the total volume of domains administered through moniker.com, so probably not enough for them to worry about.

The Several Million Dollar Bug

In case you landed here without any context and have no idea who I am or what this article is about this article should give you some background.

For years we were skating on very thin ice. Our only advantage that we had over competitors is that we had figured something out that they had not. Which is that Netscape, Microsoft and pretty much every other browser vendor had made a small but crucial mistake in implementing HTTP. It was an easy mistake to make, and a lot of code would have to be written to ensure that the bug was not present, without making any difference to normal every day HTTP requests.

RFC 1945 says:

The HTTP protocol is based on a request/response paradigm. A client
establishes a connection with a server and sends a request to the
server in the form of a request method, URI, and protocol version,
followed by a MIME-like message containing request modifiers, client
information, and possible body content. The server responds with a
status line, including the message's protocol version and a success
or error code, followed by a MIME-like message containing server
information, entity metainformation, and possible body content. 

So, let’s get this straight: Client connects, client sends request, server responds to that request. Pay attention to the hairline cracks of ambiguity in those sentences and how they are commonly interpreted. The fact that it is written in that order seems to imply that it has to be implemented in that order.

This stack overflow question gets it just right:

My question might sound stupid, but I just wanted to be sure :

    Is it possible to send an HTTP response before having the request 
    for that resource ?

Say for example you have an HTML page index.html that only shows a 
picture called img.jpg. Now, if your sever knows that a visitor will 
request the HTML file and then the jpg image every time :

Would it be possible for the server to send the image just after the 
HTML file to save time ?

I know that HTTP is a synchronous protocol, so in theory it should 
not work, but I just wanted someone to confirm it (or not).

Funny how people are still asking this question nearly 20 years later, and no, that was not a stupid question, in fact, once upon a time I was wondering about just that myself and the answer surprised me. There really are no stupid questions and checking such ‘obvious’ things sometimes pays off in a big way. Which is probably what those responding with their definitive ‘no’s’ to that SO question should have done. Now, for the moment ignoring such details as ‘keepalive’, mixing content types (the question is about html and jpg) and the maximum number of parallel connections that your browser has to the server, in theory, if you knew what the response was going to be before the request even arrived, would it work if you sent the response ahead of time?

And the answer to that question is a resounding YES, you can, it works like a dream and that’s the tiny little item that was the sum total of our competitive advantage. HTTP may be designed as a synchronous protocol but it is not implemented as a synchronous protocol! I figured this out one afternoon when making the early version of the webcam software. The frame rate absolutely sucked, it was 1 frame per second over an ISDN line with a round-trip time in the 100’s of ms or so between server and cam. Sometimes two. But at two channels bonded for a whopping 128Kbit upstream we should be able to do three, maybe more fps. And that’s a huge difference, the difference between ‘animated slideshow’ and ‘almost video’.

So I started toying with the idea of sending the response before the request was in, in fact, totally ignoring the requests! Of course, my initial response was ‘this will never work, for sure the browser will discard the responses to requests that it hasn’t sent yet’. Only it did not! So that’s exactly how we ended up doing it, set the TCP buffer size to roughly the size of a frame, check to see if the frame would fit in its entirety, if not, drop the whole thing and if it did fit then send it. Instant rate adaptation, and maximum frame rate across any kind of connectivity, all the way up to what the hardware could capture/compress/transmit. On a lan you’d get 15 fps (which was absolutely astounding at the time), 320x240, whereas over the WAN links of the day we’d scale it down to something more leisurely (and probably using a smaller image). But still quite good to look at, since it was simply jpeg frames (no incremental updates, so you got a whole frame of reasonably good quality rather than all kinds of blocky updates).

This little trick made lots of money, and I always wondered why our competitors didn’t catch on the little bit of black magic that made it work. All they really had to do was capture some packets in flight and the secret would be out. So much for having a defensible moat, score one for ‘trade secrets’, even if they’re kept in the open. Sending the response before the request is in is not exactly valid in a synchronous client/server protocol, but to this day this trick works wonders. So if you know what the request is going to be feel free to send the response ahead of time, it just might make you some money. At a minimum it will make your users happier, always a good thing.

Every time a major new browser update or such was announced I’d lose lots of sleep, being sure the game would be up, this time they’d close the loophole and we’d be out of luck. But it never did! (Of course, that’s blind luck but if you look long enough at any success I’m sure you’ll be able to identify a number of items that can only be described as lucky. I could re-write this post in terms of how good we were at engineering but that would be a total lie, we just got lucky that it worked and luckier still that it kept working as long as it did. SPDY now tries to make this kind of behaviour official.) Relying on obscure undocumented behaviour is rarely a smart move.

Of course, if you can build out your advantage beyond the trivial to copy then you should do that. But just because you don’t have a real technological advantage doesn’t mean you will be found out in time (or maybe not even at all…). (Unless you blog about it of course, so I guess the secret is definitely out now).