Jacques Mattheij

technology, coding and business

Whose Phone Is It Anyway?

When telephones were still solidly connected to the wall, either by being bolted on to it or by an umbilical (sometimes consisting of one or more phone extension cords) it was pretty clear who owned them, the phone company did. And bad things (such as disconnection) would happen to you if you connected something to the network that wasn’t authorized.

When computers were still unconnected to each other using networks it was pretty clear who owned them too, the company or person that paid for it did.

Life was simple and all was good.

Then one day in the 60’s, before I was even born the world changed in a remarkable way though it would take decades before the implications would become clear. The idea of networked computers first saw the light of day and by the early seventies the first real networks of computers were a fact.

Telephones were still mostly tethered to walls, apart from some very important people who had access to ‘mobile phone technology’. In practice this meant that they gave up the trunks of their cars to enormous radio transmitters and some pretty bulky interface components in the interior. Such was the price of being an important enough person that your thoughts had to be available to those that needed instructions or information.

For the most part life was still simple and all was good, the few people that had advance warning of a tremendous shift seemed to ignore it. There was a thing in the 60’s called ‘the mother of all demos’ and it successfully extrapolated about 30 years of the future, up to and including the world wide web.

In the late 80’s and early 90’s mobile phones underwent a revolutionary shrinking and suddenly instead of a select few a much larger number of people got access to them. These phones were still quite bulky (the size of a toaster, maybe a bit thinner) and weighed a lot but those disadvantages did not seem to slow their adaptation at all. Suddenly not only royalty, presidents and the CEOs of the very largest companies had the power of consultation and direction at every hour of the day, it was available to just about every business person and the wealthier private individuals.

Computers were still treating being networked as ‘optional’, though in offices networks were now commonplace. Phones and computers occupied very distinct networks (though for the ‘long haul’ computer networks often employed telephone lines altered specifically for that purpose, so called ‘dedicated lines’ which had their filters removed) and phone companies still viewed the gear they rented to you as their property and the contracts in place were pretty specific on that. You paid a monthly fee for the equipment. In some countries the rules had been relaxed a bit and third party gear was permitted but in most cases only after an expensive approval process. After all, the phone company did not want any weird/dangerous/unlicensed gear on its network and infrastructure, the telephone network was - and is - critical infrastructure and only trusted parties are supposed to be on it.

In ‘95 a revolution happened. Suddenly unconnected computers went from being useful and productive machinery to sad lonely devices that did not have access to the majority of the available information resources embodied in this new fangled thing called the world-wide-web.

Phones continued to shrink and mobile phones shrunk even faster. By 1998 the mobile phone revolution was well under way and even though the phone network was still very much holy there were some inroads into the phone companies territory. For one cable companies started to offer phone services (which has since been mirrored by traditional telcos offering TV channels). You were allowed to buy your equipment rather than rent it. Clever phone company, sell you a peripheral of their network at a huge mark-up and then make you pay for it again whenever you use it for its intended purpose. I think that’s called a ‘win-win’ ;). There was no more lower limit on owning a phone, everybody seemed to have one, including kids! In just 30 years the ability to communicate world wide at any given moment of the day had gone from being a privilege only afforded the wealthiest and the most powerful to children that were still learning how to spell properly, an incredible example of commoditization.

Computer owners continued to see their machines as theirs almost without exception, and the network card was clearly a peripheral with the browser just another application on the computer. The demarcations were very clear but the pattern of convergence was at least as clear and with it came a slow blurring of the lines.

By the mid ’00s we were well under way in revolutionary territory. Suddenly pocket computers with internet connections became wide-spread. And regular computers without an internet connection all but disappeared. Providers required the installation of their software on your computer in an attempt to establish a beach-head.

Phone companies selling phones from various vendors had long also established a beach-head on your phone, the so called ‘baseband processor’. It is a dedicated computer that talks to the phone company network, the software that it runs is not under the control of the owner of the phone and upgrades and or functionality are not transparent at all.

And so we arrive today. Where you think that you own the computer that you use and the phone in your pocket. You see the network as a peripheral of your computer and your phone, with your device as a node in a network operated by ‘peers’.

But that’s not the view of the phone company and your ISP. They see that phone in your pocket and that computer on your desk as a peripheral of their infrastructure.

In smartphones the line of demarcation runs somewhere in the middle of a circuit board on a channel between the baseband processor and the main processor. But memory of the mainprocessor and other peripherals may be readable/writeable/controllable by the baseband processor. You won’t know about it as long as they don’t tell you about it. That phone in your pocket is about as nice an analogy of a trojan horse as you could imagine, you’re lugging along a ton of software that runs on behalf of others and you can’t even inspect it.

Meanwhile, on the home network that demarcation line runs roughly between your router (unless you built it yourself) and the computers connected to that router. In some cases special software has been installed on your computer (which serves absolutely no other purpose than to control what you can do with your internet connection, monitoring and control).

This situation will persist as long as people accept full fledged computers running tons of software supplied by third parties with convenience rather than data ownership as the main focus. So don’t be surprised if the phone company has access to all your contacts, to each and every email you’ve ever sent via your phone, to each and every website that you have visited (even if it was with https), to the voice contents of all your calls (historically they had that last one already but with the technology at the time such massive datacollection would have been all but impossible).

So, whose phone is it anyway? You paid for it, you should own it. But the phone company and your ISP have a different view of the world than you do.

To them you’re nothing more than a money making peripheral, just like in the old days, when phones were still wired solidly to the wall.

My New Vehicle

After looking over many advertisements for all kinds of super nice looking vehicles I’ve decided to go with this baby:

It was custom built for me by my eldest son, he spent a ton of time on it and the result is really quite nice. I had a whole pile of weird requests so buying a bike off-the-rack was out (I did quite a bit of shopping around but I could not find anything with these specs).

The demands were:

  • really good all weather braking

  • fast

  • strong

  • light (or as light as possible)

  • street legal

  • comfy

The first he solved by getting some pretty weird brakes, a hybrid mechanical/hydraulical brake set. I never even realized such a thing existed! The idea here is to have very good stopping power regardless of whether or not the rims are wet without the downside of using disc brakes, which work well but are super maintenance intensive and tend to go out of whack at the slightest disturbance. Definitely impressed with these brakes, if you hit the front wheel too hard without thinking about it you’ll be flying, and in the rain they work very well indeed.

As for speed, I like to go places when I cycle, not stand in one place so I like the old fashioned racing bikes a lot. This decided the basic geometry of the bike, a regular racing bike set-up with curved handlebars. I’ve had it up to 44 Km/h and for an old guy like me that’s reasonably ok. (And let’s not factor in the two week recovery period afterwards ;) ).

The bike is strong, the frame is very stiff and it has 32 spoke wheels for lots and lots of kilometers without having to worry too much about things coming out of alignment due to potholes or other disturbances.

It’s pretty light for a bike with all the trimmings and this level of strength. The frame is recycled, carbon/aluminum, strong where you need it and light where it matters (bought on the local equivalent of ebay for a song), the rest was bought new from a variety of bike parts stores.

Being street-legal is a must, I know that you can get away with a lot (especially in NL), but I like to be seen (lights) heard (bell), to have fenders (because it always rains) and I like to be able to go out in the evenings. Besides it being good that it is street legal from a fines perspective it is also simply good to be able to see and be seen.

It’s comfy because it has a very nice saddle and cushy handlebar tape, it also works well with my posture.

I’m still going to add a pack carrier, and maybe a secondary one in front.

All in all, a super bike and I’m totally happy with it, when I held my newly born son in my hands what feels like only a very short while ago I never anticipated that one day he’d build me a bicycle. I always felt pretty good about letting him loose in the workshop when he was still small and it looks like that confidence and trust has paid off.

Thanks son, well done!

It Is Not About the Money, Silly! It Is All About the Time.

One of the oldest business sayings that stuck into my head was ‘Time is Money.’. It’s deceptive in its simplicity, it generates in the mind of the beholder of this simple 3 word adage that there is a direct equivalent between time and money, and that that equivalence runs both ways.

Ways in which you see this at work in every days life are people working (spending their time) in order to get some money. People spending their money to give them more time (for instance, by using some form of mechanised transportation to arrive at their destination sooner, bicycle, moped, car, jet etc) is another way in which it is illustrated.

And so we all seem to have blindly bought in to the fact that in order to be able to spend money we will have to expend our time and in order to be able to maximize the time we have available we have to spend more money. And thus the treadmill was born. And once you’re on that treadmill it gets very hard to get off it.

Before you know it you’re hip-deep in debt (credit cards, mortgages, car loans, student loans) and you need to work even harder (and so spend more of your precious time) in order to keep up with the now much higher speed of the treadmill.

But nobody is born into debt (at least, not in any functional society), so at some point in our lives we derail from the path where we are debt free onto a path where we are no longer debt free and take out a mortgage on our future health, well being and free time budget in order to be allowed to enter the treadmill and then we willingly assist in speeding it up.

When I look around me I see people making endless purchases of stuff they don’t actually need, brandnew cars, houses large enough to live in with two families (and then there’s only two people living there), parents that barely see their kids because of the pressure to earn money and a continuous stream of complaints about how it is absolutely impossible to make ends meet.

I don’t understand any of it. Maybe it is because I don’t allow any advertising at all into my life but the draw of all these gadgets and the need to show off wealth by buying expensive brand clothing and so on are things that I’m a stranger to. In a way I consider myself very lucky, I’ll buy what I (think) I want when I want it and because I can (easily) afford it. Not because there is some kind of peer pressure or a sense of having to keep up.

In my family there is an interesting set of divides between people with some money, people somewhere in the middle and people with no money at all. And in my circle of friends and their extended families there is a similar set of divides. Most of the complaining about the situation comes from the middle group, which really surprises me. The people that have no money are surprisingly good at managing the little bit that they have. Month after month they impress the hell out of me by making a couple of credits stretch well beyond what you’d normally think would be possible. Absolutely superb money management skills. The people that are somewhat wealthy have little reason to complain and none of them do.

But oh, that middle group. Two incomes, 160K euros / year jointly is not exceptional. That’s a very large amount of money. And yet, at the end of every year it’s all gone. They can’t get rid of their mortgages (someone explained to me in a very serious tone of voice that getting rid of your mortgage is stupid because it is a deductible, better to wait with paying it off when you sell the house in 25 years), they are supremely bad at figuring out what the cost of all this consumption driven borrowing is and how big a difference it would make if instead of buying stuff they’d be saving for a while (or at least, lowering their mortgage).

It’s almost as if common sense about finances is something that is extremely rare. People that are by their own admission barely able to make ends meet show off their new cars, smartphones, laptops, tablets and so on. And all bought on ‘cheap’ credit. But that credit really isn’t all that cheap (read the fine print on your credit card statement, if you have a 1.9% card make sure that it isn’t accidentally per month instead of annually).

Seriously. If you are making good money, you’re in debt and at the end of the year you are further in debt or the situation hasn’t improved then learn to control your spending. Saving is so much easier than earning, and it’s a habit that once built will pay you back for the rest of your life.

And once you’re out of debt, maybe have a bit of an investment strategy going you can safely spend the surplus on things that you can actually afford. But until you reach that milestone getting rid of debt should be priority #1 and the gadgets, the new car and other luxury spending should have to wait.

And then, when you are in that luxurious position you can maybe slow down that treadmill again and recover some of your precious time. That’s the only form of capital that you will NEVER have more of. So spend it wisely.

Choosing a Web Framework/Language Combo - Ruby on Rails

In case you’re tuning in late, the previous installment in this series is here. This is an absolutely ridiculously long blog post, feel free to either totally ignore it or to read only the top portion, it will likely contain all you need or want to know to keep up to date with what’s happening here. The rest is more of a detailed log of what I did to be able to get back into it quickly if I decide that ‘Ruby Is the One’.

A picture paints a thousand words I guess:

And yes, it works, you can actually learn Romanian with it, so I guess that’s ‘mission accomplished’ :) It’s also butt-ugly, works on mobile, tablets and so on (courtesy of the template, I did not do much work on that other than to resolve some conflicts).

The short version of what the last couple of weeks were like (resulting in that screenshot) is:

  • Install RubyOnRails

  • Dive headlong into rails

  • Run aground, decide to learn the Ruby language first

  • Learn Enough Ruby to make sense of rails

  • ported the PHP command line application to ruby

  • re-write all database interaction to use activerecord

  • move all the business logic into the data model

  • port the datamodel over to the demo rails application

  • drive to Amsterdam

  • re-install RoR on the machine here at the office

  • make a ‘user creation’ form

  • make a ‘login’ form

  • figure out how to log-in a user

  • install a template and chop it up

  • make the ‘available courses’ page

  • make the learning page

  • make the feedback from the server about the answer given work

  • make a top row that shows the user info + logout link

All in all it was more work than I thought it was and I ended up spending a frustrating amount of time just searching for stuff and being stuck on simple little details. The problem with rails is not the the information isn’t out there, the problem is that there is a ton of stuff out there that simply is no longer relevant but because everybody is playing the SEO game that data will forever pollute your ability to find out what is relevant and what is not. In the end it worked out though and I’m pretty happy with the result. This of course does not say anything yet about how it would be to deploy this particular solution, it only means I’ve managed to build a first version.

My first impression is that developing on RoR is (when you know what you’re doing, always helpful) fairly quick but the result is quite fragile, far more fragile than I expected. This is in part due to Ruby the language (which Rails can’t really do all that much about) which has some peculiar quirks and in part because of the way Rails is set up, it’s all ‘conventions’ and conventions translate into ‘things you should know and learn by heart’. This makes it a bit harder to get going. I’m used to reading a program much like I’d read a book, and that’s also how I normally write programs. In comparison, working with Ruby on Rails feels like someone tore the book up and scattered the pages in a hundred different places. Ruby is a funky little language that allows you to take all kinds of short-cuts, the Ruby code was on average about 25% smaller than the corresponding PHP code. And I don’t know Ruby and I know PHP like I know my back pocket so likely with some more knowledge I could save more code.

Another thing that strikes me is that Ruby is a language that almost asks for unintended side-effects to be present in your code. And that in turn is an excellent way to get security issues, if your ‘counterpart’ knows Rails (much) better than you do chances are that they’ll find something exploitable that you overlooked.

Even so, I’m fairly impressed with how concise the final solution is, how quick it got built In spite of all the hours of frustrating searching and the continuous feeling of going very slow and working ‘against’ the frame work only 16 days have passed since the last installment, when I didn’t know anything about Ruby yet, and 3 of those days were spent on the road driving from Bucharest to Amsterdam.

This code is so fresh and most likely contains tons of security holes and bugs that I’m not confident to put it out there, besides, this is only the evaluation of Ruby On Rails and the list of frameworks that remain is a long one.

The logfiles (you *really* should stop reading here)

Warning, this is an extremely long blog section detailing weeks of work to try to get up to speed with Ruby, then a couple more to work through Rails tutorials and then more still to get a minimally working version of the SRS application up and running. If you just want the ‘verdict’ you’ll have to wait until I have done this with all the platforms so I can compare them side-by-side, think of this as a progress report and a way to segment the various platforms into one-page-each. It is written the way it is so you get an idea of the work I did, in what order I did it and what I was thinking while I did it and what issues I encountered along the way and how I (hopefully) solved them. This is probably of limited use to you unless you are currently evaluating Ruby / Ruby On Rails.

I’m going to try to keep the external representation of the various implementations as close together as I can. So starting off with a ‘full fledged’ batteries included combo like RoR is a good way to find out what is missing in other offerings, and it will also get the first web based version of this working in (hopefully) short order. I don’t recall having ever written a line of Ruby before so consider me a total noob when it comes to this particular set.

While you’re reading this keep your eye on the ball and remember that I’m trying to learn some Romanian.

I start out where I imagine everybody else starting with Ruby On Rails starts out, the getting started guide and going step-by-step.

ruby -v ruby 1.9.3p484 (2013-11-22 revision 43786) [i686-linux]

sqlite3 –version 3.8.2 2013-12-06 14:53:30 27392118af4c38c5203a04b8013e1afdb1cebd0d

gem install rails

After a long pause and a whole bunch of disk grinding and fan noises:

    "ERROR:  Error installing rails:
    ERROR: Failed to build gem native extension.

    /usr/bin/ruby1.9.1 extconf.rb

/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require’: cannot load such file – mkmf (LoadError)

    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from extconf.rb:1:in `<main>'"

Some googling later it turns out that ruby-dev is also required:

apt-get install ruby-dev

Maybe someone should update that tutorial (and retry it periodically to see if it all still works as advertised). Little gotchas like this in the first few minutes can turn off newbies and will reduce adoption.

gem install rails

More disk grinding. It’s pretty hot here today and the gem installer uses a lot of CPU cycles according to the sounds coming from the machine. The machine I’m doing this all on is pretty ancient, all of 2G of ram and a regular drive. My ‘main’ machine sits back in the Netherlands under my desk there and would probably make short work of all this but I have this one to work with and I guess it is a lot better than chiseling 1’s and 0’s into stone tablets. After a good 15 minutes it is finally done. Rails is installed :)

rails -v Rails 4.1.1

Neat. So at least it looks like once you get past that ruby-dev hickup the rest works as advertised.

On with the tutorial!

rails new blog

After a couple of minutes this prompts for my root password, then after yet another couple of minutes tons of errors. Scrolling back I find that this is caused by not having sqlite3-dev installed (a header file is missing).

Easy to correct.

sudo apt-get install libsqlite3-dev

On error, retry:

rails new blog

This asks me if I want to overwrite a secrets file, I’ve yet to modify files here so I say ‘Y’.

A few minutes later the installation completes without further issues. At the end there is this line in the output:

“Warning: You’re using Rubygems 1.8.23 with Spring. Upgrade to at least Rubygems 2.1.0 and run gem pristine --all for better startup performance.”

Ok, sounds good. How do you upgrade Rubygems? More googling leads me to:

gem install rubygems-update

followed by:


Which seems to install RubyGems 2.2.2

The tutorial next advises to start the embedded rails server from within the blog directory:

rails server

Only that doesn’t work…

/usr/local/lib/site_ruby/1.9.1/rubygems/dependency.rb:298:in `to_specs’: Could not find ‘railties’ (>= 0) among 0 total gem(s) (Gem::LoadError)

    from /usr/local/lib/site_ruby/1.9.1/rubygems/dependency.rb:309:in `to_spec'
    from /usr/local/lib/site_ruby/1.9.1/rubygems/core_ext/kernel_gem.rb:53:in `gem'
    from /usr/local/bin/rails:22:in `<main>'

In a stackoverflow thread I found a hint that I should re-run ‘gem install rails’ as a regular user:

gem install rails

Rather than as root. So I tried that.

“You don’t have write permissions for the /usr/lib/ruby/gems/1.9.1 directory.”

This is a very bad idea:

chmod -R 777 /usr/lib/ruby/gems/*

So we won’t be doing that. Let’s try again as root:

sudo gem install rails

Try again:

cd blog rails server

/usr/lib/ruby/gems/1.9.1/gems/execjs-2.0.2/lib/execjs/runtimes.rb:51:in `autodetect’: Could not find a JavaScript runtime. See https://github.com/sstephenson/execjs for a list of available runtimes. (ExecJS::RuntimeUnavailable)

Hm. What does ruby or rails have to do with finding a javascript runtime?

More googling. In this stackoveflow thread I read that I have to enable a gem called ‘therubyracer’ in a file called ‘gemfile’. I load ‘gemfile’ into vi and indeed, there is a line like that. Remove the ‘#’ in front of the line (# marks are used as a comment indicator in ruby).

rails server

Could not find gem ‘therubyracer (>= 0) ruby’ in the gems available on this machine. Run bundle install to install missing gems.


bundle install

Tons of errors, the topmost of which suggests that I should install g++ first.

sudo apt-get install g++

bundle install

At this point we’re about 1.5 hours into the installation process. To save RAM I’ve killed firefox.

Lots of grinding and about 1/2 hour later:

“Your bundle is complete!”

rails server

=> Booting WEBrick => Rails 4.1.1 application starting in development on => Run rails server -h for more startup options => Notice: server is listening on all interfaces ( Consider using (–binding option) => Ctrl-C to shutdown server [2014-05-26 22:49:55] INFO WEBrick 1.3.1 [2014-05-26 22:49:55] INFO ruby 1.9.3 (2013-11-22) [i686-linux] [2014-05-26 22:49:55] INFO WEBrick::HTTPServer#start: pid=7154 port=3000

Started GET “/” for at 2014-05-26 22:50:07 +0300 Processing by Rails::WelcomeController#index as HTML Rendered /usr/lib/ruby/gems/1.9.1/gems/railties-4.1.1/lib/rails/templates/rails/welcome/index.html.erb (34.9ms) Completed 200 OK in 325ms (Views: 215.4ms | ActiveRecord: 0.0ms)

Yay! It appears we have a working RoR installation.

Excellent. It took a while but it looks like it is working.

The built in server is quite convenient when testing code on your local host.

Next the getting_started guide walks you through a bit of configuration and then you will be able to see your test page. After that you are prompted to create a ‘resource’, by making some changes to the config/routes.rb file.

The listed example shows that you should make it look like this:

Blog::Application.routes.draw do

resources :articles

root ‘welcome#index’ end

But that throws up all kinds of errors.

rake routes

rake aborted! NoMethodError: undefined method `application’ for Blog:Module

Changing ‘Blog’ back into ‘Rails’ seems to fix this for now (my understanding of the routing dsl is so minimal at this point that I have no clue what this all means, I will read the doc on the routing in a moment, maybe then I’ll have a better understanding).

Next we add a form and there is something weird about that, the tutorial tells you to change the first line of the form to:

<%= form_for :article, url: articles_path do |f| %>

But gives no explanation why the ‘url:’ has a trailing colon and the rest of the : characters in that form are at the beginning of the terms. It doesn’t explain much of anything anyway but this is a little strange.

The explanation underneath reads:

“The form needs to use a different URL in order to go somewhere else. This can be done quite simply with the :url option of form_for. Typically in Rails, the action that is used for new form submissions like this is called “create”, and so the form should be pointed to that action.”

Hopefully this will be explained in:


So time to do some reading.

So what is it? url: or :url? According to the routing docs it is indeed url: so the tutorial got that bit wrong in the text but right in the example.

So the magic works, the predicted ‘Template is missing’ error shows up.

Next we add a bit of code to allow us to see what the controller receives.

There is this sentence in the tutorial at this point:

“The render method here is taking a very simple hash with a key of text and value of params[:article].inspect.”


“render plain: params[:article].inspect”

And I really can’t make soup of this. What hash? What key of ‘text’?

I suppose the ‘key’ refers to the name of the field ‘text’, but that value then makes very little sense (after all, there are two fields, title and text), and that still does not explain the ‘hash’.

At the end of section 5.6 we’re told to insert the following bit of code to get around a ForbiddenAttributesError:

private def article_params

params.require(:article).permit(:title, :text)


But after I do this and refesh the page the error remains.


Says: you appear to be following a pre rails 4.0 tutorial with rails 4. You need to use strong params now.

But at the top the tutorial explicitly confirms that it should work with rails versions > 4.

I use the second line in the SO post answer:

# @comment = @post.comments.create!(params.require(:comment).permit(:comment_text,:link) 

Convert its format to what I think should be required to make it work for articles:

# @article = Article.create(params.require(:article).permit(:title,:text))

Comment out the original line that made a new article:

# @article = Article.new(params[:article])

And commented out the bit that section 5.6 said I should have added:

#  def article_params
#    params.require(:article).permit(:title, :text)
#  end

And try again. Now rails complains about not having a ‘new’ action.

Checking above, I did not change ‘create’ to ‘new’. That’s an easy fix.

And now it indeed gives me:

“The action ‘show’ could not be found for ArticlesController”

Whoever maintains that tutorial certainly hasn’t been paying attention lately, and without stackoverflow and google to help I would have either thrown my computer out the window at this point or de-installed rails. Maybe this is some kind of secret plot by the RoR overlords to test newbies for the joint start-up required qualities of persistance and ingenuity? Or maybe they just don’t like newbies and want them to go away. I don’t know which it is but I can’t imagine this tutorial is leaving the people following it in good spirits.

Anyway, on we go.

I add the following line to config/routes.rb:

article GET    /articles/:id(.:format)      articles#show

And hit refresh:

/home/jam/svn/src/srs/www/ror/blog/config/routes.rb:4: syntax error, unexpected ':', expecting keyword_end article GET /articles/:id(.:format) articles#show

Most helpful. I change the line according to the format listed in an example lower down in the routes file to read:

get '/articles/:id(.:format)' => 'articles#show'

With ‘rake routes’ I verify that the route is indeed present:

      Prefix Verb   URI Pattern                  Controller#Action
             GET    /articles/:id(.:format)      articles#show
    articles GET    /articles(.:format)          articles#index
             POST   /articles(.:format)          articles#create
 new_article GET    /articles/new(.:format)      articles#new
edit_article GET    /articles/:id/edit(.:format) articles#edit
     article GET    /articles/:id(.:format)      articles#show
             PATCH  /articles/:id(.:format)      articles#update
             PUT    /articles/:id(.:format)      articles#update
             DELETE /articles/:id(.:format)      articles#destroy
        root GET    /                            welcome#index

I add the action and the view, hit refresh and indeed, the ‘article’ appears.

So far so good. One of the problems undoubtedly is that I’m a total noob to both ‘ruby’ and ‘rails’, but I suspect I’m not the only person that finds this tutorial as their first point of contact. And I just can’t get over how frustrating this whole experience is. It’s like being a failed Harry Potter, you’re being told to use all these spells that are supposed to have some magic effect, only they don’t and throw back errors at you all the time implicating you’re the one doing something wrong. And then after consult it always turns out that the spells you were given were wrong. Really annoying. Anyway, if I wasn’t the persistent type I would have never gotten into programming in the first place so I’ll try real hard to stop bitching about this but it’s getting to me. I sure hope the quality of ‘ruby’ and ‘rails’ is at a higher level than this tutorial.

The main issue I have at this point is that given the fact that Ruby does a ton of things under the hood that appear to be black magic to the user you should at least build up the students confidence that the magic actually works. Right now I feel as if a single ‘.’ or ‘,’ misplaced that does not have a relevant stack-overflow message waiting for me (assuming I know what to search for) will leave me totally stranded. You’re asked to do all these things without a proper explanation of what is going on, it’s a huge exercise in faith.

I suspect that one of the problems here is that I dove head-first into RoR without learning some Ruby first. So let’s divert for a bit and get a feel for ‘Ruby’ the language without looking at the framework.


Some things to remember (when coming from some other language):

  • Expressions work like they do in most other languages
  • functions are defined using a ‘def’ … ‘end’ pair
  • if a function has no parameters you can omit the () both during the declaration and the subsequent call
  • even if a function has parameters, if the context is unambiguous then you can omit them
  • Within a string #{varname} expands varname to the contents of varname.
  • functions can be defined only once (so no erlang like tricks based on arity)
  • you can call a method of an object by using object.method
  • variable names are not prefixed by any special character
  • irb (interactive ruby) will only attempt to process the code when it thinks it is complete (this can become quite messy with multi-line function definitions containing errors)
  • you can use the respond_to? method to figure out what an object understands, this allows for code to reflect on what a class can do and adapt accordingly. For instance, you could try to see if you received a list or a simple variable by checking of the object you received understands “each”
  • attr_accessor :someattribute will generate a getter and a setter method
  • puts “Goodbye #{@names.join(“, “)}. Come back soon!” shows that you can do just about anything in between #{ }

Something weird happened during the tutorial: I defined a simple class according to what it said on the page there and then listed the instance methods:

irb(main):043:0> class Hey
irb(main):044:1> def initialize(name="World")
irb(main):045:2> @name=name
irb(main):046:2> end
irb(main):047:1> def say_hi
irb(main):048:2> puts "Hi #{@name}!"
irb(main):049:2> end
irb(main):050:1> def say_bye
irb(main):051:2>  puts "Bye #{@name}, come back again!"
irb(main):052:2> end 
irb(main):053:1> end
=> nil

=> [:say_hi, :say_bye, :h, :nil?, :===, :=~, :!~, :eql?, :hash, :<=>, :class, :singleton_class, :clone, :dup, :initialize_dup, :initialize_clone, :taint, :tainted?, :untaint, :untrust, :untrusted?, :trust, :freeze, :frozen?, :to_s, :inspect, :methods, :singleton_methods, :protected_methods, :private_methods, :public_methods, :instance_variables, :instance_variable_get, :instance_variable_set, :instance_variable_defined?, :instance_of?, :kind_of?, :is_a?, :tap, :send, :public_send, :respond_to?, :respond_to_missing?, :extend, :display, :method, :public_method, :define_singleton_method, :object_id, :to_enum, :enum_for, :==, :equal?, :!, :!=, :instance_eval, :instance_exec, :__send__, :__id__]

Spot the weird one? That ‘h’ in there is a definition I made before defining the class ‘Hey’, and yet it is a method in the class!

Let’s try that again, completely from scratch.

jam@homebox:~/svn/src/srs$ irb
irb(main):001:0> def h(name)
irb(main):002:1> puts "hello #{name}!"
irb(main):004:1* end
=> nil
irb(main):005:0> h("hey")
hello hey!
=> nil
irb(main):006:0> class Hey
irb(main):007:1> def initialize(name="World")
irb(main):008:2> @name=name
irb(main):009:2> end
irb(main):010:1> def say_hi
irb(main):011:2> puts "Hi #{@name}!"
irb(main):012:2> end
irb(main):013:1> def say_bye
irb(main):014:2> puts "Bye #{@name}, come back again!"
irb(main):015:2> end
irb(main):016:1> end
=> nil
irb(main):017:0> Hey.instance_methods
=> [:say_hi, :say_bye, :h, :nil?, :===, :=~, :!~, :eql?, :hash, :<=>, :class, :singleton_class, :clone, :dup, :initialize_dup, :initialize_clone, :taint, :tainted?, :untaint, :untrust, :untrusted?, :trust, :freeze, :frozen?, :to_s, :inspect, :methods, :singleton_methods, :protected_methods, :private_methods, :public_methods, :instance_variables, :instance_variable_get, :instance_variable_set, :instance_variable_defined?, :instance_of?, :kind_of?, :is_a?, :tap, :send, :public_send, :respond_to?, :respond_to_missing?, :extend, :display, :method, :public_method, :define_singleton_method, :object_id, :to_enum, :enum_for, :==, :equal?, :!, :!=, :instance_eval, :instance_exec, :__send__, :__id__]
irb(main):018:0> Hey.h "hello"
hello hello!
=> nil

Yup, that’s reproducible. So a class absorbs any methods defined separately just prior to that class?

Weird. I can also still call ‘h’ outside of the class:

irb(main):019:0> h("hello")
hello hello!
=> nil

irb(main):020:0> say_hi "jacques"
NoMethodError: undefined method `say_hi' for main:Object
    from (irb):20
    from /usr/bin/irb:12:in `<main>'

But the class methods are not accessible without an instance of the class. I don’t understand this, but I assume it is somehow intentional. Maybe someone more versed in Ruby can explain what is going on here.

=> [:say_hi, :say_bye]

Shows only what we defined in the body of the class.

The cliff notes to ‘Why’s poignant guid on ruby’, about as short a crash course on Ruby as you’ll get:

  • variables are all in lowercase
  • strings can be made with both single and double quotes
  • constants start with an upper case letter, so Constant is a constant
  • :symbol is a symbol
  • instance method invocation works like this: variable.method
  • class method invocation works like this: Classname::method
  • the normal argument passing looks like lots of other programming languages: variable.method(param1, param2)
  • ‘kernel methods’ are core language functions such as ‘print’ they don’t require an object to be attached to, the default object Kernel:: is always searched if you type a method name.
  • global variables have their name prefixed with a ‘$’
  • instance local variables start with an @ sign, their scope is the local object
  • class variables start with @@, their scope is the whole class
  • code blocks are { between }, or between do and end, these are equivalent
  • blocks can have 0 or more arguments between || characters just inside the block
  • ranges look like this: (1..3) or (‘x’..’z’)
  • arrays look like this: [1, 2, 4]
  • hashes http://www.ruby-doc.org/core-2.1.2/Hash.html – a ‘hash’ in ruby parlance is a dictionary that can only store definitions, they look like this: { “a” => “test”, “b” => “you know” } – you get the value stored at key ‘a’ back out of a hash like this: thehash[‘a’]
  • regexps look like this: /regexp/
  • Interesting: 5.==(6-1) returns ‘true’, so == is a method like every other
  • << concatenates
  • .reverse reverses a string

The guide lists “string”.each as a valid construct, but the interpreter complains when I try that. Apparently ‘each’ is no longer available, you now have to use ‘each_char’. I wonder why one would deprecate an existing behavior, breaking existing code and then to re-implement that behavior using a new name. It seems like a pointless exercise to me, nothing changes except that a bunch of people end up having to debug their code. And in an interpreted language such changes are extra nasty because you don’t get the benefit of a compile time check to see if your code is still at least correct at that level.

  • string search/replace is called gsub, it is a method of the string class so you use it as string.gsub(“old”,”new”)
  • affixing ! to a method name does the operation ‘in place’ instead of returning a new result, it doesn’t seem to work for all operators though, a.+!(3) is not the equivalent of a+=3 (which works fine all by itself…), so the ! is part of the method name, not some kind of modifier that makes any method destructive

  • File::read(“filename.txt”) will return a string with the contents of the file

  • File::open(“filename.txt”,’r’) do |f| block end executes the code in block with the filehandle of the open file in ‘f’
  • f << string where ‘f’ is a file descriptor will concatenate the string to the file contents (what about seek? does it write or only concatenates??)
  • very clever dir globbing example: Dir[‘*.txt’].each do |file| system “cat ” << file end
  • there are multiple ways to do the same thing. t = [:a => ‘e’] + [:a => ‘d’] is equivalent to t = [ {:a => ‘e’}, {:a => ‘d’} ]


“Yukihiro Matsumoto: Ruby inherited the Perl philosophy of having more than one way to do the same thing. I inherited that philosophy from Larry Wall, who is my hero actually. I want to make Ruby users free. I want to give them the freedom to choose. People are different. People choose different criteria. But if there is a better way among many alternatives, I want to encourage that way by making it comfortable. So that’s what I’ve tried to do. Maybe Python code is a bit more readable. Everyone can write the same style of Python code, so it can be easier to read, maybe. But the difference from one person to the next is so big, providing only one way is little help even if you’re using Python, I think. I’d rather provide many ways if it’s possible, but encourage or guide users to choose a better way if it’s possible. ”

  • t.sort_by { |e| e[:a] }.each do |t| print t end # sorts ‘t’ by key ‘a’ for each element e by retrieving the value of e[:a]. That’s pretty concise!
  • next functions like ‘continue’ in ‘C’
  • break like it does in most other languages (it breaks out of the innermost loop)
  • a switch statement looks like this in Ruby: case when value when value when value end (a bit strange how they chose to use confusing names for widely accepted constructs, it’s like exchanging ‘-’ and ‘+’ just because you can)
  • return value of a function is the value of the last expression of that function
  • (5..10) === 7 evaluates to ‘true’, because 7 is between the range start and end values

Slight detour through:

http://lambda-the-ultimate.org/node/934 http://www.tbray.org/ongoing/When/200x/2005/08/27/Ruby

Because of Ruby’s variable scoping rules.

Since the _why guide was written Ruby has evolved quite a bit, for instance, the block variable scope is now a lot more sane, the example on page 48 (fortunately!) no longer works as advertised. Variables declared outside of a block are no longer affected by what goes on inside of a block. See here:


  • classes are defined like this: class ClassName bodyoftheclass end
  • 5.next gives you the next number after 5
  • (5.5).floor.next first truncates the float to an int, then calls next
  • object.class returns the class for that object
  • b = a.class.new -> make another object ‘b’ of the same class as object ‘a’
  • you can reach the current object from within a class definition by using ‘self’
  • you can add new methods to existing classes, including kernel classes! (bad idea (tm))

For instance, this is how you can add a ‘square’ method to numbers

irb(main):152:0> class Fixnum
irb(main):153:1> def square
irb(main):154:2> self * self
irb(main):155:2> end
irb(main):156:1> end
  • respond_to?(“methodname”) applied to any class tells you if that class has a method ‘methodname’ (or should I say ‘knows how to respond to a message ‘messagename’)?
  • you can slice strings like this: “abcde”[3..4]
  • you can split strings on delimiters using “ab-cd-ef”.split(“-“)
  • new class creation: class ClassName < parentclass; end (why does it need that ‘;’ there?)
  • you can query the parent class with the superclass method: ClassName.superclass
  • modules govern namespaces
  • to define a class with all the entities in a module you use ‘extends’ as in ‘class X extends ModuleName; end’
  • a *argument to a function will be passed in as an array
  • attr_reader will create getters for the symbols passed as arguments, for instance attr_reader :test :test1 will create reader functions for variables with that same name
  • attr_accessor does both readers and writers, attr_writer of course only writers
  • class << ClassName is used to add class methods (methods that can be called on the class rather than on an instance of the class)
  • a mixin is a module ‘included’ into a class, it instantly endows the class with all the stuff in the module
  • reading and processing the contents of a web page: require ‘open-uri’ # Searching all found items containing the word `truck’. open( “http://preeventualist.org/lost/searchfound?q=truck” ) do |truck| puts truck.read end
  • similar, but line-by-line processing: require ‘open-uri’ open( “http://preeventualist.org/lost/searchfound?q=truck” ) do |truck|
     truck.each_line do |line|
     puts line if line['pickup']

    end end

I don’t particularly like the ‘if x’ conditional appended to that line, it feels a bit backwards to me. Imagine the instructions for disarming a bomb read ‘cut the red-green wire if the blue wire is still connected’. boom. Too bad, you should have parsed the instructions first before you started to execute them!

  • yielding is a way to let two blocks of code communicate, the ‘inner’ block passes information to the ‘outer’ block with every time yield is executed. Or should I say ‘inner blocks pull data from outer blocks whenever they need to’? It’s reminiscent of co-routines (which ruby also has: http://rubydoc.info/stdlib/core/1.9.2/Fiber), one routine the producer, the other the consumer.

  • ‘brief’ style versus verbose style (both are identical in function):

    open( “idea.txt” ) { |f| f.read }

    open( “idea.txt” ) do |f| f.read end

  • traits in a parent class allow for elegant subclassing without having to pass in tons of (unnamed) parameters in the initializer use them like this in the parent class ‘traits :traita, :traitb, :traitc’

  • @x ||= {} is a little trick to ensure that a variable exists before it is used. The ||= says ‘if this varible does not exist yet then make a new one of this type, but if it does exist then adding an empty list to it won’t change anything’. I’m not sure why that works, you’d say that the ||= should fail if x does not exist yet, because it’s syntactically equivalent to x = x || {}

  • I got a little lost around page 109, the metaprogramming bit. It looks like one program is writing another, and not only is it able to evaluate code that wasn’t present in the original program, it can be used to construct entirely new objects / classes and so on. This is powerful but also asking for trouble because it can become very hard to reason about such programs. It’s akin to macro expansion, but more powerful.

  • you can overload just about everything too, including the basic operators

This probably makes ruby absolutely great at writing DSLs but it will be a nightmare to debug because you’ll never know what you are looking at means without backtracking very precisely through how you got where you are. Imagine looking at some program and seeing ‘a = b * c’. In most languages it would be fairly obvious what this does, the product of b and c is stored in variable a. Maybe b and c are arrays or matrices but at a minimum you’d expect some kind of multiplication to be going on there. In Ruby this is not the case. ‘a = b * c’ could mean literally anything, the only thing you can distill from that bit of code is that b and c are somehow combined using the operator ’’ and that the result of that combination is stored in a. Context decides whether this is two variables being multiplied or two battle ships exchanging fire or two images being compared. For a lark, someone could have decided to turn * into the division operator… Powerful, but also very* dangerous, hard to debug and easy to have unintended side effects.

I should re-read the portion on meta programming when I understand Ruby a bit better.

  • string formatting in Ruby: “abc %s and %d” % [“jaja”, 3]
  • inside double quotes #{ } allows you to escape to run ruby code (security concerns here, this goes way beyond variable substitution?)
  • reminder: globals start with a $
  • you can generate exceptions using ‘raise ErrorNo, “Description”
  • you can recover from errors by using the rescue label
  • you can create an array of strings quickly by using %w{a list of words} this gets transformed into [‘a’, ‘list’, ‘of’, ‘words’]
  • %x{} -> execute in a shell, %Q{} -> acts like a regular double quoted string
  • if ‘a’ is a list and you do ‘a = b’ then b contains a reference to a, not a copy of the list so modifying b modifies a and vice-versa
  • you can make an actual copy by using the .clone method, and then there is ‘.dup’ to create copies of objects that have been locked http://ruby.about.com/od/advancedruby/a/deepcopy.htm
  • you append a value to an array using the << operator (contrary to what the stackoverflow article on the same question says!), so a << 5 appends the value 5 to a pre-existing array a.

If you run irb with

irb –readline -r irb/completion

You can type the name of an object, append a ‘.’ and then hit Tab, this will show you all the methods you could call.

For some reason the default ruby install does not come with the embedded documentation (ri).

Ok, that’s the end of the ‘Poignant Guide’, thanks Jonathan. (some joker now has _why’s domain so I won’t be linking there from here).

Porting the command line app

The PHP command line application is as good a place as any to start working with ruby. It’s not overly large and has all the business logic in it and needs to work with the database. Let’s port it to ruby to see what that feels like.

First we need to install the mysql client development libraries:

sudo apt-get install libmysqlclient-dev

Next, we need the mysql gem:

gem install mysql

Now ruby scripts should be able to access mysql databases.

A quickie check in irb if that’s true:

irb(main):001:0> require "mysql"
=> true
irb(main):002:0> con = Mysql.new('localhost', '', '', 'srs')
=> #<Mysql:0x8a48f14>
irb(main):003:0> rs = con.query('select * from users')
=> #<Mysql::Result:0x8a190c0>
irb(main):004:0> rs.each_hash { |h| puts h['nick']}
=> #<Mysql::Result:0x8a190c0>
irb(main):005:0> con.close
=> #<Mysql:0x8a48f14>

Looks like it works :)

About 20 minutes into porting the console application I hit a bit of a roadblock. The standard mysql library module does not allow you to fetch a result from a prepared statement. This results in terribly ugly code which causes a one-liner to become something that spans a couple of lines. In the PHP world I use my own wrapper library for stuff like this, but for ruby I decide to see what’s available out there (this also gives a bit of an idea about how the ruby eco system works wrt to contributed software). I find this: http://sequel.jeremyevans.net/

gem install sequel

installs it, and we’re off again. I’ll change the bits of code written up to that point to use the sequel gem, it looks pretty easy to use. I could use ‘activerecord’ as well I guess but why not give this lightweight module a shot and see how far it takes me.

hash = $dbcon.fetch(“select * from users where nick=’%s’” % nick).all[0][:pwhash]

that looks manageable. Still has an ugly * and .all[0] in there that’s technically not needed but that’s fixable:

$dbcon = Sequel.mysql("srs")

# fetches the first record of a set

def $dbcon.single (q)

# fetches the first value returned by a query

def $dbcon.simple (q)

This is a little snippet from the db initialization code. To ‘fix’ the Sequel class we add two new methods to the class, one that will simply return the first result from a set, another to return the first field from the first result from a set. This allows you to do:

hash = $dbcon.simple("select pwhash from users where nick='%s'" % nick)

That looks better. Of course modifying code from the outside like this is a bit (ok, a lot) hackish but I want to get a feeling for what Ruby can do and even if this is probably not idiomatic ruby the fact that you can extend libraries so easily is a powerful concept (and probably a very dangerous one, imagine if the vendor of ‘Sequel’ decides to add a ‘single’ or a ‘simple’ method to the base class with different meaning! So for anything but this ‘toy’ program this is probably a very bad idea.

Next up, the hash comparison. The passwords in the system are hashed using ‘bcrypt’, with a workfactor of ‘10’ and the blowfish algorithm. This results in hashes with the prefix $2y$10$ followed by the salt and the hash. Testing these passwords with the default Ruby ‘BCrypt’ library results in a non-match, even with the correct password. It turns out that the maintainer of the BCrypt gem made a decision to add ‘2x’ instead of ‘2y’ and that any ‘2y’ passwords will not match, even if the lib is perfectly capable of doing the verification.

The solution? Replace the ‘y’ with an ‘a’ and then the comparison will succeed for correct hashes. See: http://stackoverflow.com/questions/20980859/using-bcrypt-ruby-to-validate-hashed-passwords-using-version-2y and https://github.com/codahale/bcrypt-ruby/pull/91 lists an open pull request with a fix.

This little gotcha cost me about 2 hours to figure out, but for now the ‘replace-the-y-with-an-a’ workaround seems to do the trick.

Ruby does not have a ++ operator!! So no increments of integer variables without doing something like += 1.

No big deal, but I’m sort of used to this and it would come in handy when porting code. But using iterators most of the increments should disappear. Still, it wouldn’t cost anything to include this operator in ruby and it’s surprising that such a simple thing is missing.

Interesting: Porting the php code to ruby turned up a bit that I should have extracted into a function (the loop where the choice was made which course to study). The reason why I found that this should have been a function was that there simply was no elegant way to do it ‘in place’, but then once the function was made it looked a lot better!

Ruby’s inconsistent use of {} for blocks is driving me crazy. I never know when they are permitted, expected or obligatory!

loop {


Is fine but

if a {


Is not… Just check this out:

puts [1,2,3].map{ |k| k+1 }
=> nil
puts [1,2,3].map do |k| k+1; end
=> nil 

What is happening here is that the {} have different precedence than do .. end. Very confusing, even after multiple tries I can’t find a way to convert the curly braces example to a do .. end example.

Maybe something like this:

irb(main):029:0> [1,2,3].map do |k| puts k+1; k+1 end
=> [2, 3, 4]
irb(main):030:0> [1,2,3].map { |k| puts k+1; k+1 }

At least now they both behave identical.

A nice little ‘gem’, wirble, it adds syntax highlighting and auto completion:

sudo gem install wirble

see http://tuxdna.wordpress.com/2011/11/13/autocomplete-and-colorize-your-ruby-shell-irb-with-wirble/

From here on forward porting the command line version of the ‘learn’ script to ruby went fairly straightforward, a couple of minor glitches and a few functions that I had to find a work-around for but nothing that I would not have experienced going the other way (from Ruby to PHP).

I note that the lines on average are a bit shorter than in PHP, they look cleaner but there are definitely more of them, in part this is because Ruby is so terse at times that you feel the need to explain what’s going on. Code + comments = a constant…

Part of this lies in that PHP basically contains anything you would like to use as soon as you start the interpreter in terms of crypto, database, string transliteration, json and so on. With ruby you have to include that explicitly.

Of course there is a very good chance that I simply do not really know what I’m doing in Ruby just yet. Another thing I noticed is that because not everything is in the ‘base package’ you have to resolve external dependencies all the time, there is always ‘one more gem’ to install and it could be quite easy to either forget such a gem or to find that when you build up a machine that your ‘favorite gem’ is no longer being supported. For this reason I’ve always tried to rely as little as possible on external dependencies but this is the ‘ruby way’ I guess. PHP has this too (pecl) and I’ve stayed away from that for much the same reasons. But in PHP land you can get away with that, in Ruby land you need to include those external dependencies just to get basic stuff.

All in all porting the learn.php program took less than a day, I learned a bit more about how ruby works under the hood and I have a better grasp of the syntax. When I’m more proficient at Ruby I should probably re-visit this to see if I can do a better job of it, which hopefully will make the program more compact and/or easier to read.

Here are the cloc stats of both the ruby and the php version:

$ cloc learn.rb
http://cloc.sourceforge.net v 1.60  T=0.03 s (30.1 files/s, 12351.2 lines/s)
Language                     files          blank        comment           code
Ruby                             1            123             92            196
PHP                              1            105             57            193

That’s very close. In fact, if you discount the functions that were added for compatibility with php then ruby comes out slighly ahead.

To gain some more experience, I got rid of the sequel gem and re-did the whole thing using active record. This also lines me up a bit better for using Rails.

Extremely frustrating moment. Examples all over the place on how to use ActiveRecord with ruby. If I use one of these examples (the most trivial), for instance:

Most of those examples/documentation give absolutely no indication what version of activerecord / ruby / possibly rails they are for.

So this works:

>> u = User.where(:nick => 'jam').first
=> #<User id: 2, nick: "jam", pwhash: "$2y$10$Hpp9rAx9d/eWIFi.E.f7pu1YDV4qfzfAwGvMpBzF6wb...", score: 6055, email: "jacques@mattheij.com", newcardsperday: 20, cardspersession: 25, decay: 3, language: "en", initialinterval: 5>

And this does not:

>> u = User.find(:first)
ActiveRecord::RecordNotFound: Couldn't find User with 'id'=first

Even though that example is used in just about every tutorial out there.

Finally I found this guide which appears to be up-to-date and relevant:


Usercourse.joins(:user).where(user_id: 2)

Tbh, after fiddling around with activerecord for a day or so now I’m really wondering how ORMs improve over SQL, but that’s probably just the novelty and the frustration talking. For instance, if I have a database with users collection scores I could do this: “update users set score=score+10 where id=1” without having to worry about locking records or anything like that. A user logged in more than once on the system could hit the server multiple times and it would all work like clockwork. But using an ORM that o-so-simple problem suddenly gets a lot harder. Just using user.score += score; user.save is not going to cut it! After all, who is to say that in another thread the exact same thing isn’t happening, causing one of the updates to be lost! Anyway, that’s a contrived example but you can see that ORMs are not a silver bullet by any means. What they do though is make your code more compact, hopefully easier to read.

The activerecord version of the console application in Ruby clocks in at 164 lines of code, but a good part of that is the elimination the ‘grade_answer’ function and inlining it. The reason why that was feasible is that the active record code was so much more readable compared to hitting the database directly using SQL statements that the duplication of the branches in the grade_answer function and the switch statement jumped out at me. So I merged the two and eliminated the function, resulting in a slightly longer version of the process_answer function but all in all still a good chunk of code elided and a cleaner overall flow. Not that that matters in a toy project like this but in a larger project such savings can add up.

Language                     files          blank        comment           code
Ruby                             1            100             86            161

I should probably work a bit more on this code to move all the business logic to the model rather than to have it spread out all over the program. Especially the fetching of the next card to learn, grading and rescheduling are all bits that should live in the model.

Ok, back to our scheduled program of learning Romanian… Further reading to be done:


I like the concept of migrations, but for now I’ll stick with doing schema maintenance through ALTER and CREATE TABLE statements in mysql, I don’t like the idea of getting so tied into any one eco system that the price of switching becomes too high, it would make evaluating other language/framework combo’s harder skewing the choice towards ruby/rails before all the data is in.

For some reason active record never detected the relationship between the various tables that are present in the schema. Not sure what is going on there, it’s easy enough to fix using some magic in the model but this should have been handled automatically, otherwise what’s the point of defining the relationships in the schema to begin with? I also suspect that under the hood there is a lot more chatter between AR and the database going on than I’m currently aware of but since this is a prototype any kind of attention spent on that would be a serious case of premature optimization so we’ll let all of that rest for now.

Another thing that worries me is the start-up time of the ActiveRecord module. Initializing the model takes well over 1 second on this machine(!) and I’m the only person using it. Ok, it’s a slow box, but still, that seems ridiculously slow to me.

So, after a gigantic detour this is the page where we left off:


In section 1.2 I now finally understand what they meant with this snippet:

<%= link_to 'Patient Record', patient_path(@patient) %>

All it does is reverse the route, given a patient it creates the link to that specific patient.

notes for the routing guide:

  • ‘resources :photos’ generates a whole pile of default routes for standard crud operations on ‘photos’
  • “get ‘profile’, to: ‘users#show’” simple routes look like this
  • singular resources still map to plural controllers, so ‘resource :customer’ will map to the customers controller
  • you can nest controllers inside namespaces like this: namespace :xxx do resources: objects end, this makes /xxx/resources/ etc

This is extremely compact, but there is some risk here as well, you could be creating more routes than you’re aware of and with default controller actions that might cause trouble. Do controllers exhibit default actions?

Here is the spec for all the actions that we want our web application to perform:

  • *the usual account creation
  • *login
  • logout
  • *generating a list of available courses
  • *a single page with a larger description of the courses
  • *a way to start learning a course

  • *a page where the actual learning takes place

  • *a place where the answer to a question gets posted
  • a page where you can track your progress
  • a page where you can view your profile details, courses selected etc
  • a page where you can add cards

Only the actions marked with a ‘*’ are required for the evaluation.

Since it all starts with the routes, let’s lay those out first:

resources :users, :courses
post '/users/login', to: 'users#login'
get  '/learn/:filter', to: 'usercards#learn'
post '/learn/:usercardid' to: 'usercard#answer'

Ok, back in NL 2275 km of driving later…

Re-play the installation of Ruby and Rails on my machine here:

apt-get install ruby

Ah no, that installs 1.8

apt-get remove ruby apt-get install ruby1.9.3

(why don’t they make that the default?)

apt-get install sqlite3 apt-get install libsqlite3-dev gem install rubygems-update update_rubygems apt-get install g++ gem install execjs apt-get install libmysqlclient-dev gem install mysql

gem install therubyracer

for some reason therubyracer did not work for me this time around but installing nodejs seems to have done the trick. Weird dependency!

apt-get install nodejs

gem install rails gem install bcrypt

mkdir ~/test cd ~/test

rails new blog

rails server

=> Booting WEBrick

So even though that’s the second time I’m doing this it still required some fixes to the process. I’ll need to test this on a VM to make sure I really have all the requirements down now.

Fragile! Anyway, it works (again). On to more productive matters.

(verified that ruby learn.rb works again as well).

In preparation to making this into a rails website, I’m first going to move all the business logic into the model. Hopefully that will survive a transition into a web app intact.

How do you ensure that for a given ruby / ruby on rails application all the relevant gems have been installed?

Ah, that’s what ‘bundle install’ is for.

added the bootstrap gem

gem install twitter-bootstrap-rails

Update the gemfile, then run bundle install again

I installed a template from one of those template sites (http://html5up.net/verti).

Installing a template is a bit messy, you have to rip it apart and place the various pieces where rails expects them, and even then it can be quite a chore to work out why things are not doing what they’re supposed to. Rails has an ‘asset pipeline’, a piece of magic that tries to compile all your css and whatnot for speed but frankly it is more trouble than it is worth. In the end I stuffed all the css files in /css/ and did a bunch of @imports to include them in the layout. Not as nice and probably not as fast but for now it works, I can re-visit this later if I want to continue with rails.

So, now we have ‘basic navigation’ and we can add the links to the template. I need to figure out how to log in a user / log a user back out again.

What’s a real pain is that every time I find a tutorial or some piece of information on how to achieve ‘x’ in rails it is invariably out-dated, deprecated, no longer maintained or just plain wrong. For instance, a tutorial on how to create a user signup process in Rails contained a whole bunch of errors, bad advice (which could lead to security issues) and deprecated stuff. Fixing that took a long time.

You’d never guess it but that ‘comment’ at the end of the line is not a comment. This had me stumped for a long time.

validates :password, :password_confirmation => true #password_confirmation attr

So, Ruby -> Rails -> JavaScript (Unavoidable) -> Coffeescript? Wtf. So I have to learn yet another language to be able to use rails. This sucks. By the way, the information on how to do stuff is literally all over the place. Railscasts (what? I have to watch a video in order to do something with a framework? I really would simply like to see a solid example that does not involve me hunting across 12 different websites to pull all the pieces together, let alone some video that was likely out-dated on the day that it was made. Forgive me for being grumpy but if frameworks are supposed to make you more productive then I’m currently not experiencing that feeling).

Finally, after many hours of messing around and reading the answers to this stack overflow question solved it:


I still can’t really believe that that is the real solution but at least it seems to work. You’d think there would be a much more generic way of handling the response rather than to write a bunch of jquery to pull the validation errors out of the json response.

But ok, it works so let’s not complain. Well, it works if you have javascript enabled. I really dislike the way the webs is developing in this sense, javascript should not be a requirement per-se and the amount of cruft that gets added to a webpage in order to achieve something that could have just as easily been achieved by expanding the HTML standard (for instance, by adding a ‘validate’ attribute to fields with a regexp of permitted values, or even a ‘validate_src=’url” for an online validator in case you need to do backend stuff to determine if the contents of a field are valid. Anything but JavaScript!).

Another thing that would really help here is a reference application that you can simply download that shows you ‘how it’s done’ using rails version ‘x’ rather than that every developer has to replicate the effort to piece it all together. Maybe I should post the SRS app sources so people can refer to them.

Useful reading:


Little by little I cribbed together the knowledge of how things are done the ‘ruby on rails’ way, it’s incredible how fragmented the information is.

Moniker.com Is Being Murdered

For many years I was a ‘bulkregister’ customer, I landed there after Network Solutions went mad. Then Bulkregister was bought by enom.com and it went from good to bad to worse in a very short time. I called around to my colleagues to see where they were registering their domains and they pointed me to a small but scrappy upstart registrar called Moniker, the brainchild of one Monte Cahn. Monte was awesome, he worked like a demon and rightfully claimed never to have lost a customers domain.

In 2005 Moniker changed hands for the first time, Seevast Corp bought it and I thought this was going to be the end. I decided to hold out for a while to see how things would go and - surprise - in spite of how most corporate take-overs are done Moniker appeared to be in good hands and I continued to be a customer. On occasion - when the support people were not able to help - Monte would step in to protect the good name of the company that he’d created. All was good.

Then in 2008 Moniker changed hands again. This time to Oversee.net. Again, I bided my time and was rewarded with a company that even though technically a small subsidiary of a larger mothership operated with some independence, and continuity was one of their prime focal points. So I stayed. Occasionally I had issues, most of these were resolved in time and competently, where that wasn’t the case we found ways to deal with it, just another customer-supplier relationship, no longer the personal treatment from the old days but that was to be expected. As these things go, for a company that had changed hands twice it could have been a whole lot worse.

In 2012 KeyDrive SA acquired Moniker.com. And in February 2014 Bonnie Wittenburg (already CEO of KeyDRive) was named CEO of Moniker. Wittenburg has apparently not heard of the age old mantra of computing systems that have been around for a while: If it isn’t broken, don’t fix it. Full of fresh and vigourous energy the CEO declared in February that ‘they’ve gone back to the drawing board’. I wished they’d stayed there. Last week Moniker.com released it’s new and revamped interface, and it is - without exaggaration - an unmitigated disaster.

I’m a pretty loyal customer, but what’s happening at Moniker is at a scale that I’ve yet to see elsewhere. The number of problems, the severity of the issues and the incredible lack of openness with respect to what is going on has managed to completely trash the excellent reputation that the Moniker.com brand had for me in the space of a few days. I’ve been a customer there for almost a decade and I’m more than willing to excuse the occasional error but this is an extremely good example of how not to do a public roll-out of a revamped website. The words “On Monday, June 2nd, 2014, visit Moniker to experience the change for yourself.” in this moniker announcement turned out to be truly prophetic, I have unfortunately tried to visit moniker on June 2nd 2014 and ‘experienced the change for myself’. What was once one of the easiest and most reliable user interfaces used at a registrar has turned into a textbook example of how you alienate an existing customer base.

Let me give you a short listing of what I’ve been able to come across so far:

  • it’s all just change, no improvements

  • stuff has been moved around for no apparent reason, basic common sense design has been replaced by a ‘hunt and seek’ kind of user interface that is totally un-intuitive and does not in any way connect to the old interface, it’s a radical breach with how things were. Sometimes this can be a good thing but imagine Microsoft would re-arrange all the items in the menu system for ‘Word’ at random and would rename a couple for good measure.

  • it’s as if there was absolutely no testing whatsoever.

  • account history is gone (so, how do I do my bookkeeping?)

  • most commonly used functions (such as domain push) do not work

  • newly registered domains are not showing up in the interface (so how do I do my work?)

  • domains that are bought and paid for are stuck in ‘awaiting payment’ even though the invoices for those domains do show up

  • charges are made for domains that have been set to expire long ago (and even for some that had expired!)

  • domains are expiring that are set to auto-renew

  • support questions are going utterly unanswered

  • the support phone system is overloaded permanently, hours on hold without any help whatsoever

  • transfering domains out can only be done on a one-by-one basis

  • I wished I was the only person experiencing all this the comments here are telling

And on and on. Monte Cahn is probably getting quitely drunk somewhere, they’re murdering his baby and it seems there is not much that can be done about it. What a total mess. Imagine, you’re being given the reins over one of the most respected registrars and within 6 months of that you take a legacy like that and totally destroy it. There has to be a prize for something like that.

If they keep this up for a while they’ll be the proud owner of a reputation worse than GoDaddy’s, which I would not have thought possible.

Wittenburg should step down or be fired and someone competent brought back in or KeyDrive will find that they have paid the most money ever for just a single domain because there will be no customers left. As soon as I figure out how to (they definitely did not make this easier) I’ll be taking my business elsewhere. Gandi looks pretty good. I’m only about 1/1000th of the total volume of domains administered through moniker.com, so probably not enough for them to worry about.

The Several Million Dollar Bug

In case you landed here without any context and have no idea who I am or what this article is about this article should give you some background.

For years we were skating on very thin ice. Our only advantage that we had over competitors is that we had figured something out that they had not. Which is that Netscape, Microsoft and pretty much every other browser vendor had made a small but crucial mistake in implementing HTTP. It was an easy mistake to make, and a lot of code would have to be written to ensure that the bug was not present, without making any difference to normal every day HTTP requests.

RFC 1945 says:

The HTTP protocol is based on a request/response paradigm. A client
establishes a connection with a server and sends a request to the
server in the form of a request method, URI, and protocol version,
followed by a MIME-like message containing request modifiers, client
information, and possible body content. The server responds with a
status line, including the message's protocol version and a success
or error code, followed by a MIME-like message containing server
information, entity metainformation, and possible body content. 

So, let’s get this straight: Client connects, client sends request, server responds to that request. Pay attention to the hairline cracks of ambiguity in those sentences and how they are commonly interpreted. The fact that it is written in that order seems to imply that it has to be implemented in that order.

This stack overflow question gets it just right:

My question might sound stupid, but I just wanted to be sure :

    Is it possible to send an HTTP response before having the request 
    for that resource ?

Say for example you have an HTML page index.html that only shows a 
picture called img.jpg. Now, if your sever knows that a visitor will 
request the HTML file and then the jpg image every time :

Would it be possible for the server to send the image just after the 
HTML file to save time ?

I know that HTTP is a synchronous protocol, so in theory it should 
not work, but I just wanted someone to confirm it (or not).

Funny how people are still asking this question nearly 20 years later, and no, that was not a stupid question, in fact, once upon a time I was wondering about just that myself and the answer surprised me. There really are no stupid questions and checking such ‘obvious’ things sometimes pays off in a big way. Which is probably what those responding with their definitive ‘no’s’ to that SO question should have done. Now, for the moment ignoring such details as ‘keepalive’, mixing content types (the question is about html and jpg) and the maximum number of parallel connections that your browser has to the server, in theory, if you knew what the response was going to be before the request even arrived, would it work if you sent the response ahead of time?

And the answer to that question is a resounding YES, you can, it works like a dream and that’s the tiny little item that was the sum total of our competitive advantage. HTTP may be designed as a synchronous protocol but it is not implemented as a synchronous protocol! I figured this out one afternoon when making the early version of the webcam software. The frame rate absolutely sucked, it was 1 frame per second over an ISDN line with a round-trip time in the 100’s of ms or so between server and cam. Sometimes two. But at two channels bonded for a whopping 128Kbit upstream we should be able to do three, maybe more fps. And that’s a huge difference, the difference between ‘animated slideshow’ and ‘almost video’.

So I started toying with the idea of sending the response before the request was in, in fact, totally ignoring the requests! Of course, my initial response was ‘this will never work, for sure the browser will discard the responses to requests that it hasn’t sent yet’. Only it did not! So that’s exactly how we ended up doing it, set the TCP buffer size to roughly the size of a frame, check to see if the frame would fit in its entirety, if not, drop the whole thing and if it did fit then send it. Instant rate adaptation, and maximum frame rate across any kind of connectivity, all the way up to what the hardware could capture/compress/transmit. On a lan you’d get 15 fps (which was absolutely astounding at the time), 320x240, whereas over the WAN links of the day we’d scale it down to something more leisurely (and probably using a smaller image). But still quite good to look at, since it was simply jpeg frames (no incremental updates, so you got a whole frame of reasonably good quality rather than all kinds of blocky updates).

This little trick made lots of money, and I always wondered why our competitors didn’t catch on the little bit of black magic that made it work. All they really had to do was capture some packets in flight and the secret would be out. So much for having a defensible moat, score one for ‘trade secrets’, even if they’re kept in the open. Sending the response before the request is in is not exactly valid in a synchronous client/server protocol, but to this day this trick works wonders. So if you know what the request is going to be feel free to send the response ahead of time, it just might make you some money. At a minimum it will make your users happier, always a good thing.

Every time a major new browser update or such was announced I’d lose lots of sleep, being sure the game would be up, this time they’d close the loophole and we’d be out of luck. But it never did! (Of course, that’s blind luck but if you look long enough at any success I’m sure you’ll be able to identify a number of items that can only be described as lucky. I could re-write this post in terms of how good we were at engineering but that would be a total lie, we just got lucky that it worked and luckier still that it kept working as long as it did. SPDY now tries to make this kind of behaviour official.) Relying on obscure undocumented behaviour is rarely a smart move.

Of course, if you can build out your advantage beyond the trivial to copy then you should do that. But just because you don’t have a real technological advantage doesn’t mean you will be found out in time (or maybe not even at all…). (Unless you blog about it of course, so I guess the secret is definitely out now).

Charities, Direct Marketing and Your Privacy

A recent comment on Hacker News gives a little bit of insight into how charities and non-profits determine who to target for donations and how they deal with requests for removal from marketing lists.

I’m more than a little shocked at the contents. The author, Tom van Antwerp goes into some detail about how his employer, the Ayn Rand Institute operates. (Surprising that such a bastion of capitalism isn’t able to turn a profit because of the useful services it provides for which it commands a fee, rather than to rely on donations to survive, but let’s leave that aside, there’s probably a joke in there somewhere.)

I work at a 501(c)(3), and direct mail is absolutely worthwhile. Maybe 1.5-3% of recipients 
of a prospecting piece actually respond, and they probably won't make up for the cost of 
the mailing. However, those respondents then get more mailings specifically for current 
donors and those mailings make money.

So if you respond once no matter how to a mailing from the Ayn Rand Institute, you’re targeted for more mailings. So far so good, that’s just common sense.

Out of that group, donors are checked against databases of known giving history and net worth. 

Sorry? ‘databases of known giving history’ and ‘databases of net worth’? Just the existence of such databases is a fairly substantial breach of privacy and the fact that any old non-profit with which you have no prior existing relationship should have access to them makes it a lot worse.

These databases aren't always accurate or comprehensive, but it helps fundraisers to ID 
people who might be able to give more--sometimes a lot more.

Never mind the accuracy, once you’re on one of these lists you’re going to get targeted. Now I realize that Forbes has a list of wealthy individuals but that’s quite a step away from having a list of ‘net worth’ on a substantial portion of the population. And that’s still side-stepping the question of how people end up in ‘databases with known giving history’, that means that non-profits actually sell your data to companies compiling those databases, which then happily sell them back to the rest of the non-profits so they can collectively pool their knowledge about who is known to be generous, without taking into account the fact that if someone is generous for one non-profit they might be utterly shocked at learning their data ends up with a whole pile of other non-profits that they have absolutely no relationship with.

When high-value prospects are IDed, they usually get assigned to a major gift officer 
who starts building a personal relationship to solicit very large gifts. This is where 
the charity makes the majority of its revenue. 

I would prefer the term ‘donations’ here, ‘revenue’ is in my head reserved for companies that aim to make a profit.

I usually see 95% of revenue come from the top 5% of donors, largely individuals.

The Pareto principle is alive and well, and if you are a ‘high-value prospect’ you can look forward to being assigned a ‘major gift officer’ (aka an account manager or salesperson).

If you're uninterested in receiving lots of mail, most charities will try to accommodate
that request. However, resources are limited and dealing with the problems of $XX level 
donors often takes second place to the problems of $XXX,XXX level donors. I've seen 
stacks of returned direct mail several feet high from people who moved, died, wanted 
no more mailings, you name it. It was always on the to-do list to remove those names, 
but never at the top of the to-do list.

Right, why would anybody even care about honoring the requests from people that are sick and tired of being bombarded with mail from non-profits they don’t want to give to. It’s all a-ok in the name of the ‘fight for liberty’.

The lessons in here are evident: if you give, especially to the Ayn Rand Institute but this likely holds across the board do it anonymously, lest you end up in a database with ‘prior giving history’ or be ID’d as a high value target which will result in you being assigned a extraction major gift officer. Make sure you initiate the conversation with the charities of your choice, and make sure they do not end up in a situation where they can send you direct mail because you’re likely to receive it well past you meet the grim reaper, and you really don’t want those trees to end up in stacks of returned direct mail several feet high. Maybe the Ayn Rand Institute should do a little housecleaning?

Google Web/Search History Disable Does Absolutely Nothing

There seems to be some kind of misunderstanding about the Google Search/Web History disable switch that google provides to its more privacy conscious users. It’s not exactly the most advertised feature (you won’t find it in your profile page) to begin with, but once you do find it (it’s on the history page, you have to click the little ‘gear’ on the top right and then click the button to switch it off) there is no guarantee whatsoever that google does anything except for changing what they display to you. So if you are under the impression that this changes something about what data google collects on you or how they will use that data then you are likely wrong.

Behind the scenes the data is still collected, in fact, google’s privacy policy pages tells you just that:


Like most websites, our servers automatically record the page requests made when you visit our sites. These “server logs” typically include your web request, Internet Protocol address, browser type, browser language, the date and time of your request and one or more cookies that may uniquely identify your browser.

Here is an example of a typical log entry where the search is for “cars”, followed by a breakdown of its parts:

  • - 25/Mar/2003 10:15:32 -
  • http://www.google.com/search?q=cars -
  • Firefox 1.0.7; Windows NT 5.1 - 740674ce2123e969

  • is the Internet Protocol address assigned to the user by the user’s ISP; depending on the user’s service, a different address may be assigned to the user by their service provider each time they connect to the Internet;

  • 25/Mar/2003 10:15:32 is the date and time of the query;
  • http://www.google.com/search?q=cars is the requested URL, including the search query;
  • Firefox 1.0.7; Windows NT 5.1 is the browser and operating system being used; and
  • 740674ce2123a969 is the unique cookie ID assigned to this particular computer the first time it visited Google. (Cookies can be deleted by users. If the user has deleted the cookie from the computer since the last time s/he visited Google, then it will be the unique cookie ID assigned to the user the next time s/he visits Google from that particular computer).


So the only thing that fancy switch does is to limit what you see, but it definitely does not limit google’s ability (or desire) to collect data about you, and you can rest assured that that is exactly what they’ll be doing.

So all you’re being given here is a false sense of privacy.

And of course, this is just taking an example from the ‘search’ facility that google offers, you should be aware that google has so many points of contact that keeping your browsing habits hidden from them has become all but impossible. Here are a few examples of how your web activity ends up associated with your profile on google’s servers:

  • analytics: This page caused you to be served a google analytics tag, so google knows you visited here. (Come to think of it, google analytics is not worth more to me than your privacy, I should find an alternative and disable GA on my web properties).

  • fonts: That fancy font that is being used on that pretty page? That’s quite possibly served from a google server.

  • google+: On many web pages (again, including this one) you’ll find a google+ button embedded loaded from a google server.

  • google hosted libraries: Many webpages include javascript libraries from a convenient service offered by google.

  • youtube embeds: whenever a page contains a youtube video you’ve just told google you visited that page

  • adsense ads: every page that serves up an adsense ad tells google about your visit to that page. And if you click on an ad it tells google about your personal preference.

  • google app engine: A service where google hosts 3rd party web applications, in other words, sites that do not look like google has anything to do with them at all running entirely on google infrastructure, which means every request in and every answer out of those services passes through google.

  • google dns: google serves up a large number of DNS requests, instances where a certain IP is requesting what address to use to reach some service somewhere else on the internet.

Over the last two months is visited 3490 web pages according to my browsers history, across 615 domains. Of those 43 were google properties. 56 of the 572 remaining contained google+ buttons, 6 contained google platform.js, 94 contained a font served by google, 324 served up google analytics tags, 109 embedded youtube video, 13 contained adsense tags, 75 contained doubleclick tags, 22 used googletagmanager.com, 43 used googletagservices, 47 contained content from googlesyndication.com, 42 contained content hosted on googlecode.com and I’ve probably missed a couple (such as google translate, which of course knows what you translate). All in all of those 572 domains 425 served up some google content, so about 74%, not counting the 43 google properties that I excluded, if we put those back in again then it is 76%. Come to think of it, those urls that google ‘samples’ (you know, when they don’t link directly to the result but through a redirector) are of course also known but I have no idea how many of those 3490 pages I reached through google search.

And that’s just the web, we’re not even looking at mobile, where the android phones are pretty much an extension of google’s infrastructure right in your pocket, with hardware capable of telling google about every spot you visit in real life. So consider google to be riding right beside you and reading over your shoulder all the time, no matter what you’ve done to that history disable button. You’d almost think that all those free and invisible services google provides have one goal: to get you to load something from google on every page that you visit.

As an aside, if I were in law enforcement I would be paying special attention to those searches done by people who have the ‘history’ feature disabled.

edit: thanks to Thomas Bachem for reminding me of DoubleClick and the google DNS servers

Choosing a Web Framework/Language Combo - the SRS Test Application

After creating a short-list of languages and frameworks to be evaluated (see here for the first installment in this series) the next step is going to be a lot harder. It involves getting my hands dirty and at a minimum going through the website of the framework, looking at each in turn to see if they actually match the criteria and then to decide whether or not to build a ‘toy’ application in that framework or not. Time consuming and annoying because it requires you to learn the essentials of each and every framework involved. On the other hand, there is no substitute for hard work in this case, you can’t really pass judgment on anything before you have become familiar with it. The previous article in this series sparked a ton of interest and I got a large amount of very useful (and some not so useful) feedback.

Quite a few people addressed the ‘what framework to use’ question with arguments rather than agenda pushing or religion, corrected a metric ton of mistakes in the previous post and helped to guide me on my quest. To those people I’m extremely grateful for their hard work, every word got read and will be integrated into the revision of the original post. This has caused some changes to the set of languages/frameworks still under consideration.

Toy Application

Because I want to compare the frameworks in as realistic a setting as possible I’ll be building a real application with all of the ones that made it to the short-list.

The toy application will be the thing this all started with, a flash-card (or Spaced Repetition application. It will not contain any user interface components related to creating cards, the database will be ‘magically’ populated with a series of flashcards grouped into several decks that you could use to study/practice some subject. New users should be able to create accounts and select decks they want to study, and the studying process should work as well.

By the way, studying a language over a long-distance medium is not exactly new. In the 1950’s you could already learn languages over the phone!

The schema for the toy application to evaluate the various webframeworks is (relatively) simple. Iteration on the design starts before the first line of code is written. This is the 4th or so revision of what I came up with initially (which was totally different and way too rigid). Some days I feel like I’m in the Monty Python ‘Spanish Inquisition’ sketch. Every time I finish a sentence I have to back up to correct it without ever getting out of the loop and making the sentences longer/different every time!

Apologies for stealing 8 minutes of your life there. And that ‘toy’ application is starting to look pretty complex.

Database Schema

So, the final (hah!) version of the human readable version of the schema looks like this:

We have ‘users’, who use the application to learn stuff and to review and refresh what they learned afterwards.

We have ‘cards’ that have a front and a back side representing the ‘question’ and the desired ‘answer’ to the question.

Cards can be tagged, 0 or more tags are allowed, these are stored in the card separated by ‘/’ characters.

The format of the fronts and backs is json encoded strings, the interpretation of which is left up to the application. Originally I experimented with fixed fields, s-expressions and other trickery but after a short while I realized that since json is now more or less an accepted standard and present as a library module for serialization and deserialization in almost every language I would want to use that this would save me the time of having to implement/maintain a parser for the data chunks.

The trick with this field is that because it is just a text string unforeseen future developments (multimedia, for instance) can be tacked on without re-doing the whole application so it is fairly future proof. All I need to do is come up with a reasonable interpretation of the data. And if the change would be so drastic that the json format itself would need revision then that could be fairly easily done using a background job without having to take the whole thing down. It also allows for easy generation of huge amounts of cards from permutations of the various bits and pieces.

Cards have a difficulty associated with them and you’re supposed to master one level of difficulty before you’re allowed to move on to the next, this ensures that cards that build on knowledge from simpler cards will not be presented until the simpler stuff is fully grokked.

A deck is simply a pre-programmed filter that lifts out cards with certain (combinations of) tags. For instance, this filter would select for an English speaker all the cards that teach you about numbers in Romanian: “/en-ro/ & /numbers/” (note that the ‘numbers’ is in the language of the user, the ‘en’ in the first part of the tag). This also allows for quick, impromptu and unforeseen course creation (such as: learn to count to ten in all different languages: “/numbers/ & /upto10/”)

Every exposure of a card is logged, as is how long it took to get an answer back and whether or not that was the correct answer.

A user can have none, some or all of the cards of any given course in play and has the capability to de-activate specific cards.

Users acquire cards as they learn, new cards are added if fewer than a user configurable number of cards are in the ‘learning’ phase.

Cards can be reversed in a deck, but only if the card allows it.

You could pull really nifty tricks in interpreted languages, for instance, you could make a front contain code that generated a different card every time it was exposed, or have the cards adapt to circumstances (for instance, a card that explains ‘today’ could always use the actual day as an example). For a compiled language such tricks are not so easy, you’d always have to bundle some kind of interpreter along. Or, alternatively, you could assume the display environment to be capable of doing this, and embed a chunk of javascript in the code! We’re not going down that route though (because we also want to do mobile at some point).

For the future, multimedia (audio, pictures, video, LaTex, html, whatever else you can come up with) could be added to the cards.

So, for the first time ever I used mysql-workbench to create a database schema. I wished I could rave about how easy it was and how well it worked to do this simple task. But truth be told I’ve never ever seen such a piece of junk. It crashed a great many times, user interface components are all over the place, it is totally counter-intuitive and makes something simple much harder than typing in the commands by hand. The save dialog had the ‘save’ button off-screen, and it happily creates broken SQL during the export… Incredible that a company like Oracle would let a thing like this pass Q&A with their name on it.

Anyway, after multiple attempts the job was done (count 4 hours or so, not bad for a handful of tables).

Here is the machine readable version of the schema:

-- MySQL dump 10.13  Distrib 5.5.37, for debian-linux-gnu (i686)
-- Host: localhost    Database: srs
-- ------------------------------------------------------
-- Server version       5.5.37-0ubuntu0.14.04.1

/*!40101 SET NAMES utf8 */;
/*!40103 SET TIME_ZONE='+00:00' */;

-- Table structure for table `cards`

/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `cards` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `creator_id` int(10) unsigned NOT NULL COMMENT 'who made the card\n',
  `reversible` int(1) DEFAULT '0' COMMENT 'not all cards can be reversed\n',
  `front` mediumtext COLLATE utf8_bin NOT NULL,
  `back` mediumtext COLLATE utf8_bin NOT NULL,
  `difficulty` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'difficulty is a number that starts at ''0'' and ascends as far as required to order cards. Two cards can be of the same difficulty (they their order of appearance is not guaranteed).\n',
  `source` int(10) unsigned NOT NULL DEFAULT '0',
  `prompt` int(10) unsigned NOT NULL COMMENT 'which prompt to use\n',
  `tags` varchar(256) COLLATE utf8_bin DEFAULT NULL,
  `prerequisites` int(10) unsigned NOT NULL COMMENT 'up to and including which card should be learned before the user can be exposed to this one.\n\n',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`),
  KEY `source` (`source`),
  KEY `creator_id` (`creator_id`),
) ENGINE=InnoDB AUTO_INCREMENT=1505 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='cards are the ''units of learning'' in the system, cards are t /* comment truncated */ /*agged, the first tag is the name of the deck. If a user makes a card for themselves then it starts out without tags.\n\n*/';
/*!40101 SET character_set_client = @saved_cs_client */;

-- Table structure for table `courses`

/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `courses` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `description` varchar(250) COLLATE utf8_bin DEFAULT NULL COMMENT 'A description of the course, if it is a language course it should be in the language of the student\n',
  `tag` varchar(45) COLLATE utf8_bin DEFAULT NULL COMMENT 'the unique tag identifying cards in this course',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='a course is something a user could follow, for instance "Rom /* comment truncated */ /*anian for English speakers", using the unique tag "en-ro" for the cards.\n\n*/';
/*!40101 SET character_set_client = @saved_cs_client */;

-- Table structure for table `exposures`

/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `exposures` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `side` int(11) DEFAULT NULL,
  `timestamp` int(10) unsigned NOT NULL DEFAULT '0',
  `duration` int(11) NOT NULL DEFAULT '0',
  `user_id` int(10) unsigned NOT NULL,
  `card_id` int(10) unsigned NOT NULL,
  `answer` varchar(256) COLLATE utf8_bin NOT NULL COMMENT 'as long as the answer is null there was no answer from the user for this card\n',
  `score` int(10) unsigned NOT NULL DEFAULT '0',
  `grade` int(11) DEFAULT '0',
  PRIMARY KEY (`id`),
  KEY `fk_exposures_users1_idx` (`user_id`),
  KEY `card_id_idx` (`card_id`),
  KEY `grade` (`grade`),
) ENGINE=InnoDB AUTO_INCREMENT=481 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='an exposure is a single showing of a  card to a user, and wh /* comment truncated */ /*at their response was*/';
/*!40101 SET character_set_client = @saved_cs_client */;

-- Table structure for table `user_cards`

DROP TABLE IF EXISTS `user_cards`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `user_cards` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `user_id` int(10) unsigned NOT NULL,
  `card_id` int(10) unsigned NOT NULL DEFAULT '0',
  `first` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'first time this card was exposed\n',
  `last` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'last time this card was exposed\n',
  `next` int(11) NOT NULL DEFAULT '0' COMMENT 'next time this card should be exposed',
  `interval` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'the interval between showings\n',
  `reversed` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'reverse the card or not on viewing\n',
  `exposures` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'times the card has been exposed\n',
  `correct` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'times the user got it right\n',
  `wrong` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'times the user got it wrong\n',
  `score` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'score is in percents, so 10 exposures correct would yield a score of 1000\n',
  `state` int(10) unsigned NOT NULL DEFAULT '1' COMMENT '0: inactive\n1: active\n2: ignored\n\n',
  `learning` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'card is currently considered to be in the learning phase or if it should count as ''learned'', this counts down from a number to ''0''\n\n',
  `tags` varchar(250) COLLATE utf8_bin DEFAULT NULL COMMENT 'initially translated to the language of the user, these are copies of the tags the cards are created with, but the user can modify them\n',
  `difficulty` float DEFAULT '0',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`),
  KEY `user_id_idx` (`user_id`),
  KEY `card_id_idx` (`card_id`),
  CONSTRAINT `user_cards_card_id` FOREIGN KEY (`card_id`) REFERENCES `cards` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
) ENGINE=InnoDB AUTO_INCREMENT=77 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='user cards are all cards in play for a user at this moment,  /* comment truncated */ /*they are in verification or learning mode or have been set to be ignored\n*/';
/*!40101 SET character_set_client = @saved_cs_client */;

-- Table structure for table `user_courses`

DROP TABLE IF EXISTS `user_courses`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `user_courses` (
  `user_id` int(10) unsigned NOT NULL,
  `course_id` int(10) unsigned NOT NULL,
  `score` int(11) NOT NULL DEFAULT '0',
  PRIMARY KEY (`user_id`,`course_id`),
  KEY `fk_user_courses_users1_idx` (`user_id`),
  KEY `fk_user_courses_courses1_idx` (`course_id`),
  CONSTRAINT `fk_user_courses_courses1` FOREIGN KEY (`course_id`) REFERENCES `courses` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
  CONSTRAINT `fk_user_courses_users1` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='a user_course is a course that a user has started on';
/*!40101 SET character_set_client = @saved_cs_client */;

-- Table structure for table `users`

/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `users` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `nick` varchar(45) COLLATE utf8_bin NOT NULL,
  `pwhash` varchar(250) COLLATE utf8_bin NOT NULL,
  `score` int(11) DEFAULT '0',
  `email` varchar(250) COLLATE utf8_bin NOT NULL,
  `newcardsperday` int(11) DEFAULT '20',
  `cardspersession` int(11) DEFAULT '25',
  `decay` int(11) DEFAULT '3',
  `language` char(2) COLLATE utf8_bin DEFAULT 'en',
  `initialinterval` int(11) DEFAULT '5',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`),
  UNIQUE KEY `nick_UNIQUE` (`nick`),
  UNIQUE KEY `email_UNIQUE` (`email`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='users are people that use the system\n';
/*!40101 SET character_set_client = @saved_cs_client */;


-- Dump completed on 2014-05-27 10:52:39

SRS Algorithm

I’m going to do the actual SRS stuff based on ‘exponential back-off’. If you respond to a question immediately and get it right then you earn a 5x increase in schedule interval. If you get it right but after a longer pause then the scheduling interval remains unchanged, get it almost right (within 1 character of the right answer) (this only works for languages!!) the interval will be reduced by a factor of 5, get it completely wrong and you start all over again from the initial interval (set to 5 seconds for now).

Intervals below 5 seconds will be reset to 5 seconds.

So this makes the scheduled intervals 5, 25, 125, 625, 3125, 15625, 78125 seconds etc. So after answering right 7 times in a row you’re on a rough schedule of reviewing a card in another day. Of course people will use this whenever they feel like so we need to make sure that we don’t overwhelm them with a backlog of cards to review, and we need to present those cards that are most likely to be forgotten first.

To allow some personalization we can make the ‘factor’ (5 in this case) configurable on a per-user basis, maybe automatically adjust it for users that score particularly good (or bad) once we have enough data to figure out how well they retain information.

I don’t like the concept of ‘lessons’, so I’ll drop that entirely, you’re ‘learning’ a deck or you’re doing something else, you can learn as long as you want and you can ‘pause’ or ‘stop’ the process at any point in time.

I don’t want users to ‘grade themselves’, the program should do any and all grading completely automatic (this creates some very interesting challenges later on when audio support gets added). And I want people to be able to start learning without being logged in the first time they hit the site.

I’m keeping all this information nicely organized in a Leo Outline Editor file, Leo is absolutely fantastic and I wished there was a version of it that used a centralized docs store.

Command-line version

The simplest framework is of course none at all.

To make sure I understand all the business logic involved and have a working knowledge of the whole process I first built a command line version of the program playing off the same ‘magically populated’ database. That way I can very quickly test stuff without having to bother with layouts, RPC, servers, client server issues and other complications. An interactive command line version of this is about as simple as it gets (and can be built in a few hours if you have some experience).

So, that seems to work. And if it is ugly/buggy that’s fine, it did what it was supposed to do, bring out the flaws in my reasoning about the data model.

The Stack

If you’re reading this far you no doubt have an excellent understanding of web development, but just in case you don’t: The web is an ugly thing these days. What used to be a single binary serving up some static files from a directory on a server has turned into a whole slew of tecnologies, software, standards and options. Collectively, any set of specific choices from all those tools is called a ‘stack’. Typically, a stack will include an operating system, some sort of database for persistence, a back-end language (server side stuff), a front end language (typically JavaScript or something that compiles down to JavaScript), a web server (for instance, Apache, nginx) or ‘application server’, and some bits and pieces to glue it all together. The reason for this hodgepodge of loosely connected bits of tech is that we’re trying to shoehorn application interaction across a medium designed to serve static files on a fire-and-forget basis.

In my case, only the operating system (Linux) and the database (MySQL) are set in stone, the rest is variable. I do have an apache installation on this development machine so if any of the language/framework combos that we will be looking at can run under apache I’ll definitely try to get that to work since it stops me from having to learn how to configure yet another web server. I’m a lazy person.

Next up, the language/framework combos.

A Western Kid Living in Communist Poland

In this HN thread I commented on how the story about the People’s Republic of Donetsk reminded me of my days in communist Poland.

User bernardom prompted me to tell this story, so here it goes (I’ve slept on it and I don’t think it will do any harm, even though all the participants are alive (and well)).

I got my driving license in 1986. My dad and his wife were looking to buy a new car and were going to trade in their old Citroen 2CV. So I matched the offer they were getting on their trade-in and I got my first car. The 2CV (named after it’s fairly anemic engine with according to the French tax office ‘2 tax horse power’) is probably best described to people that have never been in one as an extermely cleverly designed tent on wheels. It’s about as spartan as a car can get, compared to the 2CV a Mini is luxurious.

I worked in Amsterdam west at the time and of course I would slowly expand my confidence driving to work every day and back. The car was so slow to accelerate that I did not feel comfortable on the highways so I stuck to city traffic and b-roads. This meant that longer trips were out of the question but given how crappy a driver I was in those days that’s probably a good idea. Two weeks later my good luck ran out. It was the first real frost that year, November 1986 (COBOL was going strong, I was an applications programmer for a dutch bank back then) and when I accelerated away from a traffic light just past central station in Amsterdam a Volvo in oncoming traffic skidded, carreened down a slope into one of those old style cast iron road dividers. Two metric tons of Swedish steel impacted the divider, which probably weighed no more than 80 kilos or so, accelerating it to projectile speeds.

What saved my life was probably playing a lot of pingpong. Reflexes took over and I ducked down just as the barrier swept through the car, without slowing it down much. The whole top of the car was gone, it was an instant convertible. I pretty cooly steered the car to the side of the road, started shaking like crazy and waited for the police to arrive, who asked ‘where the body was’. I realized I had been very lucky.

The wreckage was towed to my house in east Amsterdam and my mom broke down spontaneously when she saw the wreck. It was quite incredible that I walked away from that without a scratch.

But I learned a valuable lesson: not all accidents are your fault, and if you drive a tent on wheels you could very well end up as roadkill when someone else messes up. So I started looking for something a bit more sturdy. My mom had had a long time boyfriend, a man named Hans. Hans was super cool, an arts dealer in Amsterdam, fancy house on Herengracht and a super nice car, a Citroen DS.

If you’ve never seen one, prepare to be amazed, it was designed in 1955 and is by my standards still one of the most advanced cars ever to be produced. It also still looks good today:

I started hunting for one through the various DS specialist garages in Amsterdam (it is still quite a popular classic in NL today) and found one right around the corner, light blue with a white roof, built in 1973 and with ‘only’ a few hundred thousand kilometers on it. For 3750 dutch guilders it was mine, rust and hydraulic leaks and all.

Now, here was a real car! I had absolutely no problem taking it on the highway and so I decided to tour the highways near Amsterdam.

As I was about to enter the Coentunnel I noticed two pedestrians hitchhiking right at the mouth of the tunnel entrance. A very dangerous spot to put it mildly. Being my usual helpful self I stopped the car, bundled them into the backseat and drove them to the other side where I exlained to them that that was a very bad idea. The hitchhikers, two girls from Poland explained that they didn’t know much about the Netherlands, but that they were walking to a place called Egmond. From where I picked them up to Egmond was at least another day worth of walking so I offered to drive them there.

We talked a bit more, I figured out they were quite hungry and offered to get them some food. This they declined. Then I told them I was hungry and had to get some food. They agreed to come along and we stopped in Alkmaar to eat. Maybe they would eat a bit after all. After both had polished off double helpings of everything we continued on our trek to Egmond.

On arrival in Egmond we said goodbye, I gave them my phone number in Amsterdam in case they got themselves in trouble and went back home. About a week later they called me. Could they please come to Amsterdam? In Egmond they’d been pretty much enslaved by a flower bulb grower, who made them work day and deep into the night for very little money, and who would then charge them for everything (food, lodging) to get it almost all back. Modern slavery (yes, we Dutch are nice people, and if you think that such things are a thing of the past now then think again, this sort of thing still happens all the time). I went back there, picked them up and they moved into my house in Amsterdam.

Calling it a house is a bit much. Today you’d probably call it a hacker space. It was a 60 square meter partition of an industrial building. No shower, just a small sink, an electrical stove and tons of computing gear, a pingpong table, a couch and a single mattrass on the floor.

I gave them the bed, made mine on the couch and for the next week or so we talked day in day out about what life was like in Poland and what it was like in the Netherlands.

When they left they gave me their addresses in Poland and invited me over.

For months that invitation burned a hole in my pocket. I was totally enamoured with one of the girls, and made up my mind to go and visit Poland. This was a lot easier said than done.

I got my first taste of what Poland would be like when I entered the Polish embassy in the Hague. A line of about 50 people. Before I got to the head of the line the office closed and I had to go back another time. This time I took care to arrive early. I applied for a visa and filled in a stack of forms. For months my visa application was held up. More and more information was required. What did my grandparents do during the war? How much money did I have? Did I have family? Sisters, brothers, everything! At some point it occured to me they knew more about me than my mom did.

But finally, after months of filing ever more papers I got the call, I could come and give them my passport. Then they would stamp it after the office had closed and then a few days after that I could go and pick it up again.

And so it happened. I rolled into Poland somewhere in the early hours of morning August 1987. Lines of tens of people in front of non-descript shops had already formed. People waiting for the bakeries to open to get bread. Be early and you might have a chance. Initially I did not comprehend what the lines were for, my mind had been totally conditioned to ‘want something, go out and buy it’. The whole idea of shortage was something that I was not familiar with. At that time, The solidarity movement was clashing headlong with the Polish communist government and there was quite a bit of tension in the air. I was received warmly in Poznan, spent about a month there and then went back to the Netherlands. What struck me most about Poland is how serious the people my age there were. And how tremendously strong the friendships. These were in many ways the most interesting times you could have possibly picked to be behind the iron curtain, I got to observe the whole underground movement from the point of view of a spoiled western kid, retrospectively one of the formative moments in my life. Solidarity was quite something, the streets were abuzz with it. The Polish secret police (the UB or MBP) was quite active and if you didn’t know someone personally you could not trust them. This caused the Polish people to form tremendously strong friendship bonds, a thing that is probably a lot less strong today in absense of such formidable enemies. The police in Poland was tremendously corrupt at the time, a nice example of power out of control.

For months my newfound girlfriend and me corresponded by good old snail mail. We got married, lived together in Amsterdam for some years and then moved to Poznan, Poland together.

This created a problem because I could not legally work in Poland. So I applied for a work permit, and this is one of the most bizarre experiences I’ve ever had. It wasn’t so much that such permits did not exist, it was that applications were so rare that almost nobody knew how the whole process worked. It took many months, ever more interviews (my Polish at that time being next to non existant), and lots and lots of forms. Ever more forms. The most frequent question was why on earth I would want to live in Poland when everybody that could was trying to get out.

Finally I got it, after I had all but given up. I still have it in Amsterdam in a suitcase somewhere, I should probably scan it and put it in here when I’m back there. Somewhere along the way a Texaco gas station opened in the middle of Poznan.

Imagine a background of gray highrises, no colour anywhere, all the cars gray and the ancient diesels of buses belching smoke. And in the middle of that an island of bright red, a tiny patch of the west invading on all that gray, a harbinger of what was to come. A friend of mine said this: “If the future is going to be so full of bright colours I don’t know if I can stand it.”.

Right now I’m living in Romania (Bucharest to be precise) and it’s funny how I can trace back almost everything major that happened in my life to that point where I stopped the car.

So, that’s the story of the Hitchhikers, the flowerbulbs and the ancient Citroen.

Thanks G. for changing my life, for the better in all respects.

edit: and thanks I. for correcting my broken memory!