Jacques Mattheij

technology, coding and business

It Is Not About the Money, Silly! It Is All About the Time.

One of the oldest business sayings that stuck into my head was ‘Time is Money.’. It’s deceptive in its simplicity, it generates in the mind of the beholder of this simple 3 word adage that there is a direct equivalent between time and money, and that that equivalence runs both ways.

Ways in which you see this at work in every days life are people working (spending their time) in order to get some money. People spending their money to give them more time (for instance, by using some form of mechanised transportation to arrive at their destination sooner, bicycle, moped, car, jet etc) is another way in which it is illustrated.

And so we all seem to have blindly bought in to the fact that in order to be able to spend money we will have to expend our time and in order to be able to maximize the time we have available we have to spend more money. And thus the treadmill was born. And once you’re on that treadmill it gets very hard to get off it.

Before you know it you’re hip-deep in debt (credit cards, mortgages, car loans, student loans) and you need to work even harder (and so spend more of your precious time) in order to keep up with the now much higher speed of the treadmill.

But nobody is born into debt (at least, not in any functional society), so at some point in our lives we derail from the path where we are debt free onto a path where we are no longer debt free and take out a mortgage on our future health, well being and free time budget in order to be allowed to enter the treadmill and then we willingly assist in speeding it up.

When I look around me I see people making endless purchases of stuff they don’t actually need, brandnew cars, houses large enough to live in with two families (and then there’s only two people living there), parents that barely see their kids because of the pressure to earn money and a continuous stream of complaints about how it is absolutely impossible to make ends meet.

I don’t understand any of it. Maybe it is because I don’t allow any advertising at all into my life but the draw of all these gadgets and the need to show off wealth by buying expensive brand clothing and so on are things that I’m a stranger to. In a way I consider myself very lucky, I’ll buy what I (think) I want when I want it and because I can (easily) afford it. Not because there is some kind of peer pressure or a sense of having to keep up.

In my family there is an interesting set of divides between people with some money, people somewhere in the middle and people with no money at all. And in my circle of friends and their extended families there is a similar set of divides. Most of the complaining about the situation comes from the middle group, which really surprises me. The people that have no money are surprisingly good at managing the little bit that they have. Month after month they impress the hell out of me by making a couple of credits stretch well beyond what you’d normally think would be possible. Absolutely superb money management skills. The people that are somewhat wealthy have little reason to complain and none of them do.

But oh, that middle group. Two incomes, 160K euros / year jointly is not exceptional. That’s a very large amount of money. And yet, at the end of every year it’s all gone. They can’t get rid of their mortgages (someone explained to me in a very serious tone of voice that getting rid of your mortgage is stupid because it is a deductible, better to wait with paying it off when you sell the house in 25 years), they are supremely bad at figuring out what the cost of all this consumption driven borrowing is and how big a difference it would make if instead of buying stuff they’d be saving for a while (or at least, lowering their mortgage).

It’s almost as if common sense about finances is something that is extremely rare. People that are by their own admission barely able to make ends meet show off their new cars, smartphones, laptops, tablets and so on. And all bought on ‘cheap’ credit. But that credit really isn’t all that cheap (read the fine print on your credit card statement, if you have a 1.9% card make sure that it isn’t accidentally per month instead of annually).

Seriously. If you are making good money, you’re in debt and at the end of the year you are further in debt or the situation hasn’t improved then learn to control your spending. Saving is so much easier than earning, and it’s a habit that once built will pay you back for the rest of your life.

And once you’re out of debt, maybe have a bit of an investment strategy going you can safely spend the surplus on things that you can actually afford. But until you reach that milestone getting rid of debt should be priority #1 and the gadgets, the new car and other luxury spending should have to wait.

And then, when you are in that luxurious position you can maybe slow down that treadmill again and recover some of your precious time. That’s the only form of capital that you will NEVER have more of. So spend it wisely.

Choosing a Web Framework/Language Combo - Ruby on Rails

In case you’re tuning in late, the previous installment in this series is here. This is an absolutely ridiculously long blog post, feel free to either totally ignore it or to read only the top portion, it will likely contain all you need or want to know to keep up to date with what’s happening here. The rest is more of a detailed log of what I did to be able to get back into it quickly if I decide that ‘Ruby Is the One’.

A picture paints a thousand words I guess:

And yes, it works, you can actually learn Romanian with it, so I guess that’s ‘mission accomplished’ :) It’s also butt-ugly, works on mobile, tablets and so on (courtesy of the template, I did not do much work on that other than to resolve some conflicts).

The short version of what the last couple of weeks were like (resulting in that screenshot) is:

  • Install RubyOnRails

  • Dive headlong into rails

  • Run aground, decide to learn the Ruby language first

  • Learn Enough Ruby to make sense of rails

  • ported the PHP command line application to ruby

  • re-write all database interaction to use activerecord

  • move all the business logic into the data model

  • port the datamodel over to the demo rails application

  • drive to Amsterdam

  • re-install RoR on the machine here at the office

  • make a ‘user creation’ form

  • make a ‘login’ form

  • figure out how to log-in a user

  • install a template and chop it up

  • make the ‘available courses’ page

  • make the learning page

  • make the feedback from the server about the answer given work

  • make a top row that shows the user info + logout link

All in all it was more work than I thought it was and I ended up spending a frustrating amount of time just searching for stuff and being stuck on simple little details. The problem with rails is not the the information isn’t out there, the problem is that there is a ton of stuff out there that simply is no longer relevant but because everybody is playing the SEO game that data will forever pollute your ability to find out what is relevant and what is not. In the end it worked out though and I’m pretty happy with the result. This of course does not say anything yet about how it would be to deploy this particular solution, it only means I’ve managed to build a first version.

My first impression is that developing on RoR is (when you know what you’re doing, always helpful) fairly quick but the result is quite fragile, far more fragile than I expected. This is in part due to Ruby the language (which Rails can’t really do all that much about) which has some peculiar quirks and in part because of the way Rails is set up, it’s all ‘conventions’ and conventions translate into ‘things you should know and learn by heart’. This makes it a bit harder to get going. I’m used to reading a program much like I’d read a book, and that’s also how I normally write programs. In comparison, working with Ruby on Rails feels like someone tore the book up and scattered the pages in a hundred different places. Ruby is a funky little language that allows you to take all kinds of short-cuts, the Ruby code was on average about 25% smaller than the corresponding PHP code. And I don’t know Ruby and I know PHP like I know my back pocket so likely with some more knowledge I could save more code.

Another thing that strikes me is that Ruby is a language that almost asks for unintended side-effects to be present in your code. And that in turn is an excellent way to get security issues, if your ‘counterpart’ knows Rails (much) better than you do chances are that they’ll find something exploitable that you overlooked.

Even so, I’m fairly impressed with how concise the final solution is, how quick it got built In spite of all the hours of frustrating searching and the continuous feeling of going very slow and working ‘against’ the frame work only 16 days have passed since the last installment, when I didn’t know anything about Ruby yet, and 3 of those days were spent on the road driving from Bucharest to Amsterdam.

This code is so fresh and most likely contains tons of security holes and bugs that I’m not confident to put it out there, besides, this is only the evaluation of Ruby On Rails and the list of frameworks that remain is a long one.

The logfiles (you *really* should stop reading here)

Warning, this is an extremely long blog section detailing weeks of work to try to get up to speed with Ruby, then a couple more to work through Rails tutorials and then more still to get a minimally working version of the SRS application up and running. If you just want the ‘verdict’ you’ll have to wait until I have done this with all the platforms so I can compare them side-by-side, think of this as a progress report and a way to segment the various platforms into one-page-each. It is written the way it is so you get an idea of the work I did, in what order I did it and what I was thinking while I did it and what issues I encountered along the way and how I (hopefully) solved them. This is probably of limited use to you unless you are currently evaluating Ruby / Ruby On Rails.

I’m going to try to keep the external representation of the various implementations as close together as I can. So starting off with a ‘full fledged’ batteries included combo like RoR is a good way to find out what is missing in other offerings, and it will also get the first web based version of this working in (hopefully) short order. I don’t recall having ever written a line of Ruby before so consider me a total noob when it comes to this particular set.

While you’re reading this keep your eye on the ball and remember that I’m trying to learn some Romanian.

I start out where I imagine everybody else starting with Ruby On Rails starts out, the getting started guide and going step-by-step.

ruby -v ruby 1.9.3p484 (2013-11-22 revision 43786) [i686-linux]

sqlite3 –version 3.8.2 2013-12-06 14:53:30 27392118af4c38c5203a04b8013e1afdb1cebd0d

gem install rails

After a long pause and a whole bunch of disk grinding and fan noises:

    "ERROR:  Error installing rails:
    ERROR: Failed to build gem native extension.

    /usr/bin/ruby1.9.1 extconf.rb

/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require’: cannot load such file – mkmf (LoadError)

    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from extconf.rb:1:in `<main>'"

Some googling later it turns out that ruby-dev is also required:

apt-get install ruby-dev

Maybe someone should update that tutorial (and retry it periodically to see if it all still works as advertised). Little gotchas like this in the first few minutes can turn off newbies and will reduce adoption.

gem install rails

More disk grinding. It’s pretty hot here today and the gem installer uses a lot of CPU cycles according to the sounds coming from the machine. The machine I’m doing this all on is pretty ancient, all of 2G of ram and a regular drive. My ‘main’ machine sits back in the Netherlands under my desk there and would probably make short work of all this but I have this one to work with and I guess it is a lot better than chiseling 1’s and 0’s into stone tablets. After a good 15 minutes it is finally done. Rails is installed :)

rails -v Rails 4.1.1

Neat. So at least it looks like once you get past that ruby-dev hickup the rest works as advertised.

On with the tutorial!

rails new blog

After a couple of minutes this prompts for my root password, then after yet another couple of minutes tons of errors. Scrolling back I find that this is caused by not having sqlite3-dev installed (a header file is missing).

Easy to correct.

sudo apt-get install libsqlite3-dev

On error, retry:

rails new blog

This asks me if I want to overwrite a secrets file, I’ve yet to modify files here so I say ‘Y’.

A few minutes later the installation completes without further issues. At the end there is this line in the output:

“Warning: You’re using Rubygems 1.8.23 with Spring. Upgrade to at least Rubygems 2.1.0 and run gem pristine --all for better startup performance.”

Ok, sounds good. How do you upgrade Rubygems? More googling leads me to:

gem install rubygems-update

followed by:


Which seems to install RubyGems 2.2.2

The tutorial next advises to start the embedded rails server from within the blog directory:

rails server

Only that doesn’t work…

/usr/local/lib/site_ruby/1.9.1/rubygems/dependency.rb:298:in `to_specs’: Could not find ‘railties’ (>= 0) among 0 total gem(s) (Gem::LoadError)

    from /usr/local/lib/site_ruby/1.9.1/rubygems/dependency.rb:309:in `to_spec'
    from /usr/local/lib/site_ruby/1.9.1/rubygems/core_ext/kernel_gem.rb:53:in `gem'
    from /usr/local/bin/rails:22:in `<main>'

In a stackoverflow thread I found a hint that I should re-run ‘gem install rails’ as a regular user:

gem install rails

Rather than as root. So I tried that.

“You don’t have write permissions for the /usr/lib/ruby/gems/1.9.1 directory.”

This is a very bad idea:

chmod -R 777 /usr/lib/ruby/gems/*

So we won’t be doing that. Let’s try again as root:

sudo gem install rails

Try again:

cd blog rails server

/usr/lib/ruby/gems/1.9.1/gems/execjs-2.0.2/lib/execjs/runtimes.rb:51:in `autodetect’: Could not find a JavaScript runtime. See https://github.com/sstephenson/execjs for a list of available runtimes. (ExecJS::RuntimeUnavailable)

Hm. What does ruby or rails have to do with finding a javascript runtime?

More googling. In this stackoveflow thread I read that I have to enable a gem called ‘therubyracer’ in a file called ‘gemfile’. I load ‘gemfile’ into vi and indeed, there is a line like that. Remove the ‘#’ in front of the line (# marks are used as a comment indicator in ruby).

rails server

Could not find gem ‘therubyracer (>= 0) ruby’ in the gems available on this machine. Run bundle install to install missing gems.


bundle install

Tons of errors, the topmost of which suggests that I should install g++ first.

sudo apt-get install g++

bundle install

At this point we’re about 1.5 hours into the installation process. To save RAM I’ve killed firefox.

Lots of grinding and about 1/2 hour later:

“Your bundle is complete!”

rails server

=> Booting WEBrick => Rails 4.1.1 application starting in development on => Run rails server -h for more startup options => Notice: server is listening on all interfaces ( Consider using (–binding option) => Ctrl-C to shutdown server [2014-05-26 22:49:55] INFO WEBrick 1.3.1 [2014-05-26 22:49:55] INFO ruby 1.9.3 (2013-11-22) [i686-linux] [2014-05-26 22:49:55] INFO WEBrick::HTTPServer#start: pid=7154 port=3000

Started GET “/” for at 2014-05-26 22:50:07 +0300 Processing by Rails::WelcomeController#index as HTML Rendered /usr/lib/ruby/gems/1.9.1/gems/railties-4.1.1/lib/rails/templates/rails/welcome/index.html.erb (34.9ms) Completed 200 OK in 325ms (Views: 215.4ms | ActiveRecord: 0.0ms)

Yay! It appears we have a working RoR installation.

Excellent. It took a while but it looks like it is working.

The built in server is quite convenient when testing code on your local host.

Next the getting_started guide walks you through a bit of configuration and then you will be able to see your test page. After that you are prompted to create a ‘resource’, by making some changes to the config/routes.rb file.

The listed example shows that you should make it look like this:

Blog::Application.routes.draw do

resources :articles

root ‘welcome#index’ end

But that throws up all kinds of errors.

rake routes

rake aborted! NoMethodError: undefined method `application’ for Blog:Module

Changing ‘Blog’ back into ‘Rails’ seems to fix this for now (my understanding of the routing dsl is so minimal at this point that I have no clue what this all means, I will read the doc on the routing in a moment, maybe then I’ll have a better understanding).

Next we add a form and there is something weird about that, the tutorial tells you to change the first line of the form to:

<%= form_for :article, url: articles_path do |f| %>

But gives no explanation why the ‘url:’ has a trailing colon and the rest of the : characters in that form are at the beginning of the terms. It doesn’t explain much of anything anyway but this is a little strange.

The explanation underneath reads:

“The form needs to use a different URL in order to go somewhere else. This can be done quite simply with the :url option of form_for. Typically in Rails, the action that is used for new form submissions like this is called “create”, and so the form should be pointed to that action.”

Hopefully this will be explained in:


So time to do some reading.

So what is it? url: or :url? According to the routing docs it is indeed url: so the tutorial got that bit wrong in the text but right in the example.

So the magic works, the predicted ‘Template is missing’ error shows up.

Next we add a bit of code to allow us to see what the controller receives.

There is this sentence in the tutorial at this point:

“The render method here is taking a very simple hash with a key of text and value of params[:article].inspect.”


“render plain: params[:article].inspect”

And I really can’t make soup of this. What hash? What key of ‘text’?

I suppose the ‘key’ refers to the name of the field ‘text’, but that value then makes very little sense (after all, there are two fields, title and text), and that still does not explain the ‘hash’.

At the end of section 5.6 we’re told to insert the following bit of code to get around a ForbiddenAttributesError:

private def article_params

params.require(:article).permit(:title, :text)


But after I do this and refesh the page the error remains.


Says: you appear to be following a pre rails 4.0 tutorial with rails 4. You need to use strong params now.

But at the top the tutorial explicitly confirms that it should work with rails versions > 4.

I use the second line in the SO post answer:

# @comment = @post.comments.create!(params.require(:comment).permit(:comment_text,:link) 

Convert its format to what I think should be required to make it work for articles:

# @article = Article.create(params.require(:article).permit(:title,:text))

Comment out the original line that made a new article:

# @article = Article.new(params[:article])

And commented out the bit that section 5.6 said I should have added:

#  def article_params
#    params.require(:article).permit(:title, :text)
#  end

And try again. Now rails complains about not having a ‘new’ action.

Checking above, I did not change ‘create’ to ‘new’. That’s an easy fix.

And now it indeed gives me:

“The action ‘show’ could not be found for ArticlesController”

Whoever maintains that tutorial certainly hasn’t been paying attention lately, and without stackoverflow and google to help I would have either thrown my computer out the window at this point or de-installed rails. Maybe this is some kind of secret plot by the RoR overlords to test newbies for the joint start-up required qualities of persistance and ingenuity? Or maybe they just don’t like newbies and want them to go away. I don’t know which it is but I can’t imagine this tutorial is leaving the people following it in good spirits.

Anyway, on we go.

I add the following line to config/routes.rb:

article GET    /articles/:id(.:format)      articles#show

And hit refresh:

/home/jam/svn/src/srs/www/ror/blog/config/routes.rb:4: syntax error, unexpected ':', expecting keyword_end article GET /articles/:id(.:format) articles#show

Most helpful. I change the line according to the format listed in an example lower down in the routes file to read:

get '/articles/:id(.:format)' => 'articles#show'

With ‘rake routes’ I verify that the route is indeed present:

      Prefix Verb   URI Pattern                  Controller#Action
             GET    /articles/:id(.:format)      articles#show
    articles GET    /articles(.:format)          articles#index
             POST   /articles(.:format)          articles#create
 new_article GET    /articles/new(.:format)      articles#new
edit_article GET    /articles/:id/edit(.:format) articles#edit
     article GET    /articles/:id(.:format)      articles#show
             PATCH  /articles/:id(.:format)      articles#update
             PUT    /articles/:id(.:format)      articles#update
             DELETE /articles/:id(.:format)      articles#destroy
        root GET    /                            welcome#index

I add the action and the view, hit refresh and indeed, the ‘article’ appears.

So far so good. One of the problems undoubtedly is that I’m a total noob to both ‘ruby’ and ‘rails’, but I suspect I’m not the only person that finds this tutorial as their first point of contact. And I just can’t get over how frustrating this whole experience is. It’s like being a failed Harry Potter, you’re being told to use all these spells that are supposed to have some magic effect, only they don’t and throw back errors at you all the time implicating you’re the one doing something wrong. And then after consult it always turns out that the spells you were given were wrong. Really annoying. Anyway, if I wasn’t the persistent type I would have never gotten into programming in the first place so I’ll try real hard to stop bitching about this but it’s getting to me. I sure hope the quality of ‘ruby’ and ‘rails’ is at a higher level than this tutorial.

The main issue I have at this point is that given the fact that Ruby does a ton of things under the hood that appear to be black magic to the user you should at least build up the students confidence that the magic actually works. Right now I feel as if a single ‘.’ or ‘,’ misplaced that does not have a relevant stack-overflow message waiting for me (assuming I know what to search for) will leave me totally stranded. You’re asked to do all these things without a proper explanation of what is going on, it’s a huge exercise in faith.

I suspect that one of the problems here is that I dove head-first into RoR without learning some Ruby first. So let’s divert for a bit and get a feel for ‘Ruby’ the language without looking at the framework.


Some things to remember (when coming from some other language):

  • Expressions work like they do in most other languages
  • functions are defined using a ‘def’ … ‘end’ pair
  • if a function has no parameters you can omit the () both during the declaration and the subsequent call
  • even if a function has parameters, if the context is unambiguous then you can omit them
  • Within a string #{varname} expands varname to the contents of varname.
  • functions can be defined only once (so no erlang like tricks based on arity)
  • you can call a method of an object by using object.method
  • variable names are not prefixed by any special character
  • irb (interactive ruby) will only attempt to process the code when it thinks it is complete (this can become quite messy with multi-line function definitions containing errors)
  • you can use the respond_to? method to figure out what an object understands, this allows for code to reflect on what a class can do and adapt accordingly. For instance, you could try to see if you received a list or a simple variable by checking of the object you received understands “each”
  • attr_accessor :someattribute will generate a getter and a setter method
  • puts “Goodbye #{@names.join(“, “)}. Come back soon!” shows that you can do just about anything in between #{ }

Something weird happened during the tutorial: I defined a simple class according to what it said on the page there and then listed the instance methods:

irb(main):043:0> class Hey
irb(main):044:1> def initialize(name="World")
irb(main):045:2> @name=name
irb(main):046:2> end
irb(main):047:1> def say_hi
irb(main):048:2> puts "Hi #{@name}!"
irb(main):049:2> end
irb(main):050:1> def say_bye
irb(main):051:2>  puts "Bye #{@name}, come back again!"
irb(main):052:2> end 
irb(main):053:1> end
=> nil

=> [:say_hi, :say_bye, :h, :nil?, :===, :=~, :!~, :eql?, :hash, :<=>, :class, :singleton_class, :clone, :dup, :initialize_dup, :initialize_clone, :taint, :tainted?, :untaint, :untrust, :untrusted?, :trust, :freeze, :frozen?, :to_s, :inspect, :methods, :singleton_methods, :protected_methods, :private_methods, :public_methods, :instance_variables, :instance_variable_get, :instance_variable_set, :instance_variable_defined?, :instance_of?, :kind_of?, :is_a?, :tap, :send, :public_send, :respond_to?, :respond_to_missing?, :extend, :display, :method, :public_method, :define_singleton_method, :object_id, :to_enum, :enum_for, :==, :equal?, :!, :!=, :instance_eval, :instance_exec, :__send__, :__id__]

Spot the weird one? That ‘h’ in there is a definition I made before defining the class ‘Hey’, and yet it is a method in the class!

Let’s try that again, completely from scratch.

jam@homebox:~/svn/src/srs$ irb
irb(main):001:0> def h(name)
irb(main):002:1> puts "hello #{name}!"
irb(main):004:1* end
=> nil
irb(main):005:0> h("hey")
hello hey!
=> nil
irb(main):006:0> class Hey
irb(main):007:1> def initialize(name="World")
irb(main):008:2> @name=name
irb(main):009:2> end
irb(main):010:1> def say_hi
irb(main):011:2> puts "Hi #{@name}!"
irb(main):012:2> end
irb(main):013:1> def say_bye
irb(main):014:2> puts "Bye #{@name}, come back again!"
irb(main):015:2> end
irb(main):016:1> end
=> nil
irb(main):017:0> Hey.instance_methods
=> [:say_hi, :say_bye, :h, :nil?, :===, :=~, :!~, :eql?, :hash, :<=>, :class, :singleton_class, :clone, :dup, :initialize_dup, :initialize_clone, :taint, :tainted?, :untaint, :untrust, :untrusted?, :trust, :freeze, :frozen?, :to_s, :inspect, :methods, :singleton_methods, :protected_methods, :private_methods, :public_methods, :instance_variables, :instance_variable_get, :instance_variable_set, :instance_variable_defined?, :instance_of?, :kind_of?, :is_a?, :tap, :send, :public_send, :respond_to?, :respond_to_missing?, :extend, :display, :method, :public_method, :define_singleton_method, :object_id, :to_enum, :enum_for, :==, :equal?, :!, :!=, :instance_eval, :instance_exec, :__send__, :__id__]
irb(main):018:0> Hey.h "hello"
hello hello!
=> nil

Yup, that’s reproducible. So a class absorbs any methods defined separately just prior to that class?

Weird. I can also still call ‘h’ outside of the class:

irb(main):019:0> h("hello")
hello hello!
=> nil

irb(main):020:0> say_hi "jacques"
NoMethodError: undefined method `say_hi' for main:Object
    from (irb):20
    from /usr/bin/irb:12:in `<main>'

But the class methods are not accessible without an instance of the class. I don’t understand this, but I assume it is somehow intentional. Maybe someone more versed in Ruby can explain what is going on here.

=> [:say_hi, :say_bye]

Shows only what we defined in the body of the class.

The cliff notes to ‘Why’s poignant guid on ruby’, about as short a crash course on Ruby as you’ll get:

  • variables are all in lowercase
  • strings can be made with both single and double quotes
  • constants start with an upper case letter, so Constant is a constant
  • :symbol is a symbol
  • instance method invocation works like this: variable.method
  • class method invocation works like this: Classname::method
  • the normal argument passing looks like lots of other programming languages: variable.method(param1, param2)
  • ‘kernel methods’ are core language functions such as ‘print’ they don’t require an object to be attached to, the default object Kernel:: is always searched if you type a method name.
  • global variables have their name prefixed with a ‘$’
  • instance local variables start with an @ sign, their scope is the local object
  • class variables start with @@, their scope is the whole class
  • code blocks are { between }, or between do and end, these are equivalent
  • blocks can have 0 or more arguments between || characters just inside the block
  • ranges look like this: (1..3) or (‘x’..’z’)
  • arrays look like this: [1, 2, 4]
  • hashes http://www.ruby-doc.org/core-2.1.2/Hash.html – a ‘hash’ in ruby parlance is a dictionary that can only store definitions, they look like this: { “a” => “test”, “b” => “you know” } – you get the value stored at key ‘a’ back out of a hash like this: thehash[‘a’]
  • regexps look like this: /regexp/
  • Interesting: 5.==(6-1) returns ‘true’, so == is a method like every other
  • << concatenates
  • .reverse reverses a string

The guide lists “string”.each as a valid construct, but the interpreter complains when I try that. Apparently ‘each’ is no longer available, you now have to use ‘each_char’. I wonder why one would deprecate an existing behavior, breaking existing code and then to re-implement that behavior using a new name. It seems like a pointless exercise to me, nothing changes except that a bunch of people end up having to debug their code. And in an interpreted language such changes are extra nasty because you don’t get the benefit of a compile time check to see if your code is still at least correct at that level.

  • string search/replace is called gsub, it is a method of the string class so you use it as string.gsub(“old”,”new”)
  • affixing ! to a method name does the operation ‘in place’ instead of returning a new result, it doesn’t seem to work for all operators though, a.+!(3) is not the equivalent of a+=3 (which works fine all by itself…), so the ! is part of the method name, not some kind of modifier that makes any method destructive

  • File::read(“filename.txt”) will return a string with the contents of the file

  • File::open(“filename.txt”,’r’) do |f| block end executes the code in block with the filehandle of the open file in ‘f’
  • f << string where ‘f’ is a file descriptor will concatenate the string to the file contents (what about seek? does it write or only concatenates??)
  • very clever dir globbing example: Dir[‘*.txt’].each do |file| system “cat ” << file end
  • there are multiple ways to do the same thing. t = [:a => ‘e’] + [:a => ‘d’] is equivalent to t = [ {:a => ‘e’}, {:a => ‘d’} ]


“Yukihiro Matsumoto: Ruby inherited the Perl philosophy of having more than one way to do the same thing. I inherited that philosophy from Larry Wall, who is my hero actually. I want to make Ruby users free. I want to give them the freedom to choose. People are different. People choose different criteria. But if there is a better way among many alternatives, I want to encourage that way by making it comfortable. So that’s what I’ve tried to do. Maybe Python code is a bit more readable. Everyone can write the same style of Python code, so it can be easier to read, maybe. But the difference from one person to the next is so big, providing only one way is little help even if you’re using Python, I think. I’d rather provide many ways if it’s possible, but encourage or guide users to choose a better way if it’s possible. ”

  • t.sort_by { |e| e[:a] }.each do |t| print t end # sorts ‘t’ by key ‘a’ for each element e by retrieving the value of e[:a]. That’s pretty concise!
  • next functions like ‘continue’ in ‘C’
  • break like it does in most other languages (it breaks out of the innermost loop)
  • a switch statement looks like this in Ruby: case when value when value when value end (a bit strange how they chose to use confusing names for widely accepted constructs, it’s like exchanging ‘-’ and ‘+’ just because you can)
  • return value of a function is the value of the last expression of that function
  • (5..10) === 7 evaluates to ‘true’, because 7 is between the range start and end values

Slight detour through:

http://lambda-the-ultimate.org/node/934 http://www.tbray.org/ongoing/When/200x/2005/08/27/Ruby

Because of Ruby’s variable scoping rules.

Since the _why guide was written Ruby has evolved quite a bit, for instance, the block variable scope is now a lot more sane, the example on page 48 (fortunately!) no longer works as advertised. Variables declared outside of a block are no longer affected by what goes on inside of a block. See here:


  • classes are defined like this: class ClassName bodyoftheclass end
  • 5.next gives you the next number after 5
  • (5.5).floor.next first truncates the float to an int, then calls next
  • object.class returns the class for that object
  • b = a.class.new -> make another object ‘b’ of the same class as object ‘a’
  • you can reach the current object from within a class definition by using ‘self’
  • you can add new methods to existing classes, including kernel classes! (bad idea (tm))

For instance, this is how you can add a ‘square’ method to numbers

irb(main):152:0> class Fixnum
irb(main):153:1> def square
irb(main):154:2> self * self
irb(main):155:2> end
irb(main):156:1> end
  • respond_to?(“methodname”) applied to any class tells you if that class has a method ‘methodname’ (or should I say ‘knows how to respond to a message ‘messagename’)?
  • you can slice strings like this: “abcde”[3..4]
  • you can split strings on delimiters using “ab-cd-ef”.split(“-“)
  • new class creation: class ClassName < parentclass; end (why does it need that ‘;’ there?)
  • you can query the parent class with the superclass method: ClassName.superclass
  • modules govern namespaces
  • to define a class with all the entities in a module you use ‘extends’ as in ‘class X extends ModuleName; end’
  • a *argument to a function will be passed in as an array
  • attr_reader will create getters for the symbols passed as arguments, for instance attr_reader :test :test1 will create reader functions for variables with that same name
  • attr_accessor does both readers and writers, attr_writer of course only writers
  • class << ClassName is used to add class methods (methods that can be called on the class rather than on an instance of the class)
  • a mixin is a module ‘included’ into a class, it instantly endows the class with all the stuff in the module
  • reading and processing the contents of a web page: require ‘open-uri’ # Searching all found items containing the word `truck’. open( “http://preeventualist.org/lost/searchfound?q=truck” ) do |truck| puts truck.read end
  • similar, but line-by-line processing: require ‘open-uri’ open( “http://preeventualist.org/lost/searchfound?q=truck” ) do |truck|
     truck.each_line do |line|
     puts line if line['pickup']

    end end

I don’t particularly like the ‘if x’ conditional appended to that line, it feels a bit backwards to me. Imagine the instructions for disarming a bomb read ‘cut the red-green wire if the blue wire is still connected’. boom. Too bad, you should have parsed the instructions first before you started to execute them!

  • yielding is a way to let two blocks of code communicate, the ‘inner’ block passes information to the ‘outer’ block with every time yield is executed. Or should I say ‘inner blocks pull data from outer blocks whenever they need to’? It’s reminiscent of co-routines (which ruby also has: http://rubydoc.info/stdlib/core/1.9.2/Fiber), one routine the producer, the other the consumer.

  • ‘brief’ style versus verbose style (both are identical in function):

    open( “idea.txt” ) { |f| f.read }

    open( “idea.txt” ) do |f| f.read end

  • traits in a parent class allow for elegant subclassing without having to pass in tons of (unnamed) parameters in the initializer use them like this in the parent class ‘traits :traita, :traitb, :traitc’

  • @x ||= {} is a little trick to ensure that a variable exists before it is used. The ||= says ‘if this varible does not exist yet then make a new one of this type, but if it does exist then adding an empty list to it won’t change anything’. I’m not sure why that works, you’d say that the ||= should fail if x does not exist yet, because it’s syntactically equivalent to x = x || {}

  • I got a little lost around page 109, the metaprogramming bit. It looks like one program is writing another, and not only is it able to evaluate code that wasn’t present in the original program, it can be used to construct entirely new objects / classes and so on. This is powerful but also asking for trouble because it can become very hard to reason about such programs. It’s akin to macro expansion, but more powerful.

  • you can overload just about everything too, including the basic operators

This probably makes ruby absolutely great at writing DSLs but it will be a nightmare to debug because you’ll never know what you are looking at means without backtracking very precisely through how you got where you are. Imagine looking at some program and seeing ‘a = b * c’. In most languages it would be fairly obvious what this does, the product of b and c is stored in variable a. Maybe b and c are arrays or matrices but at a minimum you’d expect some kind of multiplication to be going on there. In Ruby this is not the case. ‘a = b * c’ could mean literally anything, the only thing you can distill from that bit of code is that b and c are somehow combined using the operator ’’ and that the result of that combination is stored in a. Context decides whether this is two variables being multiplied or two battle ships exchanging fire or two images being compared. For a lark, someone could have decided to turn * into the division operator… Powerful, but also very* dangerous, hard to debug and easy to have unintended side effects.

I should re-read the portion on meta programming when I understand Ruby a bit better.

  • string formatting in Ruby: “abc %s and %d” % [“jaja”, 3]
  • inside double quotes #{ } allows you to escape to run ruby code (security concerns here, this goes way beyond variable substitution?)
  • reminder: globals start with a $
  • you can generate exceptions using ‘raise ErrorNo, “Description”
  • you can recover from errors by using the rescue label
  • you can create an array of strings quickly by using %w{a list of words} this gets transformed into [‘a’, ‘list’, ‘of’, ‘words’]
  • %x{} -> execute in a shell, %Q{} -> acts like a regular double quoted string
  • if ‘a’ is a list and you do ‘a = b’ then b contains a reference to a, not a copy of the list so modifying b modifies a and vice-versa
  • you can make an actual copy by using the .clone method, and then there is ‘.dup’ to create copies of objects that have been locked http://ruby.about.com/od/advancedruby/a/deepcopy.htm
  • you append a value to an array using the << operator (contrary to what the stackoverflow article on the same question says!), so a << 5 appends the value 5 to a pre-existing array a.

If you run irb with

irb –readline -r irb/completion

You can type the name of an object, append a ‘.’ and then hit Tab, this will show you all the methods you could call.

For some reason the default ruby install does not come with the embedded documentation (ri).

Ok, that’s the end of the ‘Poignant Guide’, thanks Jonathan. (some joker now has _why’s domain so I won’t be linking there from here).

Porting the command line app

The PHP command line application is as good a place as any to start working with ruby. It’s not overly large and has all the business logic in it and needs to work with the database. Let’s port it to ruby to see what that feels like.

First we need to install the mysql client development libraries:

sudo apt-get install libmysqlclient-dev

Next, we need the mysql gem:

gem install mysql

Now ruby scripts should be able to access mysql databases.

A quickie check in irb if that’s true:

irb(main):001:0> require "mysql"
=> true
irb(main):002:0> con = Mysql.new('localhost', '', '', 'srs')
=> #<Mysql:0x8a48f14>
irb(main):003:0> rs = con.query('select * from users')
=> #<Mysql::Result:0x8a190c0>
irb(main):004:0> rs.each_hash { |h| puts h['nick']}
=> #<Mysql::Result:0x8a190c0>
irb(main):005:0> con.close
=> #<Mysql:0x8a48f14>

Looks like it works :)

About 20 minutes into porting the console application I hit a bit of a roadblock. The standard mysql library module does not allow you to fetch a result from a prepared statement. This results in terribly ugly code which causes a one-liner to become something that spans a couple of lines. In the PHP world I use my own wrapper library for stuff like this, but for ruby I decide to see what’s available out there (this also gives a bit of an idea about how the ruby eco system works wrt to contributed software). I find this: http://sequel.jeremyevans.net/

gem install sequel

installs it, and we’re off again. I’ll change the bits of code written up to that point to use the sequel gem, it looks pretty easy to use. I could use ‘activerecord’ as well I guess but why not give this lightweight module a shot and see how far it takes me.

hash = $dbcon.fetch(“select * from users where nick=’%s’” % nick).all[0][:pwhash]

that looks manageable. Still has an ugly * and .all[0] in there that’s technically not needed but that’s fixable:

$dbcon = Sequel.mysql("srs")

# fetches the first record of a set

def $dbcon.single (q)

# fetches the first value returned by a query

def $dbcon.simple (q)

This is a little snippet from the db initialization code. To ‘fix’ the Sequel class we add two new methods to the class, one that will simply return the first result from a set, another to return the first field from the first result from a set. This allows you to do:

hash = $dbcon.simple("select pwhash from users where nick='%s'" % nick)

That looks better. Of course modifying code from the outside like this is a bit (ok, a lot) hackish but I want to get a feeling for what Ruby can do and even if this is probably not idiomatic ruby the fact that you can extend libraries so easily is a powerful concept (and probably a very dangerous one, imagine if the vendor of ‘Sequel’ decides to add a ‘single’ or a ‘simple’ method to the base class with different meaning! So for anything but this ‘toy’ program this is probably a very bad idea.

Next up, the hash comparison. The passwords in the system are hashed using ‘bcrypt’, with a workfactor of ‘10’ and the blowfish algorithm. This results in hashes with the prefix $2y$10$ followed by the salt and the hash. Testing these passwords with the default Ruby ‘BCrypt’ library results in a non-match, even with the correct password. It turns out that the maintainer of the BCrypt gem made a decision to add ‘2x’ instead of ‘2y’ and that any ‘2y’ passwords will not match, even if the lib is perfectly capable of doing the verification.

The solution? Replace the ‘y’ with an ‘a’ and then the comparison will succeed for correct hashes. See: http://stackoverflow.com/questions/20980859/using-bcrypt-ruby-to-validate-hashed-passwords-using-version-2y and https://github.com/codahale/bcrypt-ruby/pull/91 lists an open pull request with a fix.

This little gotcha cost me about 2 hours to figure out, but for now the ‘replace-the-y-with-an-a’ workaround seems to do the trick.

Ruby does not have a ++ operator!! So no increments of integer variables without doing something like += 1.

No big deal, but I’m sort of used to this and it would come in handy when porting code. But using iterators most of the increments should disappear. Still, it wouldn’t cost anything to include this operator in ruby and it’s surprising that such a simple thing is missing.

Interesting: Porting the php code to ruby turned up a bit that I should have extracted into a function (the loop where the choice was made which course to study). The reason why I found that this should have been a function was that there simply was no elegant way to do it ‘in place’, but then once the function was made it looked a lot better!

Ruby’s inconsistent use of {} for blocks is driving me crazy. I never know when they are permitted, expected or obligatory!

loop {


Is fine but

if a {


Is not… Just check this out:

puts [1,2,3].map{ |k| k+1 }
=> nil
puts [1,2,3].map do |k| k+1; end
=> nil 

What is happening here is that the {} have different precedence than do .. end. Very confusing, even after multiple tries I can’t find a way to convert the curly braces example to a do .. end example.

Maybe something like this:

irb(main):029:0> [1,2,3].map do |k| puts k+1; k+1 end
=> [2, 3, 4]
irb(main):030:0> [1,2,3].map { |k| puts k+1; k+1 }

At least now they both behave identical.

A nice little ‘gem’, wirble, it adds syntax highlighting and auto completion:

sudo gem install wirble

see http://tuxdna.wordpress.com/2011/11/13/autocomplete-and-colorize-your-ruby-shell-irb-with-wirble/

From here on forward porting the command line version of the ‘learn’ script to ruby went fairly straightforward, a couple of minor glitches and a few functions that I had to find a work-around for but nothing that I would not have experienced going the other way (from Ruby to PHP).

I note that the lines on average are a bit shorter than in PHP, they look cleaner but there are definitely more of them, in part this is because Ruby is so terse at times that you feel the need to explain what’s going on. Code + comments = a constant…

Part of this lies in that PHP basically contains anything you would like to use as soon as you start the interpreter in terms of crypto, database, string transliteration, json and so on. With ruby you have to include that explicitly.

Of course there is a very good chance that I simply do not really know what I’m doing in Ruby just yet. Another thing I noticed is that because not everything is in the ‘base package’ you have to resolve external dependencies all the time, there is always ‘one more gem’ to install and it could be quite easy to either forget such a gem or to find that when you build up a machine that your ‘favorite gem’ is no longer being supported. For this reason I’ve always tried to rely as little as possible on external dependencies but this is the ‘ruby way’ I guess. PHP has this too (pecl) and I’ve stayed away from that for much the same reasons. But in PHP land you can get away with that, in Ruby land you need to include those external dependencies just to get basic stuff.

All in all porting the learn.php program took less than a day, I learned a bit more about how ruby works under the hood and I have a better grasp of the syntax. When I’m more proficient at Ruby I should probably re-visit this to see if I can do a better job of it, which hopefully will make the program more compact and/or easier to read.

Here are the cloc stats of both the ruby and the php version:

$ cloc learn.rb
http://cloc.sourceforge.net v 1.60  T=0.03 s (30.1 files/s, 12351.2 lines/s)
Language                     files          blank        comment           code
Ruby                             1            123             92            196
PHP                              1            105             57            193

That’s very close. In fact, if you discount the functions that were added for compatibility with php then ruby comes out slighly ahead.

To gain some more experience, I got rid of the sequel gem and re-did the whole thing using active record. This also lines me up a bit better for using Rails.

Extremely frustrating moment. Examples all over the place on how to use ActiveRecord with ruby. If I use one of these examples (the most trivial), for instance:

Most of those examples/documentation give absolutely no indication what version of activerecord / ruby / possibly rails they are for.

So this works:

>> u = User.where(:nick => 'jam').first
=> #<User id: 2, nick: "jam", pwhash: "$2y$10$Hpp9rAx9d/eWIFi.E.f7pu1YDV4qfzfAwGvMpBzF6wb...", score: 6055, email: "jacques@mattheij.com", newcardsperday: 20, cardspersession: 25, decay: 3, language: "en", initialinterval: 5>

And this does not:

>> u = User.find(:first)
ActiveRecord::RecordNotFound: Couldn't find User with 'id'=first

Even though that example is used in just about every tutorial out there.

Finally I found this guide which appears to be up-to-date and relevant:


Usercourse.joins(:user).where(user_id: 2)

Tbh, after fiddling around with activerecord for a day or so now I’m really wondering how ORMs improve over SQL, but that’s probably just the novelty and the frustration talking. For instance, if I have a database with users collection scores I could do this: “update users set score=score+10 where id=1” without having to worry about locking records or anything like that. A user logged in more than once on the system could hit the server multiple times and it would all work like clockwork. But using an ORM that o-so-simple problem suddenly gets a lot harder. Just using user.score += score; user.save is not going to cut it! After all, who is to say that in another thread the exact same thing isn’t happening, causing one of the updates to be lost! Anyway, that’s a contrived example but you can see that ORMs are not a silver bullet by any means. What they do though is make your code more compact, hopefully easier to read.

The activerecord version of the console application in Ruby clocks in at 164 lines of code, but a good part of that is the elimination the ‘grade_answer’ function and inlining it. The reason why that was feasible is that the active record code was so much more readable compared to hitting the database directly using SQL statements that the duplication of the branches in the grade_answer function and the switch statement jumped out at me. So I merged the two and eliminated the function, resulting in a slightly longer version of the process_answer function but all in all still a good chunk of code elided and a cleaner overall flow. Not that that matters in a toy project like this but in a larger project such savings can add up.

Language                     files          blank        comment           code
Ruby                             1            100             86            161

I should probably work a bit more on this code to move all the business logic to the model rather than to have it spread out all over the program. Especially the fetching of the next card to learn, grading and rescheduling are all bits that should live in the model.

Ok, back to our scheduled program of learning Romanian… Further reading to be done:


I like the concept of migrations, but for now I’ll stick with doing schema maintenance through ALTER and CREATE TABLE statements in mysql, I don’t like the idea of getting so tied into any one eco system that the price of switching becomes too high, it would make evaluating other language/framework combo’s harder skewing the choice towards ruby/rails before all the data is in.

For some reason active record never detected the relationship between the various tables that are present in the schema. Not sure what is going on there, it’s easy enough to fix using some magic in the model but this should have been handled automatically, otherwise what’s the point of defining the relationships in the schema to begin with? I also suspect that under the hood there is a lot more chatter between AR and the database going on than I’m currently aware of but since this is a prototype any kind of attention spent on that would be a serious case of premature optimization so we’ll let all of that rest for now.

Another thing that worries me is the start-up time of the ActiveRecord module. Initializing the model takes well over 1 second on this machine(!) and I’m the only person using it. Ok, it’s a slow box, but still, that seems ridiculously slow to me.

So, after a gigantic detour this is the page where we left off:


In section 1.2 I now finally understand what they meant with this snippet:

<%= link_to 'Patient Record', patient_path(@patient) %>

All it does is reverse the route, given a patient it creates the link to that specific patient.

notes for the routing guide:

  • ‘resources :photos’ generates a whole pile of default routes for standard crud operations on ‘photos’
  • “get ‘profile’, to: ‘users#show’” simple routes look like this
  • singular resources still map to plural controllers, so ‘resource :customer’ will map to the customers controller
  • you can nest controllers inside namespaces like this: namespace :xxx do resources: objects end, this makes /xxx/resources/ etc

This is extremely compact, but there is some risk here as well, you could be creating more routes than you’re aware of and with default controller actions that might cause trouble. Do controllers exhibit default actions?

Here is the spec for all the actions that we want our web application to perform:

  • *the usual account creation
  • *login
  • logout
  • *generating a list of available courses
  • *a single page with a larger description of the courses
  • *a way to start learning a course

  • *a page where the actual learning takes place

  • *a place where the answer to a question gets posted
  • a page where you can track your progress
  • a page where you can view your profile details, courses selected etc
  • a page where you can add cards

Only the actions marked with a ‘*’ are required for the evaluation.

Since it all starts with the routes, let’s lay those out first:

resources :users, :courses
post '/users/login', to: 'users#login'
get  '/learn/:filter', to: 'usercards#learn'
post '/learn/:usercardid' to: 'usercard#answer'

Ok, back in NL 2275 km of driving later…

Re-play the installation of Ruby and Rails on my machine here:

apt-get install ruby

Ah no, that installs 1.8

apt-get remove ruby apt-get install ruby1.9.3

(why don’t they make that the default?)

apt-get install sqlite3 apt-get install libsqlite3-dev gem install rubygems-update update_rubygems apt-get install g++ gem install execjs apt-get install libmysqlclient-dev gem install mysql

gem install therubyracer

for some reason therubyracer did not work for me this time around but installing nodejs seems to have done the trick. Weird dependency!

apt-get install nodejs

gem install rails gem install bcrypt

mkdir ~/test cd ~/test

rails new blog

rails server

=> Booting WEBrick

So even though that’s the second time I’m doing this it still required some fixes to the process. I’ll need to test this on a VM to make sure I really have all the requirements down now.

Fragile! Anyway, it works (again). On to more productive matters.

(verified that ruby learn.rb works again as well).

In preparation to making this into a rails website, I’m first going to move all the business logic into the model. Hopefully that will survive a transition into a web app intact.

How do you ensure that for a given ruby / ruby on rails application all the relevant gems have been installed?

Ah, that’s what ‘bundle install’ is for.

added the bootstrap gem

gem install twitter-bootstrap-rails

Update the gemfile, then run bundle install again

I installed a template from one of those template sites (http://html5up.net/verti).

Installing a template is a bit messy, you have to rip it apart and place the various pieces where rails expects them, and even then it can be quite a chore to work out why things are not doing what they’re supposed to. Rails has an ‘asset pipeline’, a piece of magic that tries to compile all your css and whatnot for speed but frankly it is more trouble than it is worth. In the end I stuffed all the css files in /css/ and did a bunch of @imports to include them in the layout. Not as nice and probably not as fast but for now it works, I can re-visit this later if I want to continue with rails.

So, now we have ‘basic navigation’ and we can add the links to the template. I need to figure out how to log in a user / log a user back out again.

What’s a real pain is that every time I find a tutorial or some piece of information on how to achieve ‘x’ in rails it is invariably out-dated, deprecated, no longer maintained or just plain wrong. For instance, a tutorial on how to create a user signup process in Rails contained a whole bunch of errors, bad advice (which could lead to security issues) and deprecated stuff. Fixing that took a long time.

You’d never guess it but that ‘comment’ at the end of the line is not a comment. This had me stumped for a long time.

validates :password, :password_confirmation => true #password_confirmation attr

So, Ruby -> Rails -> JavaScript (Unavoidable) -> Coffeescript? Wtf. So I have to learn yet another language to be able to use rails. This sucks. By the way, the information on how to do stuff is literally all over the place. Railscasts (what? I have to watch a video in order to do something with a framework? I really would simply like to see a solid example that does not involve me hunting across 12 different websites to pull all the pieces together, let alone some video that was likely out-dated on the day that it was made. Forgive me for being grumpy but if frameworks are supposed to make you more productive then I’m currently not experiencing that feeling).

Finally, after many hours of messing around and reading the answers to this stack overflow question solved it:


I still can’t really believe that that is the real solution but at least it seems to work. You’d think there would be a much more generic way of handling the response rather than to write a bunch of jquery to pull the validation errors out of the json response.

But ok, it works so let’s not complain. Well, it works if you have javascript enabled. I really dislike the way the webs is developing in this sense, javascript should not be a requirement per-se and the amount of cruft that gets added to a webpage in order to achieve something that could have just as easily been achieved by expanding the HTML standard (for instance, by adding a ‘validate’ attribute to fields with a regexp of permitted values, or even a ‘validate_src=’url” for an online validator in case you need to do backend stuff to determine if the contents of a field are valid. Anything but JavaScript!).

Another thing that would really help here is a reference application that you can simply download that shows you ‘how it’s done’ using rails version ‘x’ rather than that every developer has to replicate the effort to piece it all together. Maybe I should post the SRS app sources so people can refer to them.

Useful reading:


Little by little I cribbed together the knowledge of how things are done the ‘ruby on rails’ way, it’s incredible how fragmented the information is.

Moniker.com Is Being Murdered

For many years I was a ‘bulkregister’ customer, I landed there after Network Solutions went mad. Then Bulkregister was bought by enom.com and it went from good to bad to worse in a very short time. I called around to my colleagues to see where they were registering their domains and they pointed me to a small but scrappy upstart registrar called Moniker, the brainchild of one Monte Cahn. Monte was awesome, he worked like a demon and rightfully claimed never to have lost a customers domain.

In 2005 Moniker changed hands for the first time, Seevast Corp bought it and I thought this was going to be the end. I decided to hold out for a while to see how things would go and - surprise - in spite of how most corporate take-overs are done Moniker appeared to be in good hands and I continued to be a customer. On occasion - when the support people were not able to help - Monte would step in to protect the good name of the company that he’d created. All was good.

Then in 2008 Moniker changed hands again. This time to Oversee.net. Again, I bided my time and was rewarded with a company that even though technically a small subsidiary of a larger mothership operated with some independence, and continuity was one of their prime focal points. So I stayed. Occasionally I had issues, most of these were resolved in time and competently, where that wasn’t the case we found ways to deal with it, just another customer-supplier relationship, no longer the personal treatment from the old days but that was to be expected. As these things go, for a company that had changed hands twice it could have been a whole lot worse.

In 2012 KeyDrive SA acquired Moniker.com. And in February 2014 Bonnie Wittenburg (already CEO of KeyDRive) was named CEO of Moniker. Wittenburg has apparently not heard of the age old mantra of computing systems that have been around for a while: If it isn’t broken, don’t fix it. Full of fresh and vigourous energy the CEO declared in February that ‘they’ve gone back to the drawing board’. I wished they’d stayed there. Last week Moniker.com released it’s new and revamped interface, and it is - without exaggaration - an unmitigated disaster.

I’m a pretty loyal customer, but what’s happening at Moniker is at a scale that I’ve yet to see elsewhere. The number of problems, the severity of the issues and the incredible lack of openness with respect to what is going on has managed to completely trash the excellent reputation that the Moniker.com brand had for me in the space of a few days. I’ve been a customer there for almost a decade and I’m more than willing to excuse the occasional error but this is an extremely good example of how not to do a public roll-out of a revamped website. The words “On Monday, June 2nd, 2014, visit Moniker to experience the change for yourself.” in this moniker announcement turned out to be truly prophetic, I have unfortunately tried to visit moniker on June 2nd 2014 and ‘experienced the change for myself’. What was once one of the easiest and most reliable user interfaces used at a registrar has turned into a textbook example of how you alienate an existing customer base.

Let me give you a short listing of what I’ve been able to come across so far:

  • it’s all just change, no improvements

  • stuff has been moved around for no apparent reason, basic common sense design has been replaced by a ‘hunt and seek’ kind of user interface that is totally un-intuitive and does not in any way connect to the old interface, it’s a radical breach with how things were. Sometimes this can be a good thing but imagine Microsoft would re-arrange all the items in the menu system for ‘Word’ at random and would rename a couple for good measure.

  • it’s as if there was absolutely no testing whatsoever.

  • account history is gone (so, how do I do my bookkeeping?)

  • most commonly used functions (such as domain push) do not work

  • newly registered domains are not showing up in the interface (so how do I do my work?)

  • domains that are bought and paid for are stuck in ‘awaiting payment’ even though the invoices for those domains do show up

  • charges are made for domains that have been set to expire long ago (and even for some that had expired!)

  • domains are expiring that are set to auto-renew

  • support questions are going utterly unanswered

  • the support phone system is overloaded permanently, hours on hold without any help whatsoever

  • transfering domains out can only be done on a one-by-one basis

  • I wished I was the only person experiencing all this the comments here are telling

And on and on. Monte Cahn is probably getting quitely drunk somewhere, they’re murdering his baby and it seems there is not much that can be done about it. What a total mess. Imagine, you’re being given the reins over one of the most respected registrars and within 6 months of that you take a legacy like that and totally destroy it. There has to be a prize for something like that.

If they keep this up for a while they’ll be the proud owner of a reputation worse than GoDaddy’s, which I would not have thought possible.

Wittenburg should step down or be fired and someone competent brought back in or KeyDrive will find that they have paid the most money ever for just a single domain because there will be no customers left. As soon as I figure out how to (they definitely did not make this easier) I’ll be taking my business elsewhere. Gandi looks pretty good. I’m only about 1/1000th of the total volume of domains administered through moniker.com, so probably not enough for them to worry about.

The Several Million Dollar Bug

In case you landed here without any context and have no idea who I am or what this article is about this article should give you some background.

For years we were skating on very thin ice. Our only advantage that we had over competitors is that we had figured something out that they had not. Which is that Netscape, Microsoft and pretty much every other browser vendor had made a small but crucial mistake in implementing HTTP. It was an easy mistake to make, and a lot of code would have to be written to ensure that the bug was not present, without making any difference to normal every day HTTP requests.

RFC 1945 says:

The HTTP protocol is based on a request/response paradigm. A client
establishes a connection with a server and sends a request to the
server in the form of a request method, URI, and protocol version,
followed by a MIME-like message containing request modifiers, client
information, and possible body content. The server responds with a
status line, including the message's protocol version and a success
or error code, followed by a MIME-like message containing server
information, entity metainformation, and possible body content. 

So, let’s get this straight: Client connects, client sends request, server responds to that request. Pay attention to the hairline cracks of ambiguity in those sentences and how they are commonly interpreted. The fact that it is written in that order seems to imply that it has to be implemented in that order.

This stack overflow question gets it just right:

My question might sound stupid, but I just wanted to be sure :

    Is it possible to send an HTTP response before having the request 
    for that resource ?

Say for example you have an HTML page index.html that only shows a 
picture called img.jpg. Now, if your sever knows that a visitor will 
request the HTML file and then the jpg image every time :

Would it be possible for the server to send the image just after the 
HTML file to save time ?

I know that HTTP is a synchronous protocol, so in theory it should 
not work, but I just wanted someone to confirm it (or not).

Funny how people are still asking this question nearly 20 years later, and no, that was not a stupid question, in fact, once upon a time I was wondering about just that myself and the answer surprised me. There really are no stupid questions and checking such ‘obvious’ things sometimes pays off in a big way. Which is probably what those responding with their definitive ‘no’s’ to that SO question should have done. Now, for the moment ignoring such details as ‘keepalive’, mixing content types (the question is about html and jpg) and the maximum number of parallel connections that your browser has to the server, in theory, if you knew what the response was going to be before the request even arrived, would it work if you sent the response ahead of time?

And the answer to that question is a resounding YES, you can, it works like a dream and that’s the tiny little item that was the sum total of our competitive advantage. HTTP may be designed as a synchronous protocol but it is not implemented as a synchronous protocol! I figured this out one afternoon when making the early version of the webcam software. The frame rate absolutely sucked, it was 1 frame per second over an ISDN line with a round-trip time in the 100’s of ms or so between server and cam. Sometimes two. But at two channels bonded for a whopping 128Kbit upstream we should be able to do three, maybe more fps. And that’s a huge difference, the difference between ‘animated slideshow’ and ‘almost video’.

So I started toying with the idea of sending the response before the request was in, in fact, totally ignoring the requests! Of course, my initial response was ‘this will never work, for sure the browser will discard the responses to requests that it hasn’t sent yet’. Only it did not! So that’s exactly how we ended up doing it, set the TCP buffer size to roughly the size of a frame, check to see if the frame would fit in its entirety, if not, drop the whole thing and if it did fit then send it. Instant rate adaptation, and maximum frame rate across any kind of connectivity, all the way up to what the hardware could capture/compress/transmit. On a lan you’d get 15 fps (which was absolutely astounding at the time), 320x240, whereas over the WAN links of the day we’d scale it down to something more leisurely (and probably using a smaller image). But still quite good to look at, since it was simply jpeg frames (no incremental updates, so you got a whole frame of reasonably good quality rather than all kinds of blocky updates).

This little trick made lots of money, and I always wondered why our competitors didn’t catch on the little bit of black magic that made it work. All they really had to do was capture some packets in flight and the secret would be out. So much for having a defensible moat, score one for ‘trade secrets’, even if they’re kept in the open. Sending the response before the request is in is not exactly valid in a synchronous client/server protocol, but to this day this trick works wonders. So if you know what the request is going to be feel free to send the response ahead of time, it just might make you some money. At a minimum it will make your users happier, always a good thing.

Every time a major new browser update or such was announced I’d lose lots of sleep, being sure the game would be up, this time they’d close the loophole and we’d be out of luck. But it never did! (Of course, that’s blind luck but if you look long enough at any success I’m sure you’ll be able to identify a number of items that can only be described as lucky. I could re-write this post in terms of how good we were at engineering but that would be a total lie, we just got lucky that it worked and luckier still that it kept working as long as it did. SPDY now tries to make this kind of behaviour official.) Relying on obscure undocumented behaviour is rarely a smart move.

Of course, if you can build out your advantage beyond the trivial to copy then you should do that. But just because you don’t have a real technological advantage doesn’t mean you will be found out in time (or maybe not even at all…). (Unless you blog about it of course, so I guess the secret is definitely out now).

Charities, Direct Marketing and Your Privacy

A recent comment on Hacker News gives a little bit of insight into how charities and non-profits determine who to target for donations and how they deal with requests for removal from marketing lists.

I’m more than a little shocked at the contents. The author, Tom van Antwerp goes into some detail about how his employer, the Ayn Rand Institute operates. (Surprising that such a bastion of capitalism isn’t able to turn a profit because of the useful services it provides for which it commands a fee, rather than to rely on donations to survive, but let’s leave that aside, there’s probably a joke in there somewhere.)

I work at a 501(c)(3), and direct mail is absolutely worthwhile. Maybe 1.5-3% of recipients 
of a prospecting piece actually respond, and they probably won't make up for the cost of 
the mailing. However, those respondents then get more mailings specifically for current 
donors and those mailings make money.

So if you respond once no matter how to a mailing from the Ayn Rand Institute, you’re targeted for more mailings. So far so good, that’s just common sense.

Out of that group, donors are checked against databases of known giving history and net worth. 

Sorry? ‘databases of known giving history’ and ‘databases of net worth’? Just the existence of such databases is a fairly substantial breach of privacy and the fact that any old non-profit with which you have no prior existing relationship should have access to them makes it a lot worse.

These databases aren't always accurate or comprehensive, but it helps fundraisers to ID 
people who might be able to give more--sometimes a lot more.

Never mind the accuracy, once you’re on one of these lists you’re going to get targeted. Now I realize that Forbes has a list of wealthy individuals but that’s quite a step away from having a list of ‘net worth’ on a substantial portion of the population. And that’s still side-stepping the question of how people end up in ‘databases with known giving history’, that means that non-profits actually sell your data to companies compiling those databases, which then happily sell them back to the rest of the non-profits so they can collectively pool their knowledge about who is known to be generous, without taking into account the fact that if someone is generous for one non-profit they might be utterly shocked at learning their data ends up with a whole pile of other non-profits that they have absolutely no relationship with.

When high-value prospects are IDed, they usually get assigned to a major gift officer 
who starts building a personal relationship to solicit very large gifts. This is where 
the charity makes the majority of its revenue. 

I would prefer the term ‘donations’ here, ‘revenue’ is in my head reserved for companies that aim to make a profit.

I usually see 95% of revenue come from the top 5% of donors, largely individuals.

The Pareto principle is alive and well, and if you are a ‘high-value prospect’ you can look forward to being assigned a ‘major gift officer’ (aka an account manager or salesperson).

If you're uninterested in receiving lots of mail, most charities will try to accommodate
that request. However, resources are limited and dealing with the problems of $XX level 
donors often takes second place to the problems of $XXX,XXX level donors. I've seen 
stacks of returned direct mail several feet high from people who moved, died, wanted 
no more mailings, you name it. It was always on the to-do list to remove those names, 
but never at the top of the to-do list.

Right, why would anybody even care about honoring the requests from people that are sick and tired of being bombarded with mail from non-profits they don’t want to give to. It’s all a-ok in the name of the ‘fight for liberty’.

The lessons in here are evident: if you give, especially to the Ayn Rand Institute but this likely holds across the board do it anonymously, lest you end up in a database with ‘prior giving history’ or be ID’d as a high value target which will result in you being assigned a extraction major gift officer. Make sure you initiate the conversation with the charities of your choice, and make sure they do not end up in a situation where they can send you direct mail because you’re likely to receive it well past you meet the grim reaper, and you really don’t want those trees to end up in stacks of returned direct mail several feet high. Maybe the Ayn Rand Institute should do a little housecleaning?

Google Web/Search History Disable Does Absolutely Nothing

There seems to be some kind of misunderstanding about the Google Search/Web History disable switch that google provides to its more privacy conscious users. It’s not exactly the most advertised feature (you won’t find it in your profile page) to begin with, but once you do find it (it’s on the history page, you have to click the little ‘gear’ on the top right and then click the button to switch it off) there is no guarantee whatsoever that google does anything except for changing what they display to you. So if you are under the impression that this changes something about what data google collects on you or how they will use that data then you are likely wrong.

Behind the scenes the data is still collected, in fact, google’s privacy policy pages tells you just that:


Like most websites, our servers automatically record the page requests made when you visit our sites. These “server logs” typically include your web request, Internet Protocol address, browser type, browser language, the date and time of your request and one or more cookies that may uniquely identify your browser.

Here is an example of a typical log entry where the search is for “cars”, followed by a breakdown of its parts:

  • - 25/Mar/2003 10:15:32 -
  • http://www.google.com/search?q=cars -
  • Firefox 1.0.7; Windows NT 5.1 - 740674ce2123e969

  • is the Internet Protocol address assigned to the user by the user’s ISP; depending on the user’s service, a different address may be assigned to the user by their service provider each time they connect to the Internet;

  • 25/Mar/2003 10:15:32 is the date and time of the query;
  • http://www.google.com/search?q=cars is the requested URL, including the search query;
  • Firefox 1.0.7; Windows NT 5.1 is the browser and operating system being used; and
  • 740674ce2123a969 is the unique cookie ID assigned to this particular computer the first time it visited Google. (Cookies can be deleted by users. If the user has deleted the cookie from the computer since the last time s/he visited Google, then it will be the unique cookie ID assigned to the user the next time s/he visits Google from that particular computer).


So the only thing that fancy switch does is to limit what you see, but it definitely does not limit google’s ability (or desire) to collect data about you, and you can rest assured that that is exactly what they’ll be doing.

So all you’re being given here is a false sense of privacy.

And of course, this is just taking an example from the ‘search’ facility that google offers, you should be aware that google has so many points of contact that keeping your browsing habits hidden from them has become all but impossible. Here are a few examples of how your web activity ends up associated with your profile on google’s servers:

  • analytics: This page caused you to be served a google analytics tag, so google knows you visited here. (Come to think of it, google analytics is not worth more to me than your privacy, I should find an alternative and disable GA on my web properties).

  • fonts: That fancy font that is being used on that pretty page? That’s quite possibly served from a google server.

  • google+: On many web pages (again, including this one) you’ll find a google+ button embedded loaded from a google server.

  • google hosted libraries: Many webpages include javascript libraries from a convenient service offered by google.

  • youtube embeds: whenever a page contains a youtube video you’ve just told google you visited that page

  • adsense ads: every page that serves up an adsense ad tells google about your visit to that page. And if you click on an ad it tells google about your personal preference.

  • google app engine: A service where google hosts 3rd party web applications, in other words, sites that do not look like google has anything to do with them at all running entirely on google infrastructure, which means every request in and every answer out of those services passes through google.

  • google dns: google serves up a large number of DNS requests, instances where a certain IP is requesting what address to use to reach some service somewhere else on the internet.

Over the last two months is visited 3490 web pages according to my browsers history, across 615 domains. Of those 43 were google properties. 56 of the 572 remaining contained google+ buttons, 6 contained google platform.js, 94 contained a font served by google, 324 served up google analytics tags, 109 embedded youtube video, 13 contained adsense tags, 75 contained doubleclick tags, 22 used googletagmanager.com, 43 used googletagservices, 47 contained content from googlesyndication.com, 42 contained content hosted on googlecode.com and I’ve probably missed a couple (such as google translate, which of course knows what you translate). All in all of those 572 domains 425 served up some google content, so about 74%, not counting the 43 google properties that I excluded, if we put those back in again then it is 76%. Come to think of it, those urls that google ‘samples’ (you know, when they don’t link directly to the result but through a redirector) are of course also known but I have no idea how many of those 3490 pages I reached through google search.

And that’s just the web, we’re not even looking at mobile, where the android phones are pretty much an extension of google’s infrastructure right in your pocket, with hardware capable of telling google about every spot you visit in real life. So consider google to be riding right beside you and reading over your shoulder all the time, no matter what you’ve done to that history disable button. You’d almost think that all those free and invisible services google provides have one goal: to get you to load something from google on every page that you visit.

As an aside, if I were in law enforcement I would be paying special attention to those searches done by people who have the ‘history’ feature disabled.

edit: thanks to Thomas Bachem for reminding me of DoubleClick and the google DNS servers

Choosing a Web Framework/Language Combo - the SRS Test Application

After creating a short-list of languages and frameworks to be evaluated (see here for the first installment in this series) the next step is going to be a lot harder. It involves getting my hands dirty and at a minimum going through the website of the framework, looking at each in turn to see if they actually match the criteria and then to decide whether or not to build a ‘toy’ application in that framework or not. Time consuming and annoying because it requires you to learn the essentials of each and every framework involved. On the other hand, there is no substitute for hard work in this case, you can’t really pass judgment on anything before you have become familiar with it. The previous article in this series sparked a ton of interest and I got a large amount of very useful (and some not so useful) feedback.

Quite a few people addressed the ‘what framework to use’ question with arguments rather than agenda pushing or religion, corrected a metric ton of mistakes in the previous post and helped to guide me on my quest. To those people I’m extremely grateful for their hard work, every word got read and will be integrated into the revision of the original post. This has caused some changes to the set of languages/frameworks still under consideration.

Toy Application

Because I want to compare the frameworks in as realistic a setting as possible I’ll be building a real application with all of the ones that made it to the short-list.

The toy application will be the thing this all started with, a flash-card (or Spaced Repetition application. It will not contain any user interface components related to creating cards, the database will be ‘magically’ populated with a series of flashcards grouped into several decks that you could use to study/practice some subject. New users should be able to create accounts and select decks they want to study, and the studying process should work as well.

By the way, studying a language over a long-distance medium is not exactly new. In the 1950’s you could already learn languages over the phone!

The schema for the toy application to evaluate the various webframeworks is (relatively) simple. Iteration on the design starts before the first line of code is written. This is the 4th or so revision of what I came up with initially (which was totally different and way too rigid). Some days I feel like I’m in the Monty Python ‘Spanish Inquisition’ sketch. Every time I finish a sentence I have to back up to correct it without ever getting out of the loop and making the sentences longer/different every time!

Apologies for stealing 8 minutes of your life there. And that ‘toy’ application is starting to look pretty complex.

Database Schema

So, the final (hah!) version of the human readable version of the schema looks like this:

We have ‘users’, who use the application to learn stuff and to review and refresh what they learned afterwards.

We have ‘cards’ that have a front and a back side representing the ‘question’ and the desired ‘answer’ to the question.

Cards can be tagged, 0 or more tags are allowed, these are stored in the card separated by ‘/’ characters.

The format of the fronts and backs is json encoded strings, the interpretation of which is left up to the application. Originally I experimented with fixed fields, s-expressions and other trickery but after a short while I realized that since json is now more or less an accepted standard and present as a library module for serialization and deserialization in almost every language I would want to use that this would save me the time of having to implement/maintain a parser for the data chunks.

The trick with this field is that because it is just a text string unforeseen future developments (multimedia, for instance) can be tacked on without re-doing the whole application so it is fairly future proof. All I need to do is come up with a reasonable interpretation of the data. And if the change would be so drastic that the json format itself would need revision then that could be fairly easily done using a background job without having to take the whole thing down. It also allows for easy generation of huge amounts of cards from permutations of the various bits and pieces.

Cards have a difficulty associated with them and you’re supposed to master one level of difficulty before you’re allowed to move on to the next, this ensures that cards that build on knowledge from simpler cards will not be presented until the simpler stuff is fully grokked.

A deck is simply a pre-programmed filter that lifts out cards with certain (combinations of) tags. For instance, this filter would select for an English speaker all the cards that teach you about numbers in Romanian: “/en-ro/ & /numbers/” (note that the ‘numbers’ is in the language of the user, the ‘en’ in the first part of the tag). This also allows for quick, impromptu and unforeseen course creation (such as: learn to count to ten in all different languages: “/numbers/ & /upto10/”)

Every exposure of a card is logged, as is how long it took to get an answer back and whether or not that was the correct answer.

A user can have none, some or all of the cards of any given course in play and has the capability to de-activate specific cards.

Users acquire cards as they learn, new cards are added if fewer than a user configurable number of cards are in the ‘learning’ phase.

Cards can be reversed in a deck, but only if the card allows it.

You could pull really nifty tricks in interpreted languages, for instance, you could make a front contain code that generated a different card every time it was exposed, or have the cards adapt to circumstances (for instance, a card that explains ‘today’ could always use the actual day as an example). For a compiled language such tricks are not so easy, you’d always have to bundle some kind of interpreter along. Or, alternatively, you could assume the display environment to be capable of doing this, and embed a chunk of javascript in the code! We’re not going down that route though (because we also want to do mobile at some point).

For the future, multimedia (audio, pictures, video, LaTex, html, whatever else you can come up with) could be added to the cards.

So, for the first time ever I used mysql-workbench to create a database schema. I wished I could rave about how easy it was and how well it worked to do this simple task. But truth be told I’ve never ever seen such a piece of junk. It crashed a great many times, user interface components are all over the place, it is totally counter-intuitive and makes something simple much harder than typing in the commands by hand. The save dialog had the ‘save’ button off-screen, and it happily creates broken SQL during the export… Incredible that a company like Oracle would let a thing like this pass Q&A with their name on it.

Anyway, after multiple attempts the job was done (count 4 hours or so, not bad for a handful of tables).

Here is the machine readable version of the schema:

-- MySQL dump 10.13  Distrib 5.5.37, for debian-linux-gnu (i686)
-- Host: localhost    Database: srs
-- ------------------------------------------------------
-- Server version       5.5.37-0ubuntu0.14.04.1

/*!40101 SET NAMES utf8 */;
/*!40103 SET TIME_ZONE='+00:00' */;

-- Table structure for table `cards`

/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `cards` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `creator_id` int(10) unsigned NOT NULL COMMENT 'who made the card\n',
  `reversible` int(1) DEFAULT '0' COMMENT 'not all cards can be reversed\n',
  `front` mediumtext COLLATE utf8_bin NOT NULL,
  `back` mediumtext COLLATE utf8_bin NOT NULL,
  `difficulty` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'difficulty is a number that starts at ''0'' and ascends as far as required to order cards. Two cards can be of the same difficulty (they their order of appearance is not guaranteed).\n',
  `source` int(10) unsigned NOT NULL DEFAULT '0',
  `prompt` int(10) unsigned NOT NULL COMMENT 'which prompt to use\n',
  `tags` varchar(256) COLLATE utf8_bin DEFAULT NULL,
  `prerequisites` int(10) unsigned NOT NULL COMMENT 'up to and including which card should be learned before the user can be exposed to this one.\n\n',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`),
  KEY `source` (`source`),
  KEY `creator_id` (`creator_id`),
) ENGINE=InnoDB AUTO_INCREMENT=1505 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='cards are the ''units of learning'' in the system, cards are t /* comment truncated */ /*agged, the first tag is the name of the deck. If a user makes a card for themselves then it starts out without tags.\n\n*/';
/*!40101 SET character_set_client = @saved_cs_client */;

-- Table structure for table `courses`

/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `courses` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `description` varchar(250) COLLATE utf8_bin DEFAULT NULL COMMENT 'A description of the course, if it is a language course it should be in the language of the student\n',
  `tag` varchar(45) COLLATE utf8_bin DEFAULT NULL COMMENT 'the unique tag identifying cards in this course',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='a course is something a user could follow, for instance "Rom /* comment truncated */ /*anian for English speakers", using the unique tag "en-ro" for the cards.\n\n*/';
/*!40101 SET character_set_client = @saved_cs_client */;

-- Table structure for table `exposures`

/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `exposures` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `side` int(11) DEFAULT NULL,
  `timestamp` int(10) unsigned NOT NULL DEFAULT '0',
  `duration` int(11) NOT NULL DEFAULT '0',
  `user_id` int(10) unsigned NOT NULL,
  `card_id` int(10) unsigned NOT NULL,
  `answer` varchar(256) COLLATE utf8_bin NOT NULL COMMENT 'as long as the answer is null there was no answer from the user for this card\n',
  `score` int(10) unsigned NOT NULL DEFAULT '0',
  `grade` int(11) DEFAULT '0',
  PRIMARY KEY (`id`),
  KEY `fk_exposures_users1_idx` (`user_id`),
  KEY `card_id_idx` (`card_id`),
  KEY `grade` (`grade`),
) ENGINE=InnoDB AUTO_INCREMENT=481 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='an exposure is a single showing of a  card to a user, and wh /* comment truncated */ /*at their response was*/';
/*!40101 SET character_set_client = @saved_cs_client */;

-- Table structure for table `user_cards`

DROP TABLE IF EXISTS `user_cards`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `user_cards` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `user_id` int(10) unsigned NOT NULL,
  `card_id` int(10) unsigned NOT NULL DEFAULT '0',
  `first` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'first time this card was exposed\n',
  `last` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'last time this card was exposed\n',
  `next` int(11) NOT NULL DEFAULT '0' COMMENT 'next time this card should be exposed',
  `interval` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'the interval between showings\n',
  `reversed` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'reverse the card or not on viewing\n',
  `exposures` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'times the card has been exposed\n',
  `correct` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'times the user got it right\n',
  `wrong` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'times the user got it wrong\n',
  `score` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'score is in percents, so 10 exposures correct would yield a score of 1000\n',
  `state` int(10) unsigned NOT NULL DEFAULT '1' COMMENT '0: inactive\n1: active\n2: ignored\n\n',
  `learning` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'card is currently considered to be in the learning phase or if it should count as ''learned'', this counts down from a number to ''0''\n\n',
  `tags` varchar(250) COLLATE utf8_bin DEFAULT NULL COMMENT 'initially translated to the language of the user, these are copies of the tags the cards are created with, but the user can modify them\n',
  `difficulty` float DEFAULT '0',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`),
  KEY `user_id_idx` (`user_id`),
  KEY `card_id_idx` (`card_id`),
  CONSTRAINT `user_cards_card_id` FOREIGN KEY (`card_id`) REFERENCES `cards` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
) ENGINE=InnoDB AUTO_INCREMENT=77 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='user cards are all cards in play for a user at this moment,  /* comment truncated */ /*they are in verification or learning mode or have been set to be ignored\n*/';
/*!40101 SET character_set_client = @saved_cs_client */;

-- Table structure for table `user_courses`

DROP TABLE IF EXISTS `user_courses`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `user_courses` (
  `user_id` int(10) unsigned NOT NULL,
  `course_id` int(10) unsigned NOT NULL,
  `score` int(11) NOT NULL DEFAULT '0',
  PRIMARY KEY (`user_id`,`course_id`),
  KEY `fk_user_courses_users1_idx` (`user_id`),
  KEY `fk_user_courses_courses1_idx` (`course_id`),
  CONSTRAINT `fk_user_courses_courses1` FOREIGN KEY (`course_id`) REFERENCES `courses` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
  CONSTRAINT `fk_user_courses_users1` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='a user_course is a course that a user has started on';
/*!40101 SET character_set_client = @saved_cs_client */;

-- Table structure for table `users`

/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `users` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `nick` varchar(45) COLLATE utf8_bin NOT NULL,
  `pwhash` varchar(250) COLLATE utf8_bin NOT NULL,
  `score` int(11) DEFAULT '0',
  `email` varchar(250) COLLATE utf8_bin NOT NULL,
  `newcardsperday` int(11) DEFAULT '20',
  `cardspersession` int(11) DEFAULT '25',
  `decay` int(11) DEFAULT '3',
  `language` char(2) COLLATE utf8_bin DEFAULT 'en',
  `initialinterval` int(11) DEFAULT '5',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`),
  UNIQUE KEY `nick_UNIQUE` (`nick`),
  UNIQUE KEY `email_UNIQUE` (`email`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='users are people that use the system\n';
/*!40101 SET character_set_client = @saved_cs_client */;


-- Dump completed on 2014-05-27 10:52:39

SRS Algorithm

I’m going to do the actual SRS stuff based on ‘exponential back-off’. If you respond to a question immediately and get it right then you earn a 5x increase in schedule interval. If you get it right but after a longer pause then the scheduling interval remains unchanged, get it almost right (within 1 character of the right answer) (this only works for languages!!) the interval will be reduced by a factor of 5, get it completely wrong and you start all over again from the initial interval (set to 5 seconds for now).

Intervals below 5 seconds will be reset to 5 seconds.

So this makes the scheduled intervals 5, 25, 125, 625, 3125, 15625, 78125 seconds etc. So after answering right 7 times in a row you’re on a rough schedule of reviewing a card in another day. Of course people will use this whenever they feel like so we need to make sure that we don’t overwhelm them with a backlog of cards to review, and we need to present those cards that are most likely to be forgotten first.

To allow some personalization we can make the ‘factor’ (5 in this case) configurable on a per-user basis, maybe automatically adjust it for users that score particularly good (or bad) once we have enough data to figure out how well they retain information.

I don’t like the concept of ‘lessons’, so I’ll drop that entirely, you’re ‘learning’ a deck or you’re doing something else, you can learn as long as you want and you can ‘pause’ or ‘stop’ the process at any point in time.

I don’t want users to ‘grade themselves’, the program should do any and all grading completely automatic (this creates some very interesting challenges later on when audio support gets added). And I want people to be able to start learning without being logged in the first time they hit the site.

I’m keeping all this information nicely organized in a Leo Outline Editor file, Leo is absolutely fantastic and I wished there was a version of it that used a centralized docs store.

Command-line version

The simplest framework is of course none at all.

To make sure I understand all the business logic involved and have a working knowledge of the whole process I first built a command line version of the program playing off the same ‘magically populated’ database. That way I can very quickly test stuff without having to bother with layouts, RPC, servers, client server issues and other complications. An interactive command line version of this is about as simple as it gets (and can be built in a few hours if you have some experience).

So, that seems to work. And if it is ugly/buggy that’s fine, it did what it was supposed to do, bring out the flaws in my reasoning about the data model.

The Stack

If you’re reading this far you no doubt have an excellent understanding of web development, but just in case you don’t: The web is an ugly thing these days. What used to be a single binary serving up some static files from a directory on a server has turned into a whole slew of tecnologies, software, standards and options. Collectively, any set of specific choices from all those tools is called a ‘stack’. Typically, a stack will include an operating system, some sort of database for persistence, a back-end language (server side stuff), a front end language (typically JavaScript or something that compiles down to JavaScript), a web server (for instance, Apache, nginx) or ‘application server’, and some bits and pieces to glue it all together. The reason for this hodgepodge of loosely connected bits of tech is that we’re trying to shoehorn application interaction across a medium designed to serve static files on a fire-and-forget basis.

In my case, only the operating system (Linux) and the database (MySQL) are set in stone, the rest is variable. I do have an apache installation on this development machine so if any of the language/framework combos that we will be looking at can run under apache I’ll definitely try to get that to work since it stops me from having to learn how to configure yet another web server. I’m a lazy person.

Next up, the language/framework combos.

A Western Kid Living in Communist Poland

In this HN thread I commented on how the story about the People’s Republic of Donetsk reminded me of my days in communist Poland.

User bernardom prompted me to tell this story, so here it goes (I’ve slept on it and I don’t think it will do any harm, even though all the participants are alive (and well)).

I got my driving license in 1986. My dad and his wife were looking to buy a new car and were going to trade in their old Citroen 2CV. So I matched the offer they were getting on their trade-in and I got my first car. The 2CV (named after it’s fairly anemic engine with according to the French tax office ‘2 tax horse power’) is probably best described to people that have never been in one as an extermely cleverly designed tent on wheels. It’s about as spartan as a car can get, compared to the 2CV a Mini is luxurious.

I worked in Amsterdam west at the time and of course I would slowly expand my confidence driving to work every day and back. The car was so slow to accelerate that I did not feel comfortable on the highways so I stuck to city traffic and b-roads. This meant that longer trips were out of the question but given how crappy a driver I was in those days that’s probably a good idea. Two weeks later my good luck ran out. It was the first real frost that year, November 1986 (COBOL was going strong, I was an applications programmer for a dutch bank back then) and when I accelerated away from a traffic light just past central station in Amsterdam a Volvo in oncoming traffic skidded, carreened down a slope into one of those old style cast iron road dividers. Two metric tons of Swedish steel impacted the divider, which probably weighed no more than 80 kilos or so, accelerating it to projectile speeds.

What saved my life was probably playing a lot of pingpong. Reflexes took over and I ducked down just as the barrier swept through the car, without slowing it down much. The whole top of the car was gone, it was an instant convertible. I pretty cooly steered the car to the side of the road, started shaking like crazy and waited for the police to arrive, who asked ‘where the body was’. I realized I had been very lucky.

The wreckage was towed to my house in east Amsterdam and my mom broke down spontaneously when she saw the wreck. It was quite incredible that I walked away from that without a scratch.

But I learned a valuable lesson: not all accidents are your fault, and if you drive a tent on wheels you could very well end up as roadkill when someone else messes up. So I started looking for something a bit more sturdy. My mom had had a long time boyfriend, a man named Hans. Hans was super cool, an arts dealer in Amsterdam, fancy house on Herengracht and a super nice car, a Citroen DS.

If you’ve never seen one, prepare to be amazed, it was designed in 1955 and is by my standards still one of the most advanced cars ever to be produced. It also still looks good today:

I started hunting for one through the various DS specialist garages in Amsterdam (it is still quite a popular classic in NL today) and found one right around the corner, light blue with a white roof, built in 1973 and with ‘only’ a few hundred thousand kilometers on it. For 3750 dutch guilders it was mine, rust and hydraulic leaks and all.

Now, here was a real car! I had absolutely no problem taking it on the highway and so I decided to tour the highways near Amsterdam.

As I was about to enter the Coentunnel I noticed two pedestrians hitchhiking right at the mouth of the tunnel entrance. A very dangerous spot to put it mildly. Being my usual helpful self I stopped the car, bundled them into the backseat and drove them to the other side where I exlained to them that that was a very bad idea. The hitchhikers, two girls from Poland explained that they didn’t know much about the Netherlands, but that they were walking to a place called Egmond. From where I picked them up to Egmond was at least another day worth of walking so I offered to drive them there.

We talked a bit more, I figured out they were quite hungry and offered to get them some food. This they declined. Then I told them I was hungry and had to get some food. They agreed to come along and we stopped in Alkmaar to eat. Maybe they would eat a bit after all. After both had polished off double helpings of everything we continued on our trek to Egmond.

On arrival in Egmond we said goodbye, I gave them my phone number in Amsterdam in case they got themselves in trouble and went back home. About a week later they called me. Could they please come to Amsterdam? In Egmond they’d been pretty much enslaved by a flower bulb grower, who made them work day and deep into the night for very little money, and who would then charge them for everything (food, lodging) to get it almost all back. Modern slavery (yes, we Dutch are nice people, and if you think that such things are a thing of the past now then think again, this sort of thing still happens all the time). I went back there, picked them up and they moved into my house in Amsterdam.

Calling it a house is a bit much. Today you’d probably call it a hacker space. It was a 60 square meter partition of an industrial building. No shower, just a small sink, an electrical stove and tons of computing gear, a pingpong table, a couch and a single mattrass on the floor.

I gave them the bed, made mine on the couch and for the next week or so we talked day in day out about what life was like in Poland and what it was like in the Netherlands.

When they left they gave me their addresses in Poland and invited me over.

For months that invitation burned a hole in my pocket. I was totally enamoured with one of the girls, and made up my mind to go and visit Poland. This was a lot easier said than done.

I got my first taste of what Poland would be like when I entered the Polish embassy in the Hague. A line of about 50 people. Before I got to the head of the line the office closed and I had to go back another time. This time I took care to arrive early. I applied for a visa and filled in a stack of forms. For months my visa application was held up. More and more information was required. What did my grandparents do during the war? How much money did I have? Did I have family? Sisters, brothers, everything! At some point it occured to me they knew more about me than my mom did.

But finally, after months of filing ever more papers I got the call, I could come and give them my passport. Then they would stamp it after the office had closed and then a few days after that I could go and pick it up again.

And so it happened. I rolled into Poland somewhere in the early hours of morning August 1987. Lines of tens of people in front of non-descript shops had already formed. People waiting for the bakeries to open to get bread. Be early and you might have a chance. Initially I did not comprehend what the lines were for, my mind had been totally conditioned to ‘want something, go out and buy it’. The whole idea of shortage was something that I was not familiar with. At that time, The solidarity movement was clashing headlong with the Polish communist government and there was quite a bit of tension in the air. I was received warmly in Poznan, spent about a month there and then went back to the Netherlands. What struck me most about Poland is how serious the people my age there were. And how tremendously strong the friendships. These were in many ways the most interesting times you could have possibly picked to be behind the iron curtain, I got to observe the whole underground movement from the point of view of a spoiled western kid, retrospectively one of the formative moments in my life. Solidarity was quite something, the streets were abuzz with it. The Polish secret police (the UB or MBP) was quite active and if you didn’t know someone personally you could not trust them. This caused the Polish people to form tremendously strong friendship bonds, a thing that is probably a lot less strong today in absense of such formidable enemies. The police in Poland was tremendously corrupt at the time, a nice example of power out of control.

For months my newfound girlfriend and me corresponded by good old snail mail. We got married, lived together in Amsterdam for some years and then moved to Poznan, Poland together.

This created a problem because I could not legally work in Poland. So I applied for a work permit, and this is one of the most bizarre experiences I’ve ever had. It wasn’t so much that such permits did not exist, it was that applications were so rare that almost nobody knew how the whole process worked. It took many months, ever more interviews (my Polish at that time being next to non existant), and lots and lots of forms. Ever more forms. The most frequent question was why on earth I would want to live in Poland when everybody that could was trying to get out.

Finally I got it, after I had all but given up. I still have it in Amsterdam in a suitcase somewhere, I should probably scan it and put it in here when I’m back there. Somewhere along the way a Texaco gas station opened in the middle of Poznan.

Imagine a background of gray highrises, no colour anywhere, all the cars gray and the ancient diesels of buses belching smoke. And in the middle of that an island of bright red, a tiny patch of the west invading on all that gray, a harbinger of what was to come. A friend of mine said this: “If the future is going to be so full of bright colours I don’t know if I can stand it.”.

Right now I’m living in Romania (Bucharest to be precise) and it’s funny how I can trace back almost everything major that happened in my life to that point where I stopped the car.

So, that’s the story of the Hitchhikers, the flowerbulbs and the ancient Citroen.

Thanks G. for changing my life, for the better in all respects.

edit: and thanks I. for correcting my broken memory!

Ripe for Disruption

Disruption is something that you can usually only establish after the fact. A decade later or so you can look back and say ‘x’ disrupted ‘y’. Microsoft did it to IBM.

Disruption usually plays on a larger scale, for instance, the semi-conductor industry all but destroyed the vacuum tube industry (for certain purposes though vacuum tubes had hard-to-beat advantages potentially good enough for a niche revival).

On the web disruption takes place at a smaller scale. Craigslist disrupted classifieds, Facebook disrupted myspace, Google did it to Altavista and DuckDuckGo is doing it to Google (well, they’re dreaming about it anyway and I wish them luck).

So who is currently setting themselves up for disruption? The hardest part in taking on an incumbent is usually that the incumbent has had years to entrench themselves and that they may have a very solid network effect in place making it all but impossible to take them on.

The thing that can level the playing field and that can give you momentarily a window of opportunity to crawl through and get a chance when normally you would not even be able to dent the armour is a crisis. Every big company has moments of weakness and that’s when they are at their most vulnerable, like over-ripe plums waiting to fall from the tree. In part this stems from their internal sense of being invulnerable, they get sloppy and they have blind spots. So let’s analyze a bunch of the top contenders and point at their weaknesses:

  • Google Adsense. Google makes a ton of money off their ads, so much that they don’t give a damn about the publishers giving them the inventory. More and more publishers are completely fed up with this and would jump ship instantly if a viable alternative appeared. Beware, right from day 1 you’ll be the target of just about every shyster on the planet with a plan of action so ‘know your customer’ is going to be key. And that’s good because that is exactly where google fails.

  • Ebay. This blog post was quite prescient (I wrote it a while ago), but the events of the last week underline the complacency angle. Not only does ebay look terrible, it has a huge problem with scammers and now, on top of that it turns out that it was very lax with security. That’s one of those crisis moments that just might be bad enough to topple a giant.

  • LinkedIn. Linked in is safely entrenched in the business world. Or so they think. But I’m not on there and with me 100’s of thousands of others. I’m not on there because I don’t see the value of LinkedIn, I’m not on there because they are playing fast and loose with data and they spam you to death. If I’m going to be ‘the product’ at least be gracious about it. Linked in seems to believe they are too big to fail, and that their users can no longer do without them and that’s a vulnerability.

  • Flickr. Once the only really good solution for photo sharing in town (a part of that function has now been taken over by Facebook) Flickr is asking for some kick-ass competition. The site has not really thrived under Yahoo! stewardship (what ever did?) and has been steadily dropping in the rankings and in the favour with the audience.

  • Quora, stackoverflow etc. The one suffers from appropriation syndrome, the rest from inappropriatitis. Both leave me frustrated, on Quora half the content (donated freely by users) is now used to draw in visitors through search engines and then used to force them to log in to the service. If a log-in is not required, don’t force your users to do it. And on stackoverflow anything (ok, exaggaration here) that can’t be answered with a simple fact will get closed. Too many times did I have the exact same question as some former stackoverflow visitor, dropped the question into google only to land on a completely useless page. That’s an opportunity.

So, if you’re looking for targets, the web is full of them, go forth and disrupt, business opportunities galore.

Choosing a Web Framework/Language Combo for the Next Decade

Learning Romanian is rapidly turning into a huge exercise in Yak-Shaving.

You often see the question “What stack should I use to develop my application on?” posed but rarely do you get any insight in the actual process once someone decides to answer it specifically, so both as an analogy to rubber-ducking this and showing you what went on while I made my choice I’ve decided to document the whole process. Even though you might not like or agree with the conclusion (which is to a very large extent personal) you may still get some value from the method or the information presented here.

Once I decided to have a go at building my own srs/flashcard web app the question arose to figure out what to build it in. I’ve been putting of this choice for quite a while now, but it really is time I bit the bullet and updated my toolbox and got with the times. There is a lot of general wisdom floating around about this theme (stick to what you know, don’t do your own plumbing (that translates into don’t build your own framework), concentrate on the problem you are trying to solve (yeah, right, learn Romanian), old = good and so on).

So technically speaking this should be a non-choice for me (LAMP + Yii). After all ‘stick to what you know’ is by far the best choice you could make here. And it allows me to re-use my library of home-grown code, which is quite an investment in time and effort. So I had more or less settled on that. But after messing around for a day trying to get eclipse to support Yii (very frustrating, to put it mildly) and the fact that so many of the Yii bits and pieces (such as the Yii eclipse plug-in) are depending on single individuals it is getting harder and harder to defend Yii as a viable choice at this point in time. Yii 2.0 is now in ‘beta’ for however long that will take, and though they write on their website that you definitely should not stop using Yii 1.1 for your new projects at the same time they write that Yii may or may not be supported past 2015. I don’t like gambling that much. And I remember how the Osborne Computer Corporation performed Seppuku on itself.

The good news I guess is that the web is designed around protocols, not pieces of software so that’s all fixable but to require a re-write every few years is really not an option, I simply do not have the time for that. Building things 3 times (MVP, first real version, good enough for long term use) is hard enough, a forced rewrite on someone else’s timetable sounds like a very bad idea.

Let me explain that, I tend to build things 3 times, the first is sort of an exploration of the space, I fiddle around with usually just a bit of data and an extremely rudimentary UI to figure out what it is that I really want to build. Then when that’s sort of crystalized out I build the second version, one that lives for a while, days, weeks or months. This usually has actual use(rs) and maps out the space even further. Inevitably this turns up some really bad decisions made in the first or the second phase that were not apparent at the time. And so after a relatively short time the whole thing gets ripped apart again and then I build it in a way that it can last for a decade or longer while keeping the exterior unchanged as much as possible. It’s surprising how much of the stuff I built in my nearly 3 decades of programming is still in active use today. Even more surprising that the things that are still in use all felt like they were temporary at best, projected lifespan a few year maximum. In one extreme case of this a prototype I built for a company in 1986 is still in production use today. So deciding what stack to build something on is a very important decision.

Choosing not to go with the most familiar previously used combo is against just about all the good advice, but I like learning and this is as good an excuse as any I’ll get to jump out of the box.

My list of requirements is:

  • L-M-, to clarify, Linux and MySQL are non-negotiable (because I exclusively use linux for everything I do and I have a ton of data already stored in MySQL tables) This also limits the size of the Yak to some extent and given the direction this is going that is probably a good thing.
  • The software should be actively maintained
  • The license should be an open source license that is well known or one that is so simple that I can understand it.
  • Issues should preferably be solvable using google searches and stackoverflow, I don’t like making tickets so community size and having others walk the path before is important
  • The framework/language combo should play nice with other tools (editors, revision control systems and so on)
  • It can be ‘opinionated’ but not so opinionated that I have to migrate databases to new schemas to make the framework happy
  • It doesn’t have to be the fastest but it should play nice with caching to speed it up should the need arise. The reason for this requirement is that several of my little projects are now in the 50K+ and 100K+ per day visitor counts and if you eat up a lot of cpu per pageview then you’re going to spend a ton on hosting. Not all my projects are for profit so this really matters.
  • It should be possible to find collaborators at some point (another community size factor), in my experience success of my toy projects is extremely hard to predict and I’ve been caught several times now with something doing a take-off all by itself.
  • Reasonably low bar to entry (quick to get up to speed)
  • Right now this is a one-man-show so programmer productivity is important.
  • The interface with the framework should feel natural and idiomatic to the language underlying it. In my experience, if that is not the case it will sooner or later cause trouble.
  • Actively maintained does not translate into ‘was made yesterday’.
  • The team of maintainers of the language and framework should have more than 1 member.
  • The language underlying the framework should be solid, can’t be classed as experimental and should not be a moving target.
  • Sessions, authentication, routing, internationalization, MVC, caching, mobile, templating, ORM and so on should preferably all be part of the deal, so batteries included please.

Some of these requirements are no doubt contradictory and there will have to be compromise, for instance excellent programmer productivity, low bar to entry and good execution speed are rarely found together.

In other words, time for some web browsing, apply the above criteria, pick a language/framework combo and then get to work. Sounds easy, right?

This opens up the field quite a bit. Stick with PHP? (still plenty of choice in terms of frameworks) Move to a new language? (though the wisdom holds that using a new language for a project that you need or care about will make you hate both). Doing technical due-diligence exposes me to all sorts of technology, and it also helps me in making this choice because I’ve had the privilege of getting a ring-side seat in the arena of capable web start-ups and established companies. You get a ton of examples of how things could be solved and what really works (and sometimes, what really does not).

The amount of choice available to someone that is going to make this decision is absolutely incredible. This wikipedia table lists a staggering number of potentials to choose from, and on a first reading it isn’t even complete (for instance, the Go language is missing, are a whole bunch of frameworks (not just Go ones, for instance, Erlang’s Chicago Boss isn’t on that page). If you’re maintainer of some framework or care about your language ecosystem please update that page!).

On the language front there are: asp.net C, C#, C++, Clojure, coldfusion, Erlang, Elixir, F#, Go, Haskell, Java, JavaScript, Lua, Scala, Pascal, Perl, PHP, Python, Ruby, Rust, SmallTalk and Tcl. (there even is a web framework written in COBOL and even though at one point in my long career this was how I made my living nostalgia is not enough to overcome the resistance to putting that language on the initial list. Basic, Fortran, Algol etc omitted for similar reasons.

What a beautiful illustration of The Paradox of Choice!

Because the list is simply way too long to start comparing the frameworks and then letting the framework dictate the language I’m going to turn things upside down for a while, and use the languages as an indirect way to get rid of a ton of framework selection work, then when the list is small enough we’ll reverse this again and look at the frameworks left over.

Language Selection

Of those languages listed above I’m very familiar with C (old!) and PHP (ugly!), I have a working knowledge of Java (but really don’t like it), I’m interested in the Erlang ecosystem (beam and erlang/elixir), do not like Perl, do not like the Ruby ecosystem, I think Pascal has had its day, Go is interesting because it is close to C and has support from a huge company, Python has never really felt like the right choice for something that may one day need to be supported, clojure looks nice but has a huge learning curve and I got totally stumped by the errors it throws up in a toy project I built with it, SmallTalk due to its image based nature has a hard time playing nice with commonly accepted toolchains and has quite a learning curve, Tcl is not really modern web suitable (though it’s been done), Scala is unfamiliar, Lua looks interesting, Rust is too young, JavaScript feels like everything built in it is extremely temporary (but it has the advantage that you can decide to work in only one language) and Haskell seems both for people much smarter than me and totally unfamiliar.

Since this is now a process of elimination, let’s come up with some (hopefully) good criteria to winnow down the language set to something a bit more workable.

Let’s start by getting rid of the languages that are in my opinion unsuitable for modern day web development. Goodbye C, Perl, Pascal, SmallTalk and Tcl. Smalltalk also does not conform to the requirement of playing nice with others and being able to use version control tools outside of the image.

Next, let’s get rid of the languages that are too new:

That removes Rust, even though in the long term I have high hopes for the language.

Haskell gets removed because I’m not in with the smart crowd (call me a blub programmer and see if I care) and because I’m not writing some financial application (which is, apparently where Haskell really shines).

Scala gets dropped because I don’t really know it but think it is really terribly ugly. This could be another instance of me not being smart enough or not getting it but I just can’t seem to get past that.

I still haven’t forgiven Microsoft for their countless misdeeds so Asp/C#/F# are all right out. That may be shooting myself in the foot here (cut off my nose to spite my face), but contributing to this decision is that I work exclusively with linux (both on desktops, laptops and servers) so the MS ecosystem has become totally unfamiliar to me and I don’t like running windows on servers (though, of course there are some workarounds possible here and Mono is quite good). So, the L in my LAMP is firmly established. And besides, I don’t have a lawyer on the payroll to make sure that my licenses are all read, ok’d and acceptable.

I’ve once maintained a coldfusion application for a client and I still wake up in cold sweat every now and then, that may have been a very bad example but the taste has not left me so strike coldfusion.

JavaScript web frameworks seem to have a half-life of about 6 months, I really don’t feel that that is the right attitude to software development. Twitter has rebuilt most of their infrastructure in JavaScript, that’s a pretty bold move and a huge vote for JavaScript. But I don’t exactly run twitter, I’m a one-man-show, do not plan on hiring an army of programmers and have always thought of JavaScript as a terrible hack that has now somehow managed to propagate itself backwards across the client/server link to infect the server environment. I just can’t get behind it, so JavaScript goes.

So, now we’re down to 9 candidates, still way too many but at least there is some breathing room. The eight are: Erlang, Elixir, Clojure, Go, Java, Lua, PHP, Python, Ruby.

I’m going to eliminate Clojure on the grounds of it being a super nice language but I really hate the runtime error messages (see above) and I see it used far more in data-analysis roles than in web sites. In a way this is a pity because Clojure has all the good stuff of Java (tons of libraries, compiled, reasonably fast). Now we’re down to 8.

Lua… lua is a very interesting little language, it’s elegant, I really like the concepts behind it, I’m aware of some extremely impressive applications built with it. Think of lua as hitting the sweet spot between Python and Ruby, with a good link to C thrown in for good measure. The grounds for which I’m dropping it are that lua does not feel like it is a stand-alone environment, I always think of it as a plug-in to something else to give it web capability or to make it configurable/extensible. The lack of libraries is also a huge drawback.

Python has always been a mixed bag for me. What I wrote in it worked, when I wrote it. Then, a little while later it stops working, either because some module that it depends on gets deprecated or changed in an incompatible way, starts to exhibit weird behavior or even disappears. I’ve written some thousands of lines of python altogether so I would definitely not consider myself fluent in the language. But contrasted with C code that I wrote 30 years ago that I can compile today this is not a good personal record (though it may very well have to do with my picks of python modules). The significant whitespace has always felt to me like driving on the autobahn at 200 km/hour without guardrails and every time I try using it for web development I end up re-writing it in something more suited to the task. For scientific stuff it has replaced Perl in a large number of use cases and it will be here for a very long time. So Python is out for me, but this is really becoming personal and less objective. Every web project I ever built in python also felt slow to use.

With Python eliminated we are now down to 6 entries in the language category, but Erlang and Elixer should probably be combined on account of their close relationship. (A similar thing could be said for clojure and Java but I think that clojure’s integration into the JVM is less elegant than Elixir is when looking at Beam).

Ecosystem size is a huge factor when evaluating a language/framework combo, and if the ecosystem around a language is small that means that for the most part you’re going to be on your own figuring things out and that if you ever want to hire someone (a contractor, an employee) to help out with some aspect of your project that it will probably be difficult (this would have been another strike against Clojure, come to think of it). Erlang & Elixir definitely have this problem, but I can’t help but be totally fascinated by the ease with which applications can be clustered and made ultra reliable. For instance, ‘whatsapp’ is built in Erlang. So erlang sounds like a worthwhile skill to have under my belt and the eternal draw of technology for its own sake is really making it hard to keep my eye on the ball here (which is, to learn Romanian, in case you forgot). So, with deep regret, pain in my heart and apologies to Joe Armstrong, whose thesis (or should I say Magnum Opus) is a must read for any programmer I’m going to have to drop Erlang, and by strong association, Elixir as well.

Java, PHP and Ruby are all mature and stable languages, Go is new but has some very big names behind it, people that I know are capable of building for the long haul. Sites like langpop don’t even have it on their radar. Go does concurrency very well (always a handy feature), compiles wicked fast and allows me to re-use at least a little bit of my C knowledge. So I’ll keep it in, even though it is only 4 1/2 year old and still extremely small in footprint (joke intended) when compared to Java, PHP or Ruby.

The short-list of languages is now 4 entries long: Go, Java, PHP, Ruby. Two of those (Go, Java) are compiled, and two are interpreted (PHP, Ruby), though, you could compile PHP using facebooks HHVM (HipHop Virtual Machine). I think that’s short enough that we can stop eliminating languages for good reasons and start to expand the list to include frameworks to get an idea of what each language has to offer that speaks for it, rather than to get rid of it, reversing us back to where we began, selecting a framework and let that dictate the language.

Framework Selection

Infant mortality amongst frameworks and languages is ridiculously high so going with a ‘young’ framework or language is a significant risk and can turn out extremely expensive in the long run (for instance, after the framework ends up being orphaned you have to make sure your stuff remains afloat so you need to support/extend the framework and you have a serious risk of security issues going unnoticed for a longer period of time). If I’m going to place a long term bet on some chunk of code I would really like to at least have the feeling that it will be around for a while. This will make life extremely hard for Go to stay in the race, after all it is young itself so all frameworks are likely going to be young as well.

Let’s make a listing of what we’re let with at this stage (ripped from the wikipedia page):


Project Current stable version Release date License
Apache Click 2.3.0 2011-03-27 Apache Software License 2.0 (ASL 2.0)
Apache OFBiz 12.04.02 2013-07-30 Apache Software License 2.0 (ASL 2.0)
Apache Shale 1.0.4 (Retired) 2007-12-19 Apache
Apache Sling 6 2011-04-18 Apache 2.0
Apache Struts 2 2.3.16 2013-12-08 Apache 2.0
Apache Tapestry 5.3.7 2013-04-24 Apache
Apache Wicket 6.13.0 2014-01-14 Apache 2.0
AppFuse 2.1 2011-04-04 Apache
Eclipse RAP 2.0 2013-02-11 Eclipse
FormEngine 2.0.1 2011-05-08 Proprietary
Grails 2.3.5 2014-01-16 Apache
Google Web Toolkit 2.6.0 2014-01-30 Apache 2.0
Hamlets 1.7 2011-03-11 BSD
ItsNat 1.2 2011-05-24 LGPL, proprietary
JavaServer Faces (Mojarra) 2.2.6 2014-03-04 CDDL, GPL 2, Apache 2.0,
JBoss Seam 3.1.0 final 2012-01-13 LGPL
Jspx-bay 1.2 2013-02-14 Apache 2.0
JVx 1.1 2013-01-23 Apache 2.0
OpenLaszlo 4.9.0 2010-10-21 CPL
OpenXava 4.7 2013-04-02 LGPL
Oracle ADF 2013-07-11 Oracle Technology Network Developer License
Play! 2.2.3 2014-05-01 Apache 2.0
RIFE 1.6.1 2007-07-14 CDDL, LGPL
Spring 4.0.3 2014-03-27 SpringSource
Stripes 1.5.7 2012-05-17 Apache
ThinWire 1.2 2007-09-17 GPL
Vaadin 7.1.9 2013-12-04 Apache 2.0
VRaptor 4.0.0.Final 2014-04-23 Apache 2.0
Wavemaker 6.5.3 2013-03-14 Apache
WebObjects 5.4.3 2008-09-15 Proprietary
WebWork 2.2.6 2007-07-21 Apache
ZK 6.5.2 2013-03-26 LGPL, ZOL
ztemplates 2.4.0 2011-09-11 Apache
4WS.Platform 2.1.6 2014-05-07 LGPL


Project Start date Current stable version Release date License
Agavi 2005-05 1.0.7 2011-12-22 LGPL
Aiki Framework 2009-09 0.9.1 2012-02-23 GPLv3
AppFlower 2009-02 1.3 2012-10-09 GPLv3
Ayoola Framework 2011-01 1.4.1 2014-01-12 Proprietary
CakePHP 2005-08 2.5.0 2014-05-13 MIT
Cgiapp 2004-12 1.0 2004-12-01 LGPL
CodeIgniter 2006-02-28 2.1.4 2013-07-08 OSLv3
Drupal 2001-01-15 7.28 2014-05-08 GPLv2
Fat-Free 2009-09 3.1.0 2013-08-19 GPLv3
FuelPHP 2011-08 1.7.1 2013-12-01 MIT
Hazaar MVC 2012-10 1.1 2013-04-19 Apache 2.0
Joomla 2005-08-17 2.5.19 2014-03-06 GPLv2
Kajona 2006 4.2 2013-07-16 LGPLv2
Laravel 2011-06-11 4.1 2013-12-11 MIT
Logiks 2011-06-24 3.6.1 2014-05-02 MIT
Lithium 2009-10 0.11 2012-10-03 BSD
Nette Framework 2006-01 2.1.2 2014-03-17 New BSD, GPLv2, GPLv3
PHPixie 2012-07 2.1 2012-04-24 BSD
PRADO 2004-01 3.2.2 2013-07-20 New BSD
Qcodo 2005-10 0.4.22 2011-08-15 MIT
SilverStripe 2005-11 3.0.5 2013-02-20 BSD
Seagull 2003-10 1.0.4 2013-01-04 BSD
Solodev CMS 2005-11 8.0 2006-11-12 Proprietary
Symfony 2005-10 2.4.4 2014-04-27 MIT
TYPO3 Flow 2011-10 2.0.0 2013-07-12 LGPLv3
Xyster Framework 2007-09 02 Build 01 2010-10-18 BSD
Yii 2008-01 1.1.14 2013-08-11 New BSD
Zend Framework 2006-03 2.2.5 2013-10-31 New BSD


Project Current stable version Release date License
Camping 2.1 2010-08-20 MIT
Ruby on Rails 4.1.1 2014-05-06 MIT, Ruby
Ramaze 2012.12.08 2012-12-08 Ruby
Sinatra 1.4.4 2013-10-21 MIT
Merb 1.1.3 2010-07-10 MIT
PureMVC 2.0.4 2008-08-14 CC+Attribution


Project Current stable version first release License
Martini ? 2013-10-30 MIT
Negroni ? 2014-05-19 MIT
gocraft ? 2013-11-10 MIT
beego 1.2.0 2012-03-05 Apache 2.0
Gorilla ? 2012-10-03 Copyright (c) 2012 Rodrigo Moraes. All rights reserved.
Revel 0.9.1 2011-12-09 MIT
TigerTonic 2013-02-10 Copyright 2013 Richard Crowley. All rights reserved.
Go Rest 0.1 2011-08-04 New BSD License
Web MIT 2009-12-09 New BSD License

That make for 75(!!) frameworks in total for the four languages still left. That’s still a bit much to do say a quick and dirty test application in all of them to see how they feel.

More winnowing…

We can use similar criteria used above to get rid of a number of frameworks, first, let’s drop everything that has a proprietary license. I really don’t feel like expending a bunch of effort and then having the rug pulled out from under me because of some obscure passage in a license that I did not evaluate properly. So either it is open source under a well-known license that suits my needs or I’m going to have to pass on that particular framework.

This gets rid of Eclipse RAP, Formengine, Oracle ADF, Spring and Webobjects in the Java arena. For PHP we lose Ayoola, and Solodev CMS, for Ruby and Go we lose none.

7 down, 68 to go.

Next up, have I even heard of the framework? That’s quite subjective, especially for Go, but it’s a good criterium. If a project is so obscure that someone looking into as many companies as I do and a HN junkie to boot has not heard of a framework that’s reasonable grounds for dismissal. For Java we lose Apache Click, Apache OFBiz, Apache Shale, Hamlets, JavaServer Faces, Jspx-bay, JVx, OpenXava, Stripes (vaguely rings a bell somewhere though), Vaadin, ztemplates and 4WS.platform. For PHP that makes short work of Hazaar, Kajona, Logiks, Qcodo, Typo 3 Flow,

In the Ruby camp this loses us Camping and Ramaze. (Technically, I should probably delete all the frameworks in the Ruby category except for Rails, but I have heard of Sinatra, Merb and PureMVC, I just don’t remember if that was in a good or a bad context, which is anecdotal proof that there is no such thing as bad publicity). For Go this means the end of TigerTonic, it isn’t a framework anyway (more of a router) and it could be combined with another framework to do high-speed routing (which it does using a trie).

Another reason to discard frameworks is because they are stale, there is a very small chance that a framework is so perfect that it does not need maintenance for years on end but by my reading a framework that is not worked on continuously is a dead framework. This is because the web is a moving target and frameworks should keep that target in their sight at all times.

For Java this axes OpenLaszlo, RIFE, ThinWire, WebWork (now merged into struts2). For PHP we register the demise of Agavi, Aiki Framework, AppFlower, Cgiapp, PhPixie (or maybe they just ignore those github issues), PRADO (superceded by Yii), Seagull and Xyster. In the Ruby domain Merb has apparently been merged into Rails, and PureMVC looks quite dead. For GoLang it was a sad week because Martini bit the dust after some pretty well aimed criticism, to be replaced by negroni.

That was quite the slaughter, down to 35 frameworks now if I’ve counted that all correctly.

Since the start of this journey was PHP/Yii I’m really tempted to throw out Yii too, I really strongly dislike it when important chunks of code like frameworks are given vague future prospects, a framework is either eol’d or it should be in active development. Saying you may or may not sunset a framework in a years time and in the meantime to encourage users to build new software on top is very irresponsible. I have similar misgivings about Drupal, where a bi-annual re-write is pretty much the only way to stay current. So out of sheer spite and to get me out of my comfort zone I’m going to drop both Yii and Drupal.

The whole Martini/Negroni thing has me in a quandary. On the one hand, I think the response to stop developing the code and to move on to a more ‘proper’ approach is a very mature one. At the same time I’m so terribly happy that I’m not one of the people that started to develop some major chunk of code on top of Martini about 3 months ago or so. I would feel very much let down. The fact that a whole project gets axed by one well aimed blog-post is - to me - proof that one should be very distrustful of anything that has not yet withstood the test of time. Just like ‘too old’ and ‘stale’ are valid criteria for rejection, the same holds true for ‘too new’. Who is to day what will happen to Negroni 6 months from now, when another blog post by a competitor aims to derail it? (The fact that that particular blogpost aimed to also promote the product of the author was less than classy). So I’m dropping Negroni from the ‘Go’ set of frameworks on account of it being too fresh and the maintainer giving me the impression of being quite unpredictable.

Me not having heard of your framework is one thing, but google not really knowing about it is another red flag. Let’s do a little notoriety check and see what turns up:

Apache Sling44,000 results
Apache struts2170,000
Apache Tapestry18,800
Apache Wicket95,000
AppFuse 6470,
Grails 33,000,
GWT 282,000,
JBoss 27,000,
PLAY 993,000(!),
Vraptor 3980,
wavemaker 3200,
ZK 57,200.
CakePHP 98,000,
CodeIgniter 285,000,
Kohana 113,000 (not in wp, eol’d!)
Fat-Free 2,580,000 (wrong?)
FuelPHP 8110,
Joomla 513,000,
Laravel 136,000,
Lithium 6190,
Nette 354,000
Silverstripe, 25,500
Symfony 281,000
Zend 2,280,000(!!!)
Rails 258,000
Sinatra 5160
Gocraft 3820
Beego 1370
Gorilla 695
Revel 4650
Go Rest 6580

Web is impossible to google for in a normal way.

You can very clearly get an idea of the relative sizes, maturity and adoption of the various frameworks and language eco systems here. (I did not expect such a huge spread and it’s a good indication that I should not go by my gut on things like this, for instance, I would have estimated Sinatra much larger and never knew Play! was that big.)

Based on the relative sizes of the frameworks within each language category I’m going to retire the ones that are substantially smaller than the mid-range. This removes Tapestry, ApFuse, JBoss, Vraptor, wavemaker, FuelPHP, Lithium, Silverstripe, Sinatra and Gorilla, and leaves us with 21 frameworks.

Web frameworks are not all created equal and not all of them are created by teams. Each of them has its own target audience, is more or less feature rich and typically outcompete each other on some aspect but have a large chunk of overlap when it comes to the basics. Those basics had better be there though. For me, that translates into: authentication, routing, session management, multi-language, ajax, MVC, persistence, templating, input validation and testing. If those are not present in a framework then that means I’m going to have to roll my own for that aspect of the framework and in the long term I’d rather maintain the app than a chunk of framework (that’s the whole reason for choosing a framework to begin with!). I dropped my own one-man-show framework years ago because it was getting longer and longer in the tooth, I see no reason to go back, even partially to that situation if I don’t have to. Caching is optional, I assume that the makers of a framework are dedicated to keeping things secure. I also like it when frameworks have a team behind them rather than a single person as the main driver.

On account of not having testing, internationalization and an ORM I’m dropping Sling (though likely you could use ActiveRecord), Wicket does not do MVC so it goes, Grails is not written in Java, CodeIgniter has no ORM and no internationalization, Joomla is more of a CMS than a framework, Laravel has a ‘bus-factor’ suspiciously close to 1. The ‘Go’ frameworks are not documented in Wikipedia so this is a bit more work to figure out. GoRest appears to be mostly a router rather than a complete framework, GoCraft ditto (it’s a router ‘web’ and a simple orm ‘dbr’). Hoisie/web is another router. The Go world seems to be taking an extremly modular approach to the framework problem, in a way I like that (modularity is good), but it also means that you have to do a lot of hunting and picking to get a set that you like and it is extremely likely that you are then one of the few using that exact combination. It also signficantly increases the chances of one of those elements being suddenly end-of-lifed (the same thing happens in the Drupal world all the time, with every major release there are tons of orphaned modules).

12 frameworks left, and still, the variety is enormous. I think the next thing to do is to write a very small sample application in all of those (setting them up will be a total pain) to see how well it all comes together and what the friction points are. To make sure I end up with a good choice I have to at a minimum check that all my requirements listed above are fulfilled, and enough of them have to do with actually working in the environment that I won’t be able to check without getting my hands dirty.

The frameworks that are left now are Ruby on Rails, for php CakePHP, Fat-Free, Nette, Symfony and Zend, for Java Struts2, GWT, Play! ZK, for GoLang beego and Revel.

To be continued. Interesting reading along these lines on other people’s pages:

Edit: in the HN thread about this article user nemasu points out that Kohana is no longer actively supported so I will remove it from the list of candidates, also I incorrectly axed C# as being a Microsoft only product, but I feel that even though there are re-implementations of C# the language is under Microsoft control (just like Go is under Google’s control, even if someone else wrote another Go compiler). If and when Microsoft is no longer the dominant factor in the direction of C# would that situation change sufficiently for me to make C# an option for development.