Jacques Mattheij

technology, coding and business

Choosing a Web Framework/Language Combo - the SRS Test Application

After creating a short-list of languages and frameworks to be evaluated (see here for the first installment in this series) the next step is going to be a lot harder. It involves getting my hands dirty and at a minimum going through the website of the framework, looking at each in turn to see if they actually match the criteria and then to decide whether or not to build a ‘toy’ application in that framework or not. Time consuming and annoying because it requires you to learn the essentials of each and every framework involved. On the other hand, there is no substitute for hard work in this case, you can’t really pass judgment on anything before you have become familiar with it. The previous article in this series sparked a ton of interest and I got a large amount of very useful (and some not so useful) feedback.

Quite a few people addressed the ‘what framework to use’ question with arguments rather than agenda pushing or religion, corrected a metric ton of mistakes in the previous post and helped to guide me on my quest. To those people I’m extremely grateful for their hard work, every word got read and will be integrated into the revision of the original post. This has caused some changes to the set of languages/frameworks still under consideration.

Toy Application

Because I want to compare the frameworks in as realistic a setting as possible I’ll be building a real application with all of the ones that made it to the short-list.

The toy application will be the thing this all started with, a flash-card (or Spaced Repetition application. It will not contain any user interface components related to creating cards, the database will be ‘magically’ populated with a series of flashcards grouped into several decks that you could use to study/practice some subject. New users should be able to create accounts and select decks they want to study, and the studying process should work as well.

By the way, studying a language over a long-distance medium is not exactly new. In the 1950’s you could already learn languages over the phone!

The schema for the toy application to evaluate the various webframeworks is (relatively) simple. Iteration on the design starts before the first line of code is written. This is the 4th or so revision of what I came up with initially (which was totally different and way too rigid). Some days I feel like I’m in the Monty Python ‘Spanish Inquisition’ sketch. Every time I finish a sentence I have to back up to correct it without ever getting out of the loop and making the sentences longer/different every time!

Apologies for stealing 8 minutes of your life there. And that ‘toy’ application is starting to look pretty complex.

Database Schema

So, the final (hah!) version of the human readable version of the schema looks like this:

We have ‘users’, who use the application to learn stuff and to review and refresh what they learned afterwards.

We have ‘cards’ that have a front and a back side representing the ‘question’ and the desired ‘answer’ to the question.

Cards can be tagged, 0 or more tags are allowed, these are stored in the card separated by ‘/’ characters.

The format of the fronts and backs is json encoded strings, the interpretation of which is left up to the application. Originally I experimented with fixed fields, s-expressions and other trickery but after a short while I realized that since json is now more or less an accepted standard and present as a library module for serialization and deserialization in almost every language I would want to use that this would save me the time of having to implement/maintain a parser for the data chunks.

The trick with this field is that because it is just a text string unforeseen future developments (multimedia, for instance) can be tacked on without re-doing the whole application so it is fairly future proof. All I need to do is come up with a reasonable interpretation of the data. And if the change would be so drastic that the json format itself would need revision then that could be fairly easily done using a background job without having to take the whole thing down. It also allows for easy generation of huge amounts of cards from permutations of the various bits and pieces.

Cards have a difficulty associated with them and you’re supposed to master one level of difficulty before you’re allowed to move on to the next, this ensures that cards that build on knowledge from simpler cards will not be presented until the simpler stuff is fully grokked.

A deck is simply a pre-programmed filter that lifts out cards with certain (combinations of) tags. For instance, this filter would select for an English speaker all the cards that teach you about numbers in Romanian: “/en-ro/ & /numbers/” (note that the ‘numbers’ is in the language of the user, the ‘en’ in the first part of the tag). This also allows for quick, impromptu and unforeseen course creation (such as: learn to count to ten in all different languages: “/numbers/ & /upto10/”)

Every exposure of a card is logged, as is how long it took to get an answer back and whether or not that was the correct answer.

A user can have none, some or all of the cards of any given course in play and has the capability to de-activate specific cards.

Users acquire cards as they learn, new cards are added if fewer than a user configurable number of cards are in the ‘learning’ phase.

Cards can be reversed in a deck, but only if the card allows it.

You could pull really nifty tricks in interpreted languages, for instance, you could make a front contain code that generated a different card every time it was exposed, or have the cards adapt to circumstances (for instance, a card that explains ‘today’ could always use the actual day as an example). For a compiled language such tricks are not so easy, you’d always have to bundle some kind of interpreter along. Or, alternatively, you could assume the display environment to be capable of doing this, and embed a chunk of javascript in the code! We’re not going down that route though (because we also want to do mobile at some point).

For the future, multimedia (audio, pictures, video, LaTex, html, whatever else you can come up with) could be added to the cards.

So, for the first time ever I used mysql-workbench to create a database schema. I wished I could rave about how easy it was and how well it worked to do this simple task. But truth be told I’ve never ever seen such a piece of junk. It crashed a great many times, user interface components are all over the place, it is totally counter-intuitive and makes something simple much harder than typing in the commands by hand. The save dialog had the ‘save’ button off-screen, and it happily creates broken SQL during the export… Incredible that a company like Oracle would let a thing like this pass Q&A with their name on it.

Anyway, after multiple attempts the job was done (count 4 hours or so, not bad for a handful of tables).



Here is the machine readable version of the schema:

-- MySQL dump 10.13  Distrib 5.5.37, for debian-linux-gnu (i686)
--
-- Host: localhost    Database: srs
-- ------------------------------------------------------
-- Server version       5.5.37-0ubuntu0.14.04.1

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;

--
-- Table structure for table `cards`
--

DROP TABLE IF EXISTS `cards`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `cards` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `creator_id` int(10) unsigned NOT NULL COMMENT 'who made the card\n',
  `reversible` int(1) DEFAULT '0' COMMENT 'not all cards can be reversed\n',
  `front` mediumtext COLLATE utf8_bin NOT NULL,
  `back` mediumtext COLLATE utf8_bin NOT NULL,
  `difficulty` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'difficulty is a number that starts at ''0'' and ascends as far as required to order cards. Two cards can be of the same difficulty (they their order of appearance is not guaranteed).\n',
  `source` int(10) unsigned NOT NULL DEFAULT '0',
  `prompt` int(10) unsigned NOT NULL COMMENT 'which prompt to use\n',
  `tags` varchar(256) COLLATE utf8_bin DEFAULT NULL,
  `prerequisites` int(10) unsigned NOT NULL COMMENT 'up to and including which card should be learned before the user can be exposed to this one.\n\n',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`),
  KEY `source` (`source`),
  KEY `creator_id` (`creator_id`),
  CONSTRAINT `creator_id` FOREIGN KEY (`creator_id`) REFERENCES `users` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
  CONSTRAINT `source` FOREIGN KEY (`source`) REFERENCES `sourcedata` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=1505 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='cards are the ''units of learning'' in the system, cards are t /* comment truncated */ /*agged, the first tag is the name of the deck. If a user makes a card for themselves then it starts out without tags.\n\n*/';
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `courses`
--

DROP TABLE IF EXISTS `courses`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `courses` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `description` varchar(250) COLLATE utf8_bin DEFAULT NULL COMMENT 'A description of the course, if it is a language course it should be in the language of the student\n',
  `tag` varchar(45) COLLATE utf8_bin DEFAULT NULL COMMENT 'the unique tag identifying cards in this course',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='a course is something a user could follow, for instance "Rom /* comment truncated */ /*anian for English speakers", using the unique tag "en-ro" for the cards.\n\n*/';
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `exposures`
--

DROP TABLE IF EXISTS `exposures`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `exposures` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `side` int(11) DEFAULT NULL,
  `timestamp` int(10) unsigned NOT NULL DEFAULT '0',
  `duration` int(11) NOT NULL DEFAULT '0',
  `user_id` int(10) unsigned NOT NULL,
  `card_id` int(10) unsigned NOT NULL,
  `answer` varchar(256) COLLATE utf8_bin NOT NULL COMMENT 'as long as the answer is null there was no answer from the user for this card\n',
  `score` int(10) unsigned NOT NULL DEFAULT '0',
  `grade` int(11) DEFAULT '0',
  PRIMARY KEY (`id`),
  KEY `fk_exposures_users1_idx` (`user_id`),
  KEY `card_id_idx` (`card_id`),
  KEY `grade` (`grade`),
  CONSTRAINT `card_id` FOREIGN KEY (`card_id`) REFERENCES `cards` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
  CONSTRAINT `user_id` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=481 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='an exposure is a single showing of a  card to a user, and wh /* comment truncated */ /*at their response was*/';
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `user_cards`
--

DROP TABLE IF EXISTS `user_cards`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `user_cards` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `user_id` int(10) unsigned NOT NULL,
  `card_id` int(10) unsigned NOT NULL DEFAULT '0',
  `first` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'first time this card was exposed\n',
  `last` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'last time this card was exposed\n',
  `next` int(11) NOT NULL DEFAULT '0' COMMENT 'next time this card should be exposed',
  `interval` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'the interval between showings\n',
  `reversed` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'reverse the card or not on viewing\n',
  `exposures` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'times the card has been exposed\n',
  `correct` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'times the user got it right\n',
  `wrong` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'times the user got it wrong\n',
  `score` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'score is in percents, so 10 exposures correct would yield a score of 1000\n',
  `state` int(10) unsigned NOT NULL DEFAULT '1' COMMENT '0: inactive\n1: active\n2: ignored\n\n',
  `learning` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'card is currently considered to be in the learning phase or if it should count as ''learned'', this counts down from a number to ''0''\n\n',
  `tags` varchar(250) COLLATE utf8_bin DEFAULT NULL COMMENT 'initially translated to the language of the user, these are copies of the tags the cards are created with, but the user can modify them\n',
  `difficulty` float DEFAULT '0',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`),
  KEY `user_id_idx` (`user_id`),
  KEY `card_id_idx` (`card_id`),
  CONSTRAINT `user_cards_card_id` FOREIGN KEY (`card_id`) REFERENCES `cards` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
  CONSTRAINT `user_cards_user_id` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=77 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='user cards are all cards in play for a user at this moment,  /* comment truncated */ /*they are in verification or learning mode or have been set to be ignored\n*/';
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `user_courses`
--

DROP TABLE IF EXISTS `user_courses`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `user_courses` (
  `user_id` int(10) unsigned NOT NULL,
  `course_id` int(10) unsigned NOT NULL,
  `score` int(11) NOT NULL DEFAULT '0',
  PRIMARY KEY (`user_id`,`course_id`),
  KEY `fk_user_courses_users1_idx` (`user_id`),
  KEY `fk_user_courses_courses1_idx` (`course_id`),
  CONSTRAINT `fk_user_courses_courses1` FOREIGN KEY (`course_id`) REFERENCES `courses` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
  CONSTRAINT `fk_user_courses_users1` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='a user_course is a course that a user has started on';
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `users`
--

DROP TABLE IF EXISTS `users`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `users` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `nick` varchar(45) COLLATE utf8_bin NOT NULL,
  `pwhash` varchar(250) COLLATE utf8_bin NOT NULL,
  `score` int(11) DEFAULT '0',
  `email` varchar(250) COLLATE utf8_bin NOT NULL,
  `newcardsperday` int(11) DEFAULT '20',
  `cardspersession` int(11) DEFAULT '25',
  `decay` int(11) DEFAULT '3',
  `language` char(2) COLLATE utf8_bin DEFAULT 'en',
  `initialinterval` int(11) DEFAULT '5',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`),
  UNIQUE KEY `nick_UNIQUE` (`nick`),
  UNIQUE KEY `email_UNIQUE` (`email`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='users are people that use the system\n';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;

/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;

-- Dump completed on 2014-05-27 10:52:39

SRS Algorithm

I’m going to do the actual SRS stuff based on ‘exponential back-off’. If you respond to a question immediately and get it right then you earn a 5x increase in schedule interval. If you get it right but after a longer pause then the scheduling interval remains unchanged, get it almost right (within 1 character of the right answer) (this only works for languages!!) the interval will be reduced by a factor of 5, get it completely wrong and you start all over again from the initial interval (set to 5 seconds for now).

Intervals below 5 seconds will be reset to 5 seconds.

So this makes the scheduled intervals 5, 25, 125, 625, 3125, 15625, 78125 seconds etc. So after answering right 7 times in a row you’re on a rough schedule of reviewing a card in another day. Of course people will use this whenever they feel like so we need to make sure that we don’t overwhelm them with a backlog of cards to review, and we need to present those cards that are most likely to be forgotten first.

To allow some personalization we can make the ‘factor’ (5 in this case) configurable on a per-user basis, maybe automatically adjust it for users that score particularly good (or bad) once we have enough data to figure out how well they retain information.

I don’t like the concept of ‘lessons’, so I’ll drop that entirely, you’re ‘learning’ a deck or you’re doing something else, you can learn as long as you want and you can ‘pause’ or ‘stop’ the process at any point in time.

I don’t want users to ‘grade themselves’, the program should do any and all grading completely automatic (this creates some very interesting challenges later on when audio support gets added). And I want people to be able to start learning without being logged in the first time they hit the site.

I’m keeping all this information nicely organized in a Leo Outline Editor file, Leo is absolutely fantastic and I wished there was a version of it that used a centralized docs store.

Command-line version

The simplest framework is of course none at all.

To make sure I understand all the business logic involved and have a working knowledge of the whole process I first built a command line version of the program playing off the same ‘magically populated’ database. That way I can very quickly test stuff without having to bother with layouts, RPC, servers, client server issues and other complications. An interactive command line version of this is about as simple as it gets (and can be built in a few hours if you have some experience).

So, that seems to work. And if it is ugly/buggy that’s fine, it did what it was supposed to do, bring out the flaws in my reasoning about the data model.

The Stack

If you’re reading this far you no doubt have an excellent understanding of web development, but just in case you don’t: The web is an ugly thing these days. What used to be a single binary serving up some static files from a directory on a server has turned into a whole slew of tecnologies, software, standards and options. Collectively, any set of specific choices from all those tools is called a ‘stack’. Typically, a stack will include an operating system, some sort of database for persistence, a back-end language (server side stuff), a front end language (typically JavaScript or something that compiles down to JavaScript), a web server (for instance, Apache, nginx) or ‘application server’, and some bits and pieces to glue it all together. The reason for this hodgepodge of loosely connected bits of tech is that we’re trying to shoehorn application interaction across a medium designed to serve static files on a fire-and-forget basis.

In my case, only the operating system (Linux) and the database (MySQL) are set in stone, the rest is variable. I do have an apache installation on this development machine so if any of the language/framework combos that we will be looking at can run under apache I’ll definitely try to get that to work since it stops me from having to learn how to configure yet another web server. I’m a lazy person.

Next up, the language/framework combos.

A Western Kid Living in Communist Poland

In this HN thread I commented on how the story about the People’s Republic of Donetsk reminded me of my days in communist Poland.

User bernardom prompted me to tell this story, so here it goes (I’ve slept on it and I don’t think it will do any harm, even though all the participants are alive (and well)).

I got my driving license in 1986. My dad and his wife were looking to buy a new car and were going to trade in their old Citroen 2CV. So I matched the offer they were getting on their trade-in and I got my first car. The 2CV (named after it’s fairly anemic engine with according to the French tax office ‘2 tax horse power’) is probably best described to people that have never been in one as an extermely cleverly designed tent on wheels. It’s about as spartan as a car can get, compared to the 2CV a Mini is luxurious.

I worked in Amsterdam west at the time and of course I would slowly expand my confidence driving to work every day and back. The car was so slow to accelerate that I did not feel comfortable on the highways so I stuck to city traffic and b-roads. This meant that longer trips were out of the question but given how crappy a driver I was in those days that’s probably a good idea. Two weeks later my good luck ran out. It was the first real frost that year, November 1986 (COBOL was going strong, I was an applications programmer for a dutch bank back then) and when I accelerated away from a traffic light just past central station in Amsterdam a Volvo in oncoming traffic skidded, carreened down a slope into one of those old style cast iron road dividers. Two metric tons of Swedish steel impacted the divider, which probably weighed no more than 80 kilos or so, accelerating it to projectile speeds.

What saved my life was probably playing a lot of pingpong. Reflexes took over and I ducked down just as the barrier swept through the car, without slowing it down much. The whole top of the car was gone, it was an instant convertible. I pretty cooly steered the car to the side of the road, started shaking like crazy and waited for the police to arrive, who asked ‘where the body was’. I realized I had been very lucky.

The wreckage was towed to my house in east Amsterdam and my mom broke down spontaneously when she saw the wreck. It was quite incredible that I walked away from that without a scratch.

But I learned a valuable lesson: not all accidents are your fault, and if you drive a tent on wheels you could very well end up as roadkill when someone else messes up. So I started looking for something a bit more sturdy. My mom had had a long time boyfriend, a man named Hans. Hans was super cool, an arts dealer in Amsterdam, fancy house on Herengracht and a super nice car, a Citroen DS.

If you’ve never seen one, prepare to be amazed, it was designed in 1955 and is by my standards still one of the most advanced cars ever to be produced. It also still looks good today:



I started hunting for one through the various DS specialist garages in Amsterdam (it is still quite a popular classic in NL today) and found one right around the corner, light blue with a white roof, built in 1973 and with ‘only’ a few hundred thousand kilometers on it. For 3750 dutch guilders it was mine, rust and hydraulic leaks and all.

Now, here was a real car! I had absolutely no problem taking it on the highway and so I decided to tour the highways near Amsterdam.

As I was about to enter the Coentunnel I noticed two pedestrians hitchhiking right at the mouth of the tunnel entrance. A very dangerous spot to put it mildly. Being my usual helpful self I stopped the car, bundled them into the backseat and drove them to the other side where I exlained to them that that was a very bad idea. The hitchhikers, two girls from Poland explained that they didn’t know much about the Netherlands, but that they were walking to a place called Egmond. From where I picked them up to Egmond was at least another day worth of walking so I offered to drive them there.

We talked a bit more, I figured out they were quite hungry and offered to get them some food. This they declined. Then I told them I was hungry and had to get some food. They agreed to come along and we stopped in Alkmaar to eat. Maybe they would eat a bit after all. After both had polished off double helpings of everything we continued on our trek to Egmond.

On arrival in Egmond we said goodbye, I gave them my phone number in Amsterdam in case they got themselves in trouble and went back home. About a week later they called me. Could they please come to Amsterdam? In Egmond they’d been pretty much enslaved by a flower bulb grower, who made them work day and deep into the night for very little money, and who would then charge them for everything (food, lodging) to get it almost all back. Modern slavery (yes, we Dutch are nice people, and if you think that such things are a thing of the past now then think again, this sort of thing still happens all the time). I went back there, picked them up and they moved into my house in Amsterdam.

Calling it a house is a bit much. Today you’d probably call it a hacker space. It was a 60 square meter partition of an industrial building. No shower, just a small sink, an electrical stove and tons of computing gear, a pingpong table, a couch and a single mattrass on the floor.

I gave them the bed, made mine on the couch and for the next week or so we talked day in day out about what life was like in Poland and what it was like in the Netherlands.

When they left they gave me their addresses in Poland and invited me over.

For months that invitation burned a hole in my pocket. I was totally enamoured with one of the girls, and made up my mind to go and visit Poland. This was a lot easier said than done.

I got my first taste of what Poland would be like when I entered the Polish embassy in the Hague. A line of about 50 people. Before I got to the head of the line the office closed and I had to go back another time. This time I took care to arrive early. I applied for a visa and filled in a stack of forms. For months my visa application was held up. More and more information was required. What did my grandparents do during the war? How much money did I have? Did I have family? Sisters, brothers, everything! At some point it occured to me they knew more about me than my mom did.

But finally, after months of filing ever more papers I got the call, I could come and give them my passport. Then they would stamp it after the office had closed and then a few days after that I could go and pick it up again.

And so it happened. I rolled into Poland somewhere in the early hours of morning August 1987. Lines of tens of people in front of non-descript shops had already formed. People waiting for the bakeries to open to get bread. Be early and you might have a chance. Initially I did not comprehend what the lines were for, my mind had been totally conditioned to ‘want something, go out and buy it’. The whole idea of shortage was something that I was not familiar with. At that time, The solidarity movement was clashing headlong with the Polish communist government and there was quite a bit of tension in the air. I was received warmly in Poznan, spent about a month there and then went back to the Netherlands. What struck me most about Poland is how serious the people my age there were. And how tremendously strong the friendships. These were in many ways the most interesting times you could have possibly picked to be behind the iron curtain, I got to observe the whole underground movement from the point of view of a spoiled western kid, retrospectively one of the formative moments in my life. Solidarity was quite something, the streets were abuzz with it. The Polish secret police (the UB or MBP) was quite active and if you didn’t know someone personally you could not trust them. This caused the Polish people to form tremendously strong friendship bonds, a thing that is probably a lot less strong today in absense of such formidable enemies. The police in Poland was tremendously corrupt at the time, a nice example of power out of control.

For months my newfound girlfriend and me corresponded by good old snail mail. We got married, lived together in Amsterdam for some years and then moved to Poznan, Poland together.

This created a problem because I could not legally work in Poland. So I applied for a work permit, and this is one of the most bizarre experiences I’ve ever had. It wasn’t so much that such permits did not exist, it was that applications were so rare that almost nobody knew how the whole process worked. It took many months, ever more interviews (my Polish at that time being next to non existant), and lots and lots of forms. Ever more forms. The most frequent question was why on earth I would want to live in Poland when everybody that could was trying to get out.

Finally I got it, after I had all but given up. I still have it in Amsterdam in a suitcase somewhere, I should probably scan it and put it in here when I’m back there. Somewhere along the way a Texaco gas station opened in the middle of Poznan.

Imagine a background of gray highrises, no colour anywhere, all the cars gray and the ancient diesels of buses belching smoke. And in the middle of that an island of bright red, a tiny patch of the west invading on all that gray, a harbinger of what was to come. A friend of mine said this: “If the future is going to be so full of bright colours I don’t know if I can stand it.”.

Right now I’m living in Romania (Bucharest to be precise) and it’s funny how I can trace back almost everything major that happened in my life to that point where I stopped the car.

So, that’s the story of the Hitchhikers, the flowerbulbs and the ancient Citroen.

Thanks G. for changing my life, for the better in all respects.

edit: and thanks I. for correcting my broken memory!

Ripe for Disruption

Disruption is something that you can usually only establish after the fact. A decade later or so you can look back and say ‘x’ disrupted ‘y’. Microsoft did it to IBM.

Disruption usually plays on a larger scale, for instance, the semi-conductor industry all but destroyed the vacuum tube industry (for certain purposes though vacuum tubes had hard-to-beat advantages potentially good enough for a niche revival).

On the web disruption takes place at a smaller scale. Craigslist disrupted classifieds, Facebook disrupted myspace, Google did it to Altavista and DuckDuckGo is doing it to Google (well, they’re dreaming about it anyway and I wish them luck).

So who is currently setting themselves up for disruption? The hardest part in taking on an incumbent is usually that the incumbent has had years to entrench themselves and that they may have a very solid network effect in place making it all but impossible to take them on.

The thing that can level the playing field and that can give you momentarily a window of opportunity to crawl through and get a chance when normally you would not even be able to dent the armour is a crisis. Every big company has moments of weakness and that’s when they are at their most vulnerable, like over-ripe plums waiting to fall from the tree. In part this stems from their internal sense of being invulnerable, they get sloppy and they have blind spots. So let’s analyze a bunch of the top contenders and point at their weaknesses:

  • Google Adsense. Google makes a ton of money off their ads, so much that they don’t give a damn about the publishers giving them the inventory. More and more publishers are completely fed up with this and would jump ship instantly if a viable alternative appeared. Beware, right from day 1 you’ll be the target of just about every shyster on the planet with a plan of action so ‘know your customer’ is going to be key. And that’s good because that is exactly where google fails.

  • Ebay. This blog post was quite prescient (I wrote it a while ago), but the events of the last week underline the complacency angle. Not only does ebay look terrible, it has a huge problem with scammers and now, on top of that it turns out that it was very lax with security. That’s one of those crisis moments that just might be bad enough to topple a giant.

  • LinkedIn. Linked in is safely entrenched in the business world. Or so they think. But I’m not on there and with me 100’s of thousands of others. I’m not on there because I don’t see the value of LinkedIn, I’m not on there because they are playing fast and loose with data and they spam you to death. If I’m going to be ‘the product’ at least be gracious about it. Linked in seems to believe they are too big to fail, and that their users can no longer do without them and that’s a vulnerability.

  • Flickr. Once the only really good solution for photo sharing in town (a part of that function has now been taken over by Facebook) Flickr is asking for some kick-ass competition. The site has not really thrived under Yahoo! stewardship (what ever did?) and has been steadily dropping in the rankings and in the favour with the audience.

  • Quora, stackoverflow etc. The one suffers from appropriation syndrome, the rest from inappropriatitis. Both leave me frustrated, on Quora half the content (donated freely by users) is now used to draw in visitors through search engines and then used to force them to log in to the service. If a log-in is not required, don’t force your users to do it. And on stackoverflow anything (ok, exaggaration here) that can’t be answered with a simple fact will get closed. Too many times did I have the exact same question as some former stackoverflow visitor, dropped the question into google only to land on a completely useless page. That’s an opportunity.

So, if you’re looking for targets, the web is full of them, go forth and disrupt, business opportunities galore.

Choosing a Web Framework/Language Combo for the Next Decade

Learning Romanian is rapidly turning into a huge exercise in Yak-Shaving.

You often see the question “What stack should I use to develop my application on?” posed but rarely do you get any insight in the actual process once someone decides to answer it specifically, so both as an analogy to rubber-ducking this and showing you what went on while I made my choice I’ve decided to document the whole process. Even though you might not like or agree with the conclusion (which is to a very large extent personal) you may still get some value from the method or the information presented here.

Once I decided to have a go at building my own srs/flashcard web app the question arose to figure out what to build it in. I’ve been putting of this choice for quite a while now, but it really is time I bit the bullet and updated my toolbox and got with the times. There is a lot of general wisdom floating around about this theme (stick to what you know, don’t do your own plumbing (that translates into don’t build your own framework), concentrate on the problem you are trying to solve (yeah, right, learn Romanian), old = good and so on).

So technically speaking this should be a non-choice for me (LAMP + Yii). After all ‘stick to what you know’ is by far the best choice you could make here. And it allows me to re-use my library of home-grown code, which is quite an investment in time and effort. So I had more or less settled on that. But after messing around for a day trying to get eclipse to support Yii (very frustrating, to put it mildly) and the fact that so many of the Yii bits and pieces (such as the Yii eclipse plug-in) are depending on single individuals it is getting harder and harder to defend Yii as a viable choice at this point in time. Yii 2.0 is now in ‘beta’ for however long that will take, and though they write on their website that you definitely should not stop using Yii 1.1 for your new projects at the same time they write that Yii may or may not be supported past 2015. I don’t like gambling that much. And I remember how the Osborne Computer Corporation performed Seppuku on itself.

The good news I guess is that the web is designed around protocols, not pieces of software so that’s all fixable but to require a re-write every few years is really not an option, I simply do not have the time for that. Building things 3 times (MVP, first real version, good enough for long term use) is hard enough, a forced rewrite on someone else’s timetable sounds like a very bad idea.

Let me explain that, I tend to build things 3 times, the first is sort of an exploration of the space, I fiddle around with usually just a bit of data and an extremely rudimentary UI to figure out what it is that I really want to build. Then when that’s sort of crystalized out I build the second version, one that lives for a while, days, weeks or months. This usually has actual use(rs) and maps out the space even further. Inevitably this turns up some really bad decisions made in the first or the second phase that were not apparent at the time. And so after a relatively short time the whole thing gets ripped apart again and then I build it in a way that it can last for a decade or longer while keeping the exterior unchanged as much as possible. It’s surprising how much of the stuff I built in my nearly 3 decades of programming is still in active use today. Even more surprising that the things that are still in use all felt like they were temporary at best, projected lifespan a few year maximum. In one extreme case of this a prototype I built for a company in 1986 is still in production use today. So deciding what stack to build something on is a very important decision.

Choosing not to go with the most familiar previously used combo is against just about all the good advice, but I like learning and this is as good an excuse as any I’ll get to jump out of the box.

My list of requirements is:

  • L-M-, to clarify, Linux and MySQL are non-negotiable (because I exclusively use linux for everything I do and I have a ton of data already stored in MySQL tables) This also limits the size of the Yak to some extent and given the direction this is going that is probably a good thing.
  • The software should be actively maintained
  • The license should be an open source license that is well known or one that is so simple that I can understand it.
  • Issues should preferably be solvable using google searches and stackoverflow, I don’t like making tickets so community size and having others walk the path before is important
  • The framework/language combo should play nice with other tools (editors, revision control systems and so on)
  • It can be ‘opinionated’ but not so opinionated that I have to migrate databases to new schemas to make the framework happy
  • It doesn’t have to be the fastest but it should play nice with caching to speed it up should the need arise. The reason for this requirement is that several of my little projects are now in the 50K+ and 100K+ per day visitor counts and if you eat up a lot of cpu per pageview then you’re going to spend a ton on hosting. Not all my projects are for profit so this really matters.
  • It should be possible to find collaborators at some point (another community size factor), in my experience success of my toy projects is extremely hard to predict and I’ve been caught several times now with something doing a take-off all by itself.
  • Reasonably low bar to entry (quick to get up to speed)
  • Right now this is a one-man-show so programmer productivity is important.
  • The interface with the framework should feel natural and idiomatic to the language underlying it. In my experience, if that is not the case it will sooner or later cause trouble.
  • Actively maintained does not translate into ‘was made yesterday’.
  • The team of maintainers of the language and framework should have more than 1 member.
  • The language underlying the framework should be solid, can’t be classed as experimental and should not be a moving target.
  • Sessions, authentication, routing, internationalization, MVC, caching, mobile, templating, ORM and so on should preferably all be part of the deal, so batteries included please.

Some of these requirements are no doubt contradictory and there will have to be compromise, for instance excellent programmer productivity, low bar to entry and good execution speed are rarely found together.

In other words, time for some web browsing, apply the above criteria, pick a language/framework combo and then get to work. Sounds easy, right?

This opens up the field quite a bit. Stick with PHP? (still plenty of choice in terms of frameworks) Move to a new language? (though the wisdom holds that using a new language for a project that you need or care about will make you hate both). Doing technical due-diligence exposes me to all sorts of technology, and it also helps me in making this choice because I’ve had the privilege of getting a ring-side seat in the arena of capable web start-ups and established companies. You get a ton of examples of how things could be solved and what really works (and sometimes, what really does not).

The amount of choice available to someone that is going to make this decision is absolutely incredible. This wikipedia table lists a staggering number of potentials to choose from, and on a first reading it isn’t even complete (for instance, the Go language is missing, are a whole bunch of frameworks (not just Go ones, for instance, Erlang’s Chicago Boss isn’t on that page). If you’re maintainer of some framework or care about your language ecosystem please update that page!).

On the language front there are: asp.net C, C#, C++, Clojure, coldfusion, Erlang, Elixir, F#, Go, Haskell, Java, JavaScript, Lua, Scala, Pascal, Perl, PHP, Python, Ruby, Rust, SmallTalk and Tcl. (there even is a web framework written in COBOL and even though at one point in my long career this was how I made my living nostalgia is not enough to overcome the resistance to putting that language on the initial list. Basic, Fortran, Algol etc omitted for similar reasons.

What a beautiful illustration of The Paradox of Choice!

Because the list is simply way too long to start comparing the frameworks and then letting the framework dictate the language I’m going to turn things upside down for a while, and use the languages as an indirect way to get rid of a ton of framework selection work, then when the list is small enough we’ll reverse this again and look at the frameworks left over.

Language Selection

Of those languages listed above I’m very familiar with C (old!) and PHP (ugly!), I have a working knowledge of Java (but really don’t like it), I’m interested in the Erlang ecosystem (beam and erlang/elixir), do not like Perl, do not like the Ruby ecosystem, I think Pascal has had its day, Go is interesting because it is close to C and has support from a huge company, Python has never really felt like the right choice for something that may one day need to be supported, clojure looks nice but has a huge learning curve and I got totally stumped by the errors it throws up in a toy project I built with it, SmallTalk due to its image based nature has a hard time playing nice with commonly accepted toolchains and has quite a learning curve, Tcl is not really modern web suitable (though it’s been done), Scala is unfamiliar, Lua looks interesting, Rust is too young, JavaScript feels like everything built in it is extremely temporary (but it has the advantage that you can decide to work in only one language) and Haskell seems both for people much smarter than me and totally unfamiliar.

Since this is now a process of elimination, let’s come up with some (hopefully) good criteria to winnow down the language set to something a bit more workable.

Let’s start by getting rid of the languages that are in my opinion unsuitable for modern day web development. Goodbye C, Perl, Pascal, SmallTalk and Tcl. Smalltalk also does not conform to the requirement of playing nice with others and being able to use version control tools outside of the image.

Next, let’s get rid of the languages that are too new:

That removes Rust, even though in the long term I have high hopes for the language.

Haskell gets removed because I’m not in with the smart crowd (call me a blub programmer and see if I care) and because I’m not writing some financial application (which is, apparently where Haskell really shines).

Scala gets dropped because I don’t really know it but think it is really terribly ugly. This could be another instance of me not being smart enough or not getting it but I just can’t seem to get past that.

I still haven’t forgiven Microsoft for their countless misdeeds so Asp/C#/F# are all right out. That may be shooting myself in the foot here (cut off my nose to spite my face), but contributing to this decision is that I work exclusively with linux (both on desktops, laptops and servers) so the MS ecosystem has become totally unfamiliar to me and I don’t like running windows on servers (though, of course there are some workarounds possible here and Mono is quite good). So, the L in my LAMP is firmly established. And besides, I don’t have a lawyer on the payroll to make sure that my licenses are all read, ok’d and acceptable.

I’ve once maintained a coldfusion application for a client and I still wake up in cold sweat every now and then, that may have been a very bad example but the taste has not left me so strike coldfusion.

JavaScript web frameworks seem to have a half-life of about 6 months, I really don’t feel that that is the right attitude to software development. Twitter has rebuilt most of their infrastructure in JavaScript, that’s a pretty bold move and a huge vote for JavaScript. But I don’t exactly run twitter, I’m a one-man-show, do not plan on hiring an army of programmers and have always thought of JavaScript as a terrible hack that has now somehow managed to propagate itself backwards across the client/server link to infect the server environment. I just can’t get behind it, so JavaScript goes.

So, now we’re down to 9 candidates, still way too many but at least there is some breathing room. The eight are: Erlang, Elixir, Clojure, Go, Java, Lua, PHP, Python, Ruby.

I’m going to eliminate Clojure on the grounds of it being a super nice language but I really hate the runtime error messages (see above) and I see it used far more in data-analysis roles than in web sites. In a way this is a pity because Clojure has all the good stuff of Java (tons of libraries, compiled, reasonably fast). Now we’re down to 8.

Lua… lua is a very interesting little language, it’s elegant, I really like the concepts behind it, I’m aware of some extremely impressive applications built with it. Think of lua as hitting the sweet spot between Python and Ruby, with a good link to C thrown in for good measure. The grounds for which I’m dropping it are that lua does not feel like it is a stand-alone environment, I always think of it as a plug-in to something else to give it web capability or to make it configurable/extensible. The lack of libraries is also a huge drawback.

Python has always been a mixed bag for me. What I wrote in it worked, when I wrote it. Then, a little while later it stops working, either because some module that it depends on gets deprecated or changed in an incompatible way, starts to exhibit weird behavior or even disappears. I’ve written some thousands of lines of python altogether so I would definitely not consider myself fluent in the language. But contrasted with C code that I wrote 30 years ago that I can compile today this is not a good personal record (though it may very well have to do with my picks of python modules). The significant whitespace has always felt to me like driving on the autobahn at 200 km/hour without guardrails and every time I try using it for web development I end up re-writing it in something more suited to the task. For scientific stuff it has replaced Perl in a large number of use cases and it will be here for a very long time. So Python is out for me, but this is really becoming personal and less objective. Every web project I ever built in python also felt slow to use.

With Python eliminated we are now down to 6 entries in the language category, but Erlang and Elixer should probably be combined on account of their close relationship. (A similar thing could be said for clojure and Java but I think that clojure’s integration into the JVM is less elegant than Elixir is when looking at Beam).

Ecosystem size is a huge factor when evaluating a language/framework combo, and if the ecosystem around a language is small that means that for the most part you’re going to be on your own figuring things out and that if you ever want to hire someone (a contractor, an employee) to help out with some aspect of your project that it will probably be difficult (this would have been another strike against Clojure, come to think of it). Erlang & Elixir definitely have this problem, but I can’t help but be totally fascinated by the ease with which applications can be clustered and made ultra reliable. For instance, ‘whatsapp’ is built in Erlang. So erlang sounds like a worthwhile skill to have under my belt and the eternal draw of technology for its own sake is really making it hard to keep my eye on the ball here (which is, to learn Romanian, in case you forgot). So, with deep regret, pain in my heart and apologies to Joe Armstrong, whose thesis (or should I say Magnum Opus) is a must read for any programmer I’m going to have to drop Erlang, and by strong association, Elixir as well.

Java, PHP and Ruby are all mature and stable languages, Go is new but has some very big names behind it, people that I know are capable of building for the long haul. Sites like langpop don’t even have it on their radar. Go does concurrency very well (always a handy feature), compiles wicked fast and allows me to re-use at least a little bit of my C knowledge. So I’ll keep it in, even though it is only 4 1/2 year old and still extremely small in footprint (joke intended) when compared to Java, PHP or Ruby.

The short-list of languages is now 4 entries long: Go, Java, PHP, Ruby. Two of those (Go, Java) are compiled, and two are interpreted (PHP, Ruby), though, you could compile PHP using facebooks HHVM (HipHop Virtual Machine). I think that’s short enough that we can stop eliminating languages for good reasons and start to expand the list to include frameworks to get an idea of what each language has to offer that speaks for it, rather than to get rid of it, reversing us back to where we began, selecting a framework and let that dictate the language.

Framework Selection

Infant mortality amongst frameworks and languages is ridiculously high so going with a ‘young’ framework or language is a significant risk and can turn out extremely expensive in the long run (for instance, after the framework ends up being orphaned you have to make sure your stuff remains afloat so you need to support/extend the framework and you have a serious risk of security issues going unnoticed for a longer period of time). If I’m going to place a long term bet on some chunk of code I would really like to at least have the feeling that it will be around for a while. This will make life extremely hard for Go to stay in the race, after all it is young itself so all frameworks are likely going to be young as well.

Let’s make a listing of what we’re let with at this stage (ripped from the wikipedia page):

Java:

Project Current stable version Release date License
Apache Click 2.3.0 2011-03-27 Apache Software License 2.0 (ASL 2.0)
Apache OFBiz 12.04.02 2013-07-30 Apache Software License 2.0 (ASL 2.0)
Apache Shale 1.0.4 (Retired) 2007-12-19 Apache
Apache Sling 6 2011-04-18 Apache 2.0
Apache Struts 2 2.3.16 2013-12-08 Apache 2.0
Apache Tapestry 5.3.7 2013-04-24 Apache
Apache Wicket 6.13.0 2014-01-14 Apache 2.0
AppFuse 2.1 2011-04-04 Apache
Eclipse RAP 2.0 2013-02-11 Eclipse
FormEngine 2.0.1 2011-05-08 Proprietary
Grails 2.3.5 2014-01-16 Apache
Google Web Toolkit 2.6.0 2014-01-30 Apache 2.0
Hamlets 1.7 2011-03-11 BSD
ItsNat 1.2 2011-05-24 LGPL, proprietary
JavaServer Faces (Mojarra) 2.2.6 2014-03-04 CDDL, GPL 2, Apache 2.0,
JBoss Seam 3.1.0 final 2012-01-13 LGPL
Jspx-bay 1.2 2013-02-14 Apache 2.0
JVx 1.1 2013-01-23 Apache 2.0
OpenLaszlo 4.9.0 2010-10-21 CPL
OpenXava 4.7 2013-04-02 LGPL
Oracle ADF 12.1.2.0 2013-07-11 Oracle Technology Network Developer License
Play! 2.2.3 2014-05-01 Apache 2.0
RIFE 1.6.1 2007-07-14 CDDL, LGPL
Spring 4.0.3 2014-03-27 SpringSource
Stripes 1.5.7 2012-05-17 Apache
ThinWire 1.2 2007-09-17 GPL
Vaadin 7.1.9 2013-12-04 Apache 2.0
VRaptor 4.0.0.Final 2014-04-23 Apache 2.0
Wavemaker 6.5.3 2013-03-14 Apache
WebObjects 5.4.3 2008-09-15 Proprietary
WebWork 2.2.6 2007-07-21 Apache
ZK 6.5.2 2013-03-26 LGPL, ZOL
ztemplates 2.4.0 2011-09-11 Apache
4WS.Platform 2.1.6 2014-05-07 LGPL

PHP:

Project Start date Current stable version Release date License
Agavi 2005-05 1.0.7 2011-12-22 LGPL
Aiki Framework 2009-09 0.9.1 2012-02-23 GPLv3
AppFlower 2009-02 1.3 2012-10-09 GPLv3
Ayoola Framework 2011-01 1.4.1 2014-01-12 Proprietary
CakePHP 2005-08 2.5.0 2014-05-13 MIT
Cgiapp 2004-12 1.0 2004-12-01 LGPL
CodeIgniter 2006-02-28 2.1.4 2013-07-08 OSLv3
Drupal 2001-01-15 7.28 2014-05-08 GPLv2
Fat-Free 2009-09 3.1.0 2013-08-19 GPLv3
FuelPHP 2011-08 1.7.1 2013-12-01 MIT
Hazaar MVC 2012-10 1.1 2013-04-19 Apache 2.0
Joomla 2005-08-17 2.5.19 2014-03-06 GPLv2
Kajona 2006 4.2 2013-07-16 LGPLv2
Laravel 2011-06-11 4.1 2013-12-11 MIT
Logiks 2011-06-24 3.6.1 2014-05-02 MIT
Lithium 2009-10 0.11 2012-10-03 BSD
Nette Framework 2006-01 2.1.2 2014-03-17 New BSD, GPLv2, GPLv3
PHPixie 2012-07 2.1 2012-04-24 BSD
PRADO 2004-01 3.2.2 2013-07-20 New BSD
Qcodo 2005-10 0.4.22 2011-08-15 MIT
SilverStripe 2005-11 3.0.5 2013-02-20 BSD
Seagull 2003-10 1.0.4 2013-01-04 BSD
Solodev CMS 2005-11 8.0 2006-11-12 Proprietary
Symfony 2005-10 2.4.4 2014-04-27 MIT
TYPO3 Flow 2011-10 2.0.0 2013-07-12 LGPLv3
Xyster Framework 2007-09 02 Build 01 2010-10-18 BSD
Yii 2008-01 1.1.14 2013-08-11 New BSD
Zend Framework 2006-03 2.2.5 2013-10-31 New BSD

Ruby:

Project Current stable version Release date License
Camping 2.1 2010-08-20 MIT
Ruby on Rails 4.1.1 2014-05-06 MIT, Ruby
Ramaze 2012.12.08 2012-12-08 Ruby
Sinatra 1.4.4 2013-10-21 MIT
Merb 1.1.3 2010-07-10 MIT
PureMVC 2.0.4 2008-08-14 CC+Attribution

Go:

Project Current stable version first release License
Martini ? 2013-10-30 MIT
Negroni ? 2014-05-19 MIT
gocraft ? 2013-11-10 MIT
beego 1.2.0 2012-03-05 Apache 2.0
Gorilla ? 2012-10-03 Copyright (c) 2012 Rodrigo Moraes. All rights reserved.
Revel 0.9.1 2011-12-09 MIT
TigerTonic 2013-02-10 Copyright 2013 Richard Crowley. All rights reserved.
Go Rest 0.1 2011-08-04 New BSD License
Web MIT 2009-12-09 New BSD License

That make for 75(!!) frameworks in total for the four languages still left. That’s still a bit much to do say a quick and dirty test application in all of them to see how they feel.

More winnowing…

We can use similar criteria used above to get rid of a number of frameworks, first, let’s drop everything that has a proprietary license. I really don’t feel like expending a bunch of effort and then having the rug pulled out from under me because of some obscure passage in a license that I did not evaluate properly. So either it is open source under a well-known license that suits my needs or I’m going to have to pass on that particular framework.

This gets rid of Eclipse RAP, Formengine, Oracle ADF, Spring and Webobjects in the Java arena. For PHP we lose Ayoola, and Solodev CMS, for Ruby and Go we lose none.

7 down, 68 to go.

Next up, have I even heard of the framework? That’s quite subjective, especially for Go, but it’s a good criterium. If a project is so obscure that someone looking into as many companies as I do and a HN junkie to boot has not heard of a framework that’s reasonable grounds for dismissal. For Java we lose Apache Click, Apache OFBiz, Apache Shale, Hamlets, JavaServer Faces, Jspx-bay, JVx, OpenXava, Stripes (vaguely rings a bell somewhere though), Vaadin, ztemplates and 4WS.platform. For PHP that makes short work of Hazaar, Kajona, Logiks, Qcodo, Typo 3 Flow,

In the Ruby camp this loses us Camping and Ramaze. (Technically, I should probably delete all the frameworks in the Ruby category except for Rails, but I have heard of Sinatra, Merb and PureMVC, I just don’t remember if that was in a good or a bad context, which is anecdotal proof that there is no such thing as bad publicity). For Go this means the end of TigerTonic, it isn’t a framework anyway (more of a router) and it could be combined with another framework to do high-speed routing (which it does using a trie).

Another reason to discard frameworks is because they are stale, there is a very small chance that a framework is so perfect that it does not need maintenance for years on end but by my reading a framework that is not worked on continuously is a dead framework. This is because the web is a moving target and frameworks should keep that target in their sight at all times.

For Java this axes OpenLaszlo, RIFE, ThinWire, WebWork (now merged into struts2). For PHP we register the demise of Agavi, Aiki Framework, AppFlower, Cgiapp, PhPixie (or maybe they just ignore those github issues), PRADO (superceded by Yii), Seagull and Xyster. In the Ruby domain Merb has apparently been merged into Rails, and PureMVC looks quite dead. For GoLang it was a sad week because Martini bit the dust after some pretty well aimed criticism, to be replaced by negroni.

That was quite the slaughter, down to 35 frameworks now if I’ve counted that all correctly.

Since the start of this journey was PHP/Yii I’m really tempted to throw out Yii too, I really strongly dislike it when important chunks of code like frameworks are given vague future prospects, a framework is either eol’d or it should be in active development. Saying you may or may not sunset a framework in a years time and in the meantime to encourage users to build new software on top is very irresponsible. I have similar misgivings about Drupal, where a bi-annual re-write is pretty much the only way to stay current. So out of sheer spite and to get me out of my comfort zone I’m going to drop both Yii and Drupal.

The whole Martini/Negroni thing has me in a quandary. On the one hand, I think the response to stop developing the code and to move on to a more ‘proper’ approach is a very mature one. At the same time I’m so terribly happy that I’m not one of the people that started to develop some major chunk of code on top of Martini about 3 months ago or so. I would feel very much let down. The fact that a whole project gets axed by one well aimed blog-post is - to me - proof that one should be very distrustful of anything that has not yet withstood the test of time. Just like ‘too old’ and ‘stale’ are valid criteria for rejection, the same holds true for ‘too new’. Who is to day what will happen to Negroni 6 months from now, when another blog post by a competitor aims to derail it? (The fact that that particular blogpost aimed to also promote the product of the author was less than classy). So I’m dropping Negroni from the ‘Go’ set of frameworks on account of it being too fresh and the maintainer giving me the impression of being quite unpredictable.

Me not having heard of your framework is one thing, but google not really knowing about it is another red flag. Let’s do a little notoriety check and see what turns up:

Java
Apache Sling44,000 results
Apache struts2170,000
Apache Tapestry18,800
Apache Wicket95,000
AppFuse 6470,
Grails 33,000,
GWT 282,000,
JBoss 27,000,
PLAY 993,000(!),
Vraptor 3980,
wavemaker 3200,
ZK 57,200.
PHP
CakePHP 98,000,
CodeIgniter 285,000,
Kohana 113,000 (not in wp, eol’d!)
Fat-Free 2,580,000 (wrong?)
FuelPHP 8110,
Joomla 513,000,
Laravel 136,000,
Lithium 6190,
Nette 354,000
Silverstripe, 25,500
Symfony 281,000
Zend 2,280,000(!!!)
Ruby
Rails 258,000
Sinatra 5160
GoLang
Gocraft 3820
Beego 1370
Gorilla 695
Revel 4650
Go Rest 6580

Web is impossible to google for in a normal way.

You can very clearly get an idea of the relative sizes, maturity and adoption of the various frameworks and language eco systems here. (I did not expect such a huge spread and it’s a good indication that I should not go by my gut on things like this, for instance, I would have estimated Sinatra much larger and never knew Play! was that big.)

Based on the relative sizes of the frameworks within each language category I’m going to retire the ones that are substantially smaller than the mid-range. This removes Tapestry, ApFuse, JBoss, Vraptor, wavemaker, FuelPHP, Lithium, Silverstripe, Sinatra and Gorilla, and leaves us with 21 frameworks.

Web frameworks are not all created equal and not all of them are created by teams. Each of them has its own target audience, is more or less feature rich and typically outcompete each other on some aspect but have a large chunk of overlap when it comes to the basics. Those basics had better be there though. For me, that translates into: authentication, routing, session management, multi-language, ajax, MVC, persistence, templating, input validation and testing. If those are not present in a framework then that means I’m going to have to roll my own for that aspect of the framework and in the long term I’d rather maintain the app than a chunk of framework (that’s the whole reason for choosing a framework to begin with!). I dropped my own one-man-show framework years ago because it was getting longer and longer in the tooth, I see no reason to go back, even partially to that situation if I don’t have to. Caching is optional, I assume that the makers of a framework are dedicated to keeping things secure. I also like it when frameworks have a team behind them rather than a single person as the main driver.

On account of not having testing, internationalization and an ORM I’m dropping Sling (though likely you could use ActiveRecord), Wicket does not do MVC so it goes, Grails is not written in Java, CodeIgniter has no ORM and no internationalization, Joomla is more of a CMS than a framework, Laravel has a ‘bus-factor’ suspiciously close to 1. The ‘Go’ frameworks are not documented in Wikipedia so this is a bit more work to figure out. GoRest appears to be mostly a router rather than a complete framework, GoCraft ditto (it’s a router ‘web’ and a simple orm ‘dbr’). Hoisie/web is another router. The Go world seems to be taking an extremly modular approach to the framework problem, in a way I like that (modularity is good), but it also means that you have to do a lot of hunting and picking to get a set that you like and it is extremely likely that you are then one of the few using that exact combination. It also signficantly increases the chances of one of those elements being suddenly end-of-lifed (the same thing happens in the Drupal world all the time, with every major release there are tons of orphaned modules).

12 frameworks left, and still, the variety is enormous. I think the next thing to do is to write a very small sample application in all of those (setting them up will be a total pain) to see how well it all comes together and what the friction points are. To make sure I end up with a good choice I have to at a minimum check that all my requirements listed above are fulfilled, and enough of them have to do with actually working in the environment that I won’t be able to check without getting my hands dirty.

The frameworks that are left now are Ruby on Rails, for php CakePHP, Fat-Free, Nette, Symfony and Zend, for Java Struts2, GWT, Play! ZK, for GoLang beego and Revel.

To be continued. Interesting reading along these lines on other people’s pages:

Edit: in the HN thread about this article user nemasu points out that Kohana is no longer actively supported so I will remove it from the list of candidates, also I incorrectly axed C# as being a Microsoft only product, but I feel that even though there are re-implementations of C# the language is under Microsoft control (just like Go is under Google’s control, even if someone else wrote another Go compiler). If and when Microsoft is no longer the dominant factor in the direction of C# would that situation change sufficiently for me to make C# an option for development.

Learning Romanian

I currently find myself living in Romania for a bit. One thing that bothers me is that if I spend time in a place where I don’t speak the language and make the locals speak English to me all the time (provided they can!) is that it feels like I’m rude and not making an effort to meet them halfway. Why should (many) be the ones to adapt rather than me (one)?

So I make it a point to make a strong attempt to learn the language of the place where I currently live. I’ve landed a short-term contract (which hopefully will not interfere with all my other plans too much) and currently live in an apartment in Bucharest. Bucharest, Romania is a place where younger people will all speak English in various degrees of fluency, but older people and most people you will interact with in stores and other day-to-day situations do not. So there is not just a sense of balance involved in this but also a simple need. In order to get through a normal day it would very much help if I spoke the language.

Romanian is best described as modern day Latin with a number of Slavic influences. It has all the complexities of Latin as well (conjugations, declensions) and has numerous exceptions to the regular rules of the language as well as a whole bunch of extensions to the normal Latin alphabet indicated using diacriticals. Romanian is hard. It’s certainly not the hardest thing I’ve learned but it is harder than I expected it to be (normally I start picking up the basics of a language after a few weeks). I’ve been here for two months now and am not making the progress that I would like to make so somewhere last week (after delivering the first part of the job to the customer, a program that spots spammers before they can become a nuisance to a community, tons of machine learning and interesting tricks in order to output a single bit of data for every new signup) I decided to get serious about this language business.

After reading this very fortunately timed thread on HackerNews I found out about MemRise (I first tried ‘Anki’ but it wouldn’t work for some reason and I gave up after getting stuck in dll hell and versioning conflicts for longer than I have patience for).

Memrise is an instance of a technique called Spaced Repetition Software. What that does is to set you up with a set of flashcards that are presented at precisely timed intervals and various ways of responding to input the expected answer. Optionally flash-cards are accompanied by audio files or pictures that you can upload yourself as memory hints. The whole thing has been gamified to the max (the MemRise people were coached by some guy that was responsible for the farmville cancer), though the leaderboard has been shut down because there was too much cheating (who on earth would cheat to fake learning a language just to be top of a leaderboard is a mystery to me). MemRise is very slick and I took to it with total abandon. They use a ‘gardening’ metaphor where you first ‘plant’ a word or sentence and then you ‘water’ it in subsequent reviews. It works very nice in the first few days.

But after completing the first ‘course’ for Romanian (user contributed content) I noticed a few things. For one, the courses are supplied by the end-users and they do not appear to be ‘natives’ to me. Quite a few mistakes in the deck marred the experience and caused me to learn the wrong translations very rigorously. This is a real problem. The second issue with MemRise floated to the top as soon as I added a second course (deck). There was a lot of overlap with the first and yet MemRise wasn’t clever enough to realize I’d already learned a lot of those words in the previous deck (you can solve this using the ‘ignore’ option but that’s a ton of manual work that could easily be automated). Another problem that surfaced is that two decks do not necessarily agree on what a word means. Maybe both are right, maybe one of the two is wrong. Regardless, at a minimum it will cause you to not know what you’re supposed to answer because invariably the answer that will float to the top will be the one from the other deck (because that’s the one you saw last).

Another issue is that the MemRise courses vary greatly in usefullness and quality but the courses are not graded in any way, nor is there any indication of whether or not they have been made by someone fluent in the target language. There is also no way to flip or reverse the deck. Decks (courses) may contain (lots) of words of questionable usability and of course you’ld like to know the most useful words first. (why would I have to learn ‘sobolanul’ (the rat in Romanian) before learning a useful word like ‘toothpaste’ or ‘ticket’ in the first few hundred words in a new language?). I can’t search for a word in the courses that I’m learning (for instance, during the setting up of a new deck to check if a definition interferes with a previous deck) and I have no quick way to compare two decks side-by-side. So it somewhat works but is starting to frustrate me and I feel that the more time I spend on MemRise the stronger those frustrations will become. (and none of them seem to be too hard to fix).

This is written after I’ve been playing with MemRise for only a few days (but full-time, I cranked up about half a million ‘points’ in their system so I think I have some right to speak here) but still I already find myself wondering if there is a way to improve on MemRise.

MemRise is made by ‘Neuroscientists, GrandMasters of Memory and software developers’ and they are reinventing learning. I’m a-ok with that and I think it is a great product to get a first grasp on a new language but I really think they could do a much better job of this than they are doing right now when it comes to learning a new language (which is a very large part of the use-case of memrise, but when you’re on a mission to ‘reinvent learning’ you might miss that little detail).

Spaced Repetition Software not exactly a new concept (even though memrise is a nice and slickly packaged example of it the first SRS dates back decades ago). Currently there are a lot packages that attempt to do this out there, with mobile phone and tablet support and all kinds of goodies. But most of them look pretty arcane (MemRise definitely looks good), and I wonder about the quality of what’s in there. There is the PimSleur language school, of course, but after being spammed to death by them about two years ago I have made an expensive vow to never ever do business with them. One thing you have to keep in mind when using techniques like this for learning is that you are memorizing things, you are basically cramming facts but you are not in any way aiding your actual understanding, although of course those facts can later be used when you do reach a certain level of understanding. Patterns, shortcuts and other deeper modes of understanding are not easily conveyed using flash cards, so the method appears to be somewhat limited in scope. Though you might be able to get around this by crafting your flashcards in a specific manner (I haven’t seen much of that to date though).

One thing I have in common with the MemRise guys is that I’m a software developer too. And that is where I will look for my solution.

User interface issues can be resolved once you have the codebase under your own control, and one way of doing this is to go to work for MemRise. Since that’s not an option I’ll settle for door #2, write my own little flashcard program (it doesn’t look all that complicated from the outside and the basic principle of SRS does not look as though it is impossible to implement either).

The first issue to resolve though is the quality of the content. How to go about setting up a solid language course. Over the last two days I’ve been toying (more and more serious as I’m going along) with the problems of building a thing like this.

One of the first concrete issues you run into is that nobody seems to agree on what the most useful words are. So it looks as if everybody in the ‘flashcards’ industry is doing the same thing: they copy the first 20 pages our of the Berlitz language guides and call it a day (and in the case of MemRise they make their users copy that data). This is likely why you end up with a ton of identical word sets and yet miss out on lots of important words for every day use. And it is also why most or all of these courses top out at relatively low counts of words and focus on teaching you words not a language.

Language is a tremendously interesting phenomenon. It is the verbalization of concepts that can exist independently in our heads. ‘Blue’ and ‘Brother’ are concepts (the colour of the sky (a concept in its own right), and a man who has the same parents as me). They do not even need a sound to be associated with them to exist. And that sound does not necessarily have to have a written representation either. But you could attach such a sound to either one of these concepts and then you’d have the beginnings of a language. Another person might assign different sounds to them and then you’d have to learn the concepts behind the words in order to learn their language and to translate your thoughts into sounds that they would understand. So the concepts are the key.

The next issue is that concepts do not map 1:1 from one language to another. Maybe some concepts don’t exist in another language (try to translate ‘schadenfreude’ from German to another language, or the Dutch word ‘Gezellig’). The more languages I learn (Dutch, German, English, Polish, a bit of French, a bit of Latin, a bit of Spanish), the more frequently I run into words that I can’t easily translate from one language to another.

Then there are synonyms (concepts with multiple representations) and homonyms (multiple concepts that map onto the same spelling), homophones (multiple concepts that sound the same) and so on. These complicate the structure of language quite a bit. Furthermore not every language treats its words as ‘immutable elements’, many languages will modify words dependent on where they are used in a sentence or upon some other contextual element (or who the speaker is, what their relative social status is and so on). Fascinating!!

So, consider my interest engaged :) To find a starting point in all this I’d like to know what the most useful words are. That’s a suspiciously simple question, and like most questions like that the answer is extremely complex and depends on a ton of outside factors. For a lawyer visiting a university abroad to learn about the local legal system that answer is different than for a tourist visiting a museum or even a tourist in a restaurant. Context is everything. And yet, there must be some subset of language that everbody would agree is useful. And beyond mere words, there must be sentences that are more useful than others (and ditto sentence fragments).

Searching around the web for a nice body of spoken text I found this beautiful collection of spoken text. After downloading it and cleaning it up a bit here are the results for the first 30 most spoken words in English (as used in those transcripts, of course this is not) and their frequency in that corpus:

   1508 she
   1535 on
   1542 there
   1613 just
   1632 but
   1826 what
   1841 yeah
   1928 so
   1981 this
   2050 we
   2133 are
   2224 like
   2339 he
   2370 have
   2598 know
   2662 do
   2663 they
   2872 in
   3001 was
   3257 of
   4257 not
   4821 a
   4832 to
   5702 that
   5844 it
   6669 you
   7219 and
   8126 is
   8181 the
   8895 I

You can immediately see how (at least in the English language) a high frequency of use means that words are short, language has optimized itself over time to be economic with the bandwidth between the speakers and the listeners. Longer words are typically used less frequently.

So if I would use this list that I have generated as a basis for learning individual words on flashcards that would be one first step in the right direction. And that list could be cross-checked and corrected using resources such as this list of the top 1500 nouns. Next I need to translate them and come up with a datastructure that will hold the concepts and the representations in the various languages, attributes of words and so on.

This is starting to be a fun project :)! It is also of course a very nice way to procrastinate but I think that in the process of setting this up I might actually learn some Romanian too. And meanwhile I’ll continue to use MemRise, don’t throw away your old shoes before you’ve got your new ones.

When Sued Don’t Tweet

The employer of one of my coding heroes, John Carmack is currently being sued by ZeniMax over unspecified IP that he may or may not have taken with him to his new spot.

He responded to the suit and inquiries by the general public with tweets here:

Oculus uses zero lines of code that I wrote while under contract to Zenimax.

And a slightly older one:

No work I have ever done has been patented. Zenimax owns the code that I wrote, but they don’t own VR.

Now, I understand the anger, I’ve been sued a couple of times and it is hard to describe the degree to which this sort of thing can affect you if you haven’t been through it. But anger is a bad advisor. I realize that when you are quite possibly sued in a way that you consider baseless and even mean that you are tempted to set the record straight publicly. After all, there are two courts, the court of public opinion and the court of law. The problem with that strategy is that you may unwittingly be giving your enemy ammunition in the court of law, and even though you think the public opinion one is important (and that may very well be so if you’re some kind of celebrity, this does not mean ‘everybody with a twitter account’) the court of law is the only one that matters. Winning in the court of public opinion will not even get you a cookie, losing in the court of law can cost you your shirt. See here for an exception to this (Oatmeal vs FunnyJunk).

When you’re being sued or could be sued, especially for $BIGNUM: Shut up on all your public channels, do not communicate with the media and talk to your lawyer and your lawyer only (and on that subject: the only lawyer that’s yours is the one you pay). The public can wait. Everything you enter into the public record will likely stay there, and it may very well be used against you or come back to haunt you (I see several problems with Carmacks’ statements that he probably is aware of, for one, IP involves a lot more than code and second not everything that is protected needs to be patented, for example, trade secrets, designs and prototypes). IP is a minefield of special cases and subtlety, twitter of all places is not where you deal with things like this as soon as things go the legal route.

For now the arrows seem to all be pointed at Oculus Rift (follow the money on that one), but with the ‘instrument’ of the transfer of IP being John Carmack it could very well end up with him being thrown under the proverbial bus, the stakes are more than high enough.

So, hold your head cool, don’t give in to the temptation to pre-empt any lawsuits and don’t inadvertently give your opponents lawyers ‘purchase’ in the form of ill thought out tweets. They’re just about un-retractable even if you delete them and they won’t change the facts to your advantage, and the facts are really all that matter. If there is no upside to moving: don’t move. The general public can wait until the case is dealt with and they’ll definitely be understanding if you clam up in the run up to a court case. You can bet that your opponents counsel is going to go through all your public statements with a fine comb to see if they can identify pressure points or strategy hints.

edit: fixed some factual errors, such as that Carmack is not (yet) named personally as a defendant. Thanks to: HN’er TodPunk https://news.ycombinator.com/item?id=7685884

The Raspberry Pi as a Poor Mans Transputer

One of the most interesting architectures to come out of the 1980’s was the Inmos Transputer. The idea behind it is that you string together a large number of CPU’s each with their own memory and connections to neighbouring transputers (using a series of ‘links’) into a computing fabric.

A recent comment on HN suggested using a cpu-per-process as a means of achieving very high degrees of security. This made me wonder if you couldn’t use a Raspberry-Pi Compute Module to achieve this goal, and that in turn led to researching the kind of buses that would be suitable to hook up a large number of processors.

And that in turn - inevitably - led to the Transputer and the CM1 (the Connection Machine).

The 80’s were an interesting era in the field of computing. We weren’t as much invested yet in architectures as we seem to be nowadays, the field was a proverbial Cambrian explosion of architectures and subsystems. Lots of interesting research came and went before it’s time was realy ripe.

So, this comment on HN led me down some interesting rabbit holes that I had not visited in years (I played around with Transputers in the 80’s, they were quite expensive and I could not afford the number that I would have wanted to use), and I wonder if this isn’t a very fortunate alignment of the planets.

Our understanding of massive parallel systems and compute fabrics has increased considerably in the last two decades, we have an ultra cheap platform at our disposal (the RasPi compute module sells for $30 in quantity) and we have a free operating stystem to go with it (Linux).

Running the RasPi’s fabric style with hardwired links between adjacent nodes (and the occasional uplink to a switch) should be doable, there is plenty of GPIO available for that purpose. And with 256M of ram the nodes would be fairly powerful. Say 32 nodes per backplane, with backplanes connected using multiple ethernet links, or if you want to go fabric all the way, connecting them in a plane like fasion to maintain the 2d arrangement (or a hypercube of them). There was a thread a while ago about a little board that you could connect 4 ways to its neighbours (for the life of me I can’t find it…), that might be doable if the backplane edges are fashioned as connectors.

Doing this will be some work (especially the links and drivers for them) but well worth it and possibly one of the most interesting things that you could do with the module, the longer I think about the more I realize it is just about ideal for being used as a base for a compute fabric.

If anybody builds this let me know, my own hardware design skills top out at double layer hole-through so this is definitely beyond me.

Journeyman Project Trip 1 United Kingdom

In January I came up with a weird idea, to go and visit total strangers in return for ‘food and board’ in order to do useful work for them. (see: Will Work For Food And Lodging for how and why this started).

They say no battle plan ever survives first contact with the enemy, and in a slight variation on the theme, in this case the battle plan did not survive first contact with the audience. One of my major mistakes was to completely underestimate how much response there would be. I figured ‘a couple’ of responses, probably not enough to fill the first year but maybe once the project got under way there would be more takers. How wrong can you be? Well over a hundred responses and counting, they’re still trickling in! And from places too that were definitely not even close to within the radius originally mentioned (1000 km around Amsterdam) (Including Indonesia, Turkey, a few in Africa, Australia, Latin America and the United States).

The second place where the plan went wrong right from the start is that I was going to take my car along. That didn’t happen, the reason for that is that there were rather a large number of responses from the UK and several of those were the first to reply. First come, first serve but after doing some research about shipping the bus to the UK I figured the easiest way to do this trip was to take a plane and then rent a car there. That shaved two days off the duration of the trip which I could use to visit one more company.

Finally, the duration of the visits. The original idea was to end up visiting 11 companies over the course of 2014 for 14 days each, but after processing just the UK part of the applications I knew that the only way that would work is if I disappointed almost everybody by selecting only a very small part of the people that applied within the parameters set. And that just didn’t feel right.

So in concert with the people visited I ended up staying a maximum of 2 days at each location. It was quite a murderous schedule, with lots of driving on the wrong side of the road (and at night, in places where you probably shouldn’t be driving at all, in the rain, on hills, in tight curves, in an unfamiliar car, with the steering-wheel definitely placed on the wrong side of the car). In the end it worked out extremely well, I managed not to kill myself by turning into a roundabout the wrong way around, and after a day or so my right hand stopped ramming into the door to shift gears every time I forgot on which side the gear shift lever was located. Incredible how much stuff we do from habit.

One by one I visited the places that I had made appointments at, and met with a series of wonderful people doing all kinds of interesting stuff (and their pets!, the UK is full of really nice dogs).

What got discussed is of course between the parties that I went to visit and myself, I’m not going to go into details here, I don’t have any secrets worth keeping but they definitely might, suffice to say that I’m pretty sure that it was educational for me and I hope that it was useful to them, any further disclosure will have to be at their discretion, not mine. All in all this is one of the most interesting things I’ve done in the last couple of years and I’m really looking forward to doing the next round of visits.

I made a ton of pictures, only very few of them were used for this article, if you just want to browse the images you can do so via the image archive.

Below is a little bit about everybody that I ended up visiting in the order in which they were visited. A few parties disqualified themselves (not in the list below) because they either weren’t taking the project serious or because they tried to get me to do what I usually do for a living for free (and that’s not in the spirit of these visits). The companies listed run the gamut from tiny to huge but they all had one thing in common, wonderful people!

Conversion Rate Experts

With their offices in a castle in Staffordshire but with global reach through a network of remote working experts in 10 countries, Conversion Rate Experts is quite an interesting undertaking founded by Karl Blanks and Ben Jesson. They help companies to optimise from visitor to customer and they are doing an excellent job of that. Their customer range from mid-size to gigantic and they have an excellent series of articles on their company blog.

Who: Conversion Rate Experts, Karl Blanks, Ben Jesson

Where: Staffordshire, UK

What: Optimisation of the sales funnel

Looking for: More consultants to help deal with their growth (see: jobs )

On the web: Conversion Rate Experts

Contact: Contact page

Astral Dynamics

Astral Dynamics is a band of enterprising souls around Liam Kurmos, located in an old chapel in Wales. Wales seems to be a super interesting region, especially for water power based renewables, it is raining pretty much all the time there! Liam has started converting the Chapel to something that is a cross between living quarters, office space and a hacker/maker space. Super nice guys, we went for a walk nearby (and nearly got ourselves killed by being blown off a derelict staircase to the top of an old slate mine right next to a 1000’ drop).

Who: Astral Dynamics, Liam Kurmos, Dan Prince, Dyfan Searell and Noah Hall

Where: Deiniolen, Wales

What: Hacker / Maker / Office space for communal use

Looking for: More hands, some funds

on the web: AstralDynamics

contact: liam@astraldynamics.co.uk

Ryan O’Neill

One of the first people to respond to the call for locations to visit was Ryan O’Neill, an old school hacker from the mid-west of the UK, near the Welsh border. Ryan wanted to discuss a project to block TV advertising. I arrived in Ludlow on Sunday evening, spent the night in a pub/inn right next door to a church and presented myself in the morning. This trip seems to be mostly made up of classical buildings and super nice dogs, the O’Neill residence is no exception to that rule, meet meg and the ducks:

We worked throughout the day and the next on the project and I think we made good headway getting an MVP and a follow up product spec’d out to the point where it could be implemented.

Ryan is a blogger and would like to have a co-founder on this project that has technical and/or business skills to give the project more legs, and once the MVP is done funding will be sought to assist with the roll-out.

Who: Ryan O’Neill

Where: Ludlow (West Midlands)

What: TV Ad Blocker, software development

Looking for: Co-founder

On the web: Ryan O’Neill

Contact: r@ryanoneill.com

The Cats Whiskers

Two for the price of one! Christina O’Neill (Ryan’s wife) has a business called ‘The Cats Whiskers’, she makes custom gift boxes. They come in all shapes and sizes, from one just large enough to hold a few baby teeth to one the size of a trunk (and possibly beyond).

Who: The Cats Whiskers, Christina O’Neill

Where: Ludlow

What: gift boxes

On the Web: The Cats Whiskers

Looking for: people that want to give something unique to a child

Contact: christina@thecatswhiskers.biz

HoxtonVentures

HoxtonVentures is not your run-of-the-mill venture capital company. Extremely hands-on, investing in early stage ventures with a good track record they occupy a nice middle-ground between start-up incubators and later stage venture capital. They reside right in London (arguably the financial capital of Europe). Their mandate allows investment all over Europe, including Russia and other places where most venture capital companies would be too timid to tread.

Who: HoxtonVentures, Rob Kniaz

Where: London

What: Start-up / early stage capital

Looking for: start-ups all over Europe, people interested in positions for their portfolio companies

On the Web: (HoxtonVentures)[http://www.hoxtonventures.com/]

Contact: rob@hoxtonventures.com

GrantTree

Granttree is the company owned by Paulina Sygulska and her husband Daniel Tenner. Granttree’s business model is to give away money, this arguably is one of the easiest sales pitches ever. They have an office in the heart of London’s tech district in Shoreditch, where they work with a whole team of talented young people on the problem of applying for government grants for various programs.

Who: GrantTree, Paulina Zygulska, Daniel Tenner

Where: London

What: grant applications, tax credits

Looking for: Ever More Customers

on the web: GrantTree

Contact: team@granttree.co.uk

I loved doing this, a warm thank you to everybody that decided to take a chance on me and invited me into their homes/castles,chapels and other dwellings, it’s been a long time since I had this much fun being on the road. About 1,000 km got covered according to the odometer of the car, the whole trip took 12 days from start to finish. The single biggest cost factor was the car, gas, the next the plane ticket and a few overnight stays in hotels (most of which were covered by my hosts), all in all I’m out about 1300 Euros but it feels like this was money extremely well spent and I would do it again in a heartbeat. To be continued, in Germany for the second round somewhere in the coming weeks. I’ll send out a separate call for applications as soon as the dates are nailed down.

Will Work for Food and Lodging

In the past, the various professions were organized as guilds. If you wanted to become a cabinet maker you’d enter into an apprenticeship arrangement with a ‘master’ cabinetmaker and you’d be taught the tricks of the trade over a several year period. As your final examination you would be required to make a masterpiece (that’s where that word comes from) and then, if the master accepted your work you could call yourself a cabinetmaker. To avoid you setting up shop right next door and end up cannibalizing the person that taught you and to promote the flow of ideas within the guild you would then be discharged from your apprenticeship and your journeyman years started.

The journeyman years are an interesting concept. The newly minted cabinetmaker (or bricklayer, wheelwright, smith, whatever) would don travelling clothes (including a walking stick), pack up his tools and a knapsack and hit the road, travelling from town to town offering his services, telling stories and would be given food and a roof over his head during the nights. This period would last from 1 to 3 years, and upon completion the former apprentice, now journeyman would be permitted to settle in a spot that the guild had some say in and would be allowed to call himself a master.

As technology became more and more widespread the power of the guilds diminished and tradecrafts died out. In some isolated pockets of the world the journeyman concept still exists but it is really an anachronism. For instance, in Germany the carpenter trade still practices (a limited form) of the journeyman years and occasionally you can see them walking the roads or the streets there.

So, besides wishing all the readers of this blog a very happy 2014 I have an offer to make. I’ll divide 2014 into 13 equal portions of 4 weeks each. 12 of those, starting the 1st of February will have one half dedicated to the things that I normally do to make a living (maintain my websites, consultancy, technical due diligence). The other half I will give away to parties within 1,000 km from my base (Amsterdam) who think they have an interesting job for me to do. There will be no payment for this, I’ll pay for my own fuel for my trusty bus, other than in the journeymans tradition that I would expect food and lodging. Though each part will be limited to two weeks there is no reason why there could not be more parties geographically close that would all want me for a bit shorter than that so feel free to request shorter things as well.

So, if you want to take me up on this offer mail me at jacques@mattheij.com with who you are, what you (or your company) wants me to do, where you are located and if there is some specific time-frame or other conditions then you should probably tell me about that too. Every 4 week period I will select one or more of the open offers and we’ll arrange for a specific date & time to visit and work on your project.

Don’t feel too limited about the kind of work that you have, I’m pretty versatile. If any tools are required for whatever you want to do and I have them (and you don’t) then I could probably bring them with me (the bus will hold about 5 cubic meters worth of stuff), however if I don’t have the tools required then I won’t be buying them. If any materials are required then that’s your problem, and of course I would expect you to be available during the time that we agree I’ll be ‘on site’. Naturally, I reserve the right to write about my experiences on the road as well as with you / your company, suitably anonimized.

This offer does not automatically constitute an obligation to accept, I will evaluate the offers on a case-by-case basis and will select those that are in my opinion and solely at my discretion the best fit.

If you want to get an idea of what I can do the rest of this blog might give you some hand-holds. I’m not limiting this offer any more than I have to so feel free to ask me to help remodel your house but chances are that there will be something a bit more aligned with my skills. Otoh if yours is the only request and it is to help remodel your house I will probably take it :)

Some stuff I’ve done over the years: ran a mid sized company (25 employees), designed a windmill, wrote the software to help design it, built the plasmacutter/mill and associated electronics that it was made on, cad/cam software, car restoration, house rebuilding and house building, electronics design, general programming, web programming (but that’s not up to 2014 standards), general problem solving (the more difficult the bigger the chance that I’ll like the challenge :) ) and so on.

I really try to live up to my favorite line from Robert A. Heinlein’s books: “A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.”

So feel free to challenge me in unexpected ways.

The Five Year Itch

I’ve read a number of articles recently on ‘nomadic techies’, people that live as sparse as possible while moving from place to place every few days, weeks or months and how this impacted their lives, a backpack, a laptop and an online business or consultancy seem to be the main ingredients.

This spate of articles has caused me to look at my own life to date in a new light and I think I too have been bitten by the nomadic bug only with me it seems to last on average five years before I reboot my existence, and I have a rather large backpack.

The major reason for this is a very simple one: I get bored. That’s it, it is no more complicated than that. I can do any job but only for so long and once the challenges are met or tamed and things become routine I find my patience runs out and I have to change. If I start looking at this over the course of my professional life to date, since the ripe old age of 17 or so sequence looks like this:

PeriodCountryCityOccupation
1982-1986NLAmsterdamEmployed by a bank, mail room, app. programming
1987-1992NL/PLAmsterdam/Rijswijk/PoznanContract work for customers
1993-1997NLIJmuidenProduct development, licensed software
1998-2003NL/CAIJmuiden/TorontoOnline services
2004-2008CA/NLSt. Josephs Island/Groningenmanual work (various houses, windmill)
2008-2013NLGroningen, Limburg, IJmondFinancial services, project management

Give or take a few months here and there and maybe a few more sidelines. So on average every 5 years I pick up those that are dear to me and some stuff and move and/or shift into a new line of business.

This is extremely hard on others, even if to me it seems to be a very natural and easy to do thing. The risks associated with these moves and shifts, the logistics of it and the financial burden is only one part (such moves, especially international ones are very expensive). The other part is how it wreaks havoc on your relationships with other people, family and friends as well as co-workers and others that you are normally in contact with because of physical proximity. I probably don’t even have 1% of the people that I know and that are dear to me within a 10 Km radius around where I live, the majority of them are at least one or several countries away, if they’re not on a different continent altogether.

It’s also hard on my children and significant other. The flip side is that all this has given us a fairly broad perspective on life, on the (western, so far) world and on the nature of friendship. Knowing a lot of people and knowing them more than just as acquaintances has enriched my life in a large number of ways, something that holidaying in all those places would have never given me. That doesn’t mean I don’t like to travel and see places. It just means that a few weeks or months is not enough to really get a feel for a place and for how living there really is. If for instance you visited Canada from May through August you’d miss the winters and those change your view on life in Canada tremendously.

Contrary to the monthly or weekly version of the techno nomad doing this on a five year roster and with a family rather than by your lonesome (maybe with a backpack and a macbook) it is a very expensive trick to perform. You’re looking at housing, vehicles, furniture and so on as well as possibly burning up a sizeable investment in those if you can’t sell your stuff (you don’t have much time once you decide to move so you will get crappy prices).

Still, the upsides (to me) far outweigh the downsides, it keeps life interesting and varied and it gives you time enough to grow some roots but not so many that you can’t uproot and move if you feel like. I’m not sure if I’ll be able to keep this up forever, closing in on 50 now there is a tingling sense of unease, a part of me that wants to really settle down and grow roots to stay. But then, after 5 years of doing the same thing I’m pretty sure that the same urge to do something else would surely creep in.

Astute readers will have noticed that another 5 year term is up. Right now I’m not in a position to move, nor has the desire to do so announced itself in a way that I can no longer ignore. But who knows where I’ll be in two years from now, it’s impossible for me to know for sure and that by itself is part of how it starts The thing that is different this time around is that I still like the work that I’m doing, I work with people that can (and do) teach me a lot, they’re interesting and nice (both to me and to others) and there is a sense of camaraderie that i so far lacked. Currently we’re in the process of evaluating ideas that are viable for our next project/company and I’m sure that once we find something and I commit to it that I’ll be able to tame the bug for long enough that I can meet my responsibilities. But if no such project can be found all bets are off!

The nomadic life-style is marked by traveling light, I can’t say I qualify for it from that perspective. After five years in one place it is amazing how much stuff gets collected. Right now I have a warehouse full of stuff in Limburg, as part of an attempt to jumpstart a business there. Just getting rid of that is a major exercise in logistics and then when it is done we still need to sell the building. But I do find that I look at the things that I have in a different light because of all this.

Things of real value:

  • sentimental stuff (pictures, paintings)

  • tools (woo if you move to a region with a different mains voltage)

  • musical instruments

  • books

But especially books are heavy, bulky and hard to move! Even so those are the categories of things that seem to travel with me where-ever I go. Unless you’re living a minimalistic life-style trying to choose which 1 metric tonne of your family-of-four you want to take with you can be a pretty revealing exercise in priorities and sentiment discovery. (oh, and tools are quite heavy!).

Another downside to this way of life is that I really no longer know where I belong. No single place feels like home or ‘motherland’ or anything like that. You learn that the grass on the other side may look greener but that it rarely ever is, that every country has its good sides and its bad sides but that there is no country that does everything right. I sometimes miss feeling ‘home’. Until I was in my mid 20’s the center of gravity would always be Amsterdam, but having lived on a nearly uninhabited island (300 people on 200 square kilometers) and some other interesting places city life no longer holds me in the way it used to. I’m not a city boy, I’m not a country boy, I’m not even sure what kind of person I am in that respect any more. I enjoyed both and I enjoy where I am today, in fact, I’ve learned to adapt fairly easily to new situations and to be able to make friends and feel good in most places.

I have no idea where the future will lead me, I’m open to moving again at some point, maybe Asia, maybe Eastern Europe or some other place not even currently on my radar. Whatever it will be I hope it will be unlike any other place that I’ve ever lived in so far, and that it will bring me new insights, new people and new experiences. Because that’s the one thing that I think that binds all nomads: Insatiable Curiosity.