Jacques Mattheij

technology, coding and business

Why Stuff From China Is So Cheap

This is most likely the most disturbing video I have ever seen, and just to make sure that you can’t just click through I’ll make you work for it. Please take my hint and Don’t watch this video, instead, just read the text, it is disturbing enough and it will tell you all you need to know and a little bit more besides.


The original title for the video is “Chinese Workers accidentally touched High Voltage Electricity Wire, and Catch Fire”. Needless to say, they die.

This accident is one of many that happen every day in the People’s republic and it is just one very vivid illustration of what goes wrong when things are produced at cut-rate prices without any consideration whatsoever for either the environment, workers or even general ethics.

This accident was preventable in so many ways that any OHSA employee seeing the video is likely going to have nightmares about it for a long time. Pollution, a total disrespect for the environment in general and for human lives in particular is what powers the trade that brings cheap goods to the stores near us. The price differential between producing those same goods locally and producing them far away then shipping them across the globe is paid in a very real sense by the 100’s of millions of workers in factories and by the immense pollution taking place all over China (and other 3rd world countries, for that matter).

In many ways we are complicit, we vote ‘cheap’ every time we can (I’m no exception) and if we can externalize the problems in such a way that we can pretend they don’t exist then we all sleep just fine at night. Until a video like this surfaces and rams home just how unsafe worker conditions in those low wage countries really are (and electricity is a relatively well understood danger, the long term exposure to many chemicals is not).

I’m not sure how to tackle this problem, frankly I haven’t a clue. But it definitely is a wake-up call to me that we’re on the wrong track with this, by letting the companies that we buy our goods from dictate terms based solely on price we are really missing an opportunity to force some more good to come of all of this. Western companies doing business with China and other 3rd world countries should demand compliance with the client countries’ OSHA regulations, ditto when looking at things like labor laws (minimum age for instance) and so on. Sure that will eradicate some of the profits, it may even raise prices a bit.

But I think that’s a price worth paying, if only because then nobody will be able to send me video’s like this. Thanks anyway, RobB for opening my eyes.

How to Go Bankrupt

It’s fairly rare that I can write about the jobs I take on, this one is an exception to that rule and there are some lessons here that can help any entrepreneur, even the ones that are doing well. This is a tale about trust and growth, it has a very unfortunate ending.

Going bankrupt is exceptionally easy if you go down one particular route. You simply buy something for about the same or more than you’re selling it for and you forget to factor in all your variable costs and/or overhead.

Every now and then I end up advising companies or executives of companies that have hit a rough patch. Usually these stories end quite well and everybody lives to tell the tale of a close call but in this particular case there was nothing left to be done other than to pull the plug on the patient. That hurts. Here is how it happened.

A services company was growing like crazy. The CEO and CFO had bought the company several years earlier using an earn-out construction and they had been growing the business revenue wise substantially, double digit growth for three consecutive years.

The market the company operated in is highly competitive, margins are slim and customers are using their knowledge of the market and the players to negotiate agressively in order to get the best deal. In a market like that - very price sensitive - you have to be extremely aware of what your exact position is because the difference between making money on a deal and losing money on a deal is small but very significant.

In 2010 the company made a substantial profit, in 2011 the profit was still there but lower, in 2012 the company lost substantial money. The reason I got called in was because there were some irregularities with one of the current accounts but as I gained more insight into the situation I realized that the company was probably doomed. We moved for a scenario where the company would get temporary relief from their obligations in order to restructure but after a few days more it turned out that that was so unlikely to be successful that they filed for bankruptcy.

This is a pretty tough affair, bankruptcy is a bad thing to happen to a company for many reasons, not in the least because as executives you can be held liable for mis-managing the company if there are substantial reasons to suspect this. But in this particular case it was unavoidable, the company had entered into several expensive contracts increasing its overhead, had been bidding so aggressively on various deals in order to increase their turn-over that they lost sight of the bottom line and were actually losing more money on the deals they closed than they were making in the first place, once all the variable costs and overhead were accounted for.

But because the CFO was asleep at the switch and the CEO did not check up on him (trust is a good thing, but verify if you value your company and if you take your responsibilities serious) and probably did not have the financial chops required to do his job properly this wasn’t noticed until it was (much) too late.

So, in spite of double digit growth they ran out of room to manoeuver when the cash reserves built up during the years when they were still making money on the majority of their contracts ran out.

The lessons you can learn from this tale are quite simple in principle but can be hard to apply in practice:

  • Make sure you know what your counterpart is doing if you share the responsibility for the company

In this particular case, the CEO is an extremely nice, somewhat overly trusting man. By keeping him out of the loop the CFO was able to hide the problems (including some irregularities involving private accounts and the corporate current account) from his partner. This allowed the situation to continue to the point where there was nothing to be done about it.

  • Make sure you factor in all your costs when bidding on a contract

The slimmer the margins and the more competitive the field you are operating in make sure that you know what your walk away bid is, in other words, what you have to make on a deal for that deal to be profitable. Turnover is meaningless if more business decreases your profitability, the whole idea is that by increasing turnover you gain some advantages due to economies of scale working to your benefit. If the economies of scale turn against you then you are killing your business by making it grow. Know your business inside out so that you can make the right decisions.

  • If you are not able to function in a certain role, don’t take that job

A CEO should at a minimum be able to read a balance sheet, work through a liquidity prognosis and to check up on other parties actions on behalf of the corporation. He or she should be able to work fairly well under pressure and should be able to read and understand basic contracts and to estimate the consequences of actions or inactions. You should also be able to understand the exact responsibilities and if applicable the rights that come with the job. Being CEO of a company with close to a 100 employees is not something you should do on a lark or as training on the job, it means that you are indirectly responsible for close to 300 people and you should take such a responsibility extremely serious.

  • when in doubt, get professional, unbiased advice

If you are in a situation where you need legal advice make sure that you are the person paying the lawyer and pick a lawyer that is not in any way associated with one of the other parties. A lawyer that you do not pay is not working for you.

  • if you receive questionable advice then this goes double

If you receive advice that you find either curious or questionable then get a second opinion immediately, igorance of the law is a very bad defense (in fact, it is no defense at all) and a lawyer that is working just for you will be able to quickly dispell any doubt about whether or not a certain activity is (a) in the interest of the company, (b) in your own interest and (c) legal. Especially the latter can get you in hot water, more so if a company should default and you were at the helm.

Bankruptcy is something best to avoid, make sure it does not happen to you, lest you end up being witness to an asset sale and possibly subject to a personal liability to boot.

Love to Learn

My lucky break was not finishing my education. It’s probably hard for an outsider to see the upside in having your home situation implode just when school starts to get serious, to go to work for a boss doing some menial job in your teens, no diploma, no future as far as anybody could see. But this helped me in so many ways that I really should be grateful for it.

For one, the guys that I worked with in the mailroom were a pretty rough bunch, but they also were very fair and wouldn’t play favorites just because I was a scrawny kid. On my first day there - which I was pretty sure would be my last - I held them all up. Those guys were ridiculously strong, they made it look so easy. It doesn’t help when the mailbags that you need to haul around and sort weigh more than you do. But rather than doing my work for me or helping me directly the men in the mailroom helped me in a completely different way. They simply stood by chitchatting waiting patiently for me to finish my part of the shift. One memorable moment was when I really was about to give up when the shift boss (a guy that was maybe 5’ tall but also just as wide) told me: “You don’t have to move that bag in one go” hinting that if I opened the bag to do it in pieces that would be fine by him as long as it got done. Those bags were so heavy that at times I was convinced they were pranking me by having one nailed to the floorboards.

Not a bad word, no insults. When I was finally done and the last bag was in the truck to be brought to the trainstation they all said their goodbyes and did it all over again the next day. Little by little I got faster, I really owed them that much. Running mail was the easy bit for me, hauling around bags and keeping up definitely was not. It took me 6 months to get to the point where I could say I managed to keep up with the rest, I was never threatened with being fired for being slow or anything, simply the pressure of knowing that everybody is waiting for you was enough to push me. I earned just enough for a bit of food left over after paying rent once I moved to a one-room appartment in Amsterdam. 700 guilders of income, 500 guilders of rent, 200 guilders to live off. It seemed quite luxurious in one way but in another it was very depressing, not being able to save enough for new gear and especially not enough for buying books. Fortunately public libraries were stocked well with books on all kinds of subjects, and my library card saw some pretty intensive use.

So not completing my education (aside from a typing diploma and grade school, later augmented with a driving license) was in a way a lucky break. It gave me a very strong motivation to claw my way out of the situation I was in, it gave me a bunch of ersatz parents that taught me the value of honest hard work and many other things besides. I got to know most of them outside of work as well, repairing their hi-fi gear and televisions, and I got to see how a bank worked from the inside. Being a mail runner has one huge advantage over any other job which is that you get to know everybody, from the board of directors all the way to the print shop. Little by little I started to figure out by traffic analysis how the various departments interacted with each other.

I really can’t help it. Give me any piece of machinery, a computer program or a corporation and I have to figure out how it works. Nothing vexes me more than to have something that I can’t understand or figure out. On my desk a very dog eared ‘Essentials of Genetics’ by Klug & Cummings (thanks Marcela!), a book on the construction of lattice grids (Makowski), a book on heat pumps and Stirling engines and yet another on fuzzy logic. By not finishing school I never had a hard line that told me that I was ‘done’ with my education, so instead of relaxing after getting my diploma I just kept on reading and learning as much as I could. It simply never stopped, there was no ‘now I’m educated’ feeling. Rather the opposite, with every bit of knowledge you acquire you only realize all the more how much is still missing and blank. And working for a bank gave me lots of opportunities for learning. They had a course (‘NIBE’) which taught you lots more than what you could infer from moving mail from one department to another, so I worked my way (unoficially) through the course books.

And instead of hating it (like most people doing those courses) I just loved it. Every job I do I see as a paid opportunity to educate myself just a little bit further. What more could you wish for, learn and gain bits and pieces of knowledge, insight into how the world works and get paid for it to boot.

This life probably isn’t for everybody. The levels of uncertainty (other than the website income there is not much in terms of steady earnings), the lack of a safety-net (a car accident or getting ill would be a real problem) and the continuous pressure to perform have definitely taken their toll over the years, both in terms of stress and in terms of personal relationships that I have messed up. And yet, I wouldn’t trade it for anything else. I can’t imagine any other life that would give you so many opportunities to learn about so many different subjects. Another piece of luck is that I never spent a whole lot of time learning things that didn’t interest me or that did not have practical use. That kept my appetite at a healthy hungry level rather than to turn it into a dislike of learning, like I saw in so many of my school mates and co-workers.

Learning is an addiction, and I’m hooked on knowledge and I really need that next hit. So if you have any interesting books or knowledge that you intend to keep to yourself better not invite me :)

The API Paradox

APIs are great, they allow companies to expose parts of their engine for inclusion into the products of others to increase adoption and to facilitate the development of features and products around a common set of data.

In theory APIs are a win-win, both for the party that exposes the API as well as for the party that uses it and since the mid 90’s APIs have become more and more common.

API is short for ‘application program interface’. An API defines a fairly rigid (as in, not changing on a daily basis) boundary where two pieces of software (typically, on the web a service and some client software) meet and where data is exchanged based on certain criteria.

In practice though, APIs are a double edged sword, both for the exposer as well as for the user. In this article I’ll try to outline what the shadowside is of exposing an API, and why this is a potential problem for any users of that API.

When a company first gains traction there often exists a stage where the need to develop software to meet demands greatly exceeds the ability of the company to execute. The bottle-necks are typically it’s ability to attract talent fast enough, possibly the ability to pay for that talent and/or the ability to plan and execute across a wide enough swath of the problem space to serve all those in need.

Enter the API. An API is then defined that allows others to apply their talent and these others are then given (limited) access to the company data in order to be able to serve a particular need.

So far so good, this situation can persist for quite a while. But sooner or later in the life of every company there comes a time when the bottom line starts to matter more than the interests of outsiders.

Now the API gets turned on its head. Instead of a synergistic device would-be competitors that have become dependant on the company for the API it exposes can be cut off at will or can be acquired for peanuts because of the hold the company has over its API clients.

On top of that any API customer that has created a viable niche has provided the company with free validation of some market segment and unless that API customer is extremely well entrenched in its niche it will be easy to dislodge it by cutting off access and re-implementing whatever was needed to serve that niche. Small time players can be continued to be granted access to the API since they don’t consume much in terms of resources allowing the company to claim they are more open than they really are.

So the API Paradox has a timing component. A company that just starts out can use an API in order to leverage the developer community to serve markets that it does not have time or resources to serve itself, with an option right of sorts to take over those markets/companies when the time is right and resources permit a change in attitude. It is not healthy to be found in competition with the sole provider of your company’s life-blood.

If you’re a developer or start-up and you are basing your corporate future on the data provided by some company keep in mind what the future could very well bring and try to ascertain that in the long run too your interests will be aligned. If you do not heed that then chances are that at some point you will find your precious API access cut off and you’ll go the way of many before you that thought that API access is a right, instead of a privilege granted in times of plenty, easily withdrawn in times of scarcity.

Warning signs that you are basing your project/product on quicksand:

  • You are not paying for the API

  • The API is rate-limited and there is no paid option to exceed the limits

  • The company that exposes that API is still very young or in a phase of hyper-growth and is resource limited and addressing only a subset of the market they could theoretically serve

  • There are no terms of service or there are terms of service that allow the terms of service to be changed without notice and/or API access to be cut off for any reason at all

  • Exposing the API does not form a core part of the strategy of the business

  • It is possible to create a competing business using that very same API

  • The API offers no (or very little) prospect for being monetized in the future

If one or more of those are true for APIs that are crucial to the operation of your (proposed) business then you had better be very careful and you’ll need to have a ‘plan B’ in case your API access gets cut off at some point in the future.

If your business plan really needs access to certain data then you should at least try to negotiate a solid contract with terms that stipulate how much data you can consume, how much it will cost you and what kind of termination clauses are applicable. Then and only then would it make sense to build a business on someone else’s API.

If you use API exposed widgets as sugar on your cake then of course none of the above applies but for business critical API usage you have to be fully aware of the API Paradox.

AppGratis Apple AppStore Appeal

In a heartfelt appeal to Apple from AppGratis founder Simon Dawlat we gain insight into the ridiculous situation around Apple’s appstore review proces.

Seemingly on a whim Apple decided to toss AppGratis’ app from the appstore. A new reviewer is apparently all it takes to endanger the jobs of 45 employees and to kill a product that clearly filled a niche, both on the consumers side (with millions of happy users downloading the application) and on the side of the apps that are being promoted.

For a very long time it looked as though AppGratis got a pass from Apple. Only a scant few days ago Apple approved their new iPad application and - one has to agree with Dawlat here - if it was ‘ok’ a few days ago, it should be ok today.

If a new application gets submitted to the review process then Apple has every right to reject that application or to ask the developer to make changes. But once accepted into the AppStore, once established and approved (especially for 4+ years, in the case of AppGratis!) the barrier for a subsequent rejection should be MUCH higher than a simple change of personel. All it takes is for one Apple AppStore reviewer to have a bad hairday and your company could be in trouble and you too could be out of a job.

And even if your product passes under the rules of today Apple can - and does - introduce rules that specifically target certain segments of their ecosystem for annihilation with subsequent selective enforcement of those rules.

Apple handled this in a very unprofessional manner, the CEO of a multi-national corporation not being available instantly (the man was on an airplane) can’t be a reason to yank an app that has been in the AppStore as long as AppGratis has been.

Apple is acting in a tremendously irresponsible manner here. They are risking alienating the developers that made the AppStore into what it is. If Apple wishes to be left with nothing but hobbyist app developers that create a multitude of fart apps then they should definitely make a few more ‘examples’ of successful applications that have carved out a niche in the eco-system. After all, bringing an App to market that has significant back-end logic, that requires the creation of unique content in several languages and that delivers a polished user experience will require a solid team, will require significant cashflow and will likely require that such dedication will be on display over the lifetime of the App. And when new devices are added to the mix they will need to display all of that all over again.

This is not hobbyist territory. By behaving like a schoolyard bully - we do it because we can - and yanking apps willy-nilly developers the world over working for companies in the mobile space are likely reconsidering the choices they have made. Clearly one can not depend on any kind of past approval by Apple to stick.

It’s like asking for a ruling from a judge, then to use that ruling to build a business on and have the judge yank the carpet out from under you a while later saying they simply changed their minds. Apple has a responsibility here and failed to meet it.

If Apps are rejected initially then that’s all fine and good. But once approved a subsequent rejection, especially of a popular App such as AppGratis or any other should require several levels of review and clear and open communication with the company before such an App can be subsequently rejected, especially if nothing on the App side had changed. Apple owes that to its users, the mobile space in general and the employees of mobile companies the world over that are currently developing applications for the Apple eco-system. Without such a raising of the bar Apple not only puts AppGratis at risk, but risks a massive walk-out of developers and investors in mobile companies from the Apple eco system.

Available Jobs, March

I have received a number of requests to find people for IT companies, both in the Netherlands and elsewhere in Europe. If you’re interested in any of these let me know and I will put you in touch with the company.

Job 1, Location: Amsterdam, Position: CTO/VP of Engineering, mobile cloud gaming network

Job 2, Location: Cyprus, Position: Ruby Developer, gaming

Job 3, Location: Amsterdam, Position: Java/Scala Developer, online travel company (4 positions)

Job 4, Location: Amsterdam, Position: Ruby/Javascript Developer, online travel company (3 positions)

Job 5, Location: Hilversum, Position: tester, PHP & selenium, online video network

Job 6, Location: Uitgeest, Position: PHP developer, ecommerce

Job 7, Location: Uitgeest, Position: Project Manager, ecommerce

Recruiters: please do not contact me because of this posting.

If you fancy any of these please contact me, jacques@mattheij.com, full disclosure, some of these companies I have personal experience with, some not, in no way am I going to make money of referring you, directly or indirectly this is just a ‘community service’.

Come and Help Save Posterous From Oblivion!

A couple of years ago I got wind of the fact that geocities was about to die and along with many others I did my bit to save geocities, the site now lives on (or as much as I could save of it) as reocities.com.

At the time I didn’t think too much of it, a small thing to do really. But over the years, after many thousands of emails from people that are insanely happy that their old webpages, photos of loved ones (sometimes deceased) and tons of other memorabilia have not been lost it became clear to me that this really was something that had to be done. It is incredible how that project has worked out in the long term, right now about 50,000 people daily visit reocities.com.

So, when I heard that posterous was shutting down I figured I’d try to do the same. The posterous guys have sold their company to twitter and twitter has decided to pull the plug. I made an offer to continue to host posterous.com and all the stuff on it but never received an answer.

Last week I decided I’d been waiting long enough, time to do something a bit more concrete. Jason Scott and the guys at archiveteam have done a great job at putting together an easy to use VM application that you can download and run, it is literally 3 clicks to join the archive team and to help them to save what they can of the posterous.com archives.

The clock is ticking, there are only 49 days left as of this writing, so any pages that are not saved in the next 49 days will be lost forever. Please come and help the archive team, if everybody that reads this blogpost downloads a few Gs worth of data then we may be able to reduce the size of the next hole in the web!

How to help? Run your own ArchiveTeam Warrior!

Suffering From Depression? Don’t Do That Start-up!

I hope this post will save some lives.

With some regularity I read articles about some amiable, accomplished and brilliant young kid that decides to end their life in the start-up scene. Invariably they’ve managed to come a very long way along some perceived curve of success and then there is a snag. Either they plateau in their growth, the start-up tanks or there is some other hiccup that causes trouble.

To someone who isn’t suffering from depression such things can be quite devastating. Investing a significant portion of your person (your ‘ego’), of your alloted time on this pale blue dot, investing a large amount of savings or even other people’s capital (family perhaps), and then losing a large chunk of it - or even all of it - is harsh. It’s so harsh that those that fail in their endeavour typically need a long time to recharge to try again, if they ever try at all.

But when you are prone to depression chances are that you will find yourself in the kind of funk the likes of which you’ve never seen, and which you may find too much to deal with. Start-ups are - for want of a better word - an extension of their founders. If a start-up fails, then to some degree it may feel like you yourself have failed, even if that is absolutely not the case. This is the kind of introspection that can be really difficult to deal with. The more you’re associating yourself with the company that you’ve founded, the more money you took from friends, family and investors, the more in the public eye your company is, the more you will pressured to succeed. And then if your company should fail or even simply plateau the pressure can become overwhelming - even if at a rational level you know that such failure is actually to be expected, and that there is no shame at all in this failure.

It is precisely because of this that you should avoid (co-)founding a start-up like the plague if you are depression-prone.

I can name off the top of my head five people all in their twenties or early thirties that ended up taking their own life because they found themselves at the focal point of simultaneous significant negative forces related to the ideas they were executing on. There are many more similar cases, I just don’t remember their names because I’ve had no interaction with them, and I’m not elaborating on these five because that will re-open old wounds. Rationally they did not have a single good reason between them to end things that way. They left huge gaping holes, a tremendous amount of grief and shocked surroundings behind. In some cases nobody ever even suspected that they took their troubles so seriously (or even that there was trouble), keeping up outward appearances of being cheerful and happy, right up to the moment when they took their lives.

If you’re prone to depression and you are in a start-up as a (co-)founder or if you are considering starting a start-up please, please, please don’t do it. Please get out in an orderly fashion if you’re already on-board, please don’t even think of starting a start-up if you feel that this applies to you.

Please get yourself a great job at some stable company, a research position somewhere, or any one of a great multitude of ways in which you can succeed in life and have impact - without being subjected to the stresses that start-up life will inevitably bring. And please get help to deal with your depression (these things are never easy, and never simple). But for the love of $DEITY don’t get into start-ups as a founder, and if you’re in right now get out while the getting out is good. Only you know your little secret, only you have the means to chart your future paths. Purposefully, maybe even defiantly going exactly where you shouldn’t be will only lead to trouble for all involved.

I’ve had just about all I can take of young and brilliant people ending their lives because they couldn’t cope with some perceived failure. Failure is the norm in start-up land, and that means that you have to have the mental disposition that shrugs off failure and treats it like just another step on the way to success. That’s not an optional item, or a nice to have, it is essential, a must have.

For some reason start-ups are a magnet for people that suffer from depression. As though by going up the Eiffel tower it will magically make your fear of heights go away, maybe to harden yourself against it. But start-ups are nothing at all like the Eiffel tower, standing there, serenely still while you practice your mastery by going up the stairs one at a time. Start-ups are more like rodeo rides, doing their damndest not to give you a thrill ride, but to rough you up and spit you out. Do not place yourself knowingly in positions or situations where you will have to deal with extreme stress if you know that you find it hard to deal with extreme stress!

Start-ups are not a place to get therapy, or to learn to deal with (perceived) failure if you’re prone to depression. The failure rate of start-ups being what it is, if you are suffering from depression or if failure is not your thing then start-ups are the very last place that you should be. Start-ups tend to focus a lot of energy - sometimes extremely negative, but even success is extremely stressful - on the individuals running them. And if you don’t have a natural disposition to stand at the nexus of something like that without burning up like a fuse that has to conduct too much current then you shouldn’t be there in the first place. The end result is all too predictable.

No start-up is worth a life, especially not yours.

tx Kai!.

Domain Knowledge or a Lack Thereof

I believe that a lack of domain knowledge is the root cause of a lot of very bad software that gets developed and I think that it is up to computer programmers and their managers to deal with this. Acquiring domain knowledge is an essential component in the development of software that really works well for its users. A programmer that has to automate a warehouse but that has never picked an order and doesn’t have a clue about actual logistics is going to be writing far less effective software than someone that has done a few shifts on the floor. And that’s purely practical knowledge in a field that is reasonably well understood. There are many other examples where the effects of the differences between someone automating some solution with or without domain knowledge would be far harder to see.

Programming and systems analysis are like architecture in many ways. An architect has to have a reasonably good understanding of what a customer is going to do with a building if they’re going to do a good job of designing that building. An enclosure for a machine would have to match the guts of that machine just like a building encloses a company. Having the loading docks of a warehouse on the fourth floor is usually not what you want.

Software is much more intertwined with the institution commissioning it and the people using it than any building ever will be, with the exception of engineered structures such as the CERN ring. If all should fail a building, no matter how mismatched can always at least be used for storage. I’ve seen fire-stations converted to houses and industrial buildings converted to office space. But software that doesn’t match is unusable.

Imagine architects designing a fire station without consulting with fire-men and fire station chiefs and without understanding the basic operations of a fire-station and how the situation develops when a fire is called in. When utility matters, domain knowledge is very valuable.

Over the years I’ve worked on a large number of projects in a variety of industries. One of the things that I absolutely love about software is that it is everywhere, there isn’t an industry that doesn’t have software component in it nowadays. And for each and every one of those projects there was a large amount of domain knowledge to acquire. That’s hard work, and you probably can’t put it on the bill but it is an absolute requirement if you want to deliver a half decent product.

What that translates into is that when I worked for a company making CNC equipment that I learned how to use a mill and a lathe manually. Only when I understood how you work metal by hand did I reach a level where I could confidently write software to work metal by numerical control.

Working for a manufacturer of sails I soaked up piles of books on sail making and sailing and I actually spent some time on a boat. Half the library on that subject was digested for breakfast, lunch and dinner. Add another helping on fabrics and their properties and on stitching and cutting techniques! Knowledge like that helps a lot when you have to lay out your ‘panels’ on the cloth, and then afterwards when those panels have to be combined to form a sail. In that particular case I took great care to make sure that the panels had their major lines of stress run along the threads in the weave, even if that meant that we might get one or two fewer panels from a piece of fabric the resulting sail would be stronger and it would last longer. Quantity or quality is a business decision and if you’re not aware of this then you might accidentally make that decision for your employer without anybody being aware that a decision has been made. The software used the exact same terminology that the sailmakers used, and used it consistently.

When working for a company that did colour calibration software and pattern checking I read and learned about colour spaces and imaging hardware. All kinds of history there, from manual inspection to semi automatic inspection all the way to the present day. Industry is funny that way, you start out with a simple assignment (can you read values from this sensor) to sucking you in into a project that has a century of history behind it with lifetimes of domain knowledge that you need to absorb before you can even begin to properly automate the solution to some problem. It is also funny in that it all ends with knowledge about physics.

If there is a message in this blog post then it probably is that if you are currently working on a project that requires domain knowledge but you don’t have any of that knowledge yourself that you should probably take the shortcut of stopping your work and to first educate yourself about the field that you are operating in, and then to return to whatever problem you’ve been assigned to solve.

Programming without domain knowledge for a discipline that is steeped in it is like an architect designing a building without knowing its true purpose.

If you are working in some industry spend time on the shop floor, become a user of your software if there is any opportunity to do so. It will make you a far more effective programmer and it will reflect itself in the usability and the perceived quality of your software. The combined knowledge about the domain you are operating in and your IT knowledge is far more valuable than any of the two in isolation.

Domain knowledge is not optional, it is essential. The one common factor between the failed projects I come across is that they were automated by people that had little love for and little or no knowledge of the domain they were operating in. Companies and institutions should make time available to get their IT staff up to speed on the inner workings of the company and should allow their IT staff to be embedded on the floor for a taste of what the real world looks and feels like.

So stay clear of the trap of little or no domain knowledge and get your hands dirty, learn to see through your users eyes by acquiring the knowledge associated with the field you are working in, don’t treat IT as a world that only communicates with its users through tickets and requirement documents, step down from that ivory tower.

How to Make a Quick Buck

In 1992 I had bought a license to a little known real time micro kernel based operating system called QnX.

QnX was a pretty interesting development for me, it was the first time that I used an OS that used message passing for its internals and that ran the file system, the network system, the scheduler and all other i/o as separate processes. Quirky, but interesting and an opportunity to learn. Because QnX was mostly used in Industry I was a pretty strange customer, a single person shop that bought a license stood out as an oddity for the importer of the software, LS Technologies in the dutch harbour town of IJmuiden.

After a couple of visits there to talk over strategies on how to solve certain problems using QnX, one fine monday morning I got a reasonably panicked phone call from the owner of the QnX import business, his brother had a bug they couldn’t solve and he thought maybe I should go there and have a look.

So I drove to IJmuiden in my rickety old car (a 25 year old Citroen DS that was just as likely to deliver you at your destination as it was likely to get you stranded), and found myself in front of an office building at the waterfront.

Housed there was a company that brokered cargo capacity for a very large percentage of the worlds shipping companies. If you wanted to ship 12 containers from Shanghai to Boston you made a request to your shipping broker. He would then send a telex to the company in IJmuiden which would then fan out your telex to a few hundred shipping companies within a specified amount of time. That telex would then be responded to within another specified amount of time and the bid criteria were evaluated and the results would be sent back to the broker. On a good monday morning a million or so telexes would not be an exception.

After arriving there I was ushered into a room with a pretty tense atmosphere. Four guys in suits stood behind a single man behind a computer typing frantically at the keyboard, sweating profusely (and it wasn’t all that warm). ‘Who is Jan?’, I ask.

The guy behind the keyboard looks up and says ‘You’re the guy my brother sent, the ‘whizz kid’?’.

So he jokingly tells me that his brother suggests I can fix this, he doesn’t believe it, but he’s willing to give me a chance if I can fix it in the next five minutes. It’s pretty clear that he feels he’s wasting his time and that he wants to get back instantly. The four suits stand aside for a bit so I can get closer to the screen and the keyboard. So I ask him to explain the situation.

He goes: “it’s worth 2500 guilders on the spot if you solve this problem we’re having, this morning at 8 all our systems froze. Yesterday we did a huge update, tested everything and it looked pretty good. We have our own home built message switch on top of QnX, the whole thing is a large constellation of C programs, yesterday it worked flawless, now it is a total lock up, and we can’t find the cause”.

So I spent a few minutes interviewing the guy and then did one thing, I typed ‘date’ on the command line. And it came up with yesterdays date, whereas all the messages that are being queued are set to be transmitted today… The system took each and every message as being embargoed. What had happened is that someone unplugged the box and plugged it back in again. Somehow the bios clock must have been set to the wrong date & time (no NTP) and when it came back up again it simply used the wrong date. Whoever installed the machine set the date properly the last time after boot but never bothered to update the system hardware clock.

The four guys in suits were laughing pretty hard and relieved. ‘Quick money!’, one said, and another started teasing Jan about having to pay up and make good on his promise.

They were so focused on the big software change they had pushed yesterday that they completely forgot the most basic check. So he went to the bank, got me 2500 guilders in cash (I’d never even seen a thousand guilder note before that day) and gave it to me saying ‘win win’.

Good guy. No, great guy. We ended up working together for years, I became lead dev on the project at some point and we are still in contact, though he is retired now.

When you’re looking for a bug it’s terribly hard to not get bogged down in details at the expense of seeing the bigger picture. An outsider doesn’t have those details so that helps tremendously. The lesson I took away from that day is that when I’m stuck I ask an outsider.