In 1992 I had bought a license to a little known real time micro kernel based operating system called QnX.
QnX was a pretty interesting development for me, it was the first time that I used an OS that used message passing for its internals and that ran the file system, the network system, the scheduler and all other i/o as separate processes. Quirky, but interesting and an opportunity to learn. Because QnX was mostly used in Industry I was a pretty strange customer, a single person shop that bought a license stood out as an oddity for the importer of the software, LS Technologies in the dutch harbour town of IJmuiden.
After a couple of visits there to talk over strategies on how to solve certain problems using QnX, one fine monday morning I got a reasonably panicked phone call from the owner of the QnX import business, his brother had a bug they couldn’t solve and he thought maybe I should go there and have a look.
So I drove to IJmuiden in my rickety old car (a 25 year old Citroen DS that was just as likely to deliver you at your destination as it was likely to get you stranded), and found myself in front of an office building at the waterfront.
Housed there was a company that brokered cargo capacity for a very large percentage of the worlds shipping companies. If you wanted to ship 12 containers from Shanghai to Boston you made a request to your shipping broker. He would then send a telex to the company in IJmuiden which would then fan out your telex to a few hundred shipping companies within a specified amount of time. That telex would then be responded to within another specified amount of time and the bid criteria were evaluated and the results would be sent back to the broker. On a good monday morning a million or so telexes would not be an exception.
After arriving there I was ushered into a room with a pretty tense atmosphere. Four guys in suits stood behind a single man behind a computer typing frantically at the keyboard, sweating profusely (and it wasn’t all that warm). ‘Who is Jan?’, I ask.
The guy behind the keyboard looks up and says ‘You’re the guy my brother sent, the ‘whizz kid’?‘.
So he jokingly tells me that his brother suggests I can fix this, he doesn’t believe it, but he’s willing to give me a chance if I can fix it in the next five minutes. It’s pretty clear that he feels he’s wasting his time and that he wants to get back instantly. The four suits stand aside for a bit so I can get closer to the screen and the keyboard. So I ask him to explain the situation.
He goes: “it’s worth 2500 guilders on the spot if you solve this problem we’re having, this morning at 8 all our systems froze. Yesterday we did a huge update, tested everything and it looked pretty good. We have our own home built message switch on top of QnX, the whole thing is a large constellation of C programs, yesterday it worked flawless, now it is a total lock up, and we can’t find the cause”.
So I spent a few minutes interviewing the guy and then did one thing, I typed ‘date’ on the command line. And it came up with yesterdays date, whereas all the messages that are being queued are set to be transmitted today… The system took each and every message as being embargoed. What had happened is that someone unplugged the box and plugged it back in again. Somehow the bios clock must have been set to the wrong date & time (no NTP) and when it came back up again it simply used the wrong date. Whoever installed the machine set the date properly the last time after boot but never bothered to update the system hardware clock.
The four guys in suits were laughing pretty hard and relieved. ‘Quick money!’, one said, and another started teasing Jan about having to pay up and make good on his promise.
They were so focused on the big software change they had pushed yesterday that they completely forgot the most basic check. So he went to the bank, got me 2500 guilders in cash (I’d never even seen a thousand guilder note before that day) and gave it to me saying ‘win win’.
Good guy. No, great guy. We ended up working together for years, I became lead dev on the project at some point and we are still in contact, though he is retired now.
When you’re looking for a bug it’s terribly hard to not get bogged down in details at the expense of seeing the bigger picture. An outsider doesn’t have those details so that helps tremendously. The lesson I took away from that day is that when I’m stuck I ask an outsider.