10 Days of Errors

The location was a server room, somewhere up on the 4th or 5th floor of an office in Portsmouth (I think it was) by the docks. One day the main Unix DB server fell over. It was rebooted but it happily fell over again and again, so they called out the support company. So, the support gadgie – Mark I think his name was, not that it matters – gets there a few hours later. Leeds to Portsmouth, you see… it’s a long way. He switches on the server and everything works without error. Typical bloody support really, client gets upset over naff all. He goes through the log files and finds pretty much nothing that would make the box fall over. Mark then gets back on the train and heads back up to Leeds after a pointless waste of a day. Later that evening, the server falls over. Exactly the same story… it won’t come back up. Mark goes through all the usual remote support stuff but the client can’t get the server to run. The pattern continues for a few days. Server working, then after about ten hours it falls over and won’t run for another two hours. They checked the cooling, they checked for memory leaks, they checked everything but came up with nowt. Then it all stopped. Then, a week without problems. Everybody was happy… until it started again. The same pattern: 10 hours on, 2-3 hours off. And then somebody (I seem to remember he said that the person had nothing to do with IT) said: “It’s high tide!” Which was met with blank looks and probably a wavering hand over the intercom to Security. “It stops working at high tide.” This, it would seem, is a fairly alien concept to IT support staff, who are not likely to be found studying the tide almanac during the coffee breaks. So they explained that it couldn’t be anything to do with the tide, because it was working for a week. “Last week was Neaps, this week it’s Springs.” Here’s a bit of jargon busting for those of you who don’t have any RYA qualifications. The tides run on a lunar cycle, and you get high tide every 12.5 hours as the Earth turns. But as the moon’s orbit changes, so does the difference in high and low tide. When the moon is between us and the sun or on the opposite side of the planet, we get “Springs”. These are the highest highs, and the lowest lows. When there’s a half moon, we get “Neaps”. The difference between high and low is greatly reduced. The lunar cycle is 28 days, so Springs – Neaps – Springs – Neaps. Sure enough, they were right. Two weeks previously, a Navy destroyer or something had moored up nearby. Every time the tide got to a certain height, the crow’s nest was in direct line with the floor that the servers were sitting on. It seems that the radar (or radar jamming, or whatever the military have on their toy dinghies) was playing havoc with the computers.

The Second Life desktop app for Windows had a built-in updater that worked in the simplest way possible: it would download an updater program from a pre-set URL on the Second Life website, save it to the user’s hard drive as updater.exe, and run it. One day, for reasons lost in time, requests to that URL returned a 404 Not Found page. Perhaps the server was misconfigured, or the file was deleted… who knows? These things happen to every website. In most situations like this, the user trying to access that page is a human who gets annoyed and does something else, and that’s the end of the story. But when the user is a program, the outcome is less predictable. In this case, the desktop app didn’t check what it was getting. It just downloaded the 404 page, saved it as a program updater.exe and ran it. Or rather, it used a system call to tell Windows to run it. Windows looked at updater.exe and immediately saw that it wasn’t in the standard format for .exe files. But instead of just quitting with an error, this system call decided to try harder. Windows is known for its dedication to legacy support: you can still run programs from back in the MS-DOS days, ones that had the suffix .COM. Unfortunately, this system call didn’t check what it was getting, and ran updater.exe as if it was a .COM program. Binary data is binary data, and can be interpreted in infinite ways. It seems that – for entirely accidental reasons – when the Second Life website’s 404 page is interpreted as a .COM file, it can be read as a set of instructions to open the computer’s LPT device and yell gibberish into it. Which is what it did. Those of you old enough to have used MS-DOS may remember that LPT is the route to the printer. Windows, once again, went the extra mile to help legacy DOS software work. It received the gibberish spewed by the 404-page-run-as-a-program, didn’t check what it was getting (you may be sensing a theme here), and helpfully sent it along to whatever printer it knew about. Fortunately, plenty of printers do check what they’re getting! Unfortunately, that still leaves plenty that don’t, especially if they’re cheap inkjet printers from the 90s. And those printers went wild. In Mark’s own words: “… they would freak out, spew paper, and in one case, physically break.”

It was a dark and stormy night, and all was still in the Etsy office… save for an eager developer trying to ship a change on their first week at the company. They were deleting a harmless, unused style sheet — originally needed to support the ancient IE6 — and all the files that included it. Tested out on staging, everything looks good! Tests pass! Time to ship it. But lo and behold, there was a bug! A race condition on deploy meant that some of the updated pages weren’t deployed properly, and the old versions tried to include the now non-existent CSS file, which threw an error. This wouldn’t have been so bad, except the error page also included the aforementioned, not-so-harmless CSS file. So errors begat errors begat errors, resulting in a hard loop on every Apache process on every box. The looping Apache processes locked up the servers so hard that they couldn’t be restarted remotely. Some poor Etsy employee had to go to the datacenter and reboot the servers manually.

The task was to set up an e-commerce store that sold industrial parts for a mine site, think Grainger catalog, but for civic corporations. The development team came together, split up the tasks and went to work building. Pretty straight forward project; a catalogue of items, authentication and a checkout with credit card. The team finished the project and felt pretty good about it. They shipped it and were proud that the project went so smoothly. Fast forward 18 months after shipping the website, when a bug was discovered. The developer was worried about real charges being incurred while the credit card payments were being tested. So their code checked if the environment was set to dev in which case, regardless of the amount calculated at checkout, no actual transactions would be sent to the credit card processor. When the project was shipped, that environment flag was never changed to prod, so even though everything looked good on the customer end for adding items to their cart, checking out and receiving the invoice for the correct amount, the credit card was never charged for the said amount. Instead, it consistently charged the customer a whole amount of ZERO dollars. By the time this was discovered, they estimated it was something like 2 million dollars of goods that were shipped and sold, with no payments collected.

10 years ago, I was working for F5 Networks as a Solution Architect. Our hardware had been deployed at an ISP to improve web performance for mobile users. Every day at approximately the same time the internet stopped working for about 5 minutes, and system processes were restarting. I was sent on-site for the day to troubleshoot. As the day wore on I realized the quick day trip was turning into an overnight. All diagnostic and troubleshooting was showing no problems, the problem occurred at 3am. I had to be on-site to see what was going on. A last minute hotel was booked, and I went shopping for a change of clothes. I got a few hours sleep so I could be on-line at the witching hour to see what was wrong. Sure enough as expected, the system failed. Within 5 minutes everything was working again. Everything was fine with the hardware. I went sleuthing on the internet, armed with the timestamps of when the failures were occurring. None of the standard searches were resulting in a solution. Finally, I had an epiphany! Eureka! With some more in-depth research, I realized the problem. The outages were aligned with reported solar flares in the region. The flares were knocking out satellite connectivity taking our systems offline. When the flare passed the systems came back on-line.

As IMified grew, most Monday mornings we’d have an unexplained crash of our core XMPP server. We could see there was lots of network traffic around 9am PST, but it was encrypted and commercial XMPP servers didn’t give tools to analyze at any level of detail. Solution? We built our own XMPP server to instrument and analyze the traffic. And then we saw the source of the traffic. Presence packets. 9am PST is a magical time: West Coast workers sign on, East Coasters leave for lunch, Europeans leave work for the day, and finally, it’s bedtime in India. For an instant messaging platform, this is the time when the most people change state around the world. So the question remained: why Monday? We got just as many packets other weekdays, but spread over a larger amount of time. Our theory was that people seemed a bit more likely to be on time for work on Mondays. So to help combat this traffic spike, we made our XMPP server fork presence packets to another server that just served a static bit of xml for the XMPP packet. Voila! We were online and available. Always.

Computer-controlled characters in video games have behavioral logic so they can react appropriately to player actions. But this reactiveness is deliberately limited so that characters don’t react to everything; they should only react to the kinds of events that real people would react to in real life. Imagine a shopkeeper character: it shouldn’t care about the player moving objects around in the house next door, but it should act when the player takes something from the shop without paying. In most games, characters only react to events that occur in their immediate vicinity. More advanced games limit this reactiveness to events that happen in front of the character; this is important in stealth-based games so that players can, for example, sneak quietly behind a guard. For Skyrim, Bethesda took this even further: characters react to events which they can see with their eyes. It means that characters can turn their heads to notice more events. Unfortunately, the Skyrim developers didn’t implement the ability for characters to realise when their head is covered with a bucket, put there by a player who knows it’s the easiest way to steal things from shopkeepers:

To enjoy this bug: find a heavy bucket. Stand on it, in exactly the right place. Then grab the bucket. Now you’re up, up and away! We couldn’t find a definitive explanation. One speedrunner suggested that it’s caused by the game’s faulty implementation of Newton’s third law of motion. We find that unlikely, but the outcome of this bug is so bizarre that we can’t rule it out entirely.

One of the most valuable product categories in the Second Life (SL) economy is virtual pets. There’s enough profit in selling digital animals and their supplies to sustain several small real-world businesses. Take, for example, the tragic story of Ozimals, a company selling virtual rabbits which met with a messy legal end. Among the other kinds of virtual pets in SL were Arabian horses. Just like the bunnies, you had to buy them food. (The food is where most of the revenue lives in the virtual pets business. It’s just the Xerox model made weirder.) You’d put the food out, they’d find it and eat it, and some time later they’d want more. The “find it” part of the previous sentence is key to this story. For the horse to get to the food, they couldn’t just move in a straight line as there might be obstacles in the way. They had algorithms for finding a route around those obstacles and propelling themselves to the food. In video game terminology, this kind of algorithm is known as pathfinding. The SL developer platform now has built-in pathfinding functions which any developer can use, but they were added after this story took place. Back then, the horses used custom pathfinding code written by the horse creator. And, like most code written in SL’s haphazardly-designed scripting language, it wasn’t very good. As mentioned earlier, almost all the content in Second Life is created by its users. This includes tens of thousands, maybe hundreds of thousands of unique scripts, embedded in hundreds of millions of virtual objects. These scripts are executed by the underlying platform code, maintained by Linden Lab, and updated constantly. When there are new releases of the SL platform, it isn’t fully tested with all the scripts that content creators write. There are far, far too many to do that. One release had a tiny tweak to the physics engine, related to friction of objects moving on the ground. You may guess what’s coming next. The horse pathfinding logic was using the old friction rules. As soon as the SL region code updated, horses started skating past their food. In some cases, horses living on high-altitude platforms started falling off them. (I imagine them whinnying as they pirouetted into space.) This all seems really comical until you realise how many people owned these horses, and how much they’d spent on them. Within a couple of hours of the code going live, staff realised that user possessions worth tens of thousands of US dollars were being destroyed by a bug. (Yes, US dollars. Not Linden dollars. Just in virtual horses. You have no idea how much the SL economy is still worth.) Once the size of the problem was realised, the SL platform release was rolled back. But that still took several hours over the thousands of servers. (No feature flags, see.) So, during the rollback, several dedicated QA engineers stayed up much of the night, saving virtual horses from starving to death.

Back in the mid-90s, two middle-aged finance executives, Kenneth Pasternak and Walter Raquet, saw NASDAQ going digital and said: what if the traders were digital too? What if it was just computers? They could do thousands of trades a minute.  It was 1995 when they founded Knight Trading Group (later renamed to Knight Capital Group) with a bunch of smart engineers and immediately started making ridiculous amounts of money. Within three years they had gone public, their IPO (Initial Public Offering) raising $145 million on top of a market capitalization of $725 million. It took only 18 months for their market capitalization to grow over ten times as big. However, those who remember the period know what came next: the dot-com bubble burst, and the Knight Capital rollercoaster ride went screaming downwards. On top of the losses, Knight was caught breaking many stock market regulations and doing naughty things like intercepting customer trades so that the firm could profit off them first. As Knight floundered, the founders who oversaw these problems were replaced. Thomas Joyce, an experienced industry leader, took over as CEO and returned the firm to financial success. By 2011 they had a net income of $115 million and over 100 software developers working on Knight’s future: an overhauled, optimised version of its trading systems. The new technology was known as Smart Market Access Routing System (SMARS). When launched, it accelerated Knight Capital’s transactions to thousands of trades per second. In 2012, the New York Stock Exchange (NYSE) announced the Retail Liquidity Program (RLP), a new private market for stock trading. The RLP was highly controversial because it would allow big market players to trade for fractions of pennies above or below the displayed prices. For big players, those fractions of pennies would rapidly add up to millions of dollars in profits. Knight Capital, being a huge player, had to be in the RLP. But first, they had to tweak SMARS to work with the new market. Thanks to the controversy around it, the SEC had taken a while to approve the RLP. When the approval was announced in June 2012, the NYSE followed up with a rapidly approaching launch date: August 1st, 2012. The trading firms had just over a month to prepare. Knight’s engineers scrambled to make SMARS ready for the RLP. It needed a set of new code to handle the RLP’s rules, which were different from the rest of the NYSE markets. For SMARS to know whether to use the standard NYSE rules or the new RLP rules, they repurposed an existing feature flag that was previously used for some old testing routines. If this flag was enabled, SMARS would know that it was trading on the RLP.

Back when Knight Capital was creating its initial trading algorithms – the ones that gave it such a massive lead in the market – it wanted to test them in a kind of virtual stock exchange; like a video game version of the stock market. But Knight’s own trading algorithms wouldn’t be enough to simulate market activity, because they also wanted to test how the algorithms would fare when the market behaves weirdly. Their virtual market neededotheralgorithms that would do irrational things. For example, they needed a market player that would buyhighand selllow, which is exactly the opposite of what a sensible player would do. They created this exact algorithm and called it Power Peg.

Yep.

They did. However, this deletion happened at the same time as the RLP code was added. Power Peg hadn’t been used since 2003, yet lingered in the code ever since.

Nope.

But in this case, the Power Peg code was eliminated in the new version of SMARS. So as long as the previous version was completely removed from the production servers, there would be no danger of accidentally triggering Power Peg. The new SMARS code was completed a week before RLP’s launch. Knight deployed the new version of SMARS to its servers. However, Knight’s infrastructure team was still doing deployments manually. The engineer responsible for the deployment had to install the new code across eight servers. Unfortunately, this engineer missed one of the servers, and it still contained the old code. Knight’s infrastructure team had no standard process for verifying that code deployments were correct, so nobody noticed.

Yep. At 8:00am on August 1st, SMARS switched on to handle pre-market orders, and immediately started sending automated warning emails to Knight’s engineers. These emails, containing the error message “Power Peg disabled”, should have been enough to warn staff that something was going wrong. But few engineers noticed the emails, and none of them realised the danger.

At 9:30am, the new RLP market opened for business. Power Peg, still present on one production server, flew into action and started spending the firm’s money in the stupidest way possible, as fast as possible. Within a couple of minutes it had performed more trades on certain stocks than they would usually see in a month, and the total volume of trading was 12% higher than usual. At 9:34am, NYSE engineers investigating the trading spike traced it back to Knight Capital. Knight’s own engineers hadn’t noticed the extent of what was happening; they could see the higher trading volume, but didn’t realise that SMARS was now shoving Knight Capital’s market position into serious debt. Somehow, Power Peg’s activity evaded all their internal alarms other than the aforementioned warning emails, which had still gone unnoticed. By 9:45am, Knight’s engineers had looked at the problem, conferred, and leaped into action.

Nope. SMARS, for some reason,didn’t have a kill switch.

The engineers believed that the dangerous trading was due to the new RLP code. They decided to immediately roll SMARS back to the previous version.

Yep. Now, all eight servers had Power Peg enabled, churning away, trading as dangerously as possible. It wasn’t until 9:58am, that Knight’s engineers finally traced the problem to Power Peg and shut it down. But in that time, it had purchased $7 billion worth of shares. By the end of the day, Knight had frantically reduced it to $4.6 billion, but it was still far, far more than Knight were allowed to hold, because they didn’t have the cash to cover it. In the end, Knight Capital made a loss of about $440 million on Power Peg’s trades. The company immediately lost most of its biggest clients, its reputation destroyed. Knight was acquired a few months later by a rival trader for less than a third of its previous share price.