Bitwise Read online

  Code Work

  Anyone who has had the misfortune to write his first computer program remembers the humiliation in conversing with a servant or master that insists on a language unworthy of the dullest of intelligences and the lowest of men.

  —WAN-LEE YIN

  At Microsoft, as at Google, I worked in C++, a language that exudes compromises between efficiency and elegance. They are often ugly compromises, but in software engineering, an ugly compromise trumps beautiful purity every time.

  C++ is a significant step up from assembly, but considerably closer to the hardware than most other commonly used languages (including BASIC and Logo), which means that C++ is hard to beat for sheer speed—crucial when you’re writing a server and your code has to execute on thousands of machines simultaneously. A 10 percent speedup can mean a 10 percent decrease in capital expenditure.

  C++ is a “mid-level language,” in between the low-level language of assembly and high-level languages like Java and Python. The higher the level of the language, the less control it provides. Assembly lets a programmer specify exactly where in the CPU every bit of information is going. Higher-level languages offer less control over the guts of the computer, which is managed by compilers, interpreters, and virtual machines. These programs are exceedingly good at managing things automatically. They don’t make mistakes (unlike humans), but they do have their limits. In particular, they cannot understand the overall intent of a program, like an automatic transmission, which can’t quite match the manual transmission for performance because it can’t anticipate the driver’s intent. If you pile on too many abstraction layers and automated tools, performance suffers. The downfall of Microsoft’s notorious Vista operating system came about because the majority of the code was written in a new language of Microsoft’s own design, called C#. Like Java, C# was considerably higher level than C or C++, and the code responsible for taking care of the lower-level nastiness didn’t perform well enough. So Microsoft had to scrap the C# code and started over in C++, costing them an extra two years of work. Lesson learned.

  Any computer language like C++, Java, or Python consists of a certain number of commands (like repeat in Logo or GOTO in BASIC), often not more than a few dozen, and a certain number of numerical and logical operators, like + and &&. Many languages offer similar command sets; where they differ is in the methods they provide for structuring programs and in the amount of abstraction they provide from the underlying computer fundamentals.

  Early on in my time at Microsoft, the code was all that mattered to me. The Messenger client was the first piece of shipped, professional code I worked on. The Messenger client at Microsoft at the time was about a hundred thousand lines of code, all in C++. These implemented the display of the user’s list of contacts, online and offline; pop-up notifications when a buddy logged in; uninstallation code to remove the program if people hated it; IM archival functions; connections to the Messenger servers (and, for a while, the AOL servers); and more. In its early versions, it was an efficient, lean little program.

  I no longer have access to the Messenger code, which remains the private intellectual property of Microsoft. Instead, here is a piece of the open-source C code for the chat program Pidgin. This function, update_typing_icon, is called to update the “typing indicator,” which pops up to show when a buddy is typing out a message. I built this feature into the Messenger client, and I still have a fondness for it.

  The function takes a parameter called gtkconv that contains information about the chat session (PidginConversation) being updated. It calls another function called purple_conv_im_get_typing_state, passing it to the chat session in question. That function then returns one of three possible values: PURPLE_NOT_TYPING, PURPLE_TYPING, or PURPLE_TYPED. (“Purple” is the name of the core chat library interface. Coders use silly names.) A user interface function, update_typing_message, then changes what message is displayed on the screen. In the case of PURPLE_TYPING, a message with “[Buddy name] is typing” is shown. If PURPLE_TYPED, meaning that text has been entered but your buddy hasn’t typed anything for a bit, “[Buddy name] has stopped typing” is shown. And if no text has been entered and the buddy isn’t typing (i.e., PURPLE_NOT_TYPING), then no message is shown at all.

  update_typing_icon calls other functions like update_typing_message and purple_conversation_get_title. Most of these other functions are also part of the Pidgin program, separated into modular chunks so that each can be isolated, tested, and perhaps reused. One exception is the g_strdup_printf function, which creates the text string containing the message to be displayed. g_strdup_printf is part of the open-source GNOME user interface library. It is sufficiently generic to be of use to many programs, not just Pidgin.

  All this C code is compiled into assembly by a C compiler. Microsoft had its own, marketed as a part of Visual C++, while there also exists the popular, free gcc compiler, which I used at Google. The resulting executable file of assembly code can then run natively on the processor for which the compiler was designed. And there you have yourself a chat client.

  The Buffer Overflow

  Progress is the exploration of our own error.

  —JACOB BRONOWSKI

  The Messenger war was a rush. Coming in to work each morning to see whether the client still worked with AOL was thrilling. If it wasn’t, I’d have to look through reams of protocol messages to figure out what had changed, fix the client, and try to get an update out the same day. I felt that I was in an Olympic showdown. I had no idea who my adversaries were, but I had been challenged and I wanted to win. Our users cared too. They wanted us to win.

  AOL tried different tactics. At one point I suspected that they identified the Microsoft client because it wasn’t downloading the advertising that the AOL client downloaded. I updated our client to download it all and then throw it away. AOL included mysterious protocol messages that didn’t seem to affect their client but broke ours. I fixed that. One day, I came in to see this embedded in the messages from the AOL server: “HI. —MARK.” It was a little missive from engineer to engineer, hidden from the corporate, media, and PR worlds that were arguing over us. I felt solidarity with him even though we were on opposing sides.

  AOL was publishing propaganda about how Microsoft was behaving like an evil hacker by asking for AOL passwords. This wasn’t true, but we were allowed to respond only through our PR department. My team was sealed off—but our code wasn’t.

  Then AOL stopped blocking us. It was strange to encounter sudden silence, as though the enemy had abruptly yielded the battlefield, and while I wanted to believe we’d won, I suspected AOL wouldn’t give up without a word.

  A week later, we found that Messenger had been blocked again, but this time was different. The AOL server was sending a huge chunk of new gobbledygook that I could not understand. It looked approximately like this:

  The first couple of lines are the standard AOL instant message protocol header. But from 90 90 90 90 onward, it’s incomprehensible, bearing no relation to anything the AOL servers had sent their client or ours. The vast expanse of double zeros in the middle was also mysterious, since a bunch of zeros couldn’t contain much meaning.

  Our client ignored it, but the AOL client responded to this gobbledygook with a shorter version of the same gobbledygook. I didn’t know what it was. It was maddening. After staring at it for half a day, I went over to Jonathan, a brilliant server engineer on our team, and asked what he thought. He looked at it for a few minutes and said, “This is code.” It was actual x86 assembly code. The repeated 90s tipped him off: they signify a “no-op” no-operation instruction in x86 assembler, telling the processor to do nothing for one cycle.

  The pieces then came together. Normally, protocol messages sent from the server to the client are read as data, not as code. But AOL’s client had a security bug in it, called a buffer overflow. A buffer is a place in mem
ory where a program temporarily stores data during execution. It’s all too easy in lower-level languages to allow in more input than the buffer can accommodate. In this case, large protocol messages flooded over the end of the buffer. In the computer’s memory, the locations past the end of the buffer are often filled with things that aren’t just stored data. The actual assembly code to be executed is often nearby. This buffer overrun could overwrite the client code currently being executed and control the functioning of AOL’s program.

  It’s a huge security hole, since it gives the server control of the client PC. In the wrong hands, the server can shut down or spy on a computer. AOL knew about this bug in their program, and now they were exploiting it! The double zeros were filling up space in the program’s buffer so that once the large message hit the end of the AOL client’s buffer, it would overwrite executable code with the remainder of the protocol message. The remainder of the protocol message contained new code to be executed, which the client promptly did. AOL caused the users’ client to look up a particular address in memory and send it back to the server.

  This was tricky—vastly trickier than anything they’d done so far. It was also a bit outside the realm of fair play: hacking into their own client using an unfixed security hole—a hole our client didn’t possess. A “Rommel, you magnificent bastard” moment. I was out of my depth.

  Someone else at Microsoft—I never found out who—told the press about the buffer overflow, figuring that if people knew that AOL’s client had a huge security hole in it, AOL would be forced to patch their client and would no longer be able to exploit it.

  According to security expert Richard M. Smith, a certain “Phil Bucking” of “Bucking Consulting” sent him a message, alerting him to the buffer overflow in the AOL client:

  Mr. Smith,

  I am a developer who has been working on a revolutionary new instant messaging client that should be released later this year. Because of that, I have followed with interest the battle between AOL and Microsoft and have been trying to understand exactly what AOL is doing to block MS and how MS is getting around the blocks, etc. Up until very recently, it’s been pretty standard stuff, but now I fear AOL has gone too far.

  It appears that the AIM client has a buffer overflow bug. By itself this might not be the end of the world, as MS surely has had its share. But AOL is now *exploiting their own buffer overflow bug* to help in its efforts to block MS Instant Messenger.

  Getting the name of MSN Messenger Service wrong was a nice touch, but the rest of it is rather ham-fisted. This developer of a revolutionary new app takes sides in the Microsoft-AOL war without promoting his own app? The email also includes a trace of the buffer overflow message itself, which I remember vividly from the hours I spent staring at it. If Phil Bucking’s text—and his name—weren’t suspicious enough, he’d also sent the message (via a Yahoo account) from one of Microsoft’s computers at a Microsoft IP address. The IP address showed up in the email headers. Microsoft’s digital fingerprints were all over the email.

  Smith accused Microsoft of sending the email. Microsoft fessed up. The news story that emerged covered Microsoft’s attempt to bad-mouth AOL under a fake identity—an easier sell than explaining the buffer overflow. People on various security forums ascertained that the buffer overflow was real and inveighed further against AOL, but the press wasn’t paying attention. The buffer overflow persisted into several later versions of AOL’s client. AOL never admitted a thing, and the press never did understand it.

  We gave up. I licked my wounds. I switched to the server team, which I preferred to the client, but working with MSN Messenger Service gradually became dreary and politicized. Those were the years of Microsoft’s long, slow decline (which reversed only when Steve Ballmer was removed and replaced by Satya Nadella as CEO in 2014). Bureaucracy and overhead ballooned. Getting approval for any idea required running a gauntlet of meetings with management. The infamous “stack rank” review system pitted teams and individual engineers against one another. There was an incredible thirst for “headcount” within divisions, so managers would lobby aggressively to move independent groups under their control. The ambitious and forward-looking NetDocs, an internet-based document-editing suite, gobbled up a number of small groups in the late nineties. But then NetDocs got eaten by Office, which killed NetDocs before it released. The market was open when Google debuted its own online word processor, Google Docs, in the mid-2000s. And so it went. Multiyear projects with hundreds of engineers died without the public ever hearing a word.

  On September 11, 2001, my wife was away on a trip. She woke me up early with a phone call to tell me about the World Trade Center attacks. I struggled to process the news with my foggy brain, then drove to work. The televisions in the lobby were broadcasting MSNBC’s reporting. A few people stood in front of them, but no one was talking about it. Employees were quieter than usual, but the endless parade of meetings otherwise continued unchanged. It was the first hint I had that the heads-down mentality at Microsoft was removing its employees from other important aspects of the world. Complacent from success, Microsoft had turned inward and was attacking itself in much the same way one’s immune system can attack healthy cells in the absence of external threats. The ecology had gone wrong.

  During the bad days, I got to see how average people performed under conditions that encouraged them to behave unethically and dishonestly. Some stuck to their principles, trying to do good work on the hope that rationality and good arguments would prevail over groupthink and nepotism. Others caved, sacrificing their integrity in order to line up behind some manager’s empty rhetoric or secretly playing both sides in political disputes. I saw old boys protecting and boosting their own. I saw cowardice and deceit. When my team opposed the impossible ambitions of an ambitious project manager, he responded by creating another, hand-picked team to take over our work and chase his dreams. (It didn’t work out.)

  For my part, I didn’t betray my principles, but I failed to defend them against corporate inertia. My greatest regret is toward the people I managed, whom I was not able to protect from corporate caprice. Like many low-level managers, I had more responsibility than actual authority, leaving me the deliverer of bad news I couldn’t control. It’s a dilemma I have not managed to answer to this day. At the age of twenty-four, I had no idea at all. My direct reports, all of them good people, got lower review scores than they deserved, and I told them that they were doing well even as I wanted to tell them that the system had gone rotten. The stress turned me brittle and harsh. I gave others too hard a time, to say nothing of myself.

  I endured through the friendship of a couple others on my team. We reassured one another that it was the system that was crazy, not us. We were right; we all became much happier after we left. The personal and professional wounds of corporate life may not sound dramatic, but the power an employer has to define a person’s identity should not be underestimated. When I left Microsoft and went to Google, corporate paranoia lingered over me for months, until the healthier and less cutthroat environment gradually subsumed my emotional miasma.

  After a year or two of ongoing malaise, bitter jokes, and festering anxiety, I quit Microsoft for a job at Google. My boss threatened me with litigation for leaving. I packed up my things in a box and walked out the same day. I heard similar stories from other engineers who left for Google. When Windows architect Mark Lucovsky took a job at Google, Steve Ballmer allegedly threw a chair across the room (or at least shook it). Microsoft was hemorrhaging hundreds of top engineers to Google, and the combination of the talent loss plus the insult to the executives’ egos made for very bad blood.

  Despite the toxic politics, MSN Messenger Service did pretty well. Messenger acquired tens of millions of users. Millions used Messenger at any one time. I added emoticons in 2000—it was the first American chat program to turn a colon and a close-parenthesis into a smiley face (t
he South Koreans may have preceded us)—and people loved it. We added internet phone calls to the client. I helped redesign the server architecture, which was the greatest technical challenge I’d faced. I was glad to have worked on it.

  After I left Microsoft, Messenger puttered along for years. When Microsoft’s purchase of Skype rendered it both redundant and obsolete, Messenger was finally retired in 2014. By that point, I hadn’t used it in years.

  The larger lesson of the Microsoft-AOL war was that the workings of code were no longer a private matter known only to those steeped in the lore of computing. They were increasingly becoming public affairs that impacted national and even global events. The chat wars were, to me, a duel between a handful of people within two gargantuan companies, and yet they had become a national news story for a brief period. Looking back, it was the first inkling I had that it wasn’t just computers that were permeating all our lives, but the code itself. Yet few grasped code’s function and its impact—something that is still true today. The ultimate outcome of the conflict resulted from a buffer overflow whose existence was never agreed upon, because neither reporters nor the public understood it.