The Story Behind a Windows Security Patch Recall

bheer writes "Raymond Chen's blog has always been popular with Win32 developers and those interested in the odd bits of history that contribute to Windows' quirks. In a recent post, he talks about how an error he committed led to the recall of a Windows security patch."

The Money Quote

[SixFactor]

You're about to be Slashdotted.

Seriously, it's good to get a glimpse of the interactions in the dev side of MS. It's astonishing that MS even allows this to happen at all. The March 07 Wired had a feature on Channel 9 [wired.com] that humanized the MS organization quite a bit, IMO. It's not just about chair-throwing, marketing hyperbole, and world domination after all... oh wait.

Re:Fascinating

[Timesprout]

Raymond has touched on the complexity of their software before and noted that oftentimes the complexity was not acually a product of the fuctionality but due to fixes, patches and additions to the code over time. To his credit he has in the past admitted that issues similar to this one were introduced because the core problem ie loading faulty shell extensions was not addressed directly for reasons of time/money/too scared to touch it/whatever and the hacks and workarounds only served to pointlessly bloat the complexity of the whole system. It's also worth noting that this complexity creep was not entirely due to MS. They had 10s of millions users with god knows how many applications which the MS dev teams struggled to support with backwards compatability etc. Raymond has admitted in the past that specific checks were put in the OS for certain applications to keep them functioning. Nice if you are a third party developer but just asking for trouble for your OS.

Revealing

[Anonymous Coward]

And points out how their anti-competitive lockin approach has not only bitten them in the ass repeatedly, but only gotten worse as they incur additional scars on the scar tissue. The only reason they have to have their developers struggling to support third party apps is because they have never released proper APIs, and do not follow their own published interface methods when implementing features in Windows. With publicly available, stable APIs and consistent implementation the third party developers could deal with it themselves and everyone would benefit.

Re:Fascinating

[Quantam]

You should read The Old New Thing site (or the book by the same name, which is basically just a cleaned up and edited version of everything on the site). For those not familiar with him, Raymond Chen is THE backward compatibility guy at MS. He and his minions have to find all the badly programmed programs that break when Windows improvements are ready to ship (for examples of just how bad some of these programs are, read the site; little things like walking the stack from a window callback function to find some data value used earlier, etc.), and figure out kludges to make them still work despite improvements to the OS. Not surprisingly, he has a massive amount of knowledge to share about how Windows has become so complex and so warped.

He also is there to give advice on things NOT to do in coding (and yes, he does indeed talk about bad things MS employees have done, although he never refers to exactly where he encountered a particular bad thing he discusses).

Re:An error he committed?

[Helldesk Hound]

> Okay, he made an error. Why the HELL wasn't it caught in QA? Microsoft
> wants us to believe that the reason that we have to wait for patches
> is that they are getting some kind of exhaustive QA.

M$ doesn't have "QA". It has QC.

Quality Assurance is the fence at the top of the cliff that at each stage prevents faults from arising, and thus from impacting on later stages of development.

Quality Control is the ambulance at the bottom of the cliff that responds to the emergency call once the fault has been discovered.

Most places do not implement QA. They frequently have only the most rudimentary QC.

Given the phenomenal number of security flaws and other bug fixes that have to be applied every month (that's every month since WindowsXP was released six years ago), what type of quality management do you think M$ uses in it's development process?

Re:Fascinating

[Chops]

It's not just that -- there's a whole little gang of design flaws responsible here, each of them egging the others on like adolescent boys with dangerous tools at their disposal. To all of the people saying, "well, it has to be that way because Microsoft has 5 billion trillion gazillion apps to support and they're not responsible for third party blah blah blah," I say this: No. Shut up. Linux vendors release updates for a body of software that is a massive superset of what Windows Update covers, often with a tiny fraction of the QA manpower, and problems like this are still quite rare. Why? Well, let's take a look at design flaws that caused the vulnerability Chen was solving:

• Windows handles OS extensions by loading DLLs into Explorer's address space, instead of introducing a layer of separation and interacting with a separate process (a design which would have allowed graceful handling of arbitrary errors in the extension).
• It's apparently easy to accidentally construct code which Explorer believes is a usable shell extension, and likewise impossible to add checks to Explorer such that it will only use, as a shell extension, something which was deliberately intended to be a shell extension. Stop and think about that for a long, long second. That's shockingly shitty design even for Microsoft. Chen blames the application programmers (saying, "lots of people mess up IUnknown::QueryInterface [msdn.com]" and linking to a page containing what appears to be an excerpt from the Nag Hammadi scriptures in the original Coptic), instead of realizing that if professional programmers show a consistent pattern of making particular mistakes interacting with an interface, the interface is probably poorly designed.
• As you mention, this seems like a design flaw in the implementation of threading, DLL handling, and process exit. The same construct in Linux does not hang -- it waits until the thread terminates, and then exits normally:

  #include <pthread.h>
  #include <stdio.h>
  #include <stdlib.h>
  
  volatile int stop_now;
  
  void stop()
  {
  printf("Stopping, sort of.\n");
  while(!stop_now);
  printf("All done.\n");
  }
  
  void *start(void *arg)
  {
  sleep(10);
  printf("Deciding to stop.\n");
  stop_now = 1;
  }
  
  int main(int argc, char **argv)
  {
  atexit(stop);
  stop_now = 0;
  
  pthread_t thread;
  pthread_create(&thread, NULL, start, NULL);
  
  exit(0);
  }

I can feel Chen's pain -- it must have been awful trying to botch around the fallout from the first two stupidities, and then getting screwed by the third. I don't think that lets Microsoft off the hook at all, though. The responsibility for the hacked-together, poorly-planned, teetering heap of an OS that they now have to support (at tremendous cost) lies nowhere but at their own metaphorical feet.

Re:Backwards compatibility

[Peaker]

There's a reason all the important open source APIs (gtk, glibc, alsa, X etc) have "gone stable" in the past 5 years, and it's simply a better approach.

But they only have to maintain source-level compatibility. Microsoft has to maintain binary-level compatibility.

Also, when specific things are extremely and seriously broken, compatibility can be dropped altogether, and some buggy programs broken. Microsoft cannot afford to break buggy programs, even if those are few and far between - nobody can fix them.

Multi-threading is never trivial.

Using a multi-threaded approach here, when SMP scalability is not an issue, suggests that either their API design is crap, and requires threading, or that their engineers are incompetent and use threads unnecessarily. Threads are never trivial - but what they were trying to do was quite trivial. Its their fault they involved threads in there.

The complexity of Linux, Windows and MacOS X are all much the same

Compare the complexity of APIs. fork/exec vs CreateProcess. open vs CreateFile. To use either of the Windows ones you have to call multiple APIs with multiple complicated structures documented upon pages and pages of explanations you must do more than skim through.

Windows may have similar complexity in some subsets of its APIs, but the Windows APIs I had the misfortune to use were insanely complicated unnecessarily.

Re:Backwards compatibility

[Peaker]

Threading is used in GUI's so that redraws and responding to the user can still take place while it's processing the user's last command or commands. This is how "incompetent engineers" for the Win32 and Java platforms do it, at least. What pray tell are the "far simpler approaches"?

The far-simpler approach is to use asynchronous programming. Never use blocking API calls. All good API's always provide non-blocking interfaces.
If long computations are required, split them up into short computations and run the short computations asynchronously from the reactor (event loop).

And how do you write a program that hosts plug-in executable code, to check whether it will hang its host, without the "unnecessary complication" of another thread of execution, either in the program itself, or in another process, watching it?

Firstly, a process is not a thread. It is much safer and cannot cause another process's exit to hang or overwrite its memory/etc. Secondly, I would explicitly declare the interface the plugin must adhere to, and verify that it does, rather than just trying IUknown and watch it for "hanging" via a timeout heuristic.