Message Storm Knocks NYSE Offline

ninjee writes "The New York Stock Exchange is re-examining its network after it was forced to close four minutes early at 3:56pm on Wednesday (1 June) because of a communications glitch. Trading opened on time (09:30 EDT) the following morning but the outage irked traders and raised questions about the reliability of a network described as 'ultra reliable' following improvements made in the wake the September 11 terrorist attacks. The outage stemmed from a fault in a system designed to distribute market data and operate computer trading systems. NYSE Chief Executive John Thain said that both the main system and its backup were swamped with error messages, Reuters reports. He added that the exchange would carry out remedial work designed to prevent any repetition of the problem."

no final print

[gbasin]

I as well as many others in my office got royally screwed here, getting stuck with quite sizeable unhedged positions overnight. It's bad enough that order routing went down, but they failed to open up for a final print (as originally proposed) later in the afternoon. Very bad.

What is the System?

[putko]

It sounds like a distributed systems failure, alright.

Here [nysedata.com] is something about the system that might have broken. I'm wondering if the thing that failed really is the thing mentioned here -- the stuff the stuff Birman [simc-inc.org] did. His new book on distributed systems is out [amazon.com], by the way.

Somone will get flying ninja-kicked in the nuts for this, you can be sure.

Reuters?

[ajs]

"the main system and its backup were swamped with error messages, Reuters reports"

Which is kinda funny, since it was *probably* a reuters feed [reuters.com] that was spewing the errors in the first place....

Re:no final print

[wayne606]

Was your problem a "fill on close" issue? I.e. did you have trades that were supposed to go through at the closing price but didn't because there was no close that day? If so I can see how this would screw a lot of traders.

Revenge of the Dick!

[Joe Jarvis]

Somewhere, in a secret underground lair wallpapered with 100 dollar bills, Dick Grasso is laughing maniacally.

Re:no final print

[gbasin]

Yes, on the average day I completely hedge myself using MOC (market on-close) orders, as well as open up some new positions against market imbalances on close to be closed off on the ECNs a few minutes after the close.

Re:no final print

[gbasin]

The final closing price on the NYSE for a stock. Anyone who wants to trade on the final print, can, using market on close orders (to buy or sell). Primarily, it allows one to get out of any positions and get flat for the night at whatever the final closing price is (decided upon by the specialist). Secondly, it allows traders like me to "help out the specialist" meet the demands of a buy or sell imbalance on close, getting paid to take on the same risks the specialists do. Finally, professional traders have their daily profit and loss "marked to market", where even if they have positions open, receive credit/debit to their account overnight based on where their positions closed (the final print) - this is also used to calculate overnight interest in positions as well as margin needed to maintain the position. The fact that they simply stopped trading for the day forced me to keep several positions overnight, costing me some change (maybe $100) in interest but more importantly tieing up close to $5mil in buying power which I didn't get back until after the open the next day hurting my ability to participate in the opening.

Re:Any Linux role?

[Anonymous Coward]

No, linux had nothing to do with it. I work for SIAC and I was there when the crash happened. Most of our operations systems are HP|UX and Linux with a Windows box here and there. I can't say what exactly happened (being fired for a /. posting isn't really what I want) but I can say that it had to do with a bridge type connection.....