How one piece of hardware forced a full-day shutdown of Tokyo Stock Market

 

The full-day shutdown that ensued was the longest since the exchange switched to a fully electronic trading system in 1999.

At 7:04 a.m. on an autumn Thursday in Tokyo, the stewards of the world’s third-largest equity market realized they had a problem.

A data device critical to the Tokyo Stock Exchange’s trading system had malfunctioned, and the automatic backup had failed to kick in. It was less than an hour before the system, called Arrowhead, was due to start processing orders in the $6 trillion equity market. Exchange officials could see no solution.

The full-day shutdown that ensued was the longest since the exchange switched to a fully electronic trading system in 1999. It drew criticism from market participants and authorities and shone a spotlight on a lesser-discussed vulnerability in the world’s financial plumbing -- not software or security risks but the danger when one of hundreds of pieces of hardware that make up a trading system decides to give up the ghost.

“Exchanges are a crucial part of market infrastructure and it’s unacceptable that trading opportunities were denied,” Finance Minister Taro Aso told reporters in Tokyo. “You’re dealing with machines so it’s always possible they will break. They need to create the infrastructure with that possibility of a breakdown in mind.”

The TSE’s Arrowhead system launched to much fanfare in 2010, billed as a modern-day solution after a series of outages on an older system embarrassed the exchange in the 2000s. The “arrow” symbolizes speed of order processing, while the “head” suggests robustness and reliability, according to the exchange. The system of roughly 350 servers that process buy and sell orders had had a few hiccups but no major outages in its first decade.

That all changed on Thursday, when a piece of hardware called the No. 1 shared disk device, one of two square-shaped data-storage boxes, detected a memory error. These devices store management data used across the servers, and distribute information such as commands and ID and password combinations for terminals that monitor trades.