BSE Shutdown: Is Cisco to blame?

[caption id=“attachment_84869” align=“aligncenter” width=“640”] Reuters[/caption]

A technology glitch led to the complete shutdown of trading on the Bombay Stock Exchange , the world’s largest exchange in terms of listed members, for over three hours on July 3, resulting in egg on BSE’s face even as it competes hard with the National Stock Exchange.

Even though the exchange claimed there was no monetary loss, the first complete network outage in the last four years that stopped trading for almost three hours did raise some serious concerns about the exchange’s IT infrastructure as well as its technology partners. After all, stock exchanges are not supposed to fail and regulators mandate that exchanges should have required redundant backup systems in place to prevent failures and market losses. The embarrassment caused by such failures can also result in increased reputation risk.

Incidentally, this was the fourth disruption at BSE this year; however the earlier three snags lasted for just few minutes and hence didn’t get widespread media coverage. Soon after the shut down,BSE, in its release, said: “Several users were logged out abruptly due to misbehaviour of some of the network components in the BSE network, which connects to more than 10,000 primary connections with all telecom vendors in India participating in the network in addition to BSE’s own LAN network and VSAT network provided by Hughes.”

The exchange’s management has obviously taken the failure very seriously and a committee is looking into the matter. BSE, along with its network technology providers HCL Comnet and Cisco, is preparing a root cause analysis, which will be submitted to BSE’s Technology Advisory Committee and Board of Directors. “The Root Cause Analysis report will contain the details of the incident, the reasons for the same, how it was repaired, steps suggested and being undertaken with timelines to eliminate recurrence of such a specific event or similar events.”

Incidentally, BSE MD & CEO Ashish Chauhan is not only a respected business leader but also renowned for his technology management skills. He was the Group CIO of the Reliance group and has been ranked amongst the top 50 CIOs in the world by independent groups. But even as BSE is still preparing a root cause analysis of the network failure, there are several reports doing the rounds as to what could have led to the sudden shut down. Telecom industry website TeleAnalysis has said that Cisco equipment is to be blamed for the outage.

It was initally assumed that the outage could be a network failure from the bandwidth provider, thus putting bandwidth provider Hughes Communications under the scanner. However, Hughes issued a statement saying, “In the context of the BSE network outage, Hughes Communications would like to state that post a thorough check of our systems, no fault was detected and the entire VSAT network was operational. This was further verified through other institutional customers relying on our network support, none of who reported any problems or disruptions yesterday. Our VSAT network guarantees users of at least 99.9 percent uptime at all times and through all weather conditions."

And, that’s where Cisco came into the picture.

A TeleAnalysis report says that Cisco’s Nexus 7000 Series Switch was the real reason behind BSE’s 3 hour and 2 minute network failure. Some BSE officials were quoted as saying that the Cisco switch failed due to looping and configuration problem.TeleAnalysis also quoted an unnamed industry expert as saying that users globally have faced challenges regarding stability of Cisco’s Nexus 7000 Series Switch, with issues related to configuration problem, testing and scalability issues.

Firstbiz also reached out to Cisco asking for comment, but Cisco declined comment, with a Cisco PR agency representative verbally informing us that the cause would be available in BSE’s report later this week.

If the TeleAnalysis version turns out to be true, it would be interesting to see whether BSE will continue to rely on the Nexus 7000 series switch or would immediately consider other vendors.