SpaceX probes for cause of Starlink's global satellite network outage

(Reuters) -SpaceX's Starlink satellite network was back up and running on Friday as engineers hunted for the root cause of one of its biggest international outages the night before, a rare disruption for the powerful internet system set off by an internal software failure.

Users in the U.S. and Europe began experiencing the outage at around 3 p.m. EDT (1900 GMT) Thursday, according to Downdetector, a crowdsourced outage tracker that said as many as 61,000 user reports to the site were made.

In Ukraine, where troops rely heavily on Starlink for battlefield communications, the outage affected combat operations as service was "down across the entire front," said Robert Brovdi, the commander of Ukraine's drone forces.

Starlink, active in roughly 140 countries and territories and used by a growing number of militaries and government agencies, is a key source of revenue for Elon Musk's SpaceX. The network has grown rapidly since 2020 into a disruptive force in the satellite communications industry.

Starlink acknowledged the outage on its X account Thursday and said "we are actively implementing a solution."

The service mostly resumed after 2.5 hours, Michael Nicolls, SpaceX vice president of Starlink Engineering, wrote on X. By 8 p.m., the company wrote on X that the "network issue has been resolved, and Starlink service has been restored."

"The outage was due to failure of key internal software services that operate the core network," Nicolls said, apologizing for the disruption and vowing to find its cause.

Musk also apologized: "Sorry for the outage. SpaceX will remedy root cause to ensure it doesn’t happen again," the SpaceX CEO wrote on X.

The outage was a rare hiccup for SpaceX's most commercially sensitive business. Experts speculated whether the service, known for its resilience and speedy development, was beset by a glitch, a botched software update or perhaps a cyberattack.

Doug Madory, an expert at the internet analysis firm Kentik, said such a sweeping global outage was unusual.

"This is likely the longest outage ever for Starlink, at least while it became a major service provider," Madory said.

As Starlink amasses more than 6 million users, SpaceX has focused in recent months on updating its network to accommodate demands for higher speed and  bandwidth. 

The company, in a partnership with T-Mobile, is also expanding the constellation with larger, more powerful satellites to offer direct-to-cell text messaging services, a line of business in which mobile phone users can send emergency text messages through the network in rural areas.

SpaceX has launched more than 8,000 Starlink satellites since 2020, building a uniquely distributed network in low-Earth orbit that has attracted intense demand from militaries, transportation industries and consumers in rural areas with poor access to traditional, fiber-based internet.

"I'd speculate this is a bad software update, not entirely dissimilar to the CrowdStrike mess with Windows last year, or a cyberattack," said Gregory Falco, director of a space and cybersecurity laboratory at Cornell University.

An update to CrowdStrike's widely used cybersecurity software led to worldwide flight cancellations and impacted industries around the globe in July last year. The outage disrupted internet services, affecting 8.5 million Microsoft Windows devices.

It was unclear whether Thursday's outage affected SpaceX's other satellite-based services that rely on the Starlink network. Starshield, the company's military satellite business unit, has billions of dollars' worth of contracts with the Pentagon and U.S. intelligence agencies.

Separately, Reuters reported on Friday that Musk ordered a partial shutdown of Starlink during a pivotal push by Ukraine to retake territory in its war with Russia in late September 2022.

(Reporting by Juby Babu in Mexico City, Raphael Satter and Joey Roulette in Washington, and Mrinmay Dey in Bengaluru; Editing by Anil D'Silva, Matthew Lewis, Toby Chopra and David Gregorio)