[UPDATED] China Snuck A Tiny Microchip Inside US Top Secret Servers Used By The DoD And CIA — NSA Struggles To Assess Risk

[UPDATED SEVERAL TIMES FROM DIFFERENT SOURCES – SCROLL DOWN
You find all relevant reporting on the matter on this page;
including the implant built into the server’s Ethernet connector at a US Telecom company.]

The Chinese military surreptitiously inserted tiny microchips no larger than single grains of rice into servers on local assembly lines in order to gain access to data networks run by U.S. government agencies ranging from the Department of Defense to the Central Intelligence Agency, according to an explosive investigation from Bloomberg.

• A three-year investigation by U.S. government officials found that servers assembled for startup Elemental Technologies by San Jose-based company Supermicro reportedly contained tiny microchips “inserted at factories run by manufacturing subcontractors in China,” Bloomberg reported.
• The chips, independently discovered by engineers at Amazon and Apple in 2015, purportedly allowed hackers to “create a stealth doorway into any network that included the altered machines,” per Bloomberg, a Trojan horse that gave hackers a direct line into any sensitive network.
• Elemental servers assembled by Supermicro are “found in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships,” per Bloomberg, and the revelation prompted DoD officials at the time to request a small group of technologists “to think about creating commercial products that could detect hardware implants.”
• “Public documents, including the company’s own promotional materials, show that the servers have been used inside Department of Defense data centers to process drone and surveillance-camera footage, on Navy warships to transmit feeds of airborne missions, and inside government buildings to enable secure videoconferencing,” Bloomberg reports. “NASA, both houses of Congress, and the Department of Homeland Security have also been customers.”
• News of the years-long infiltration of secure networks through the lowest levels of the global industrial supply chain  — China still manufactures the majority of the raw tech behind the world’s mobile phones and personal computers — reflects not just a coup for the Chinese intelligence community, but an alarming vulnerability of the U.S. industrial base.
• Technologist Joe Grand put it best in an interview with Bloomberg: “Having a well-done, nation-state-level hardware implant surface would be like witnessing a unicorn jumping over a rainbow … Hardware is just so far off the radar, it’s almost treated like black magic.”

August Cole, a coauthor of the novel “Ghost Fleet” — which features an eerily similar scenario involving Chinese chips hidden inside an F-35 that ruin its stealth capabilities, wrote on Twitter, “Hey Siri, what is my #ghostfleet moment of the day?”

from: https://taskandpurpose.com/china-hacking-microchips-dod-cia/

In 2015, Amazon.com Inc. began quietly evaluating a startup called Elemental Technologies, a potential acquisition to help with a major expansion of its streaming video service, known today as Amazon Prime Video. Based in Portland, Ore., Elemental made software for compressing massive video files and formatting them for different devices. Its technology had helped stream the Olympic Games online, communicate with the International Space Station, and funnel drone footage to the Central Intelligence Agency. Elemental’s national security contracts weren’t the main reason for the proposed acquisition, but they fit nicely with Amazon’s government businesses, such as the highly secure cloud that Amazon Web Services (AWS) was building for the CIA.

To help with due diligence, AWS, which was overseeing the prospective acquisition, hired a third-party company to scrutinize Elemental’s security, according to one person familiar with the process. The first pass uncovered troubling issues, prompting AWS to take a closer look at Elemental’s main product: the expensive servers that customers installed in their networks to handle the video compression. These servers were assembled for Elemental by Super Micro Computer Inc., a San Jose-based company (commonly known as Supermicro) that’s also one of the world’s biggest suppliers of server motherboards, the fiberglass-mounted clusters of chips and capacitors that act as the neurons of data centers large and small. In late spring of 2015, Elemental’s staff boxed up several servers and sent them to Ontario, Canada, for the third-party security company to test, the person says.

Nested on the servers’ motherboards, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design. Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community. Elemental’s servers could be found in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships. And Elemental was just one of hundreds of Supermicro customers.

During the ensuing top-secret probe, which remains open more than three years later, investigators determined that the chips allowed the attackers to create a stealth doorway into any network that included the altered machines. Multiple people familiar with the matter say investigators found that the chips had been inserted at factories run by manufacturing subcontractors in China.

This attack was something graver than the software-based incidents the world has grown accustomed to seeing. Hardware hacks are more difficult to pull off and potentially more devastating, promising the kind of long-term, stealth access that spy agencies are willing to invest millions of dollars and many years to get.

There are two ways for spies to alter the guts of computer equipment. One, known as interdiction, consists of manipulating devices as they’re in transit from manufacturer to customer. This approach is favored by U.S. spy agencies, according to documents leaked by former National Security Agency contractor Edward Snowden. The other method involves seeding changes from the very beginning.

One country in particular has an advantage executing this kind of attack: China, which by some estimates makes 75 percent of the world’s mobile phones and 90 percent of its PCs. Still, to actually accomplish a seeding attack would mean developing a deep understanding of a product’s design, manipulating components at the factory, and ensuring that the doctored devices made it through the global logistics chain to the desired location—a feat akin to throwing a stick in the Yangtze River upstream from Shanghai and ensuring that it washes ashore in Seattle. “Having a well-done, nation-state-level hardware implant surface would be like witnessing a unicorn jumping over a rainbow,” says Joe Grand, a hardware hacker and the founder of Grand Idea Studio Inc. “Hardware is just so far off the radar, it’s almost treated like black magic.”

But that’s just what U.S. investigators found: The chips had been inserted during the manufacturing process, two officials say, by operatives from a unit of the People’s Liberation Army. In Supermicro, China’s spies appear to have found a perfect conduit for what U.S. officials now describe as the most significant supply chain attack known to have been carried out against American companies.

One official says investigators found that it eventually affected almost 30 companies, including a major bank, government contractors, and the world’s most valuable company, Apple Inc. Apple was an important Supermicro customer and had planned to order more than 30,000 of its servers in two years for a new global network of data centers. Three senior insiders at Apple say that in the summer of 2015, it, too, found malicious chips on Supermicro motherboards. Apple severed ties with Supermicro the following year, for what it described as unrelated reasons.

In emailed statements, Amazon (which announced its acquisition of Elemental in September 2015), Apple, and Supermicro disputed summaries of Bloomberg Businessweek’s reporting. “It’s untrue that AWS knew about a supply chain compromise, an issue with malicious chips, or hardware modifications when acquiring Elemental,” Amazon wrote. “On this we can be very clear: Apple has never found malicious chips, ‘hardware manipulations’ or vulnerabilities purposely planted in any server,” Apple wrote. “We remain unaware of any such investigation,” wrote a spokesman for Supermicro, Perry Hayes. The Chinese government didn’t directly address questions about manipulation of Supermicro servers, issuing a statement that read, in part, “Supply chain safety in cyberspace is an issue of common concern, and China is also a victim.” The FBI and the Office of the Director of National Intelligence, representing the CIA and NSA, declined to comment.

The companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation. One of those officials and two people inside AWS provided extensive information on how the attack played out at Elemental and Amazon; the official and one of the insiders also described Amazon’s cooperation with the government investigation. In addition to the three Apple insiders, four of the six U.S. officials confirmed that Apple was a victim. In all, 17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information.

One government official says China’s goal was long-term access to high-value corporate secrets and sensitive government networks. No consumer data is known to have been stolen.

The ramifications of the attack continue to play out. The Trump administration has made computer and networking hardware, including motherboards, a focus of its latest round of trade sanctions against China, and White House officials have made it clear they think companies will begin shifting their supply chains to other countries as a result. Such a shift might assuage officials who have been warning for years about the security of the supply chain—even though they’ve never disclosed a major reason for their concerns.

How the Hack Worked, According to U.S. Officials

Back in 2006, three engineers in Oregon had a clever idea. Demand for mobile video was about to explode, and they predicted that broadcasters would be desperate to transform programs designed to fit TV screens into the various formats needed for viewing on smartphones, laptops, and other devices. To meet the anticipated demand, the engineers started Elemental Technologies, assembling what one former adviser to the company calls a genius team to write code that would adapt the superfast graphics chips being produced for high-end video-gaming machines. The resulting software dramatically reduced the time it took to process large video files. Elemental then loaded the software onto custom-built servers emblazoned with its leprechaun-green logos.

Elemental servers sold for as much as $100,000 each, at profit margins of as high as 70 percent, according to a former adviser to the company. Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not.

Elemental also started working with American spy agencies. In 2009 the company announced a development partnership with In-Q-Tel Inc., the CIA’s investment arm, a deal that paved the way for Elemental servers to be used in national security missions across the U.S. government. Public documents, including the company’s own promotional materials, show that the servers have been used inside Department of Defense data centers to process drone and surveillance-camera footage, on Navy warships to transmit feeds of airborne missions, and inside government buildings to enable secure videoconferencing. NASA, both houses of Congress, and the Department of Homeland Security have also been customers. This portfolio made Elemental a target for foreign adversaries.

Supermicro had been an obvious choice to build Elemental’s servers. Headquartered north of San Jose’s airport, up a smoggy stretch of Interstate 880, the company was founded by Charles Liang, a Taiwanese engineer who attended graduate school in Texas and then moved west to start Supermicro with his wife in 1993. Silicon Valley was then embracing outsourcing, forging a pathway from Taiwanese, and later Chinese, factories to American consumers, and Liang added a comforting advantage: Supermicro’s motherboards would be engineered mostly in San Jose, close to the company’s biggest clients, even if the products were manufactured overseas.

Today, Supermicro sells more server motherboards than almost anyone else. It also dominates the $1 billion market for boards used in special-purpose computers, from MRI machines to weapons systems. Its motherboards can be found in made-to-order server setups at banks, hedge funds, cloud computing providers, and web-hosting services, among other places. Supermicro has assembly facilities in California, the Netherlands, and Taiwan, but its motherboards—its core product—are nearly all manufactured by contractors in China.

The company’s pitch to customers hinges on unmatched customization, made possible by hundreds of full-time engineers and a catalog encompassing more than 600 designs. The majority of its workforce in San Jose is Taiwanese or Chinese, and Mandarin is the preferred language, with hanzi filling the whiteboards, according to six former employees. Chinese pastries are delivered every week, and many routine calls are done twice, once for English-only workers and again in Mandarin. The latter are more productive, according to people who’ve been on both. These overseas ties, especially the widespread use of Mandarin, would have made it easier for China to gain an understanding of Supermicro’s operations and potentially to infiltrate the company. (A U.S. official says the government’s probe is still examining whether spies were planted inside Supermicro or other American companies to aid the attack.)

With more than 900 customers in 100 countries by 2015, Supermicro offered inroads to a bountiful collection of sensitive targets. “Think of Supermicro as the Microsoft of the hardware world,” says a former U.S. intelligence official who’s studied Supermicro and its business model. “Attacking Supermicro motherboards is like attacking Windows. It’s like attacking the whole world.”

Well before evidence of the attack surfaced inside the networks of U.S. companies, American intelligence sources were reporting that China’s spies had plans to introduce malicious microchips into the supply chain. The sources weren’t specific, according to a person familiar with the information they provided, and millions of motherboards are shipped into the U.S. annually. But in the first half of 2014, a different person briefed on high-level discussions says, intelligence officials went to the White House with something more concrete: China’s military was preparing to insert the chips into Supermicro motherboards bound for U.S. companies.

The specificity of the information was remarkable, but so were the challenges it posed. Issuing a broad warning to Supermicro’s customers could have crippled the company, a major American hardware maker, and it wasn’t clear from the intelligence whom the operation was targeting or what its ultimate aims were. Plus, without confirmation that anyone had been attacked, the FBI was limited in how it could respond. The White House requested periodic updates as information came in, the person familiar with the discussions says.

Apple made its discovery of suspicious chips inside Supermicro servers around May 2015, after detecting odd network activity and firmware problems, according to a person familiar with the timeline. Two of the senior Apple insiders say the company reported the incident to the FBI but kept details about what it had detected tightly held, even internally. Government investigators were still chasing clues on their own when Amazon made its discovery and gave them access to sabotaged hardware, according to one U.S. official. This created an invaluable opportunity for intelligence agencies and the FBI—by then running a full investigation led by its cyber- and counterintelligence teams—to see what the chips looked like and how they worked.

The chips on Elemental servers were designed to be as inconspicuous as possible, according to one person who saw a detailed report prepared for Amazon by its third-party security contractor, as well as a second person who saw digital photos and X-ray images of the chips incorporated into a later report prepared by Amazon’s security team. Gray or off-white in color, they looked more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment. Depending on the board model, the chips varied slightly in size, suggesting that the attackers had supplied different factories with different batches.

Officials familiar with the investigation say the primary role of implants such as these is to open doors that other attackers can go through. “Hardware attacks are about access,” as one former senior official puts it. In simplified terms, the implants on Supermicro hardware manipulated the core operating instructions that tell the server what to do as data move across a motherboard, two people familiar with the chips’ operation say. This happened at a crucial moment, as small bits of the operating system were being stored in the board’s temporary memory en route to the server’s central processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information queue, injecting its own code or altering the order of the instructions the CPU was meant to follow. Deviously small changes could create disastrous effects.

Since the implants were small, the amount of code they contained was small as well. But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.

This system could let the attackers alter how the device functioned, line by line, however they wanted, leaving no one the wiser. To understand the power that would give them, take this hypothetical example: Somewhere in the Linux operating system, which runs in many servers, is code that authorizes a user by verifying a typed password against a stored encrypted one. An implanted chip can alter part of that code so the server won’t check for a password—and presto! A secure machine is open to any and all users. A chip can also steal encryption keys for secure communications, block security updates that would neutralize the attack, and open up new pathways to the internet. Should some anomaly be noticed, it would likely be cast as an unexplained oddity. “The hardware opens whatever door it wants,” says Joe FitzPatrick, founder of Hardware Security Resources LLC, a company that trains cybersecurity professionals in hardware hacking techniques.

U.S. officials had caught China experimenting with hardware tampering before, but they’d never seen anything of this scale and ambition. The security of the global technology supply chain had been compromised, even if consumers and most companies didn’t know it yet. What remained for investigators to learn was how the attackers had so thoroughly infiltrated Supermicro’s production process—and how many doors they’d opened into American targets.

Unlike software-based hacks, hardware manipulation creates a real-world trail. Components leave a wake of shipping manifests and invoices. Boards have serial numbers that trace to specific factories. To track the corrupted chips to their source, U.S. intelligence agencies began following Supermicro’s serpentine supply chain in reverse, a person briefed on evidence gathered during the probe says.

As recently as 2016, according to DigiTimes, a news site specializing in supply chain research, Supermicro had three primary manufacturers constructing its motherboards, two headquartered in Taiwan and one in Shanghai. When such suppliers are choked with big orders, they sometimes parcel out work to subcontractors. In order to get further down the trail, U.S. spy agencies drew on the prodigious tools at their disposal. They sifted through communications intercepts, tapped informants in Taiwan and China, even tracked key individuals through their phones, according to the person briefed on evidence gathered during the probe. Eventually, that person says, they traced the malicious chips to four subcontracting factories that had been building Supermicro motherboards for at least two years.

As the agents monitored interactions among Chinese officials, motherboard manufacturers, and middlemen, they glimpsed how the seeding process worked. In some cases, plant managers were approached by people who claimed to represent Supermicro or who held positions suggesting a connection to the government. The middlemen would request changes to the motherboards’ original designs, initially offering bribes in conjunction with their unusual requests. If that didn’t work, they threatened factory managers with inspections that could shut down their plants. Once arrangements were in place, the middlemen would organize delivery of the chips to the factories.

The investigators concluded that this intricate scheme was the work of a People’s Liberation Army unit specializing in hardware attacks, according to two people briefed on its activities. The existence of this group has never been revealed before, but one official says, “We’ve been tracking these guys for longer than we’d like to admit.” The unit is believed to focus on high-priority targets, including advanced commercial technology and the computers of rival militaries. In past attacks, it targeted the designs for high-performance computer chips and computing systems of large U.S. internet providers.

Provided details of Businessweek’s reporting, China’s Ministry of Foreign Affairs sent a statement that said “China is a resolute defender of cybersecurity.” The ministry added that in 2011, China proposed international guarantees on hardware security along with other members of the Shanghai Cooperation Organization, a regional security body. The statement concluded, “We hope parties make less gratuitous accusations and suspicions but conduct more constructive talk and collaboration so that we can work together in building a peaceful, safe, open, cooperative and orderly cyberspace.”

The Supermicro attack was on another order entirely from earlier episodes attributed to the PLA. It threatened to have reached a dizzying array of end users, with some vital ones in the mix. Apple, for its part, has used Supermicro hardware in its data centers sporadically for years, but the relationship intensified after 2013, when Apple acquired a startup called Topsy Labs, which created superfast technology for indexing and searching vast troves of internet content. By 2014, the startup was put to work building small data centers in or near major global cities. This project, known internally as Ledbelly, was designed to make the search function for Apple’s voice assistant, Siri, faster, according to the three senior Apple insiders.

Documents seen by Businessweek show that in 2014, Apple planned to order more than 6,000 Supermicro servers for installation in 17 locations, including Amsterdam, Chicago, Hong Kong, Los Angeles, New York, San Jose, Singapore, and Tokyo, plus 4,000 servers for its existing North Carolina and Oregon data centers. Those orders were supposed to double, to 20,000, by 2015. Ledbelly made Apple an important Supermicro customer at the exact same time the PLA was found to be manipulating the vendor’s hardware.

Project delays and early performance problems meant that around 7,000 Supermicro servers were humming in Apple’s network by the time the company’s security team found the added chips. Because Apple didn’t, according to a U.S. official, provide government investigators with access to its facilities or the tampered hardware, the extent of the attack there remained outside their view.

American investigators eventually figured out who else had been hit. Since the implanted chips were designed to ping anonymous computers on the internet for further instructions, operatives could hack those computers to identify others who’d been affected. Although the investigators couldn’t be sure they’d found every victim, a person familiar with the U.S. probe says they ultimately concluded that the number was almost 30 companies.

That left the question of whom to notify and how. U.S. officials had been warning for years that hardware made by two Chinese telecommunications giants, Huawei Corp. and ZTE Corp., was subject to Chinese government manipulation. (Both Huawei and ZTE have said no such tampering has occurred.) But a similar public alert regarding a U.S. company was out of the question. Instead, officials reached out to a small number of important Supermicro customers. One executive of a large web-hosting company says the message he took away from the exchange was clear: Supermicro’s hardware couldn’t be trusted. “That’s been the nudge to everyone—get that crap out,” the person says.

Amazon, for its part, began acquisition talks with an Elemental competitor, but according to one person familiar with Amazon’s deliberations, it reversed course in the summer of 2015 after learning that Elemental’s board was nearing a deal with another buyer. Amazon announced its acquisition of Elemental in September 2015, in a transaction whose value one person familiar with the deal places at $350 million. Multiple sources say that Amazon intended to move Elemental’s software to AWS’s cloud, whose chips, motherboards, and servers are typically designed in-house and built by factories that Amazon contracts from directly.

A notable exception was AWS’s data centers inside China, which were filled with Supermicro-built servers, according to two people with knowledge of AWS’s operations there. Mindful of the Elemental findings, Amazon’s security team conducted its own investigation into AWS’s Beijing facilities and found altered motherboards there as well, including more sophisticated designs than they’d previously encountered. In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips. That generation of chips was smaller than a sharpened pencil tip, the person says. (Amazon denies that AWS knew of servers found in China containing malicious chips.)

China has long been known to monitor banks, manufacturers, and ordinary citizens on its own soil, and the main customers of AWS’s China cloud were domestic companies or foreign entities with operations there. Still, the fact that the country appeared to be conducting those operations inside Amazon’s cloud presented the company with a Gordian knot. Its security team determined that it would be difficult to quietly remove the equipment and that, even if they could devise a way, doing so would alert the attackers that the chips had been found, according to a person familiar with the company’s probe. Instead, the team developed a method of monitoring the chips. In the ensuing months, they detected brief check-in communications between the attackers and the sabotaged servers but didn’t see any attempts to remove data. That likely meant either that the attackers were saving the chips for a later operation or that they’d infiltrated other parts of the network before the monitoring began. Neither possibility was reassuring.

When in 2016 the Chinese government was about to pass a new cybersecurity law—seen by many outside the country as a pretext to give authorities wider access to sensitive data—Amazon decided to act, the person familiar with the company’s probe says. In August it transferred operational control of its Beijing data center to its local partner, Beijing Sinnet, a move the companies said was needed to comply with the incoming law. The following November, Amazon sold the entire infrastructure to Beijing Sinnet for about $300 million. The person familiar with Amazon’s probe casts the sale as a choice to “hack off the diseased limb.”

As for Apple, one of the three senior insiders says that in the summer of 2015, a few weeks after it identified the malicious chips, the company started removing all Supermicro servers from its data centers, a process Apple referred to internally as “going to zero.” Every Supermicro server, all 7,000 or so, was replaced in a matter of weeks, the senior insider says. (Apple denies that any servers were removed.) In 2016, Apple informed Supermicro that it was severing their relationship entirely—a decision a spokesman for Apple ascribed in response to Businessweek’s questions to an unrelated and relatively minor security incident.

That August, Supermicro’s CEO, Liang, revealed that the company had lost two major customers. Although he didn’t name them, one was later identified in news reports as Apple. He blamed competition, but his explanation was vague. “When customers asked for lower price, our people did not respond quickly enough,” he said on a conference call with analysts. Hayes, the Supermicro spokesman, says the company has never been notified of the existence of malicious chips on its motherboards by either customers or U.S. law enforcement.

Concurrent with the illicit chips’ discovery in 2015 and the unfolding investigation, Supermicro has been plagued by an accounting problem, which the company characterizes as an issue related to the timing of certain revenue recognition. After missing two deadlines to file quarterly and annual reports required by regulators, Supermicro was delisted from the Nasdaq on Aug. 23 of this year. It marked an extraordinary stumble for a company whose annual revenue had risen sharply in the previous four years, from a reported $1.5 billion in 2014 to a projected $3.2 billion this year.

One Friday in late September 2015, President Barack Obama and Chinese President Xi Jinping appeared together at the White House for an hourlong press conference headlined by a landmark deal on cybersecurity. After months of negotiations, the U.S. had extracted from China a grand promise: It would no longer support the theft by hackers of U.S. intellectual property to benefit Chinese companies. Left out of those pronouncements, according to a person familiar with discussions among senior officials across the U.S. government, was the White House’s deep concern that China was willing to offer this concession because it was already developing far more advanced and surreptitious forms of hacking founded on its near monopoly of the technology supply chain.

In the weeks after the agreement was announced, the U.S. government quietly raised the alarm with several dozen tech executives and investors at a small, invite-only meeting in McLean, Va., organized by the Pentagon. According to someone who was present, Defense Department officials briefed the technologists on a recent attack and asked them to think about creating commercial products that could detect hardware implants. Attendees weren’t told the name of the hardware maker involved, but it was clear to at least some in the room that it was Supermicro, the person says.

The problem under discussion wasn’t just technological. It spoke to decisions made decades ago to send advanced production work to Southeast Asia. In the intervening years, low-cost Chinese manufacturing had come to underpin the business models of many of America’s largest technology companies. Early on, Apple, for instance, made many of its most sophisticated electronics domestically. Then in 1992, it closed a state-of-the-art plant for motherboard and computer assembly in Fremont, Calif., and sent much of that work overseas.

Over the decades, the security of the supply chain became an article of faith despite repeated warnings by Western officials. A belief formed that China was unlikely to jeopardize its position as workshop to the world by letting its spies meddle in its factories. That left the decision about where to build commercial systems resting largely on where capacity was greatest and cheapest. “You end up with a classic Satan’s bargain,” one former U.S. official says. “You can have less supply than you want and guarantee it’s secure, or you can have the supply you need, but there will be risk. Every organization has accepted the second proposition.”

In the three years since the briefing in McLean, no commercially viable way to detect attacks like the one on Supermicro’s motherboards has emerged—or has looked likely to emerge. Few companies have the resources of Apple and Amazon, and it took some luck even for them to spot the problem. “This stuff is at the cutting edge of the cutting edge, and there is no easy technological solution,” one of the people present in McLean says. “You have to invest in things that the world wants. You cannot invest in things that the world is not ready to accept yet.”

from: https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies

‘We Have No Way Of Addressing This’: Ex-NSA Scientist Reacts To China Sneaking Microchips Into DoD Servers

After an explosive Bloomberg report revealed that China was surreptitiously inserting small microchips into servers that later ended up being used by the Department of Defense, CIA, and many large American companies, an ex-NSA scientist warned there was “no way of addressing this risk” from a strategic standpoint.

“We can find a couple of them, but we’re not gonna find the next generation version,” said Dave Aitel, a former computer scientist for the National Security Agency now working as the Chief Security Technical Officer for Cyxtera. “That makes it very hard to trust computers in general.”

U.S. government investigators found that servers assembled by American companies contained motherboards — made by Chinese subcontractors — with tiny microchips that could allow hackers to “create a stealth doorway into any network that included the altered machines,” according to Bloomberg.

“They are literally in between the layers of the board,” Aitel said, adding that in order to see it, “you would have to take a board, strip it down, and X-ray it” to find the suspect chip.

“That’s just not a thing we should expect corporations to be able to do, even the biggest organizations.”

The machines are found inside DoD data centers, on Navy warships, and at the CIA, the site reported.

The Pentagon declined to comment on whether the suspect chips were found on DoD networks, citing operational security reasons. Still, Department spokeswoman Heather Babb told Task & Purpose, the U.S. military “has policies in place to address software assurance and supply chain risk management, as well as established security standards to ensure all procured commercial products and services are rigorously inspected for security vulnerabilities. As threats within the cyberspace domain change, DOD looks for solutions that provide more capability.”

“The protection of the National Security Innovation Base is a priority for the Department. Working closely with Congress and private industry, DOD is already advancing to elevate security within the supply chain,” she added.

China isn’t the only nation-state working to infiltrate hardware as a means to hack its enemies. The U.S. does much the same thing — intercepting network hardware and secretly installing beacons that call back to NSA — except it doesn’t seem to get or can legally force the cooperation of the factory making the product.

China doesn’t seem to have that problem.

“The question becomes can we move to a trusted supply chain or not?” Aitel asked. He added that “tin foil” hat thinking that foreign-made hardware should be treated as suspect isn’t so conspiratorial after all.

Still, he did offer some more positive news: “The good news is we caught it, and we’re on it,” Aitel said. “That’s actually phenomenally good news. That does send a message of deterrence. That does send a message that you can’t get away with it.”

President Barack Obama and Chinese President Xi Jinping agreed in 2015 that neither government would “conduct or knowingly support cyber-enabled theft of intellectual property” and said they would work together on other cybersecurity issues.

This latest disclosure of cyber-espionage adds fuel to the fire that China has clearly violated the agreement, which the Trump administration accused Beijing of doing earlier this year.

Aitel said it was more than likely that DoD and other governmental organizations were pulling the suspect servers if they haven’t done so already. Still, the risk will likely remain as long as the hardware is not manufactured in the U.S.

from: https://taskandpurpose.com/china-hacking-microchips-nsa-reaction/

 

Ready for information about what may be one of the largest corporate espionage programs from a nation-state? The Chinese government managed to gain access to the servers of more than 30 U.S. companies, including Apple, according to an explosive report from Bloomberg published today.

Bloomberg reports that U.S-based server motherboard specialist Supermicro was compromised in China where government-affiliated groups are alleged to have infiltrated its supply chain to attach tiny chips, some merely the size of a pencil tip, to motherboards which ended up in servers deployed in the U.S.

The goal, Bloomberg said, was to gain an entry point within company systems to potentially grab IP or confidential information. While the micro-servers themselves were limited in terms of direct capabilities, they represented a “stealth doorway” that could allow China-based operatives to remotely alter how a device functioned to potentially access information.

Once aware of the program, the U.S. government spied on the spies behind the chips but, according to Bloomberg, no consumer data is known to have been stolen through the attacks. Even still, this episode represents one of the most striking espionage programs from the Chinese government to date.

The story reports that the chips were discovered and reported to the FBI by Amazon, which found them during due diligence ahead of its 2015 acquisition of Elemental Systems, a company that held a range of U.S. government contracts, and Apple, which is said to have deployed up to 7,000 Supermicro servers at peak. Bloomberg reported that Amazon removed them all within a one-month period. Apple did indeed cut ties with Supermicro back in 2016, but it denied a claim from The Information which reported at the time that it was based on a security issue.

Amazon, meanwhile, completed the deal for Elemental Systems — reportedly worth $500 million — after it switched its software to the AWS cloud. Supermicro, meanwhile, was suspended from trading on the Nasdaq in August after failing to submit quarterly reports on time. The company is likely to be delisted once the timeframe for an appeal is over.

Amazon, Apple, Supermicro and China’s Ministry of Foreign Affairs all denied Bloomberg’s findings with strong and lengthy statements — a full list of rebuttals is here. The publication claims that it sourced its information using no fewer than 17 individuals with knowledge of developments, including six U.S. officials and four Apple “insiders.”

from: https://techcrunch.com/2018/10/04/china-reportedly-infiltrated-apple-and-other-us-companies-using-spy-chips-on-servers/

Chinesische Spionage

Apple und Amazon sollen Spionagechips in Servern gefunden haben

Dutzende US-Unternehmen und Regierungseinrichtungen haben Server eingesetzt, die vom selben Hersteller stammen: Supermicro. Doch deren Platinen waren einem Medienbericht zufolge in China manipuliert worden.

Donnerstag, 04.10.2018   14:40 Uhr

Eine Einheit der chinesischen Armee soll dafür gesorgt haben, dass winzige Spionagechips in Tausenden Servern für große Unternehmen wie Amazon und Apple verbaut wurden. Diese Chips, zum Teil so klein wie die Spitze eines Bleistifts, sollen einen heimlichen Verbindungsaufbau zu den Tätern und das unbemerkte Nachladen von Code ermöglicht haben. Das berichtet “Bloomberg Businessweek” unter Berufung auf insgesamt 17 anonyme Informanten aus Unternehmens- und US-Regierungskreisen. Dem gegenüber stehen scharfe Dementis von Amazon, Apple, dem Hersteller der Server und der chinesischen Regierung.

Gefunden wurden die Chips dem Bericht zufolge sowohl von Amazon, als auch von Apple selbst, und zwar auf den Hauptplatinen von Servern, die das in Kalifornien beheimatete Unternehmen Supermicro zusammenbaut oder von Auftragsfirmen zusammenbauen lässt. Diese Auftragshersteller, das erste Glied der Lieferkette, befänden sich im Fall von Supermicro in China. Dort sollen Militärs die Manager bestochen oder bedroht haben, bis diese einwilligten, die Bauteile in das Design der Platinen einzuschmuggeln und zu verbauen.

Zu den betroffenen US-Kunden von Supermicro gehörten “fast 30 Unternehmen”, neben Apple und Amazon auch eine große Bank und mehrere Auftragnehmer der US-Regierung.

Apple: “Wir haben niemals bösartige Chips gefunden”

Amazons Cloudsparte teilte auf Anfrage von “Bloomberg” allerdings mit, es sei unwahr, dass man von der kompromittierten Lieferkette wusste. Apple schrieb: “Wir haben niemals bösartige Chips, manipulierte Hardware oder in Servern versteckte Schwachstellen gefunden”. In ihren vollständigen Dementis führen beide Unternehmen das länger und unmissverständlich aus.

Die insgesamt 17 Quellen der Journalisten widersprechen dieser Darstellung. Sie konnten der Zeitung detailliert darlegen, wie Amazon und Apple die Chips unabhängig voneinander im Jahr 2015 fanden.

Im Fall von Amazon geschah das angeblich bei einer externen Überprüfung von Servern der Firma Elemental, die eine spezielle Software zum Komprimieren und Formatieren von Videos entwickelt hatte und zusammen mit passenden Supermicro-Servern verkaufte, unter anderem an das US-Verteidigungsministerium, die CIA und die US-Marine. Elemental galt damals als möglicher Übernahmekandidat für Amazon, das deshalb die Sicherheitsüberprüfung veranlasst hatte.

Apple beendete 2016 Geschäftsbeziehungen zu Supermicro

Später kaufte Amazon das Start-up zwar wirklich auf, doch kompromittierte Supermicro-Server sollen nur in Amazons chinesischen Cloudzentren zum Einsatz gekommen sein, bis dessen Inventar wieder an ein einheimisches Unternehmen verkauft wurde. An anderen Standorten wollte Amazon nur Elementals Software einsetzen – auf den eigenen Maschinen.

Apple indes habe nach seinem eigenen Fund innerhalb weniger Wochen rund 7000 Supermicro-Server ersetzt, heißt es. Auch das bestreitet das Unternehmen, allerdings räumt es ein, 2016 alle Geschäftsbeziehungen zu Supermicro beendet zu haben, wenn auch aus einem anderen Grund, der etwas mit einem “vergleichsweise kleinen Sicherheitsproblem” zu tun gehabt habe.

Das Ziel der Chinesen sei ein dauerhafter Zugang zu Unternehmens- und Regierungsnetzwerken sowie Geschäftsgeheimnissen gewesen, sagte eine Quelle aus Regierungskreisen “Bloomberg”. Die entsprechenden Untersuchungen dauerten bis heute an.

from: http://www.spiegel.de/netzwelt/web/apple-und-amazon-laut-medienbericht-spionagechips-in-servern-gefunden-a-1231543.html

Today’s bombshell Bloomberg story has the internet split: either the story is right, and reporters have uncovered one of the largest and jarring breaches of the U.S. tech industry by a foreign adversary… or it’s not, and a lot of people screwed up.

To recap, Chinese spies reportedly infiltrated the supply chain and installed tiny chips the size of a pencil tip on the motherboards built by Supermicro, which are used in data center servers across the U.S. tech industry — from Apple to Amazon. That chip can compromise data on the server, allowing China to spy on some of the world’s most wealthy and powerful countries.

Apple, Amazon and Supermicro — and the Chinese government — strenuously denied the allegations. Apple also released its own standalone statement later in the day, as did Supermicro. You don’t see that very often unless they think they have nothing to hide. You can — and should — read the statements for yourself.

Welcome to the murky world of national security reporting.

I’ve covered cybersecurity and national security for about five years, most recently at CBS, where I reported exclusively on several stories — including the U.S. government’s covert efforts to force tech companies to hand over their source code in an effort to find vulnerabilities and conduct surveillance. And last year I revealed that the National Security Agency had its fifth data breach in as many years, and classified documents showed that a government data collection program was far wider than first thought and was collecting data on U.S. citizens.

Even with this story, my gut is mixed.

Where reporters across any topic and beat try to seek the truth, tapping information from the intelligence community is near impossible. For spies and diplomats, it’s illegal to share classified information with anyone and can be — and is — punishable by time in prison.

As a security reporter, you’re either incredibly well sourced or downright lucky. More often than not it’s the latter.

Naturally, people are skeptical of this “spy chip” story. On one side you have Bloomberg’s decades-long stellar reputation and reporting acumen, a thoroughly researched story citing more than a dozen sources — some inside the government and out — and presenting enough evidence to present a convincing case.

On the other, the sources are anonymous — likely because the information they shared wasn’t theirs to share or it was classified, putting sources in risk of legal jeopardy. But that makes accountability difficult. No reporter wants to say “a source familiar with the matter” because it weakens the story. It’s the reason reporters will tag names to spokespeople or officials so that it holds the powers accountable for their words. And, the denials from the companies themselves — though transparently published in full by Bloomberg — are not bulletproof in outright rejection of the story’s claims. These statements go through legal counsel and are subject to government regulation. These statements become a counterbalance — turning the story from an evidence-based report into a “he said, she said” situation.

That puts the onus on the reader to judge Bloomberg’s reporting. Reporters can publish the truth all they want, but ultimately it’s down to the reader to believe it or not.

In fairness to Bloomberg, chief among Apple’s complaints is a claim that Bloomberg’s reporters were vague in their questioning. Given the magnitude of the story, you don’t want to reveal all of your cards — but still want to seek answers and clarifications without having the subject tip off another news agency — a trick sometimes employed by the government in the hope of lighter coverage.

Yet, to Apple — and Amazon and other companies implicated by the report — they too might also be in the dark. Assuming there was an active espionage investigation into the alleged actions of a foreign government, you can bet that only a handful of people at these companies will be even cursorily aware of the situation. U.S. surveillance and counter-espionage laws restrict who can be told about classified information or investigations. Only those who need to be in the know are kept in a very tight loop — typically a company’s chief counsel. Often their bosses, the chief executive or president, are not told to avoid making false or misleading statements to shareholders.

It’s worth casting your mind back to 2013, days after the first Edward Snowden documents were published.

In the aftermath of the disclosure of PRISM, the NSA’s data pulling program that implicated several tech companies — including Apple, but not Amazon — the companies came out fighting, vehemently denying any involvement or connection. Was it a failure of reporting? Partially, yes. But the companies also had plausible deniability by cherry picking what they rebuffed. Despite a claim by the government that PRISM had “direct access” to tech companies’ servers, the companies responded that this wasn’t true. They didn’t, however, refute indirect access — which the companies wouldn’t be allowed to say in any case.

Critics of Bloomberg’s story have rightfully argued for more information — such as more technical data on the chip, its design and its functionality. Rightfully so — it’s entirely reasonable to want to know more. Jake Williams, a former NSA hacker turned founder of Rendition Infosec, told me that the story is “credible,” but “even if it turns out to be untrue, the capability exists and you need to architect your networks to detect this.”

I was hesitant to cover this at first given the complexity of the allegations and how explosive the claims are without also seeking confirmation. That’s not easy to do in an hour when Bloomberg’s reporters have been working for the best part of a year. Assuming Bloomberg did everything right — a cover story on its magazine, no less, which would have gone through endless editing and fact-checking before going to print — the reporters likely hit a wall and had nothing more to report, and went to print.

But Bloomberg’s delivery could have been better. Just as The New York Times does — even as recently as its coverage of President Trump’s tax affairs, Bloomberg missed an opportunity to be more open and transparent in how it came to the conclusions that it did. Journalism isn’t proprietary. It should be open to as many people as possible. If you’re not transparent in how you report things, you lose readers’ trust.

That’s where the story rests on shaky ground. Admittedly, as detailed and as well-sourced as the story is, you — and I — have to put a lot of trust and faith in Bloomberg and its reporters.

And in this day and age where “fake news” is splashed around wrongly and unfairly, for the sake of journalism, my only hope is they’re not wrong.

from: https://techcrunch.com/2018/10/04/bloomberg-spy-chip-murky-world-national-security-reporting/

Supply Chain Security is the Whole Enchilada, But Who’s Willing to Pay for It?

From time to time, there emerge cybersecurity stories of such potential impact that they have the effect of making all other security concerns seem minuscule and trifling by comparison. Yesterday was one of those times. Bloomberg Businessweek on Thursday published a bombshell investigation alleging that Chinese cyber spies had used a U.S.-based tech firm to secretly embed tiny computer chips into electronic devices purchased and used by almost 30 different companies. There aren’t any corroborating accounts of this scoop so far, but it is both fascinating and terrifying to look at why threats to the global technology supply chain can be so difficult to detect, verify and counter.

In the context of computer and Internet security, supply chain security refers to the challenge of validating that a given piece of electronics — and by extension the software that powers those computing parts — does not include any extraneous or fraudulent components beyond what was specified by the company that paid for the production of said item.

In a nutshell, the Bloomberg story claims that San Jose, Calif. based tech giant Supermicro was somehow caught up in a plan to quietly insert a rice-sized computer chip on the circuit boards that get put into a variety of servers and electronic components purchased by major vendors, allegedly including Amazon and Apple. The chips were alleged to have spied on users of the devices and sent unspecified data back to the Chinese military.

It’s critical to note up top that Amazon, Apple and Supermicro have categorically denied most of the claims in the Bloomberg piece. That is, their positions refuting core components of the story would appear to leave little wiggle room for future backtracking on those statements. Amazon also penned a blog post that more emphatically stated their objections to the Bloomberg piece.

Nevertheless, Bloomberg reporters write that “the companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation.”

The story continues:

Today, Supermicro sells more server motherboards than almost anyone else. It also dominates the $1 billion market for boards used in special-purpose computers, from MRI machines to weapons systems. Its motherboards can be found in made-to-order server setups at banks, hedge funds, cloud computing providers, and web-hosting services, among other places. Supermicro has assembly facilities in California, the Netherlands, and Taiwan, but its motherboards—its core product—are nearly all manufactured by contractors in China.

Many readers have asked for my take on this piece. I heard similar allegations earlier this year about Supermicro and tried mightily to verify them but could not. That in itself should be zero gauge of the story’s potential merit. After all, I am just one guy, whereas this is the type of scoop that usually takes entire portions of a newsroom to research, report and vet. By Bloomberg’s own account, the story took more than a year to report and write, and cites 17 anonymous sources as confirming the activity.

Most of what I have to share here is based on conversations with some clueful people over the years who would probably find themselves confined to a tiny, windowless room for an extended period if their names or quotes ever showed up in a story like this, so I will tread carefully around this subject.

The U.S. Government isn’t eager to admit it, but there has long been an unofficial inventory of tech components and vendors that are forbidden to buy from if you’re in charge of procuring products or services on behalf of the U.S. Government. Call it the “brown list, “black list,” “entity list” or what have you, but it’s basically an indelible index of companies that are on the permanent Shit List of Uncle Sam for having been caught pulling some kind of supply chain shenanigans.

More than a decade ago when I was a reporter with The Washington Post, I heard from an extremely well-placed source that one Chinese tech company had made it onto Uncle Sam’s entity list because they sold a custom hardware component for many Internet-enabled printers that secretly made a copy of every document or image sent to the printer and forwarded that to a server allegedly controlled by hackers aligned with the Chinese government.

That example gives a whole new meaning to the term “supply chain,” doesn’t it? If Bloomberg’s reporting is accurate, that’s more or less what we’re dealing with here in Supermicro as well.

But here’s the thing: Even if you identify which technology vendors are guilty of supply-chain hacks, it can be difficult to enforce their banishment from the procurement chain. One reason is that it is often tough to tell from the brand name of a given gizmo who actually makes all the multifarious components that go into any one electronic device sold today.

Take, for instance, the problem right now with insecure Internet of Things (IoT) devices — cheapo security cameras, Internet routers and digital video recorders — sold at places like Amazon and Walmart. Many of these IoT devices have become a major security problem because they are massively insecure by default and difficult if not also impractical to secure after they are sold and put into use.

For every company in China that produces these IoT devices, there are dozens of “white label” firms that market and/or sell the core electronic components as their own. So while security researchers might identify a set of security holes in IoT products made by one company whose products are white labeled by others, actually informing consumers about which third-party products include those vulnerabilities can be extremely challenging. In some cases, a technology vendor responsible for some part of this mess may simply go out of business or close its doors and re-emerge under different names and managers.

Mind you, there is no indication anyone is purposefully engineering so many of these IoT products to be insecure; a more likely explanation is that building in more security tends to make devices considerably more expensive and slower to market. In many cases, their insecurity stems from a combination of factors: They ship with every imaginable feature turned on by default; they bundle outdated software and firmware components; and their default settings are difficult or impossible for users to change.

We don’t often hear about intentional efforts to subvert the security of the technology supply chain simply because these incidents tend to get quickly classified by the military when they are discovered. But the U.S. Congress has held multiple hearings about supply chain security challenges, and the U.S. government has taken steps on several occasions to block Chinese tech companies from doing business with the federal government and/or U.S.-based firms.

Most recently, the Pentagon banned the sale of Chinese-made ZTE and Huawei phones on military bases, according to a Defense Department directive that cites security risks posed by the devices. The U.S. Department of Commerce also has instituted a seven-year export restriction for ZTE, resulting in a ban on U.S. component makers selling to ZTE.

Still, the issue here isn’t that we can’t trust technology products made in China. Indeed there are numerous examples of other countries — including the United States and its allies — slipping their own “backdoors” into hardware and software products.

Like it or not, the vast majority of electronics are made in China, and this is unlikely to change anytime soon. The central issue is that we don’t have any other choice right now. The reason is that by nearly all accounts it would be punishingly expensive to replicate that manufacturing process here in the United States.

Even if the U.S. government and Silicon Valley somehow mustered the funding and political will to do that, insisting that products sold to U.S. consumers or the U.S. government be made only with components made here in the U.S.A. would massively drive up the cost of all forms of technology. Consumers would almost certainly balk at buying these way more expensive devices. Years of experience has shown that consumers aren’t interested in paying a huge premium for security when a comparable product with the features they want is available much more cheaply.

Indeed, noted security expert Bruce Schneier calls supply-chain security “an insurmountably hard problem.”

“Our IT industry is inexorably international, and anyone involved in the process can subvert the security of the end product,” Schneier wrote in an opinion piece published earlier this year in The Washington Post. “No one wants to even think about a US-only anything; prices would multiply many times over. We cannot trust anyone, yet we have no choice but to trust everyone. No one is ready for the costs that solving this would entail.”

The Bloomberg piece also addresses this elephant in the room:

“The problem under discussion wasn’t just technological. It spoke to decisions made decades ago to send advanced production work to Southeast Asia. In the intervening years, low-cost Chinese manufacturing had come to underpin the business models of many of America’s largest technology companies. Early on, Apple, for instance, made many of its most sophisticated electronics domestically. Then in 1992, it closed a state-of-the-art plant for motherboard and computer assembly in Fremont, Calif., and sent much of that work overseas.

Over the decades, the security of the supply chain became an article of faith despite repeated warnings by Western officials. A belief formed that China was unlikely to jeopardize its position as workshop to the world by letting its spies meddle in its factories. That left the decision about where to build commercial systems resting largely on where capacity was greatest and cheapest. “You end up with a classic Satan’s bargain,” one former U.S. official says. “You can have less supply than you want and guarantee it’s secure, or you can have the supply you need, but there will be risk. Every organization has accepted the second proposition.”

Another huge challenge of securing the technology supply chain is that it’s quite time consuming and expensive to detect when products may have been intentionally compromised during some part of the manufacturing process. Your typical motherboard of the kind produced by a company like Supermicro can include hundreds of chips, but it only takes one hinky chip to subvert the security of the entire product.

Also, most of the U.S. government’s efforts to police the global technology supply chain seem to be focused on preventing counterfeits — not finding secretly added spying components.

Finally, it’s not clear that private industry is up to the job, either. At least not yet.

“In the three years since the briefing in McLean, no commercially viable way to detect attacks like the one on Supermicro’s motherboards has emerged—or has looked likely to emerge,” the Bloomberg story concludes. “Few companies have the resources of Apple and Amazon, and it took some luck even for them to spot the problem. ‘This stuff is at the cutting edge of the cutting edge, and there is no easy technological solution,’ one of the people present in McLean says. ‘You have to invest in things that the world wants. You cannot invest in things that the world is not ready to accept yet.’”

For my part, I try not to spin my wheels worrying about things I can’t change, and the supply chain challenges definitely fit into that category. I’ll have some more thoughts on the supply chain problem and what we can do about it in an interview to be published next week.

But for the time being, there are some things worth thinking about that can help mitigate the threat from stealthy supply chain hacks. Writing for this week’s newsletter put out by the SANS Institute, a security training company based in Bethesda, Md., editorial board member William Hugh Murray has a few provocative thoughts:

1. Abandon the password for all but trivial applications. Steve Jobs and the ubiquitous mobile computer have lowered the cost and improved the convenience of strong authentication enough to overcome all arguments against it.
2. Abandon the flat network. Secure and trusted communication now trump ease of any-to-any communication.
3. Move traffic monitoring from encouraged to essential.
4. Establish and maintain end-to-end encryption for all applications. Think TLS, VPNs, VLANs and physically segmented networks. Software Defined Networks put this within the budget of most enterprises.
5. Abandon the convenient but dangerously permissive default access control rule of “read/write/execute” in favor of restrictive “read/execute-only” or even better, “Least privilege.” Least privilege is expensive to administer but it is effective. Our current strategy of “ship low-quality early/patch late” is proving to be ineffective and more expensive in maintenance and breaches than we could ever have imagined.

from: https://krebsonsecurity.com/2018/10/supply-chain-security-is-the-whole-enchilada-but-whos-willing-to-pay-for-it/

04 OCT 2018

Decoding the Chinese Super Micro super spy-chip super-scandal: What do we know – and who is telling the truth?

Analysis Chinese government agents sneaked spy chips into Super Micro servers used by Amazon, Apple, the US government, and about 30 other organizations, giving Beijing’s snoops access to highly sensitive data, according to a bombshell Bloomberg report today.

The story, which has been a year in the making and covers events it says happened three years ago, had a huge impact on the markets: the company at the center of the story, San Jose-based Super Micro, saw its share price drop by nearly 50 per cent; likewise Apple’s share price dropped by just under two per cent, and Amazon’s dropped by more than two per cent.

But the article has been strongly denied by the three main companies involved: Apple, Amazon, and Super Micro. Each has issued strong and seemingly unambiguous statements denying the existence and discovery of such chips or any investigation by the US intelligence services into the surveillance implants.

These statements will have gone through layers of lawyers to make sure they do not open these publicly traded corporations to lawsuits and securities fraud claims down the line. Similarly, Bloomberg employs veteran reporters and layers of editors, who check and refine stories, and has a zero tolerance for inaccuracies.

So which is true: did the Chinese government succeed in infiltrating the hardware supply chain and install spy chips in highly sensitive US systems; or did Bloomberg’s journalists go too far in their assertions? We’ll dig in.

The report

First up, the key details of the exclusive. According to the report, tiny microchips that were made to look like signal conditioning couplers were added to Super Micro data center server motherboards manufactured by sub-contractors based in China.

Those spy chips were not on the original board designs, and were secretly added after factory bosses were pressured or bribed into altering the blueprints, it is claimed. The surveillance chips, we’re told, contained enough memory and processing power to effectively backdoor the host systems so that outside agents could, say, meddle with the servers and exfiltrate information.

The Bloomberg article is not particularly technical, so a lot of us are having to guesstimate how the hack worked. From what we can tell, the spy chip was designed to look like an innocuous component on the motherboard with a few connector pins – just enough for power and a serial interface, perhaps. One version was sandwiched between the fiberglass layers of the PCB, it is claimed.

The spy chip could have been placed electrically between the baseboard management controller (BMC) and its SPI flash or serial EEPROM storage containing the BMC’s firmware. Thus, when the BMC fetched and executed its code from this memory, the spy chip would intercept the signals and modify the bitstream to inject malicious code into the BMC processor, allowing its masters to control the BMC.

The BMC is a crucial component on a server motherboard. It allows administrators to remotely monitor and repair machines, typically over a network, without having to find the box in a data center, physically pull it out of the rack, fix it, and re-rack it. The BMC and its firmware can be told to power-cycle the server, reinstall or modify the host operating system, mount additional storage containing malicious code and data, access a virtual keyboard and terminal connected to the computer, and so on. If you can reach the BMC and its software, you have total control over the box.

With the BMC compromised, it is possible the alleged spies modified the controller’s firmware and/or the host operating system and software to allow attackers to connect in or allow data to flow out. We’ve been covering BMC security issues for a while.

Here is Bloomberg’s layman explanation for how that snoop-chip worked: the component “manipulated the core operating instructions that tell the server what to do as data move across a motherboard… this happened at a crucial moment, as small bits of the operating system were being stored in the board’s temporary memory en route to the server’s central processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information queue, injecting its own code or altering the order of the instructions the CPU was meant to follow.”

There are a few things to bear in mind: one is that it should be possible to detect weird network traffic coming from the compromised machine, and another is that modifying BMC firmware on the fly to compromise the host system is non-trivial but also not impossible. Various methods are described, here.

“It is technically plausible,” said infosec expert and US military veteran Jake Williams in a hastily organized web conference on Thursday morning. “If I wanted to do this, this is how I’d do it.”

The BMC would be a “great place to put it,” said Williams, because the controller has access to the server’s main memory, allowing it to inject backdoor code into the host operating system kernel. From there, it could pull down second-stage spyware and execute it, assuming this doesn’t set off any firewall rules.

A third thing to consider is this: if true, a lot of effort went into this surveillance operation. It’s not the sort of thing that would be added to any Super Micro server shipping to any old company – it would be highly targeted to minimize its discovery. If you’ve bought Super Micro kit, it’s very unlikely it has a spy chip in it, we reckon, if the report is correct. Other than Apple and Amazon, the other 30 or so organizations that used allegedly compromised Super Micro boxes included a major bank and government contractors.

A fourth thing is this: why go to the bother of smuggling another chip on the board, when a chip already due to be placed in the circuitry could be tampered with during manufacture, using bribes and pressure? Why not switch the SPI flash chip with a backdoored one – one that looks identical to a legit one? Perhaps the disguised signal coupler was the best way to go.

And a fifth thing: the chip allegedly fits on a pencil tip. That it can intercept and rewrite data on the fly from SPI flash or a serial EEPROM is not impossible. However, it has to contain enough data to replace the fetched BMC firmware code, that then alters the running operating system or otherwise implements a viable backdoor. Either the chip pictured in Bloomberg’s article is incorrect and just an illustration, and the actual device is larger, or there is state-of-the-art custom semiconductor fabrication involved here.

One final point: you would expect corporations like Apple and Amazon to have in place systems that detect not only unexpected network traffic, but also unexpected operating system states. It should be possible that alterations to the kernel and the stack of software above it should set off alarms during or after boot.

Bloomberg claims the chip was first noticed in 2015 in a third-party security audit of Super Micro servers that was carried out when Amazon was doing due diligence into a company called Elemental Technologies that it was thinking of acquiring. Elemental used Super Micro’s servers to do super-fast video processing.

Big problem

Amazon reported what it found to the authorities and, according to Bloomberg, that “sent a shudder” through the intelligence community because similar motherboards were in use “in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships.”

Around the same time, Apple also found the tiny chips, according to the report, “after detecting odd network activity and firmware problems.” Apple contacted the FBI and gave the agency access to the actual hardware. US intelligence agencies then tracked the hardware components backwards through the supply chain, and used their various spying programs to sift through intercepted communications, eventually ending up with a focus on four sub-contracting factories in China.

According to Bloomberg, the US intelligence agencies were then able to uncover how the seeding process worked: “Plant managers were approached by people who claimed to represent Super Micro or who held positions suggesting a connection to the government. The middlemen would request changes to the motherboards’ original designs, initially offering bribes in conjunction with their unusual requests. If that didn’t work, they threatened factory managers with inspections that could shut down their plants. Once arrangements were in place, the middlemen would organize delivery of the chips to the factories.”

This explanation seemingly passes the sniff test: it fits what we know of US intelligence agencies investigative approaches, their spy programs, and how the Chinese government works when interacting with private businesses.

The report then provides various forms of circumstantial evidence that adds weight to the idea that this all happened by pointing to subsequent actions of both Apple and Amazon. Apple ditched Super Micro entirely as a supplier, over the course of just a few weeks, despite planning to put in a massive order for thousands of motherboards. And Amazon sold off its Beijing data center to its local partner, Beijing Sinnet, for $300m.

from: https://www.theregister.co.uk/2018/10/04/supermicro_bloomberg/

07 SEP 2018

Supermicro wraps crypto-blanket around server firmware to hide it from malware injectors

Researchers claim to have discovered an exploitable flaw in the baseboard management controller (BMC) hardware used by Supermicro servers.

Security biz Eclypsium today said a weakness in the mechanism for updating a BMC’s firmware could be abused by an attacker to install and run malicious code that would be extremely difficult to remove.

A BMC is typically installed directly onto the motherboard of a server where it is able to directly control and manage the various hardware components of the server independent of the host and guest operating systems. It can also repair, alter, or reinstall the system software, and is remotely controlled over a network or dedicated channel by an administrator. It allows IT staff to manage, configure, and power cycle boxes from afar, which is handy for people looking after warehouses of machines.

Because BMCs operate at such a low level, they are also valuable targets for hackers.

In this case, Eclypsium says the firmware update code in Supermicro’s BMCs don’t bother to cryptographically verify whether or not the downloaded upgrade was issued by the manufacturer, leaving them vulnerable to tampering. The bug could be exploited to execute code that would then be able to withstand OS-level antivirus tools and reinstalls.

To do this, an attacker already on the data center network, or otherwise able to access the controllers, would need to intercept the firmware download, meddle with it, and pass it on to the hardware that will then blindly install it. Alternatively, a miscreant able to eavesdrop on and fiddle with internet traffic feeding into an organization could tamper with the IT team’s BMC firmware downloads, which again would be accepted by the controller.

“We found that the BMC code responsible for processing and applying firmware updates does not perform cryptographic signature verification on the provided firmware image before accepting the update and committing it to non-volatile storage,” says Eclypsium.

“This effectively allows the attacker to load modified code onto the BMC.”

In addition to running malware code beneath the OS level, the researchers said the flaw could also be used to permanently brick the BMC or even the entire server. Even worse, a potential attack wouldn’t even necessarily require physical access to the server itself.

“Because IPMI communications can be performed over the BMC LAN interface, this update mechanism could also be exploited remotely if the attacker has been able to capture the admin password for the BMC,” Eclypsium warned.

“This requires access to the systems management network, which should be isolated and protected from the production network. However, the implicit trust of management networks and interfaces may generate a false sense of security, leading to otherwise-diligent administrators practicing password reuse for convenience.”

Fortunately, Eclypsium says it has already reported the bug to Supermicro, who responded by adding signature verification to the firmware update tool, effectively plugging this vulnerability. Admins are being advised to get in touch with their Supermicro security contacts to get the fix in place.

from: https://www.theregister.co.uk/2018/09/07/supermicro_bmcs_hole/

18 OCT 2018

Supply Chain Security 101: An Expert’s View

Earlier this month I spoke at a cybersecurity conference in Albany, N.Y. alongside Tony Sager, senior vice president and chief evangelist at the Center for Internet Security and a former bug hunter at the U.S. National Security Agency. We talked at length about many issues, including supply chain security, and I asked Sager whether he’d heard anything about rumors that Supermicro — a high tech firm in San Jose, Calif. — had allegedly inserted hardware backdoors in technology sold to a number of American companies.

The event Sager and I spoke at was prior to the publication of Bloomberg Businessweek‘s controversial story alleging that Supermicro had duped almost 30 companies into buying backdoored hardware. Sager said he hadn’t heard anything about Supermicro specifically, but we chatted at length about the challenges of policing the technology supply chain.

Below are some excerpts from our conversation. I learned quite bit, and I hope you will, too.

Brian Krebs (BK): Do you think Uncle Sam spends enough time focusing on the supply chain security problem? It seems like a pretty big threat, but also one that is really hard to counter.

Tony Sager (TS): The federal government has been worrying about this kind of problem for decades. In the 70s and 80s, the government was more dominant in the technology industry and didn’t have this massive internationalization of the technology supply chain.

But even then there were people who saw where this was all going, and there were some pretty big government programs to look into it.

BK: Right, the Trusted Foundry program I guess is a good example.

TS: Exactly. That was an attempt to help support a U.S.-based technology industry so that we had an indigenous place to work with, and where we have only cleared people and total control over the processes and parts.

BK: Why do you think more companies aren’t insisting on producing stuff through code and hardware foundries here in the U.S.?

TS: Like a lot of things in security, the economics always win. And eventually the cost differential for offshoring parts and labor overwhelmed attempts at managing that challenge.

BK: But certainly there are some areas of computer hardware and network design where you absolutely must have far greater integrity assurance?

TS: Right, and this is how they approach things at Sandia National Laboratories [one of three national nuclear security research and development laboratories]. One of the things they’ve looked at is this whole business of whether someone might sneak something into the design of a nuclear weapon.

The basic design principle has been to assume that one person in the process may have been subverted somehow, and the whole design philosophy is built around making sure that no one person gets to sign off on what goes into a particular process, and that there is never unobserved control over any one aspect of the system. So, there are a lot of technical and procedural controls there.

But the bottom line is that doing this is really much harder [for non-nuclear electronic components] because of all the offshoring now of electronic parts, as well as the software that runs on top of that hardware.

BK: So is the government basically only interested in supply chain security so long as it affects stuff they want to buy and use?

TS: The government still has regular meetings on supply chain risk management, but there are no easy answers to this problem. The technical ability to detect something wrong has been outpaced by the ability to do something about it.

BK: Wait…what?

TS: Suppose a nation state dominates a piece of technology and in theory could plant something inside of it. The attacker in this case has a risk model, too. Yes, he could put something in the circuitry or design, but his risk of exposure also goes up.

Could I as an attacker control components that go into certain designs or products? Sure, but it’s often not very clear what the target is for that product, or how you will guarantee it gets used by your target. And there are still a limited set of bad guys who can pull that stuff off. In the past, it’s been much more lucrative for the attacker to attack the supply chain on the distribution side, to go after targeted machines in targeted markets to lessen the exposure of this activity.

BK: So targeting your attack becomes problematic if you’re not really limiting the scope of targets that get hit with compromised hardware.

TS: Yes, you can put something into everything, but all of a sudden you have this massive big data collection problem on the back end where you as the attacker have created a different kind of analysis problem. Of course, some nations have more capability than others to sift through huge amounts of data they’re collecting.

BK: Can you talk about some of the things the government has typically done to figure out whether a given technology supplier might be trying to slip in a few compromised devices among an order of many?

TS: There’s this concept of the “blind buy,” where if you think the threat vector is someone gets into my supply chain and subverts the security of individual machines or groups of machines, the government figures out a way to purchase specific systems so that no one can target them. In other words, the seller doesn’t know it’s the government who’s buying it. This is a pretty standard technique to get past this, but it’s an ongoing cat and mouse game to be sure.

BK: I know you said before this interview that you weren’t prepared to comment on the specific claims in the recent Bloomberg article, but it does seem that supply chain attacks targeting cloud providers could be very attractive for an attacker. Can you talk about how the big cloud providers could mitigate the threat of incorporating factory-compromised hardware into their operations?

TS: It’s certainly a natural place to attack, but it’s also a complicated place to attack — particularly the very nature of the cloud, which is many tenants on one machine. If you’re attacking a target with on-premise technology, that’s pretty simple. But the purpose of the cloud is to abstract machines and make more efficient use of the same resources, so that there could be many users on a given machine. So how do you target that in a supply chain attack?

BK: Is there anything about the way these cloud-based companies operate….maybe just sheer scale…that makes them perhaps uniquely more resilient to supply chain attacks vis-a-vis companies in other industries?

TS: That’s a great question. The counter positive trend is that in order to get the kind of speed and scale that the Googles and Amazons and Microsofts of the world want and need, these companies are far less inclined now to just take off-the-shelf hardware and they’re actually now more inclined to build their own.

BK: Can you give some examples?

TS: There’s a fair amount of discussion among these cloud providers about commonalities — what parts of design could they cooperate on so there’s a marketplace for all of them to draw upon. And so we’re starting to see a real shift from off-the-shelf components to things that the service provider is either designing or pretty closely involved in the design, and so they can also build in security controls for that hardware. Now, if you’re counting on people to exactly implement designs, you have a different problem. But these are really complex technologies, so it’s non-trivial to insert backdoors. It gets harder and harder to hide those kinds of things.

BK: That’s interesting, given how much each of us have tied up in various cloud platforms. Are there other examples of how the cloud providers can make it harder for attackers who might seek to subvert their services through supply chain shenanigans?

TS: One factor is they’re rolling this technology out fairly regularly, and on top of that the shelf life of technology for these cloud providers is now a very small number of years. They all want faster, more efficient, powerful hardware, and a dynamic environment is much harder to attack. This actually turns out to be a very expensive problem for the attacker because it might have taken them a year to get that foothold, but in a lot of cases the short shelf life of this technology [with the cloud providers] is really raising the costs for the attackers.

When I looked at what Amazon and Google and Microsoft are pushing for it’s really a lot of horsepower going into the architecture and designs that support that service model, including the building in of more and more security right up front. Yes, they’re still making lots of use of non-U.S. made parts, but they’re really aware of that when they do. That doesn’t mean these kinds of supply chain attacks are impossible to pull off, but by the same token they don’t get easier with time.

BK: It seems to me that the majority of the government’s efforts to help secure the tech supply chain come in the form of looking for counterfeit products that might somehow wind up in tanks and ships and planes and cause problems there — as opposed to using that microscope to look at commercial technology. Do you think that’s accurate?

TS: I think that’s a fair characterization. It’s a logistical issue. This problem of counterfeits is a related problem. Transparency is one general design philosophy. Another is accountability and traceability back to a source. There’s this buzzphrase that if you can’t build in security then build in accountability. Basically the notion there was you often can’t build in the best or perfect security, but if you can build in accountability and traceability, that’s a pretty powerful deterrent as well as a necessary aid.

BK: For example….?

TS: Well, there’s this emphasis on high quality and unchangeable logging. If you can build strong accountability that if something goes wrong I can trace it back to who caused that, I can trace it back far enough to make the problem more technically difficult for the attacker. Once I know I can trace back the construction of a computer board to a certain place, you’ve built a different kind of security challenge for the attacker. So the notion there is while you may not be able to prevent every attack, this causes the attacker different kinds of difficulties, which is good news for the defense.

BK: So is supply chain security more of a physical security or cybersecurity problem?

TS: We like to think of this as we’re fighting in cyber all the time, but often that’s not true. If you can force attackers to subvert your supply chain, they you first off take away the mid-level criminal elements and you force the attackers to do things that are outside the cyber domain, such as set up front companies, bribe humans, etc. And in those domains — particularly the human dimension — we have other mechanisms that are detectors of activity there.

BK: What role does network monitoring play here? I’m hearing a lot right now from tech experts who say organizations should be able to detect supply chain compromises because at some point they should be able to see truckloads of data leaving their networks if they’re doing network monitoring right. What do you think about the role of effective network monitoring in fighting potential supply chain attacks.

TS:  I’m not so optimistic about that. It’s too easy to hide. Monitoring is about finding anomalies, either in the volume or type of traffic you’d expect to see. It’s a hard problem category. For the US government, with perimeter monitoring there’s always a trade off in the ability to monitor traffic and the natural movement of the entire Internet towards encryption by default. So a lot of things we don’t get to touch because of tunneling and encryption, and the Department of Defense in particular has really struggled with this.

Now obviously what you can do is man-in-the-middle traffic with proxies and inspect everything there, and the perimeter of the network is ideally where you’d like to do that, but the speed and volume of the traffic is often just too great.

BK: Isn’t the government already doing this with the “trusted internet connections” or Einstein program, where they consolidate all this traffic at the gateways and try to inspect what’s going in and out?

TS: Yes, so they’re creating a highest volume, highest speed problem. To monitor that and to not interrupt traffic you have to have bleeding edge technology to do that, and then handle a ton of it which is already encrypted. If you’re going to try to proxy that, break it out, do the inspection and then re-encrypt the data, a lot of times that’s hard to keep up with technically and speed-wise.

BK: Does that mean it’s a waste of time to do this monitoring at the perimeter?

TS: No. The initial foothold by the attacker could have easily been via a legitimate tunnel and someone took over an account inside the enterprise. The real meaning of a particular stream of packets coming through the perimeter you may not know until that thing gets through and executes. So you can’t solve every problem at the perimeter. Some things only because obvious and make sense to catch them when they open up at the desktop.

BK: Do you see any parallels between the challenges of securing the supply chain and the challenges of getting companies to secure Internet of Things (IoT) devices so that they don’t continue to become a national security threat for just about any critical infrastructure, such as with DDoS attacks like we’ve seen over the past few years?

TS: Absolutely, and again the economics of security are so compelling. With IoT we have the cheapest possible parts, devices with a relatively short life span and it’s interesting to hear people talking about regulation around IoT. But a lot of the discussion I’ve heard recently does not revolve around top-down solutions but more like how do we learn from places like the Food and Drug Administration about certification of medical devices. In other words, are there known characteristics that we would like to see these devices put through before they become in some generic sense safe.

BK: How much of addressing the IoT and supply chain problems is about being able to look at the code that powers the hardware and finding the vulnerabilities there? Where does accountability come in?

TS: I used to look at other peoples’ software for a living and find zero-day bugs. What I realized was that our ability to find things as human beings with limited technology was never going to solve the problem. The deterrent effect that people believed someone was inspecting their software usually got more positive results than the actual looking. If they were going to make a mistake – deliberately or otherwise — they would have to work hard at it and if there was some method of transparency, us finding the one or two and making a big deal of it when we did was often enough of a deterrent.

BK: Sounds like an approach that would work well to help us feel better about the security and code inside of these election machines that have become the subject of so much intense scrutiny of late.

TS: We’re definitely going through this now in thinking about the election devices. We’re kind of going through this classic argument where hackers are carrying the noble flag of truth and vendors are hunkering down on liability. So some of the vendors seem willing to do something different, but at the same time they’re kind of trapped now by the good intentions of open vulnerability community.

The question is, how do we bring some level of transparency to the process, but probably short of vendors exposing their trade secrets and the code to the world? What is it that they can demonstrate in terms of cost effectiveness of development practices to scrub out some of the problems before they get out there. This is important, because elections need one outcome: Public confidence in the outcome. And of course, one way to do that is through greater transparency.

BK: What, if anything, are the takeaways for the average user here? With the proliferation of IoT devices in consumer homes, is there any hope that we’ll see more tools that help people gain more control over how these systems are behaving on the local network?

TS: Most of [the supply chain problem] is outside the individual’s ability to do anything about, and beyond ability of small businesses to grapple with this. It’s in fact outside of the autonomy of the average company to figure it out. We do need more national focus on the problem.

It’s now almost impossible to for consumers to buy electronics stuff that isn’t Internet-connected. The chipsets are so cheap and the ability for every device to have its own Wi-Fi chip built in means that [manufacturers] are adding them whether it makes sense to or not. I think we’ll see more security coming into the marketplace to manage devices. So for example you might define rules that say appliances can talk to the manufacturer only. 

We’re going to see more easy-to-use tools available to consumers to help manage all these devices. We’re starting to see the fight for dominance in this space already at the home gateway and network management level. As these devices get more numerous and complicated, there will be more consumer oriented ways to manage them. Some of the broadband providers already offer services that will tell what devices are operating in your home and let users control when those various devices are allowed to talk to the Internet.

Since Bloomberg’s story broke, The U.S. Department of Homeland Security and the National Cyber Security Centre, a unit of Britain’s eavesdropping agency, GCHQ, both came out with statements saying they had no reason to doubt vehement denials by Amazon and Apple that they were affected by any incidents involving Supermicro’s supply chain security. Apple also penned a strongly-worded letter to lawmakers denying claims in the story.

Meanwhile, Bloomberg reporters published a follow-up story citing new, on-the-record evidence to back up claims made in their original story.

from: https://krebsonsecurity.com/2018/10/supply-chain-security-101-an-experts-view/

09 OCT 2018 – Bloomberg Follow-Up

New Evidence of Hacked Supermicro Hardware Found in U.S. Telecom

A major U.S. telecommunications company discovered manipulated hardware from Super Micro Computer Inc. in its network and removed it in August, fresh evidence of tampering in China of critical technology components bound for the U.S., according to a security expert working for the telecom company.

The security expert, Yossi Appleboum, provided documents, analysis and other evidence of the discovery following the publication of an investigative report in Bloomberg Businessweek that detailed how China’s intelligence services had ordered subcontractors to plant malicious chips in Supermicro server motherboards over a two-year period ending in 2015.

Appleboum previously worked in the technology unit of the Israeli Army Intelligence Corps and is now co-chief executive officer of Sepio Systems in Gaithersburg, Maryland. His firm specializes in hardware security and was hired to scan several large data centers belonging to the telecommunications company. Bloomberg is not identifying the company due to Appleboum’s nondisclosure agreement with the client. Unusual communications from a Supermicro server and a subsequent physical inspection revealed an implant built into the server’s Ethernet connector, a component that’s used to attach network cables to the computer, Appleboum said.

The executive said he has seen similar manipulations of different vendors’ computer hardware made by contractors in China, not just products from Supermicro. “Supermicro is a victim — so is everyone else,” he said. Appleboum said his concern is that there are countless points in the supply chain in China where manipulations can be introduced, and deducing them can in many cases be impossible. “That’s the problem with the Chinese supply chain,” he said.

Supermicro, based in San Jose, California, gave this statement: “The security of our customers and the integrity of our products are core to our business and our company values. We take care to secure the integrity of our products throughout the manufacturing process, and supply chain security is an important topic of discussion for our industry. We still have no knowledge of any unauthorized components and have not been informed by any customer that such components have been found. We are dismayed that Bloomberg would give us only limited information, no documentation, and half a day to respond to these new allegations.”

Bloomberg News first contacted Supermicro for comment on this story on Monday at 9:23 a.m. Eastern time and gave the company 24 hours to respond.

Supermicro said after the earlier story that it “strongly refutes” reports that servers it sold to customers contained malicious microchips. China’s embassy in Washington did not return a request for comment Monday. In response to the earlier Bloomberg Businessweek investigation, China’s Ministry of Foreign Affairs didn’t directly address questions about the manipulation of Supermicro servers but said supply chain security is “an issue of common concern, and China is also a victim.”

Supermicro shares plunged 41 percent last Thursday, the most since it became a public company in 2007, following the Bloomberg Businessweek revelations about the hacked servers. They fell as much as 27 percent on Tuesday after the latest story.

The more recent manipulation is different from the one described in the Bloomberg Businessweek report last week, but it shares key characteristics: They’re both designed to give attackers invisible access to data on a computer network in which the server is installed; and the alterations were found to have been made at the factory as the motherboard was being produced by a Supermicro subcontractor in China.

Based on his inspection of the device, Appleboum determined that the telecom company’s server was modified at the factory where it was manufactured. He said that he was told by Western intelligence contacts that the device was made at a Supermicro subcontractor factory in Guangzhou, a port city in southeastern China. Guangzhou is 90 miles upstream from Shenzhen, dubbed the `Silicon Valley of Hardware,’ and home to giants such as Tencent Holdings Ltd. and Huawei Technologies Co. Ltd.

The tampered hardware was found in a facility that had large numbers of Supermicro servers, and the telecommunication company’s technicians couldn’t answer what kind of data was pulsing through the infected one, said Appleboum, who accompanied them for a visual inspection of the machine. It’s not clear if the telecommunications company contacted the FBI about the discovery. An FBI spokeswoman declined to comment on whether it was aware of the finding.

AT&T Inc. spokesman Fletcher Cook said, “These devices are not part of our network, and we are not affected.” A Verizon Communications Inc. spokesman said “we’re not affected.”

The manipulation of the Ethernet connector appeared to be similar to a method also used by the U.S. National Security Agency, details of which were leaked in 2013. In e-mails, Appleboum and his team refer to the implant as their “old friend,” because he said they had previously seen several variations in investigations of hardware made by other companies manufacturing in China.

In Bloomberg Businessweek’s report, one official said investigators found that the Chinese infiltration through Supermicro reached almost 30 companies, including Amazon.com Inc. and Apple Inc. Both Amazon and Apple also disputed the findings. The U.S. Department of Homeland Security said it has “no reason to doubt” the companies’ denials of Bloomberg Businessweek’s reporting.

People familiar with the federal investigation into the 2014-2015 attacks say that it is being led by the FBI’s cyber and counterintelligence teams, and that DHS may not have been involved. Counterintelligence investigations are among the FBI’s most closely held and few officials and agencies outside of those units are briefed on the existence of those investigations.

Appleboum said that he’s consulted with intelligence agencies outside the U.S. that have told him they’ve been tracking the manipulation of Supermicro hardware, and the hardware of other companies, for some time. 

In response to the Bloomberg Businessweek story, the Norwegian National Security Authority said last week that it had been “aware of an issue” connected to Supermicro products since June.  Trond Ovstedal, a spokesman for the agency, later added to that statement, saying the agency was alerted to the concerns by someone who had heard of them via Bloomberg’s news gathering efforts. In its initial statement, the authority couldn’t confirm the details of Bloomberg’s reporting, but said that it has recently been in dialogue with partners over the issue.

Hardware manipulation is extremely difficult to detect, which is why intelligence agencies invest billions of dollars in such sabotage. The U.S. is known to have extensive programs to seed technology heading to foreign countries with spy implants, based on revelations from former CIA employee Edward Snowden. But China appears to be aggressively deploying its own versions, which take advantage of the grip the country has over global technology manufacturing.

Three security experts who have analyzed foreign hardware implants for the U.S. Department of Defense confirmed that the way Sepio’s software detected the implant is sound. One of the few ways to identify suspicious hardware is by looking at the lowest levels of network traffic. Those include not only normal network transmissions, but also analog signals — such as power consumption — that can indicate the presence of a covert piece of hardware.

In the case of the telecommunications company, Sepio’s technology detected that the tampered Supermicro server actually appeared on the network as two devices in one. The legitimate server was communicating one way, and the implant another, but all the traffic appeared to be coming from the same trusted server, which allowed it to pass through security filters.  

Appleboum said one key sign of the implant is that the manipulated Ethernet connector has metal sides instead of the usual plastic ones. The metal is necessary to diffuse heat from the chip hidden inside, which acts like a mini computer. “The module looks really innocent, high quality and ‘original’ but it was added as part of a supply chain attack,” he said.

The goal of hardware implants is to establish a covert staging area within sensitive networks, and that’s what Appleboum and his team concluded in this case. They decided it represented a serious security breach, along with multiple rogue electronics also detected on the network, and alerted the client’s security team in August, which then removed them for analysis. Once the implant was identified and the server removed, Sepio’s team was not able to perform further analysis on the chip.

The threat from hardware implants “is very real,” said Sean Kanuck, who until 2016 was the top cyber official inside the Office of the Director of National Intelligence. He’s now director of future conflict and cyber security for the International Institute for Strategic Studies in Washington. Hardware implants can give attackers power that software attacks don’t.

“Manufacturers that overlook this concern are ignoring a potentially serious problem,” Kanuck said. “Capable cyber actors — like the Chinese intelligence and security services — can access the IT supply chain at multiple points to create advanced and persistent subversions.”

One of the keys to any successful hardware attack is altering components that have an ample power supply to them, a daunting challenge the deeper into a motherboard you go. That’s why peripherals such as keyboards and mice are also perennial favorites for intelligence agencies to target, Appleboum said.

In the wake of Bloomberg’s reporting on the attack against Supermicro products, security experts say that teams around the world, from large banks and cloud computing providers to small research labs and startups, are analyzing their servers and other hardware for modifications, a stark change from normal practices. Their findings won’t necessarily be made public, since hardware manipulation is typically designed to access government and corporate secrets, rather than consumer data.

National security experts say a key problem is that, in a cybersecurity industry approaching $100 billion in revenue annually, very little of that has been spent on inspecting hardware for tampering. That’s allowed intelligence agencies around the world to work relatively unimpeded, with China holding a key advantage.

“For China, these efforts are all-encompassing,” said Tony Lawrence, CEO of VOR Technology, a Columbia, Maryland-based contractor to the intelligence community. “There is no way for us to identify the gravity or the size of these exploits — we don’t know until we find some. It could be all over the place — it could be anything coming out of China. The unknown is what gets you and that’s where we are now. We don’t know the level of exploits within our own systems.”

from: https://www.bloomberg.com/news/articles/2018-10-09/new-evidence-of-hacked-supermicro-hardware-found-in-u-s-telecom