The AI boom is running into a copper problem

Silicon Valley's AI infrastructure boom has created a surprisingly physical problem: there's only so much data you can push through copper wires before heat, distance, and power consumption become overwhelming.

That's why investors, chipmakers, and cloud giants are suddenly paying close attention to photonics, using light instead of electrical signals to move data between AI chips and servers.

I recently visited Lightmatter's Silicon Valley headquarters, where the startup demonstrated its latest photonics hardware for AI data centers. After the event, I sat down with Lightmatter CEO Nick Harris to talk about why optics may become essential infrastructure for the AI era.

Harris looks annoyingly young and he's annoyingly bright, with a Ph.D. from MIT. And Lightmatter is already annoyingly successful, having raised $850 million from huge backers including Google, Fidelity, and T. Rowe Price.

On Tuesday, Lightmatter joined Nvidia's NVLink Fusion ecosystem, which should help the startup's technology work better with Nvidia's dominant AI hardware.

Here's my conversation with Harris, lightly edited for clarity and length.

Q: Why are AI companies suddenly interested in photonics?

Harris said the AI industry has reached a point where scaling performance is less about making individual chips faster and more about connecting huge numbers of GPUs together efficiently.

Today's AI systems rely heavily on copper connections between GPUs. That works fine at smaller scales. But as companies connect hundreds or thousands of GPUs together for frontier AI models, copper becomes a bottleneck because electrical signals weaken over relatively short distances and generate huge amounts of heat.

Photonics uses light traveling through fiber instead. That allows data to move farther, faster, and with less energy.

"Let's say you have 500 GPUs and you have copper that's linking those together so they can talk in what's known as a scale up domain. People run model training workloads on those systems."

With copper wiring, you need four separate racks of GPU servers to get to 500, but "when you switch that to being all optical instead of copper, you can connect all 500 GPs directly," Harris told me. "Your time to train the AI model drops dramatically. Think about frontier models like Claude. It's 3 times faster."

"Whoever gets the technology first and the frontier race is going to be releasing models faster. They have two choices. One is I can release the models every month, or I can take three months but have a way bigger model," he added.

"Say I'm energy constrained. For the same amount of power I'm getting 3x the performance. This makes that gigawatt feel like three gigawatts. Or, you use the same amount of power for a third the time," Harris explained.

Q: What's wrong with copper connections inside AI data centers?

"Copper only goes about a meter," Harris said, because electrical signals weaken rapidly as they travel through copper cables.

"The signal launches the electrical signals in the wire and it gets smaller and smaller as it goes. After about a meter, the data's lost."

That physical limitation creates another problem: heat. Racks of GPU servers in AI data centers are now packed tightly together because copper cables can only reach short distances.

"They're jammed together," Harris said. "The problem is I need them to be all on top of each other so the copper can reach. But the downside of that is the cooling's really hard."

Photonics changes that because light signals can travel much farther and faster without degrading, so GPU servers and racks can be spaced out more.

"Optics doesn't care how far things are," Harris said. "They could be a kilometer away."

That gives data center operators more flexibility in how they design and cool AI clusters, potentially saving more money on electricity for cooling these systems.

Q: What is BiDi, and why does it matter?

One of the more practical innovations Lightmatter is working on involves reducing the sheer amount of cabling required inside AI data centers.

Harris said some next-generation AI clusters require about 300 miles of cables. Lightmatter aims to cut that in half with a technology called BiDi, short for bidirectional communication.

"Normally with either copper or optical, if I want to make a connection between this GPU and that GPU, I have to do two wires," Harris explained. "One of them is transmit, the other one is receive."

Lightmatter's approach combines both directions into a single cable.

For hyperscale data centers, that reduction matters because cables take up space, generate heat, complicate maintenance, and add cost. Cutting total fiber requirements from 300 miles to 150 miles could significantly simplify the construction of massive AI clusters, according to Lightmatter

Q: Why wasn't photonics adopted earlier?

Harris said the main issue was cost. "Photonics was too expensive," he said.

That's changing because manufacturing techniques have improved and AI infrastructure needs have exploded, Harris explained.

"The people who architected these systems, they look for a reliable 2x improvement. 2x the bandwidth, 2x the performance, and they do it on a regular cadence. There was still a little bit of performance to squeeze out a copper and now that's over. And more than that, there's gas on the fire, which is that people realize the first to adopt and deploy photonics — Nvidia will probably be the one — has a huge performance advantage."

"So it used to be 'only switch out of necessity.' Now it's switching for competitive advantage," Harris said.

Sign up for BI's Tech Memo newsletter here. Reach out to me via email at [email protected].