Richard Ho, who helped create Google’s first TPU chip, built OpenAI’s semiconductor unit from scratch. Now he’s dealing with deploying its first chips, doomerism and “competing among friends” with Nvidia.
Two years ago, shortly after Richard Ho, OpenAI’s vice president of hardware, joined the AI giant, he traveled to Asia with CEO Sam Altman and a cadre of researchers to meet with the biggest chip manufacturers in the world: TSMC, Samsung and SK Hynix.
Altman’s message was blunt. OpenAI’s already insatiable demand for computing power was only going to grow and the semiconductor industry had to start moving faster — a lot faster. “Look, in a few years we’re going to need a lot more silicon,” Ho recalls Altman telling them. “Please start building more fabs.”
Those companies did build more fabs, the fabrication plants where computer chips are physically assembled. But it wasn’t enough to satisfy OpenAI’s bottomless appetite, Ho tells Forbes. It was partly because of that crunch that OpenAI decided to just build its own hardware, and Altman tapped Ho to build a team from scratch. Two years later, the company announced its first semiconductor, an inference chip named Jalapeno. In June, OpenAI said it had built the chip in nine months and plans to start getting them into data centers by the end of the year.
“One of the constraints that we’re under now is that if we had started building [more fabs] like two and a half years ago, we would be in a much better position,” Ho tells Forbes.
Before joining OpenAI, Ho, a Google veteran, helped design the search giant’s first TPU chip, custom-built silicon tailored specifically for artificial intelligence. He sat down with Forbes for a wide-ranging conversation, where he talked about building the team, choosing Broadcom as its semiconductor partner over Nvidia and AMD, and how the doomerism debate has trickled into his world of building chips. This transcript has been edited for length and clarity.
Richard Nieva: I’ll start with a simple one: Why is the chip named Jalapeno?
Richard Ho: We don’t take ourselves too seriously. We were in the cafeteria in our office and there’s a spice bar with all the sauces, and our team always gravitates to that. Our team is actually called the Hot Peppers. We named ourselves after that. And so it sort of seemed natural we should name our device and our project after something that we love, which is spicy stuff.
Nieva: Why does OpenAI need to build a chip?
Courtesy OpenAI
Ho: I think it is really about owning the full stack of that infrastructure. So obviously, OpenAI has the models, does a good job with that, and has the software stack to take the information from the models and take it all the way down into the chips, which for the most part have been GPUs up to now. If we’re able to control [the stack], this does give us the ability to do more and do things better. So if we have control of our data centers, we have control of the chips, the systems that go into it, and the model and the software, this lets us do things more efficiently. Instead of at every layer there being a very hard interface, you can merge and move stuff around to get a little bit better on efficiency. And that ultimately leads to more intelligence at a lower power point for the user, which is our key metric here: how efficiently can we deliver intelligence per watt of energy that we’re going to be consuming?
Nieva: It’s an expensive and intensive process to build your own chip. Can you take me back to how the idea came about? What were those conversations like, and what did Sam Altman say to you?
Ho: My initial conversations with him were really like, “Do we want to build our own device?” It wasn’t a given. It was like, let’s look at what our options are. Can we get what we need by asking the existing vendors — AMD, Nvidia, Intel — to do something that we would want? And it turned out, because they have a very general market, it was very hard to get exactly what we wanted out of them. And they had legacy architectures that they had to preserve. And what we decided — and what we saw after we did that very neutral analysis — was that if we took a blank slate approach to designing a chip for this, it would become much more efficient than anything that was built on legacy architecture. And so that was when Sam basically allowed us to say, “Okay, let’s start it.”
We’re going to do a small team. We’re not going to build an Intel-sized team or even an Nvidia-sized team, but can we do something with a small, elite team, and use our own AI to accelerate that and make it good? And at that time, which was in like early ‘24 when we started talking about this, the models weren’t great at hardware, to be very honest. And so there was a little bit of doubt in our heads as to how much benefit the models were going to give us. But over the course of the following months and the following year, the models improved a lot, and then we managed to get a lot of benefit from using the models to accelerate the timeline.
Nieva: You said it’s a small team. How big is it exactly?
Ho: It’s about 100 people. But during the path of this, it was close to 70 or so during the average. [Compared to teams at Nvidia or Intel, which can number 500 more more.] But it’s not just chip people. It’s the whole system: rack, board design, power, thermal, supply chain software. So in terms of the number of chip people, it’s like at least two orders of magnitude smaller than a typical chip team.
Nieva: When you talk about efficiency, do you mean it’s tailored specifically to your workloads?
Ho: It’s a very flexible, programmable chip. It’s not specific to OpenAI models. It can take almost any model, and that’s important because, as the researchers keep working on it and as we see the models get better and better and better, you want to have a system that’s very adaptable to changes in the model that the researchers come up with. So that when we do [future models], this hardware will still be very efficient at running those models. That was really a design goal, and I think we’ve achieved it with the results that we’ve demonstrated here.
Nieva: Why did you decide to go that more general inference route? OpenAI is such a compute intensive company. Why not tailor it more to your specific needs?
Ho: This is where I put on the hat of a hardware person. Hardware takes a long time to do, right? We said it took nine months, and that is one of the fastest, if not the fastest, demonstrated processes of going from initial design or no design to getting a device out to manufacture. But even nine months in the world of AI is a long time. And it still takes us time to then put it into the data centers and deploy it. And so, whatever hardware you put in has to be adaptable and ready to change with better models and new ideas that are coming constantly in the world of AI research.
And so as a hardware person, making it very specific is super scary because you’re then looking every week: What’s the latest research? What’s the latest paper that’s come out? Are we able to adapt to it? And so by making a more programmable, more general-purpose device, you’re able to have that reassurance that you can adapt to it. Having said that, it is a little bit more specialized than a GPU — a standard GPU, which is very, very programmable in all ways. We are specialized towards transformers and LLMs. So within that category of devices, we should be very adaptable. But if we’re doing other stuff, maybe there’s better hardware for that.
Nieva: With such general inference capability, is part of the plan to sell the chips to other companies, or to rent them out via a cloud?
Ho: We’re not going to say that’s off the table, but we have so much demand internally that it’s going to take us a while to fulfill our own internal demand. Because it is so fungible, I think it’s worth having a discussion about, but it’s not something we’re focused on at all right now.
Nieva: Why did you choose to work with Broadcom for this chip, and not Nvidia, AMD or Marvell?
Ho: They’re all very good options. I would any day go ahead and use them. With Broadcom in particular, we actually ran a very neutral partner evaluation process. And for us, key things are some of the analog IP [intellectual property]. So when you talk about an accelerator, it’s not just a single chip. In the old days when we talked about a PC, it was the CPU. How good was that CPU? How much compute did that CPU have? In these days of AI, it’s clusters. It’s massive supercomputers that have many, many chips together, all interconnected in a rack and in multiple racks together to make the entire pod that is actually serving this stuff. So for that, you need a lot of communication. You need a lot of very advanced memory — memory that basically allows you to store the data, get access to it really quickly, and then be able to get a lot of that data really quickly.
Nieva: So Broadcom’s IP was better suited for you than, say, Nvidia’s?
Ho: There is that IP available from the other vendors. I think we just evaluated this, and on the overall balance, Broadcom was a really good choice for us. And we also have experience with them. So my team, myself, we came from Google. I was there [when it built] the first TPU. Many of my team actually comes from that original TPU team. Designing the TPU, we spent a lot of time working with Broadcom. So it’s the same set of people. So if we wanted to do this project really fast, that was a little bit of lubrication to make that happen.
Nieva: You mentioned one factor in building the chip was cost efficiency. Obviously, Nvidia machines are expensive. Does this lower your dependence on Nvidia?
Ho: I think that Nvidia will continue to be part of the fleet. So the main thing here is that Sam saw the vision for compute for OpenAI. We have such a strong demand for more compute that this is going to supplement things. So we’re continuing to be a very strong partner of Nvidia. We have contracts with AMD, with Cerebras, [with] a lot of companies. Where does Jalapeno fit in that? It is adding more capacity.
Nieva: How did you navigate the relationship with Nvidia, going from customer to fellow chip builder as well?
Ho: The way I view it — and I’m not sure this is the way [CEO] Jensen [Huang] or Nvidia view it — we’re competing among friends. We’re giving them a lot of business, and they’re a critical part of us. But if we can do better and prod them to do better, we should do that, right? And so that’s kind of what we’re saying here. We got this. You can do better. And if you do better, we benefit from that. So that’s all good. I would say that there was a lot of skepticism in the industry in general about a first-generation chip from an AI lab, that it could be ready to go to production, and so I’m very proud of the team. I’m proud of the support we got within OpenAI to be able to do that.
Nieva: Do you have a projection of how much money OpenAI will save over the next decade because of this?
Ho: I haven’t done that analysis. I’m sorry. Maybe we can try to get back to you with an answer on that one. That one is harder for me to come up with live. [In a follow up email, an OpenAI spokesperson said the company doesn’t have an estimate.]
Nieva: What’s the biggest difference between designing chips for Google versus versus OpenAI?
Ho: I was very privileged to have been there and seeing [the TPU] grow up. I think the main difference I see is how close we are with the researchers here, and with the software team here, because it is a much smaller company. So as a smaller company, we are very much still in startup mode. That level of interaction and that level of intellectual ping-ponging is amazing here. And having that kind of high-level vision that we really need to do this — it is existential for the company to have really good infrastructure, which is coming from Sam and [OpenAI Cofounder and President] Greg Brockman — has really driven everybody to to work in this tight way and to go really fast.
Nieva: Jalapeno is an inference chip. Would you do a training chip?
Ho: Training means a lot. I mean, let’s not kid ourselves. It needs a lot of compute power. But where the growth we see happening is on the inference side, and that’s the area where we see a bit of a gap. And so being able to bring down the cost of inference really helps with our entire product portfolio. And so that’s where our focus is now.
Specifically for pre-training, it’s not out of the question. It’s a matter of priorities and a matter of trying to get to the things that matter most for the company and for our users first, and then moving on to that. So I wouldn’t put it out of question, but I’m not focused on that right now.
Nieva: OpenAI has said it will start to get Jalapeno into data centers by the end of the year. When will it be used in full production?
Ho: Our intent is to have something in production in the first half of ‘27. So some users on some models may be on Jalapeno by that time. That’s our goal. It’s a stretch goal because there’s a lot of software work and a lot of other preparation work that’s happening.
Nieva: How many gigawatts?
Ho: I don’t think we’ve shared that publicly.
Nieva: There’s been a lot of talk about doomerism and slowdowns. I’m curious how that trickles into your world at all. If there’s a slowdown in development, does it affect how you think about designing chips and building out data centers?
Ho: Super critical point here. I think the discussion is super important generally, and I think the company is very focused on making sure that we have the right guidelines in place, the right safety guardrails, and all the rest of the monitoring that we need for that. So that’s absolutely critical. And by the way, that needs a lot of compute. And so the way to think about it is that the infrastructure is a thing that’s allowing people to make these choices. And so you can choose to use some of the compute for more safety, more guardrails. More monitoring and stuff like that. Or you can choose it to advance the models. And I think that’s a balance that is done at the policy level for the company.
And so, from our viewpoint, the infrastructure needs to continue to become more efficient and continue to grow. And I see that as really expanding what we thought we needed. We thought we needed a certain amount of compute just to serve users. And if we’re going to have a lot more safety guardrails and a lot more monitoring, boy, I better add a lot more compute to do that as well, right?
Want to see more Forbes articles on your feed? Tap here to make Forbes Australia a preferred source on Google.
Look back on the week that was with hand-picked articles from Australia and around the world. Sign up to the Forbes Australia newsletter here or become a member here.
