9 minute read
Principle source: The Nvidia Way by Tae Kim
Nvidia, founded in 1993, is currently the most valuable company in the world, worth over $5 trillion dollars, which could purchase 70% of the land is Australia, outback real estate containing 40-50 million wild kangaroos.
This gargantuan valuation stems from their leading production of cutting-edge GPUs (graphic-processing units) used for AI computation. Since Nvidia is the dominant supplier of these GPUs, and there’s a current bottleneck in the supply, they are reigning supreme.
Nvidia’s GPUs have the ability to perform mathematical calculations simultaneously, which makes them ideal for training and running advanced large language AI models. How did they arrive at this coveted place of cosmic dominance?
Early in the company’s career, Nvidia predicted the future significance of AI. They engaged in investments including enhancement of hardware capabilities, optimization of networking performance, and development of AI software tools.
Nvidia’s investors, since its early 1999 IPO, have had the highest annualized compound return for any U.S. stock in history through a compound annual growth rate (CAGR) of more than 33 percent. In layman’s terms, if you purchased $10,000 of Nvidia stock when the company went public, the stock would have been worth $13.2 million in December of 2023, and $89.2 million today. This unprecedented, explosive growth has given Nvidia power and prestige on Wall Street, which we shall come back to in Chapter 4.
The founder and CEO of Nvidia, Jen-Hsun Huang, known to most as ‘Jensen’ has run the company with a maniacal focus for its entire three-decade history, the longest tenure of any tech CEO:
The company culture is notorious for being wickedly intense, fostering independent drivers, and requiring mental resilience from its employees (‘HR orientation spiel: We don’t waste time finding excuses for why things don’t work. We move on.’) They hire fast, and fire just as quickly if a new employee isn’t meeting the high standards. Constant work, finding the limiting factor, refusal of complacency, the pressure of ‘always being 30 days from bankruptcy.’ The work ethic is contagious. When competing engineers at 3dfx were hired post 3dfx bankruptcy and they expected to find some secret sauce, a superior technological advantage, they were surprised to find the key to NVIDIA’s success was this: ‘Really hard work and intense execution on schedules.’
One of the three co-founders, Curtis Priem, came up with the name by looking up the Latin word for envy; Invidia. Then they combined it with N, because of the NV1 chip (Next Version) they were working on.
Source of initial funding? Back in the early 1990s, venture capital in general and especially Silicon Valley was slim pickings. The entire VC sector was just over $1 billion a year ($2 billion today) and Silicon Valley accounted for 20% of it. In 2024, Bay Area VC would make up the majority of VC, investing more than half of the $170 billion in funding that gets doled out every year.
While Nvidia had no product or working revenue yet, the founders had connections in the industry that were strong enough to justify their lack of revenue streams.
When Jensen, at age 31, gave his resignation to his boss at LSI, his boss asked, ‘Can I invest?’ Yet he was concerned about the potential market. ‘Who plays games? Give me an example of a gaming company.’ Jensen’s responses are indicative of how Nvidia would become prophetic for generations to come. He said that if they built the technology (superior 3D graphics cards), more companies would be founded, as 3D graphics were only beginning to take off. ‘You’ll be back soon,’ said Jensen’s boss, ‘I’ll hold your desk.’
After receiving $1 million in Series A funding from Sequia Capital (Don Valentine: ‘…I’m going to give you money. But if you lose my money, I will kill you’), and $1 million from Sutter Hill Ventures, Nvidia set out to build a chip that could display graphics at a resolution of 640×480 pixels. Today’s best chips display graphics at 7,680×4,320 or 3.98 billion pixels per second.
One of NVIDIA’s first innovations (to overcome the high cost of memory) was forward texturing, a software process for handling textures. Instead of using the traditional inverse texturing, which was based on triangles, Nvidia would render 3-D polygons by using quadrilateral shapes. Using quadrilaterals involved less computational power, thus lower memory. The drawback to this process was that software developers would have to completely remake their games in order to use NVIDIA’s forward texture mapping. Nvidia bet that, due to the fragmented world of PC video-game graphics, their technically efficient process would win out in the end. In addition, they added to their graphics cards new audio, high-quality wavetable synthesis, which recreated digitized sound from recordings of actually instruments (before this, sound was made synthetically by the company SoundBlaster and sounded silly). They were betting that software makers, which previously worked on software that was run on computers with separate cards for sound and visual, would choose a technically advanced multifunction card, even though it would mean more work to design the software.
More smart moves: Jensen convinced Intel to support the new card, Jensen convinced SGS-Thompson (a European chipmaker) to fund Nvidia’s software division ($1 million/year) to secure the right of manufacturing the NV1 chip. Presented the NV1 chip at Comdex in Las Vegas in the fall of 1994, one of the biggest computer trade shows in the world.
At the trade show, they met the representatives of Sega. In May 1995, Nvidia and Sega signed a 5-year partnership, where Nvidia planned to build its next generation chip, NV2, exclusively for Sega’s next video game console.
But the NV1 chip was a failure. In a nutshell, video game players simply wanted the fastest graphics performance for their favourite games at a reasonable price – not an over designed, superior technology with features nobody cared about. Sales were lackluster. Games needed SoundBlaster audio capability, and NV1 didn’t have it.
‘We were diluted across too many different areas,’ Jensen remembered. ‘We learned that it was better to do fewer things well than to do too many things even though it looked good on a PPT slide.’
Nvidia had never answered the supremely deep, important question about their product: ‘How would you position this?’
In the book Positioning: The Battle for Your Mind (which Jensen read), the authors argue that positioning is not about the product itself but rather about the mind of the customer, which is shaped by precious experience and knowledge.
Sega nixed the NV2 deal (but due to a clause worked into the contract by Jensen, Sega had to pay Nvidia $1 million of Nvidia could produce a working prototype), money woes, so Jensen fired 60 members of his staff, taking the company from 100 employees down to 40.
Everything was riding on the NV3 (they had $3 million in the bank, which would last them 9 more months): they wouldn’t over design, but go big with what the market wanted: 128-bit memory bus and graphics pipeline that could generate pixels at unprecedented speeds…the most powerful chip in existence.
RIVA 128 (new name, new opportunity) was showcased (after tight deadlines) with flying colors at the 1997 Computer Games Developers Conference. Now, with an impressive, working prototype, they could ask for more money.
The RIVA 128 was a wild success: the fastest 3D chip for PCs available, within 4 months more than a million units were sold, a fifth of the PC graphics market. In the 4th quarter of 1997: a $1.4 million profit, their first profitable quarter since their founding 4 years earlier. ‘The RIVA 128 was a miracle,’ Jensen said.
Another crisis during production of RIVA 128ZX: defects caused by residue known as titanium stringers required manual testing of thousands of chips. The money was running out again, and their IPO was delayed due to a financial crisis in East and Southeast Asia. To survive, they’d have to engage in bridge financing: receiving $11 million worth of loans from their three largest customers: Diamond Multimedia, Creative Labs, and STB Systems.
Moving forward, Nvidia began to build a commercial moat with various strategems: emulation and backwards-compatible drivers; these allowed them to begin a new accelerated production schedule, matching the PC maker’s buying cycles with “Three Teams, Two Seasons.”
Their IPO in 1999, during the Internet boom, raised $42 million and valued the company at $626 million. Now they could breathe (financially).
Nvidia coined the term GPUs (via a chip that wasn’t actually GPU, the next generation was) and made the first programmable GPU in existence.
Xbox deal ($1.8 billion in revenue), Apple Mac computer deal…life’s good.
But setbacks continued. ‘NV30 was an architectural disaster. It was an architectural tragedy,’ said Jensen because Microsoft was withholding information about their Direct3D API so NVIDIA’s software team, architecture team, and chip-design barely talked with one another. In addition, there was a loud fan put on the NV30; sales during the holiday quarter fell 30% vs. the previous year.
In 2002, a computer science researcher named Mark Harris noticed that a number of other computer scientists were using GPUs for nongraphics applications. These scientists were hacking their GPUs (which were faster than CPUs) to run simulations. They could study things like protein folding, MRI scans, and stock prices. Harris created a website outlining these GPGPUs (General Purpose GPUs) and was hired by Nvidia.
Jensen saw the opportunity to open up the market for GPUs way beyond computer or video game graphics. He was especially intrigued by the work in medical imaging.
Inside Nvidia, Harris discovered a secret team working on the NV50, which wouldn’t be released for a few years: it would use extensions to the C programming language, a widely adopted general purpose language, enabling parallel compute, allowing the GPU to perform all the actions of a secondary CPU that might be required via scientific, industrial or technical computing. Basically, people will no longer need to hack the GPUs. Lesson to be acknowledged: deeply analyze people who are hacking your products for particular purposes and lean into their motivations for doing it.
This new, programming model for chips was called CUDA (Compute Unified Device Architecture) and allowed users to leverage the GPU’s computing power. Jensen realized that new software, rather than hardware, would revolutionize the company.
Now here come the data centers…
Around 2008, about a year after the release of the G80 chip which had CUDA but failed to gain traction, Jensen was giving a presentation to a group of disgruntled, Wall Street analysts. They didn’t like NVIDIA’s shrinking profit margins and short term contraction. Jensen assuaged their concerns by stating that the GPU computing market would rise to $6 billion in a few years, even though it was nothing at the time. Nvidia predicted demand for enterprise data centers powered mostly by their GPUs.
‘We were convinced that accelerated computing would solve problems that normal computers couldn’t,’ said Jensen. ‘We had to make sacrifices. I had deep belief in its potential.’
The Era of computing had arrived. Everything else, financial concerns included, was secondary.
Strategic moves: donate to prestigious universities (Caltech, Stanford) and have them use Nvidia hardware in graphics programming classes. Build an academic training pipeline for CUDA via teaching, putting the classes online, and a textbook. Two-day tech summits where Nvidia could talk and learn from scientists. Deep listening and feedback from developers.
A launch of a new GPU-powered seismic software suite for investment banking and oil prospecting. Widespread solution of AMBER revolutionized how the field of molecular dynamics did research.
From the beginning of the GPU era, Nvidia invested significantly in deep learning, putting enormous resources into creating CUDA-enabled frameworks and tools. Being proactive worked out when A.I. became common parlance in the early 2020s, since Nvidia was already the preferred chip for AI developers all over the world.
Other AI chip vendors couldn’t compete with Nvidia, since AI developers want to create AI applications as quickly as possible, with limited technical risk, and NVIDIA’s established platform was likely to have much less technical problems. Why? Because the user community had already, over the last decade, ironed out the bugs and discovered the optimizations.
Moving from Nvidia to other chips like Cerebras, when you have already built AI applications on top of CUDA, is a massive task. Add to this the fact that Nvidia quickly integrated essential AI software libraries into CUDA.
Amir Salek put it well: ‘The moat is CUDA.’ CUDA opened the doors to the era of general-purpose computing.
One day Bill Dally met Andrew Ng for lunch. Ng shared his research of combining the enormous dataset of YouTube clips with the raw power of tens of thousands of traditional processors. Dally replied with, ‘I bet GPUs would be much better at doing that.’
Dally assigned Bryan Catanzaro at Nvidia to help Ng use GPUs for deep learning. Now, Ng could consolidate the work of 2000 CPUs to 12 Nvidia GPUs. Catanzaro showed that with some skilled software work, GPUs could create the “spark that ignited the AI revolution.” (According to Dally, and Dally was right.)
Enter Gary Hinton (later named the godfather of AI, who believes that humanity has a 1/3 chance of being wiped out by it) and his two students, Alex Krizhevsky and Ilya Sutskever (Ilya would go on to work for OpenAI, try to sack Sam Altman, then fail): through AlexNet (a program they created) they used NVIDIA’s GPUs in a ImageNet Large Scale Visual Recognition Challenge to support a small-scale deep-learning neural network. AlexNet was fed ImageNet content then “learned” how to build relationships between images and their associated tags. They won the contest with AlexNet correctly identifying 85% of the images. Big PR boost for Nvidia.
The paper that Ilya and Alex published about their win is not a groundbreaking mathematical formula; it is a “systems paper.” They used accelerated computing to significantly expand the dataset and the model that they were using to solve a problem.
Despite resistance from some key executives at Nvidia, Jensen embraced deep learning. ‘Deep learning is going to be really big,’ he said. ‘We should go all in on it.’
The entire 20 year history of Nvidia was leading up to this seminal choice. Everything was geared toward bringing about the AI-powered future. The company started to swarm towards machine learning.
We are now in the year 2024: Jensen says that AI is going to be more important than anything else they could possibly be doing.
Despite work already being done on the Volta GPU, several years in, they jumped into development of an entirely new type of processor: the Tensor Core, which they integrated into the Volta. Tensors are a type of data container that encodes multiple dimensions of information. Tensors were ideal for AI because they were optimized to run specialized subsets of tasks at even higher efficiencies than typical GPUs. Dally referred to them as “matrix multiple engines” made specifically for deep learning. They were 3x faster than GPUs with CUDA cores at training deep learning models.
“Jensen realized that data center scale computing requires really good high performance networking, and Mellanox was the best in the world at that.”
In the past decade, NVIDIA’s share of the add-in board of discrete GPU market has hovered around 80%.
But in 2023, despite Jensen spending the previous decade positioning Nvidia as an AI company, data-center revenue, which included AI GPUs, were only 55% of overall sales.
Then on May 24 2023, the markets took notice. NVIDIA’s 2Q revenue was $11 billion, which was $4 billion over Wall Street’s estimates. It was the largest dollar revenue upside in industrial history, according to Morgan Stanley journalist, Joseph Moore.
The next day, Jensen gave a speech at the Taiwan Computex technology conference. He announced the new DGX GH200 AI supercomputer, which had 256 GPUs in one system, an increase of 32x compared to the previous model.
Note: NVIDIA’s salespeople had been agressive in creating demand for generative AI, convincing customers that they needed to embrace it or be left behind by their competitors. Jensen labeled AI a “universal function operator” that could predict the future with accuracy. He says that it will work with anything that has structure.
No surprise, you can only gain this competitive advantage through NVIDIA’s technology.
Now NVIDIA’s data center business for fiscal 2024 rose by 427%, to $22.6 billion. This rise was driven almost exclusively by AI chip demand: these products are complex, with 35,000 parts, and Nvidia is the leader.
Jensen sensed the explosive rise of LLMs. One month before the release of ChatGPT, Hopper chip architect was released (which had been in development since the late 2010s), which included a library called the Transformer Engine. Price tag for the DGX-1, the first GPU server optimized with Tensor Cores and Transformer Engine for AI research? $149,000. Big money.
So when demand for AI explodes in 2023, Nvidia was the ONLY hardware manufacturer prepared to completely support it.
Note: Microsoft, Amazon, Google, Intel, and Advanced Micro Devices are developing their own AI chips. But Nvidia had proven, in the past 31 years, that it can perpetually stay ahead of the competition.
NVIDIA’s management has consistently stated that the $1 trillion that was already invested in global data center computer infrastructure over the years, which is is currently powered by traditional CPU servers, will someday transition to GPUs capable of parallel computations necessary for AI. This represents El Dorado for Nvidia.
(What does Jensen think the future is after AI? Digital biology. But that’s a different book.)
Now, in the next chapter on Data Centers, let us dig into the nature of this El Dorado.
Thank you for reading.