Warren Buffett is running out of ways to beat the market. His problem isn’t age. It’s size

Open this photo in gallery:

This is a story about a family that made a modest investment, which grew into a whale. It’s about where they invested their money. It’s about how, as easy as clicking a button on your online brokerage’s website, you can do likewise.

But it’s also a story about why, even if you do as they did, you will never receive the same returns. The man behind the investment that made them so much money has been saying so for years, and last weekend he doubled down on the warning.

First, the fairy tale part of the story.

On Monday, 93-year old Ruth Gottesman announced that she was giving US$1-billion to Albert Einstein College of Medicine in New York, where she has taught since 1968 and is a professor emerita.

Credit goes to a 55-year-old, buy-and-hold investment in Berkshire Hathaway BRK-B-N.

In 1963, Ms. Gottesman’s late husband, David Gottesman, was introduced to another young investor, a guy from Omaha named Warren Buffett. The two became fast friends, and eventually went into business together. In 1966, they bought a Baltimore-based department store – but as Mr. Buffett told the New York Times in 2022, they soon realized they had made a “terrible mistake.” They sold at a loss.

Mr. Gottesman chose to leave his proceeds from the sale in Berkshire, Mr. Buffett’s then-tiny company.

“There probably has never been a better return on any stock held for 44 years in the history of Wall Street,” he wrote in 2012. He estimated that the value of his shares had increased roughly 6,000-fold. (Since then, Berkshire’s Class A shares have roughly quintupled.)

Mr. Gottesman also founded his own successful business, the wealth management firm First Manhattan. In 2022, the year he died, Forbes estimated his net worth at more than US$3-billion.

However, the medical school donation appears to be funded entirely by the long-ago investment in Berkshire. Ms. Gottesman said that her gift was possible because her husband, “left me, unbeknownst to me, a whole portfolio of Berkshire Hathaway stock.”

On Saturday, Berkshire released its annual report, including the latest edition of Mr. Buffett’s plain-English letter to shareholders. As always, it contains a page showing Berkshire’s performance since 1965. At the end of 2023, the stock was up 4,384,748 per cent. Over the same period, the S&P 500 index including dividends reinvested was up 31,223 per cent.

Am I saying that you too can buy Berkshire stock (and full disclosure, I’m a long-time shareholder), wait 55 years, and become a billionaire?

Not exactly.

One bar is that Mr. Buffett, now 93 years old, is mortal. He has has chosen several capable successors to run his giant conglomerate, but even if they are as wise as him, or wiser, they face an insurmountable obstacle.

Berkshire’s big problem is bigness. Its market capitalization on Thursday afternoon was US$886-billion. That’s roughly a third of the value of the entire TSX. To outperform the market, it must not only find exceptional investment opportunities, but enormous ones. Think eleven figures, minimum.

Berkshire has underperformed the S&P 500 in three of the past five years.

When Mr. Buffett was starting out, his investment universe was infinite, since a small, successful investment could have an impact on his returns. But each year that Berkshire outperformed, increasing its value and earnings – it ended 2023 sitting on US$167.6-billion in cash – its universe of potentially outperforming investments got smaller and smaller.

“There remain only a handful of companies in this country capable of truly moving the needle at Berkshire, and they have been endlessly picked over by us and by others,” Mr. Buffett wrote in his annual letter. “All in all, we have no possibility of eye-popping performance.”

Mr. Buffett still believes that “Berkshire should do a bit better than the average American corporation and, more important, should also operate with materially less risk of permanent loss of capital. Anything beyond ‘slightly better,’ though, is wishful thinking.”

“This modest aspiration wasn’t the case” when people like the Gottesmans invested, “but it is now.”

That’s why the greatest investor in history has consistently recommended most people invest in the entire market, via a low-cost index fund. They’ll earn the average market return – which after fees will outperform the vast majority of professional money managers.

Even a tiny bit of outperformance adds up over time. Between 1965 and 2023, Berkshire’s market value rose 140 times more than the S&P 500. But that huge gap comes from the compounding of a relatively small difference in annual returns – 19.8 per cent versus 10.2 per cent.