NEW YORK (Fortune) -- Atlanta hedge fund manager Michael Masters has been a star witness in two recent Congressional hearings on how speculators are supposedly driving up oil prices. Masters and I don't see eye-to-eye on this issue, so I was surprised to get a call from him after my "Don't Blame The Oil Speculators" column went up on Fortune.com last week.

Masters contends that without speculators, the price of oil would be $65 or $70 a barrel. He points out that the amount invested in commodities index products has risen from $13 billion to $260 billion in five years, a fact he thinks is key to understanding oil prices.

"When a trader sends a buy order to the exchange floor or presses the 'buy' key on their trading terminal, if he or she is attempting to buy more contracts than are currently offered for sale at the market price, then the market price will rise," Masters told a House subcommittee in June.

My own view is that speculators can't materially impact prices if all they're doing is making bets on the direction of oil prices by trading futures and not taking delivery of actual oil - hoarding stuff that would otherwise go to consumers.

People don't fill up their tanks with futures contracts, and there's no evidence investors are putting more oil into storage as a bet on higher future prices.

In the end, Masters and I simply agreed to disagree. But there was one thing he said that really piqued my interest. "What do you think would happen," Masters asked, "if the market went into liquidation-only mode [i.e. if speculators started unloading their futures contracts], like we saw with the Hunt brothers in 1980?"

Nelson Bunker Hunt and William Herbert Hunt were famous for cornering the silver market in the 1970s, eventually driving silver prices from $2 to $50 an ounce. When the Hunts liquidated their portfolio, silver crashed to $10. So the question Masters was asking seemed relevant: Would the oil market go into a silver-esque tailspin if oil-futures traders turned bearish?

I told him it was my understanding that investors in silver and gold frequently take delivery of the physical asset. In other words, investments in precious metals affect supply and demand in ways investments in oil futures do not. What I didn't know - and neither did Masters - was whether the Hunt brothers acquired actual silver or only silver futures. I figured the answer might help make my point.

Well, it turns out that the Hunt brothers did take delivery of silver - a massive amount in fact. In 1973 and 1974, the Hunts inventoried 55 million ounces of silver - about 10% of global supply, according to "Kingdom: The Story of the Hunt Family of Texas" by Jerome Tuccille.

The Hunts arranged to have most of their stash flown to Switzerland, where the silver was stored in the vaults of Credit Suisse, UBS and other Swiss banks. (An author of several best-selling business books, Tuccille is a vice president at T. Rowe Price.)

The point is that the Hunts were not merely flipping futures contracts. As I wrote last week, pension funds, index funds, hedge funds and other so-called oil speculators aren't actually buying oil. They're buying futures. The typical investment fund will buy, say, the August oil future and then sell it days before it comes due - typically rolling over the proceeds into the next month's contract.

If a hedge fund manager correctly bets that oil prices will keep rising, his investment profit comes out of the hide of the investor on the other side of the trade - not the consumer. Unless the hedge fund manager takes oil off the physical market, his trading will have little bearing on the price.

Circling back to Masters' question - what would happen if speculators turned negative on oil futures, the way the Hunts eventually did on silver - my answer is almost nothing. Futures market speculators did turn massively negative on oil November 2005 when crude was $57 a barrel. What happened? Oil was $61 by year-end. "It's a different ballgame with oil than it was with silver," Tuccille told me in an interview. "As you said, they're not taking delivery."

The story of the Hunts cornering the silver market confirms what most academics have been saying all along about oil.

Severin Borenstein, a Berkeley economist and the director of the University of California Energy Institute, contends that in order to push oil prices 30% above fair market value, speculators would have to hoard the equivalent of 2.5 million barrels a day.

"At that rate," Borenstein writes in a new paper, "in less than a year this secret market manipulator would have built an inventory larger than the entire U.S. Strategic Petroleum Reserve."

This manipulator would have had to escape the attention of the U.S. Department of Energy and the International Energy Agency, both of which report that oil inventories are declining, not rising. "That much oil," Borenstein concludes, "would be very difficult to hide."

I sent Masters an e-mail with my findings on the Hunts but have yet to hear back. The question I now have for him - or for anyone else who believes speculators are responsible for $140 oil - is simple:

Where are the Hunt brothers of today stashing all their oil?