A UK exit from the European Union could mean the UK misses out on up to 5.6% of GDP growth by 2019, the IMF has warned.
Brexit is the "largest near-term risk" to the UK economy, the IMF said in its annual UK economic outlook.
It added that the net economic effects would probably be "negative and substantial".
But the Economists for Brexit campaign said the consensus that a UK exit would be bad for the economy was "based on flawed EU-centric models".
There are five days to go until the UK decides on its future in the European Union, in a referendum on 23 June.
In other developments:
Field Marshall Lord Guthrie, a former Chief of Defence Staff, has switched sides to Leave, saying he is worried by the prospect of "a European army"
The Times newspaper has said it supports Remain
The Remain and Leave groups have suspended their campaigning until Sunday in light of MP Jo Cox's death
The IMF said that under its least adverse scenario for Brexit, by 2019 UK GDP would be 1.4% below what it would be should the UK vote to stay in the EU.
Its most adverse Brexit scenario predicts 2019 growth 5.6% below what it would otherwise have been, and also a drop in GDP in 2017 of 0.8%, which an IMF official described as a "recession".
Under this scenario the UK would return to GDP growth of 2.9% in 2021. But the UK would have missed out on 4.5% of growth by then, it said.
'Substantial Brexit costs'
Following a Brexit, the IMF said the UK would have to negotiate new trade terms with the EU if it wanted to stay in the single market.
If not, the UK could rely on World Trade Organisation rules, but this would "significantly raise trade barriers", the IMF said.
A long period of uncertainty would also lead to market volatility and discourage investment, it said.
Concerns about the economic effects of a Brexit may have already begun to affect markets such as the housing sector, the IMF added.
"On balance, the net economic costs of an exit are likely negative and substantial," it said.
What do these losses predicted by the IMF mean?
If the analysis is right then we would be worse off than we would be if we were to remain in the EU.
In the adverse short term scenario we would for a while be worse off than we are now, but even in that situation growth would resume and take average living standards to a level higher than today.
It's just that they think if we remained they would be even higher.
Which raises (at least) two questions. Is that missed opportunity to be better off a price worth paying for whatever other advantages you might think leaving would bring?
And the other question, of course — is the analysis right? Most economists would say, broadly speaking, "Yes, it is".
But there are some serious members of the profession who say no; ultimately we would do better economically outside.
And many campaigners say this is a profession that didn't see the financial crisis coming.
'Doom and gloom'
Campaign group Economists for Brexit said the widespread view that Brexit will be bad for the economy in the long term because of bad trade effects and in the short term because of uncertainty was "based on flawed EU-centric models".
Campaign co-chair Patrick Minford said: "The IMF report, like the Treasury's, uses flawed models and makes wrong, deceitful assumptions to project doom and gloom from Brexit whereas with solidly based models and assumptions Brexit gives the UK more growth and better living standards."
Mr Minford said the best outcome would be for the UK to use World Trade Organisation rules "under which the UK would leave the EU, freed from both Single Market regulations and from free EU movement of people."
"Under this option the UK would get rid of EU trade barriers on the rest of the world, so ridding our consumers of a huge burden of EU protectionism of food and manufacturing; consumer prices would fall 8% and GDP would rise 4% in the long term due to the dynamic response of the economy to lower costs."
He added that the UK can compensate farmers and manufacturers for the loss of EU support, and said manufacturers would raise productivity to compete with the rest of the world.
Matthew Elliot, chief executive of Vote Leave said: "The IMF has chosen to ignore the positive benefits of leaving the EU and instead focused only on the supposed negatives.
"If we Vote Leave we can create 300,000 jobs by doing trade deals with fast growing economies across the globe."
Vote Leave said that the IMF analysis was "partial" and did not provide any analysis of a new bilateral free trade agreement between the UK and the EU.
It added that "the real risk to the economy is staying tied to the failing eurozone".