Shadow foreign secretary Hilary Benn has been sacked from the shadow cabinet amid claims he was encouraging ministers to resign should Jeremy Corbyn ignore a vote of no confidence.
Labour's leader faces a vote of no confidence over claims he was "lacklustre" during the EU referendum.
A Labour source told the BBC Mr Corbyn had "lost confidence" in Mr Benn.
Mr Benn said there was "widespread concern" about Mr Corbyn's "leadership and his ability to win an election".
He added: "There is no confidence to win the next election if Jeremy continues as leader.
"In a phone call to Jeremy I told him I had lost confidence in his ability to lead the party and he dismissed me."
The Labour party campaigned for Remain during the referendum, which saw the UK voting to leave the EU by 52% to 48% on Thursday.
Secret ballot
Senior Labour sources also told the BBC that a significant number of shadow cabinet resignations were likely if Mr Corbyn were to ignore the result of the confidence vote.
In other developments:
Writing in the Sunday Times business secretary Sajid Javid reveals he is to hold a round-table meeting this week with two dozen business leaders to shore up confidence after the EU vote
UKIP MP Douglas Carswell told the BBC there was "absolutely no need to rush" the process of the UK leaving the EU. He said: "Why give ourselves an artificial deadline?"
Robert Halfon, the deputy chairman of the Conservatives said the Tories have to "reconnect with working class voters" as the race to find a new prime minister continues
Labour MPs Dame Margaret Hodge and Ann Coffey submitted a motion of no confidence against Mr Corbyn — who campaigned on the losing Remain side — in a letter to the Parliamentary Labour Party (PLP) chairman John Cryer on Friday.
The motion has no formal constitutional force but calls for a discussion at the next PLP meeting on Monday.
The chairman will decide whether it is debated. If accepted, a secret ballot of Labour MPs could be held on Tuesday.
Labour MPs reacted with criticism to Mr Corbyn's move to sack Mr Benn.
Shadow housing minister Roberta Blackman-Woods said: "This is sad news indeed and I cannot understand how Jeremy thinks it will help his worsening position with the PLP."
Wes Streeting, Labour's MP for Ilford North, said: "Lots of good people chose to serve in shadow cabinet to keep the show on the road. There are no longer good reasons for good people to stay."
So what is significant about Hilary Benn's departure?
Although Jeremy Corbyn was close to Mr Benn's late father Tony, the political differences with his shadow foreign secretary became startlingly clear last December.
Mr Corbyn and Mr Benn spoke on opposite sides of the debate on the bombing of Syria.
Mr Benn's support for air strikes lost him some support on his party's left but he is hardly a Blairite, so when he publicly loses confidence in Jeremy Corbyn he might well cause others to have — or articulate — concerns of their own.
His friends say he holds no leadership ambitions himself — he simply thinks that a disastrous referendum campaign has convinced him that Mr Corbyn is not up to the job.
And other shadow cabinet members are now considering their positions.
Some tell me their local party members — including previous supporters of Mr Corbyn — are becoming more hostile to the leadership.
So the question is whether Hilary Benn's departure will strengthen Jeremy Corbyn's position or, in turn, hasten his own?
Meanwhile, Labour former culture secretary Ben Bradshaw urged the shadow cabinet to act swiftly to "save" the party.
He said: "The Labour shadow cabinet must now act to save the party and for the sake of the country. Otherwise we will never be forgiven."
Dame Margaret, the MP for Barking, said Mr Corbyn should resign because the EU referendum had been a "test of leadership" that he had "failed".
'Abundantly clear'
She argued that Labour voters were left "not getting a clear message". In a piece for the New York Times former Prime Minister Tony Blair described Mr Corbyn as "lukewarm about remaining in the union".
On Saturday, Mr Corbyn vowed to fight off any leadership challenges and told one activist who heckled him as he attended a Pride march in London: "I did all I could."
The Labour leader vowed to ensure the party's voice was heard on workers' rights, protecting the environment and human rights in the negotiations on Britain's exit from the EU — and he suggested those negotiations should happen soon.
Quizzed afterwards about claims he had run a "half-hearted" campaign for a Remain vote, he said: "Two-thirds of Labour voters voted for Remain in response to our party's call for that."
He added: "There are some people in the Parliamentary Labour Party who would probably want somebody else being the leader of this party, they have made that abundantly clear in the past few days."
Tory leadership
Mr Corbyn was confronted by Labour Party activist Tom Mauchline at the Pride event, who shouted: "It's your fault, Jeremy. When are you resigning? You need to resign."
However, an online petition on the website of campaign organisation 38 Degrees calling for Mr Corbyn to remain as leader has attracted more than 160,000 signatures from the general public.
In a joint statement, union leaders have also backed Mr Corbyn to continue as leader, saying the "last thing Labour needs is a manufactured leadership row of its own".
They called for Labour to "unite as a source of national stability" and challenge any attempt to use the referendum result to "introduce a more right-wing Conservative government by the backdoor".
Meanwhile, the Conservative Party faces a leadership battle of its own with former London Mayor Boris Johnson and Home Secretary Theresa May emerging as the frontrunners to replace the outgoing Prime Minister David Cameron, who announced he would stand down on Friday shortly after losing the EU vote.
The Sunday Times reports that Michael Gove has thrown his support behind Mr Johnson, while Cameron loyalists will favour Mrs May, who is expected to enter the leadership race in the coming days.