Labour MP John Mann has called on Gordon Brown to agree to step down as party leader before its annual conference in September.
The Bassetlaw MP said the “credibility” of a potential Lib Dem-Labour alliance would be damaged if he carried on.
Mr Mann said Labour shed votes because the PM appeared to be “out of touch”.
Labour’s 258 seats put them in second place behind the Tories, who are now trying to seek a deal with the Lib Dems after failing to gain a majority share.
Although the Conservative Party secured 306 of the 649 constituencies contested on 6 May, they are just short of the 326 MPs needed for an outright majority.
Mr Mann added it was “extraordinary” that Labour had not been convened in this “emergency situation”.
He told the BBC that it would be a “disaster” for Labour to try to “stagger on” while Mr Brown remained in charge.
Mr Mann told the BBC it was not “tenable” for Mr Brown to continue as leader nor for Nick Clegg to “reach an accord and prop up” the Labour leader.
He said: “He would be destroyed. That won’t happen. For the good of the country and for the good of his own party, Gordon Brown needs to be making plans to stand down as leader.”
We need a new leader over the summer elected to be running the Labour Party
John Mann: Profile
BBC political correspondent Iain Watson said Mr Mann was a “credible figure” within the party.
This is the first overt Labour challenge to Mr Brown’s authority since the party lost 91 seats in the election.
Mr Mann said that “literally thousands” of voters had told him they wanted to support him during the election campaign but that did not mean they wanted Gordon Brown in office.
“[The Labour voters] still stayed with us but there’s no way Gordon Brown can in any credibility carry on as prime minister for long and as Labour leader. He needs to go.
“We need a new leader over the summer elected to be running the Labour Party,” he said.
He said he imagined a “large number” of fellow Labour MPs privately agreed with him.
But he added: “I’m not involved in any cabal or any plotting – we’ve had too much of that in the Labour Party.”
Mr Mann said: “Gordon Brown has had a good run and whilst he was an excellent chancellor he has been seen as a poor prime minister who is out of touch and aloof. Labour lost votes because of this.
“Gordon Brown’s continuation as the party’s leader rules out the credibility of a Lib-Lab pact that has to prioritise the modernisation and reform of the antiquated UK political systems, the continued stabilisation of the economy in partnership with the need to protect front line public sector jobs and services.”
Mr Brown has left Downing Street for his home in Scotland. On Friday he spoke to Nick Clegg on the telephone offering talks if the Lib Dems cannot agree with the Conservatives.
Labour finished with 258 MPs, down 91, the Lib Dems 57, down five, and other parties 28. The Conservatives got 36.1% of votes (up 3.8%), Labour 29.1% (down 6.2%) and the Lib Dems 23% (up 1%). BBC