The US said the reports underscored the need to remove the regime of President Bashar al-Assad.
A Syrian spokesman said the report had no credibility as it was commissioned by Qatar, which funds rebel groups.
The report comes a day before peace talks are due to begin in Switzerland.
Delegations are now arriving for the talks, known as Geneva II, which open in Montreux on Wednesday, and continue in Geneva two days later.
The conference is seen as the biggest diplomatic effort yet to end a three-year conflict that has left more than 100,000 dead and millions displaced.
The report, by three former war crimes prosecutors, is based on the evidence of a defected military police photographer, referred to only as Caesar, who along with others reportedly smuggled about 55,000 digital images of some 11,000 dead detainees out of Syria.
US state department spokeswoman Marie Harf said it “underscores that it makes it even more important that we make progress [at Geneva II]. The situation on the ground is so horrific that we need to get a political transition in place, and we need to get the Assad regime out of power.”
She added: “Obviously we condemn these reports in the strongest possible terms.
“These most recent images are extremely disturbing; they are horrible to look at and they illustrate apparent actions that would be serious international crimes, and we have long said that those responsible for these kinds of serious violations in Syria must be held to account.”
Earlier, UK Foreign Secretary William Hague expressed similar sentiments, telling the House of Commons: “I’ve seen a lot of this evidence, it is compelling and horrific. And it is important that those who have perpetrated these crimes are one day held to account.”
Rupert Colville, spokesman for UN human rights chief Navi Pillay, told AFP: “This report is extremely alarming, and the alleged scale of the deaths in detention, if verified, is truly horrifying.
“Allegations this serious cannot be ignored and further investigation is clearly necessary.”
The photographer, “Caesar”, said his job had been to take photographs of corpses, both to allow a death certificate to be produced and to confirm that execution orders had been carried out.
“There could be as many as 50 bodies a day to photograph which require 15 to 30 minutes of work per corpse,” he is quoted as saying.
The photographs cover the period from the start of the uprising in March 2011 until August last year.
All but one of the bodies shown are male. Investigators say most were emaciated; many had been beaten or strangled. Some had no eyes, and some showed signs of electrocution.
One of the authors of the report, Prof Sir Geoffrey Nice, told the BBC’s Newsday programme that the scale and consistency of the killings provided strong evidence of government involvement that could support a criminal prosecution.
However, a spokesman for the Syrian ministry of information, Bassam Abu Abdullah, questioned the report’s evidence, telling the BBC it was unclear where the information had come from or if the photographs were “from Syria or from outside Syria”.
He said he was “astonished” at the figure of 11,000 victims, saying it had not been raised before this report.
He said: “I doubt this report. We should check these photos. Who are these people? Where are the names? From which prisons? Who is this person who has the authority to have these photos?”
Mr Abdullah said the international courts should direct their questions to Qatar: “If Qatar financed this report, there is no credibility because Qatar is one of the states who financed international terrorism and who sent killers to Syria.”
Ahead of the talks in Switzerland, US President Barack Obama telephoned Russian counterpart Vladimir Putin to “discuss the issues of the conference”, the Kremlin said.
It said the conversation was “businesslike and constructive”.
Foreign ministers Sergei Lavrov and John Kerry are to meet later on Tuesday in Montreux.
Meanwhile, the UN defended its decision to withdraw an invitation to Iran – a key ally of Syrian President Bashar al-Assad – over its refusal to endorse the Geneva Communique, the plan for a transitional governing body agreed at a UN-backed meeting in 2012.
The UN said an oral understanding did not become a written one.
“In fact the opposite is what happened, that Iran stated the same positions that it had held previously,” UN deputy spokesman Farhan Haq said.
The invitation to Iran had angered the US, while the main Syrian opposition National Coalition had threatened to pull out if the invitation was not rescinded.
Iran’s Foreign Minister Javad Zarif said the withdrawal was “beneath the dignity of the UN’s secretary general”.