The report says that in some cases troops used torture, including threats of rape and execution.
More than 40 people have died since protests began in early February, with more than 2,000 people arrested.
Protests began with students demanding action to tackle high crime rates, inflation and basic food shortages.
They grew into a wider opposition movement – demanding the resignation of President Nicolas Maduro – but demonstrations have slowed in recent weeks.
For its report “Punished for Protesting”, New-York based HRW says it interviewed more than 90 people including victims, their relatives and lawyers who advised them.
“In most of the cases we documented, security forces employed unlawful force, including shooting and severely beating unarmed individuals,” the report says.
“Nearly all of the victims were also arrested and, while in detention, subjected to physical and psychological abuse.”
HRW alleges 10 cases of torture including incidents of electric shocks, burns and threats of rape or execution.
The report also noted that some protesters had used violence and that members of the security forces were among those killed or injured.
Jose Miguel Vivanco, Americas director at Human Rights Watch, said the abuse of protesters was carried out with the tacit approval of security forces.
“Participating in the carrying out of these abuses are armed gangs of civilians who act with the permission – in many times in co-ordination, the toleration, of security forces,” he said.
The Venezuelan government has not yet responded to the report.
However, last month, the chief of Venezuela’s military strategic command, Gen Vladimir Padrino, said there had been “some excesses” committed by members of the National Guard.
He said there were 97 investigations into cases of alleged abuses.
Most of those detained during the demonstrations have since been released but a number of high-profile opposition politicians are still in detention.
Venezuela has also seen huge rallies in support of the government.