The protesters, mostly Eritreans and Sudanese, are angry about a law that allows illegal immigrants to be detained for a year without trial.
They demanded the right to work legally and for their asylum bids to be processed.
An Israeli police spokeswoman said the march was peaceful.
“We are all refugees” and “Yes to freedom, no to prison!” they chanted.
The BBC’s Richard Galpin in Jerusalem says the protest was prompted by the new law on detention, which had dashed hopes that a recent crackdown would be halted.
The migrants, some of whom have been in Israel for years and have taken low-paid jobs, say they are all refugees who have fled persecution and conflict back home in Eritrea and Sudan.
They say their only choice will be either to spend time in prison in Israel or agree to return home.
The dangers in their home countries mean they cannot be forcibly returned.
The migrants said they would hold further demonstrations outside UN offices and foreign embassies in Tel Aviv on Monday.
An Israeli official said the authorities would continue to deal with people who, as she put it, had infiltrated Israel, adding that more than 2,600 had agreed to leave voluntarily last year.
Ultra-Orthodox MP Eli Yishai said the migrants had been encouraged by “anti-Zionist human rights organisations”.
He told the Maariv daily the protest was “a sharp and clear cry for the state of Israel and judicial and law enforcement authorities to apply all the means at their disposal to return the infiltrators to their countries”.
Error: No connected account.
Please go to the Instagram Feed settings page to connect an account.