The caretaker president, a former exile, said his rival’s declaration of victory was “undemocratic”.
Mr Essebsi, 88, has been celebrating with supporters, telling them all Tunisians now need to “work together”, and promising to bring stability.
Critics say his success marks the return of a discredited establishment.
Official results are not expected until Monday evening. One exit poll gave Mr Essebsi 55.5%, with several others showing similar figures.
The election marks the last stage of Tunisia’s move to democracy, after the 2011 overthrow of President Zine el-Abedine Ben Ali.
It is the first time Tunisians have been able vote freely for their president since independence from France in 1956.
Mr Marzouki, a 67-year-old human rights activists forced into exile by the Ben Ali government, said the election was too early to call.
“The announcement of victory is undemocratic and we should wait if we want to be a state that respects the rule of law,” he told supporters.
“What I want to tell you is that we are victorious, we are victorious , we are victorious. Tunisia has won and you have won. You have won for Tunisia, for democracy and for human rights.”
Mr Marzouki has been interim president since 2011 and is more popular in the conservative, poorer south.
He was thought likely to attract support from the moderate Islamist Ennahda party, which has played a key role in Tunisian politics since the Arab Spring but did not field a candidate.
Mr Essebsi appeared on local television after polls closed on Sunday, saying, “I dedicate my victory to the martyrs of Tunisia.”
“I thank Marzouki, and now we, should work together without excluding anyone,” he added.
Supporters danced and let off fireworks outside the headquarters of Mr Essebsi’s secular-leaning Nidaa Tounes party.
Mr Essebsi held office under both deposed President Zine el-Abedine Ben Ali and Tunisia’s first post-independence leader, Habib Bourguiba.
He won the first round of voting last month with 39% of the vote.
He is popular in the wealthy, coastal regions, and based his appeal to voters on stability and experience.
Whoever wins faces restricted powers under a constitution passed earlier this year.
The president will be commander-in-chief of the armed forces but can appoint or sack senior officers only in consultation with the prime minister.
The president will also set foreign policy in consultation with the prime minister, represent the state and ratify treaties.
Tunisia boosted security for the elections and closed border posts with Libya, which has been plagued by unrest.
A group of at least three attackers targeted a polling station near the city of Kairouan on Sunday morning. Security forces say they killed one attacker and arrested three.
Error: No connected account.
Please go to the Instagram Feed settings page to connect an account.