Comey, fired by Trump in May last year, was worried the president may be open to blackmail by Russia given claims he was present when prostitutes urinated on each other during a 2013 Moscow visit.
Comey's firing came as the Federal Bureau of Investigation was probing possible connections between Trump's 2016 presidential campaign and Russia's meddling in the U.S. election.
Russia has denied interfering in the election and Trump has denied any collusion or improper activity.
Comey said in the exclusive interview with ABC News' George Stephanopoulos, which aired at 10pm on Sunday (0200 GMT on Monday), that it is "possible, but I don't know" whether Russia has evidence to back up the allegations about Trump's Moscow trip.
Trump told Comey that he had not stayed overnight in the Moscow hotel and that the claims related to the prostitutes were not true, Comey said.
"A person ... who talks about and treats women like they're pieces of meat, who lies constantly about matters big and small and insists the American people believe it, that person's not fit to be president of the United States, on moral grounds. And that's not a policy statement," Comey said.
"He is morally unfit to be president," he added.
Comey has a tell-all book, "Higher Loyalty," due out on Tuesday.
The book's imminent release - and the slated ABC News interview - prompted Trump to hurl a new set of insults at Comey earlier on Sunday, challenging accusations made in the book, and insisting that he never pressed Comey to be loyal to him.
"Slippery James Comey, a man who always ends up badly and out of whack (he is not smart!), will go down as the WORST FBI Director in history, by far!" Trump wrote early on Sunday in one of five Twitter posts aimed directly at Comey.
Reuters and other news outlets have obtained copies of Comey's book before its formal release. In it, Comey wrote that Trump, in a private meeting, pressed the then-FBI director for his loyalty.
Comey told ABC News that the title of the book came from that "bizarre conversation" he had with Trump at the White House in January 2017, shortly after his inauguration.
"He asked for my loyalty personally as the F.B.I. director. My loyalty's supposed to be to the American people and to the institution," Comey said in the interview.
The FBI has long tried to operate as an independent law enforcement agency.
"I never asked Comey for Personal Loyalty. I hardly even knew this guy. Just another of his many lies," Trump said on Twitter.
Comey is now a crucial witness for Special Counsel Robert Mueller's investigation into whether Trump has tried to obstruct the Russia probe.
Comey told ABC News that he believes there is "certainly some evidence of obstruction of justice."
Comey also defended his decision to publicly disclose the FBI's re-opening of its investigation into Democratic presidential candidate Hillary Clinton's handling of email when she was secretary of state.
The Clinton probe was already public, Comey said, whereas the FBI's examination of possible collusion between the Trump campaign and Russia was in its early stages. It did not become publicly known until after the 2016 presidential election.
Despite his myriad of reservations about Trump, Comey told ABC News that he did not believe the U.S. Congress should impeach him, as it would let the American people "off the hook" for something "they're duty bound to do directly."
"People in this country need to stand up and go to the voting booth and vote their values," he said.
U.S. House of Representatives Speaker Paul Ryan, interviewed on NBC's "Meet the Press" on Sunday morning, expressed qualified support for Comey. Asked whether Comey was a man of integrity, the Republican speaker said: "As far as I know," but added that he did not know him well.
Asked about Trump's use last week of the words "slime ball" to describe Comey, Ryan said: "I don't use words like that."
Republican National Committee Chairwoman Ronna McDaniel said the ABC interview reaffirms that Comey's "higher loyalty is to himself."
"He has no credibility and President Trump was right to follow through on the bipartisan calls for him to be fired," McDaniel said in a statement.
