"We must have spoken about ethics. Yes, ethics." Elie Wiesel, the Nobel Peace prizewinner, recounted with a chuckle the ironic dinner conversation he had 20 years ago with Bernard Madoff, the man who went on to swindle him out of $15.2m (¤12m).

Away from Hollywood, which has also been touched by Wall Street's biggest fraud, Wiesel is the most prominent victim of
Madoff's $50bn pyramid scheme. He has spoken for the first time since his tormentors confessed, to damn him as "a crook, a thief, a scoundrel, a criminal".

The 80-year-old Holocaust survivor, whose writing, teaching and charitable work led the Nobel committee to call him a "messenger to mankind", said Madoff had made no heavy sales pitch to him when they met. "We spoke about education, not the economy," he said. "Was he already a crook? Probably."

It was left to friends of Madoff to suggest Wiesel put first his own fortune and then the money from his charitable foundation with Madoff Investment Securities. Now it is all gone, just like the savings of thousands of others, from celebrities to charities, on several continents.

"We checked the people who have business with him and they were among the best minds on Wall Street, the geniuses of finance," Wiesel said. "I am not a genius of finance. I teach philosophy and literature – and so it happened."

Madoff, it is now known, constructed an elaborate fiction over more than a decade, declaring himself to have found a secret way to make double-digit percentage profits. But the complicated statements he sent to his clients were nonsense. For at least 13 years, he never invested a cent, using money coming in from new clients to pay those cashing out.

Wiesel was speaking at a panel discussion chaired by the editor of Portfolio magazine and was asked what he wants to happen to Madoff, who is under house arrest at his Manhattan penthouse.

"I would like him to be in a solitary cell with a screen," the philanthropist said. "And on that screen for at least five years of his life, every day there should be pictures of his victims, one after the other."