The governor of New York was on CNN the other week talking about closets. By way of refuting allegations that he was recently caught en flagrante with a woman not his wife at the Executive Mansion, David Paterson offered a defence based on intimate knowledge of the lay-out of his adopted home. He couldn't have been snagged in such an embarrassing situation because, according to him, no such closet exists. Within hours of this denial, newspapers were carrying forensic reports from insiders, detailing just how many such closets are in the house. This is the edifying point that politics in New York has reached.

Whether or not Paterson was up to no good in the utility closet or canoodling in a restaurant with another woman papers have helpfully dubbed "the leggy Latina", those yarns are probably the least salacious of the whispers about him that have been doing the rounds. They pale next to the stories portraying him and his wife as swingers who like to swap partners, and to host sex and drug parties at the mansion. Not to mention the one where he arrived late at the scene of a plane crash in Buffalo because he had to be disentangled from the arms of a lover first. That would be a different lady too from the transvestite being whispered about in other circles.

"For the last couple of weeks, I have been the subject of what, even by Albany [the state capital] standards, has been a spate of outrageous rumours about me," said the 55-year old. "There is an accountability that should exist in the media. How do I get my reputation back? Because I don't believe I have done anything to deserve this kind of bashing."

In his defence, the legally-blind Paterson can at least assert that he never sold himself to the public as Mr Perfect. When first promoted from lieutenant-governor to take the place of the disgraced Eliot Spitzer just under two years ago (his predecessor having been caught spending lavishly on prostitutes), he was forthright about the fact both he and his wife had enjoyed extramarital affairs. Just to offset any future attempts by the New York tabloids to do him in, he even went as far as to confess about past recreational use of cocaine and marijuana. Those frank admissions, a self-deprecating media persona and 22 years of service in the state senate meant most people were willing to give the new guy a chance.

As the first black governor of a state with a population of 20 million people, Paterson can rightly lay claim to the title of the second-most powerful African-American politician in the country.

Unfortunately for him, the title-holder in that division has, long before the whiff of scandal began to attend Paterson's every move, deemed him radioactive to the touch and surplus to requirements. As far back as last September, President Obama's administration had asked Paterson, a fellow Democrat, not to run in this coming November's gubernatorial election rather than risk losing the office to the Republicans.

While that interference from the White House was mostly down to the fact Paterson's poll numbers had dropped precipitously and the Democrats can ill-afford to concede the influential New York governorship, there was another factor too. When Hillary Clinton resigned as the state's junior senator to take over as Obama's secretary of state, choosing her replacement was the governor's job. It was not a task Paterson handled deftly. En route to finally selecting little-known Kirsten Gillibrand, he managed to make Caroline Kennedy (the highest-profile applicant and daughter of the late president John F Kennedy) look like a complete fool. Not a wise move for any Democrat.

"When the dust settled, the public was left with an underwhelming Senate pick, and a clearer than ever picture of Paterson's weaknesses: a chronic problem with making decisions, a habit of telling people what they want to hear, a sometimes shaky relationship with the truth – and a consistent inability to follow through," wrote Bill Hammond in the New York Daily News. "He compounded those perceptions in last year's budget fiasco. Through late 2008 and early 2009, he pushed hard for serious cuts in politically popular but costly social programmes. He opposed raising income taxes as a job killer. When Washington offered billions in stimulus aid, he refused to treat it as an excuse for more spending. For a moment there, it almost looked like we had a leader. Then the legislature pushed back, as everyone knew it would, and Paterson caved – on virtually all fronts."

The only thing he hasn't caved on is Obama's wish that he admit he's not done a great job, and give the president's choice Andrew Cuomo a clear run at the gig. This stance briefly improved his popularity by establishing a new reputation as an outspoken maverick willing to take on the leadership of his own party. Unfortunately, that's a difficult position from which to raise the crucial millions needed to run an election campaign, and if his chances of retaining his position appeared slim from the moment he crossed swords with the Oval Office, they all but disappeared once the tawdry rumour mill cranked into gear last month.

On 18 January, the New York Post published an account of a lengthy lunch Paterson enjoyed with a woman at a New Jersey steakhouse where eyewitnesses say he was "neck-kissing, nuzzling and cooing like a smitten schoolboy". Twelve days later, the first whispers began about the time he was caught in the closet by a state trooper. Suddenly it was open season, replete with newspaper front pages boasting classic Clintonian headlines like: "I did not have sex with that woman!"

Next up was the main course. A tweet from a New York Observer reporter was enough to prompt frenzied speculation that the New York Times was working on an in-depth Paterson story that was so explosive it was going to force him to resign immediately. That was about the juncture the internet caught fire with unsubstantiated allegations of cross-dressing, wife-swapping and all manner of other indiscretions.

"What I think is that I have been depicted in a way that has been racialised, sexualised, hyper-sexualised and dissolute," said Paterson last week. "For a person that has such weak poll numbers and hasn't raised enough money and has diminishing political support, someone is going very far out of their way to see that I'm not a candidate this year. I don't know who it is; I just know that it's a well-orchestrated attempt to do this."

Whatever the motivation and whoever the authors of the ongoing character assassination, the damage has been done. Last week's appearance on the Larry King Show, a risible programme whose only purpose in the modern media is as a safe haven where under-fire public figures can be assured of an easy ride, was a standard-issue response from a beleaguered politician. It will take much more than a soft-soaping from the worst interviewer on television though to salvage Paterson's career.

"The only way I'm not going to be governor next year is at the ballot box," said Paterson, "and the only way I'm leaving before that is in a box."

Them's fighting words but it didn't help his cause that the New York Times finally addressed the phantom story last Sunday, in a typically tortured manner that many reckon did more to fan the flames rather than to extinguish them.

"For the Times to issue a statement saying, 'We are not investigating rumours about the sex life or drug use or financial shenanigans of Public Figure X' doesn't clear the good name of Public Figure X," said editor Bill Keller. "It simply announces that we've heard the rumours and for some reason chose not to look into them."

The entire brouhaha must be viewed in the context of the especially murky arena that is New York state politics. Apart from boasting any number of local politicians under indictment for fraud at any given time, just last week the state senate voted to expel Hiram Monserrate, a Democratic senator who'd been convicted of domestic assault against his girlfriend. On a national level, moral standards are in even more decay.

Apart from Washington DC boasting a litany of high-profile sexual miscreants in the House and Senate, Paterson's travails follow on from those of South Carolina governor Mark Sanford. That Republican's future aspirations of a bid for the White House were shuttered into the siding last year by the exposure of an affair with an Argentine woman. Then there's the ongoing melodrama involving one-time vice-presidential candidate John Edwards. Yet another who sold himself as the consummate family man on the campaign trail, Edwards' fall from grace now includes the existence of a sex tape that makes him much more Paris Hilton than presidential.

"I'm black, I'm blind and I'm still alive," declared Paterson last week. "How much better do they want me to be?" One more unanswered question in a drama already too full of them.