Like that great conflict almost a century ago, the Iraqi war has been marked by repeated claims that progress is being made and that a final breakthrough is in the offing.

In 1917, the French commander General Robert Nivelle proudly announced that "we have the formula for victory" before launching the French armies on a catastrophic offensive in which they were massacred. Units ordered to the front brayed like donkeys to show they saw themselves as being like animals led to the slaughter. Soon, the soldiers broke into open mutiny.

On 10 January this year, President Bush announced that he too now believed he had the formula for victory. In an address to the American nation, he announced a new strategy for Iraq that became known as "the surge". He said he was sending a further 20,000 US troops to Iraq. With the same misguided enthusiasm as General Nivelle had expressed in his plan, President Bush explained why "our past efforts to secure Baghdad failed" and why the new American formula would succeed: in the past, US and Iraqi troops had cleared areas, but when they moved on guerrillas returned.

In future, said Bush, American and allied troops would stay put.

Six months after the surge was actually launched, in mid-February, it has failed as dismally as so many world war one offensives. The US defense department says that, this June, the average number of attacks on US and Iraqi forces, civilian forces and infrastructure peaked at 177.8 per day, higher than in any month since the end of May 2003. The US has failed to gain control of Baghdad, as Tuesday's attacks on Mosul showed. More than 250 people were killed and 200 wounded when suicide bombers detonated three fuel tankers in different areas of the city.

The surge is now joining a host of discredited formulae for success and fake turning-points that the US (with the UK tripping along behind) has promoted in Iraq over the past 52 months. Fresh US initiatives in Iraq seemed to succeed each other about every six months. Just as it was becoming evident in the US that the surge was not going anywhere very fast, there came good news from Anbar province in western Iraq. The Sunni tribes were rising against al-Qaeda, which had overplayed its hand by setting up an umbrella organisation for insurgents called the Islamic State of Iraq. In Sunni areas, it was killing rubbish collectors on the grounds that they worked for the government, shooting women in the face because they were not wearing veils, and trying to draft one young man from each family into its forces. Sunni tribal militiamen backed by the US fought al-Qaeda in insurgent strongholds such as Ramadi, and attacks on American troops there fell away dramatically.

The US administration could portray this as a fresh turning-point but, as with many a development in Iraq portrayed as a sign of progress by the White House, the recruitment of Sunni tribal militias by the US is not quite what it seems. In practice, it is a tactic fraught with dangers. In areas where they operate, police are finding more and more bodies, according to the interior ministry. Victims often appear to have been killed solely because they were Shia. The gunmen from the tribes are under American command, and this weakens the authority of the Iraqi government, army and police . . . institutions that the US is supposedly seeking to foster.

The White House says it is too early to know if the surge is succeeding, and that it will wait for a security report due next month from General David Petraeus, the top US military commander in Iraq, and the US ambassador to the country, Ryan Crocker. But the new strategy was never going to turn the tide in Iraq. Its main advantage for Bush is that it puts off the moment when failure has to be admitted, a potentially disastrous confession for Republicans standing for election next year. If an American withdrawal can be postponed until after the poll, then the neo-cons can blame the Democrats for a stab in the back, pulling out the troops at the very moment when victory was almost in their grasp.

I was in Baghdad in January, when Bush made his State of the Union speech outlining his plans for the surge. Iraqis were pessimistic from the beginning about its chances of success. A friend called Ismail remarked gloomily: "An extra 16,000 [sic] US troops are not going to be enough." A Sunni, he had recently fled his house in the west of the capital because he was frightened of being arrested and tortured by the paramilitary police commandos . . .like most Sunni, he regarded them as uniformed Shia death squads.

Baghdad was paralysed by fear.

Drivers were terrified of being stopped at impromptu checkpoints where they might be dragged out of their cars and killed for belonging to the wrong religion. Conversation was dominated by accounts of narrow escapes. Most people had at least one fake ID card so they could claim, depending on circumstance, to be either Sunni or Shia. This might not be enough; some Shia checkpoints had a list of theological questions drawn up by a religious scholar that they would use to interrogate people.

It was extraordinary how little control the US forces and the Iraqi army exercised over the very centre of the capital. There was black smoke rising from Haifa Street, a two-mile-long Sunni corridor just north of the Green Zone, which US forces had repeatedly invaded but failed to secure. When a helicopter belonging to the security company Blackwater was shot down or crashlanded in the al-Fadhil district in the centre of Baghdad, the survivors were executed by insurgents before US forces could get to them.

By the end of May, I found it a little easier to drive through Baghdad, but the danger was still extreme. I sat in the back of the car with my jacket hanging inside the window so it was difficult for other drivers to see me. We were pulled over by an army checkpoint. A soldier leaned in and asked who I was.

We were lucky. He looked surprised when told I was a foreign journalist, and said softly: "Keep well hidden."

Back in my hotel, I phoned an Iraqi friend in the Green Zone who was close to the government. "Be very careful, " he warned. "Above all, do not trust the army and police."

There was an example of what he meant a few days later when a convoy of 19 vehicles carrying 40 uniformed policemen arrived in the forecourt of the finance ministry.

They entered the building and calmly abducted five British security men, who have not been seen since.

The kidnappers may be linked to a unit of the Mehdi Army.

The surge has changed very little in Baghdad. It was always a collection of tactics rather than a strategy. All the main players . . . Sunni insurgents, Shia militiamen, Iraqi government, Kurds, Iran and Syria . . . are still in game.

One real benchmark of progress . . . or lack of it . . . is the number of Iraqis who have fled for their lives.

This figure is still going up. Over one million Iraqis have become Internally Displaced Persons (IDPs) since the Samarra bombing, according to the Red Crescent. A further 2.2 million people have fled the country. This exodus is bigger than anything ever seen in the Middle East, exceeding in size even the flight or expulsion of the Palestinians in 1948.

A true sign of progress in Iraq will be when the number of refugees, inside and outside the country, starts to go down.

Fudging the numbers: the real death toll in Iraq MORE lies have been told about casualties in Iraq and the general level of violence there than at almost any time since World War One. In that conflict, a British minister remarked sourly that he suspected the military authorities of keeping three sets of casualty figures: "One to deceive the Cabinet, a second to deceive the people and a third to achieve themselves."

The American attitude to Iraqi civilian casualties is along much the same lines. The Baker-Hamilton report drawn up by senior nonpartisan Democrats and Republicans last year examined one day in July 2006, when the US military had reported 93 attacks on US and Iraqi forces. Investigation by US intelligence agencies revealed that the real figure was about 1,100.

The Iraqi government has sought to conceal civilian casualty figures by banning journalists from the scenes of bombings, and banned hospitals and the health ministry from giving information. In July, AP reported, 2,024 Iraqis died violently, a 23% rise on June, which was the last month for which the government gave a figure.

This is almost certainly an underestimation. In a single bombing in the district of Karada in Baghdad on 26 July, Iraqi television and Western media cited the police as saying that there were 25 dead and 100 wounded. A week later, a list of the names of 92 dead and 127 wounded, compiled by municipal workers, was pinned up on shuttered shopfronts in the area.

The US military began the war by saying that it was not keeping count of Iraqi civilians killed by its troops.

It often describes bodies found after a US raid as belonging to insurgents when the local Iraqi police say they are civilians killed by the immense firepower deployed by the American forces.

Almost the only time a real investigation of such killings is carried out is when the local staff of Western media outlets are among the dead.