A long story, it begins on a wild night in New York with Joe Strummer, Allen and artists Damien Hirst and Douglas Gordon forming a marching band and busking in the restaurants of Manhattan and ends with 15-year-old Lily in the basement of the London Astoria, singing backing vocals to a jazz version of Strummer's punk anthem, 'White Riot', at the 2000 Poetry Olympics. Jump forward six months and Allen is in Paris on one end of a mobile phone while Lily waits to go onstage with Strummer in front of 15,000 fans at Wembley Arena. Allen, the concerned dad, asks whether she is nervous but Lily replies that she is not. "Well, " counsels her father, "phone me back after you've been on."

When the call comes, Lily is revved up with excitement. Keith tells her: "Remember that moment because you're going to spend the rest of your life chasing it."

Allen speaks about his popstar daughter with a mixture of pride and concern. He never worried about her becoming a performer.

"Lily went around India on her own when she was 15. I worried then. She's very resilient but that was before the whole enterprise got hold of her. She's finding it tough but I think she'll get through it."

His concern arises from the way Lily's habit of blogging her innermost thoughts can be misinterpreted. Her recent comments about her weight insecurities made national headlines and her playground spat with Cheryl Tweedy of Girls Aloud rumbles on absurdly (in the latest bulletins Gordon Ramsay, Beth Ditto and a Kaiser Chief have pledged allegiance with Lily).

"The blogs are personal, " Allen says. "And I think Lily realises that she's lost sight of that because of the nature of the beast. Then she starts to feed the beast and to lose the quality of the original blogs.

There's a great danger that you can be too honest.

"I spoke to her before she got on the plane to go back to America and she was so down. You have to remember that Lily is a girl. She's a little girl, man. She's out there on her own with a pick-up band and they're all guys. And she's constantly being used as a comparative figure in the press. They generate this dreadful stuff by comparing her to Kate Moss . . . and she quite rightly points out: what are you doing comparing me to Kate Moss? It's ludicrous. Why compare her to Amy Winehouse?

They're two entirely different artists. The media do want to generate this war. And Lily . . . she's kind of snapped with this Girls Aloud thing but there's a part of me that's really glad that she did it. What she said was nasty but if Lily gives it out she's got to learn to take it.

Simple as that.

"I just hate to think that she will lose sight of her songwriting capabilities and get involved with that shit. Lily takes after me in many respects. A lot of my bravado and fun was about insecurity and fear."

We meet in the snooker room of the Groucho Club, an institution which has been his second home.

He is dressed in a windcheater with jeans and serious sandals.

"That, " he says, peering over red glasses at the green baize, "is the very snooker table I f***ed Janet Street-Porter on." He can do this kind of talk in his sleep but, apart from a gratuitous raising of the middle finger in the direction of AA Gill, showbiz revelations are largely absent from his new autobiography.

Perhaps there wasn't room.

Allen's story is a riot of incident, from a childhood as a petty thief to spells in borstal and prison, as he journeys from the squats of Notting Hill through punk to the birth pangs of alternative comedy.

Before his first period of notoriety with a Channel 4 show and the Comic Strip movies (a period which fizzles into hedonistic underachievement with bit parts in Shallow Grave . . . the corpse . . . and Trainspotting) he washes up in Glasgow, at the Citizens' Theatre.

Cast as Lady Macduff, he breaks a leg playing poker in his rehearsal skirt and has to take to the stage wearing a plaster cast. He has a lunch of cottage pie with director Giles Havergal ("a more apt dish you couldn't have wished for since the Glasgow Citizens' Theatre Company was largely gay") and blags a temporary Equity card so he can take his part in a show he describes as "pretentious middleclass b******s".

Disenchanted with theatre, he returns to post-punk Notting Hill and falls into alternative comedy with a stream-of-consciousness routine at the emerging Comedy Store. He supports the Clash on tour and is cast as Joe Orton in Stephen Frears's film Prick Up Your Ears. The cover image on Allen's book of a sneering Allen, in trunks, on a deck chair, with oiled torso and legs apart, was done for the film, only for him to be replaced at a later date by "the safer option", Gary Oldman.

And so it goes. In Allen's version of his life, he is a kind of punk Oliver Reed, raising hell, never quite fitting in and self-destructing before his due rewards can be denied him. His one incontrovertible success was the World Cup song he recorded with Damien Hirst and Blur's Alex James.

"The only dough I ever got was from 'Vindaloo'. That's another myth about me, that I'm rich." He admits to being "genuinely flabbergasted" when he sees the houses of his former Comic Strip colleagues. "They do commercials.

That's where the money is. I've done one. I was the Listerine tooth fairy. I did it because I owed �110,000 in tax."

The central figure in Allen's story is his father, who is often absent, posted abroad on a submarine. He admits he hated him for a while. "I wanted the approval. My dad, very much a man of his generation, didn't give it. But that's nothing new."

His father hasn't read the book.

"I think he might be appalled by some of the stories. But hopefully he'll know that I love him." I tell Allen that he doesn't seem to have linked his father's absence with his own attention-grabbing behaviour.

"If I'd come from the most wonderful, warm, open, compassionate liberal environment, I've got a feeling I'd be exactly the same."

His childhood thieving was opportunistic, he says. "The only thing I ever planned to do was to rob the supermarket I worked in.

But I was watching too much Mission Impossible and there was a telephone relay box on the floor: I thought it was lasers, so I didn't get into the safe! Me and my mate crawled round the front and just nicked all the fags. F***in' idiot!"

Borstal sorted him out. His PE teacher put him on an outward bound course, giving him the chance to become a team leader, and he began to concentrate on his exams. "My experience of institutions up to that point had been a comprehensive school, a public school, a detention centre, remand homes and hostels. And all of them were shit. Compared to public school, borstal was wonderful, and very funny."

There has been much human wreckage along the way, notably two broken marriages: the first to film producer Alison Owen (mother of Lily and actor Alfie OwenAllen); the second to producer Nira Park. His BBC biography counts eight children . . . the latest being his year-old baby girl, Teddie, with his Bodies co-star Tamzin Malleson.

I attempt an inventory of at least some of his offspring and Allen's exuberance flattens to a brief sulk. Before he met Owen, he had two children as the result of one-night stands. Kevin's mother was 18, Allen was 27. She wanted a baby, he didn't. Soon after that, another fling produced Grace.

"Alison and me get on fantastically well. Myself, Alfie and Lily are very close. And that's because of who we were and what we've done, not in spite of it. I don't feel guilty . . . I just know that if I had my time again I'd do it in a different way.

"Kevin is the only birth I feel bad about. Grace's mother was a much older woman, in her 30s; she wanted another child as company for her son Philip. She got that. She never contacted me, she got on with it, she lived with another guy who brought up Grace. In fact, Grace phoned me the day before yesterday. I'm going to her wedding."

Allen first met Kevin when Lily invited him and Grace to his 50th birthday party. "She has a sense of drama, Lily." That meeting "opened negotiations" between him and Kevin. "When we first met, I said, 'I'm going to be brutally honest with you. I'm not going to pretend that I love you. I can't, it's impossible. I'm your biological father and of course I'll do anything to make your life easier than it is.' That's all you can do. He has said it's just recognition that's important to him. I can't do any more than that."

Asked about his latest baby, he coos. "I don't want to sound like Woman's Own but it's fantastic. Of course it's different this time around. I guess as you get older you grow up. You come to appreciate and value time." During his first marriage, his partying lifestyle impacted on the time he spent with his children. "Even if you're sober, you've got a hangover. It's just shit. It's not real time.

It's not fair on them. It can be so much more fun if it's just real."

So, at 53, has Keith Allen grown up? Well, professionally, he is stable, doing good, mainstream work, such as his turn as the Sheriff of Nottingham in the BBC's Robin Hood. "I'm much more comfortable about who I am, now more complete."

He is, he concedes, a late developer. By his own account he did nothing until he was 28, "apart from have a brilliant time". He was 34 when he first took cocaine (he would get it out of the way earlier, if he had his time again).

"Sometimes I can't grasp how Lily copes with it, " he says suddenly, "because I've always been an outsider. It took me years to work out that I never went for gold because I was too scared. I could easily argue, coming third, it's being in the race, not winning it.

Whereas Lily . . . I don't think I could cope with what she's got."

Keith Allen's book, 'Grow Up', is published by Ebury, /24.99