In the smoking rubble of market fundamentalism, we are all being forced to rethink the principles that order our societies – and one small, shining idea is rising again from the wreckage. It is the idea of human equality.

The need for us to return to this, our best and most basic instinct, is spelled out in a new book by Professor Richard Wilkinson and Dr Kate Pickett called The Spirit Level: Why More Equal Societies Almost Always Do Better. It is the culmination of 25 years of scientific research. The truths it contains provides us with a compass to rebuild our societies, and a reason to be optimistic. There is a way we can make our societies dramatically better – and the impulse to do it is hard-wired into each of our brains.

For millennia, there was one obvious way to improve human life: raise material living standards. If you are hungry, you will be made a lot happier by food. If you are thirsty, you will be made a lot happier by water. The human impulse for self-improvement was simple: give us more. But we now know from reams of studies that once your basic needs are met – once you pass the magic number of $25,000 a year – something changes.

We carry on accumulating, because it's what we've grown to think will give us happiness, but it works less and less. And, after a while, this unhindered chasing of more, more, more by the very richest begins to make us miserable – and corrodes some of the other basics we need as humans.

One of our most basic psychological needs is for status – to feel that we are a valued member of our tribe. We evolved in small, very egalitarian tribes of hunter-gatherers, and have only lived outside them for a few minutes in evolutionary terms. So when we feel our status is threatened – or there is no way of becoming respected by the rest of the tribe – we begin to malfunction.

Indeed, almost nothing makes humans more anxious than panic about our status. Our bodies lock into a 'fight-or-flight' response, where our heart and lungs work harder, our blood vessels constrict, and we burn up our energy stores fast.

Our systems flood with a hormone called cortisol. If this lasts only a short period, it can be good: it helps you escape that lion chasing after you. But if it goes on for weeks or months, we begin to suffer all sorts of dysfunction.

Yet we have built our societies on exaggerating this status panic, and we have been ratcheting it up over the past 30 years. The more unequal a society is, the more intense it becomes. When there is a long social ladder, and the bottom rung means humiliation and poverty, everyone at every rung feels a sweatier need to cling to their place – and the society starts to go wrong. This isn't left-wing speculation: it is an empirical fact.

Japan and Sweden are very different societies, but they are consistently at the top of the charts for every indicator of social success. They have low violence, low mental illness, low teenage pregnancy, low drug-addiction, low obesity, low prison populations, high life expectancy, and high levels of friendship and trust. They are economically highly equal societies.

The US and Portugal are also very different societies, but they are consistently at the bottom of the charts. They are highly unequal societies.

Permanent status panic is an unnatural state for humans. High cortisol levels corrode our insides and massively increase the risk of heart-attack. We eat more – and our bodies store fat differently. It hugs them to our middles, rather than storing them lower down, in our hips and thighs. We look for ways to soothe ourselves – like drug-addiction. We are far more likely to break down into depression or mental illness. And when we are locked in stress, we become more suspicious of the people around us. In highly equal Sweden, 66% of people feel they can trust their fellow citizens, and as a result have the highest levels of friendship in the developed world. In highly unequal Portugal, only 10% of the population trust the rest: see the bars on the windows.

Our élites have adopted an ideology – the extreme inequality of market fundamentalism – that simply doesn't suit our species. It doesn't have to be this way. By democratically taxing the rich and using the money to lift up the poor, we can make life better for all of us. Of course, there must be some income differentials, but nothing like our own grotesque rates.

The evidence is in: a more equal society is a happier, safer, and healthier one. (The obvious exception to this rule is communist societies. They were incredibly miserable: if equality is imposed by crazed tyrants then it has none of these positive effects.)

Wilkinson and Pickett show how the country would change over time if we taxed and invested our way to the same levels of economic equality as Sweden: "Levels of trust might be expected to be two-thirds higher, mental illness more than halved, everyone would get an additional year of life, teenage birth rates would fall by a third, homicide rates would fall by 75%, everyone could get the equivalent of almost seven weeks' extra holiday a year, and the government would be closing prisons all over the country."

It's a shining vision – and not utopian. It exists now in a free, democratic country.

And there is another even more sombre reason why we need to democratically equalise our societies. We are now highly likely to face a series of destabilising and dangerous climate shocks. In his book Collapse: How Societies Choose To Fail Or Survive, the author Jared Diamond looks at the societies throughout history that have faced similar disasters. The biggest single factor determining whether they survived was equality. If the élite stands far above the population and can initially insulate itself from the effects of the shock, then the society doesn't make it through.

At the end of the failed age of market fundamentalism, the long-suppressed cry for equality is emerging once again. Its glow should be at the core of how we move beyond this cold, cold depression.

Diarmuid Doyle is on leave