The Eurofighter is a beautiful aircraft, perfectly designed for air-to-air combat or to attack tanks and military installations on the ground. But the conflicts it was designed for are not those now being fought. Other than as an overpriced bomb truck, it is almost useless for the insurgent conflicts in Iraq and Afghanistan. The Eurofighter is obsolete even as it enters service.

Conventional embassies and their ambassadors are equally ill-suited to today’s challenges. The European foreign service, whose embryonic form already exists in the Council Secretariat, is awaiting its first orders once the EU’s Lisbon treaty is ratified. Like the Eurofighter, it will be elegantly constructed, very expensive, and heading for obsolescence even before day one.

Just like weapons designers, those who construct political bureaucracies and institutions must ask what kind of world are we trying to deal with? The 20th century was dominated by states, but this century is already shaping up with an altogether more anarchic prospect.

Non-state groups dominate military conflicts, with over 80% of the conflicts now before the UN Security Council involving non-state actors. Al Qaeda is the primary concern of security experts, rather than hostile tanks or nuclear missiles. In the field of development policy, philanthropic foundations are putting unprecedentedly large resources into such efforts as to end malaria, almost rivaling the aid flows of governments. George Soros’s Open Society Institute has in my view been as important as the European Union in fostering civil society and building the pillars of democracy in post-Soviet eastern Europe. The private sector’s foreign direct investment and speculative flows outweigh both official and philanthropic funds, in determining the economic fate of countries. In global politics, non-governmental movements and influentlial figureheads like Bono are proving almost, although not yet wholly, as important as governments, and according to surveys, are already more trusted.

Given the familiar nature of this analysis, it is surprising that culturally there is still a pervasive belief that it is governments, with their diplomats and embassies that will sort out the world. It is curious that even global campaigning organisations, like Live8 or Greenpeace, still see governments as the objects of their activism. “End debt” text messages campaigns have told G-8 leaders, but in truth the means to affect the world’s affairs are slipping from the governments’ hands. Take any global problem from the oil price to migration to climate change and it is clear that governments have less power than before to address them.

When I was a diplomat in the 1990s and the early years of this century, I found that the methods of conventional diplomacy seemed almost deliberately constructed to separate the diplomat from reality – and also from the people diplomats claim to represent. By and large, diplomats speak to other diplomats. And thanks to ballooning bureaucracy, email, and security constraints, they are more and more confined to their embassies, dealing with the real world by computer and telephone rather than directly. Most foreign services are still horribly hierarchical, with the grey heads at the top and most of the energy and ideas ignored at the bottom. Few such institutions make the necessary effort (and it takes an effort) to encourage innovative and contrary thinking. Many of the diplomats I know feel frustrated and stultified by mounting bureaucracy, and some will admit to a creeping feeling of irrelevance. This wasn’t what they signed up for.

In democratic terms, the actions and the views of diplomats are only tenuously connected to those people whom they allegedly represent. I found it ludicrous to pretend in negotiations that my views, which had in fact been invented by a small group of officials like myself, truly represented those of my whole country. This problem will of course be aggravated for the European foreign service (or European External Action Service, to give it its dreadful full name). As for accountability, one reason why governments are so little trusted is because its officials seem never to take responsibility for the failures they perpetrate in their country’s name – and in recent years there have been many. Diplomatic colleagues regarded it as naïve to believe that somehow they personally were morally responsible for actions they undertook on behalf of their government. This sort of raison d’état may have convinced earlier generations, but when I talk to the so-called Millennial generation born after 1980, they are far from impressed. Less loyal to the nation state than previous generations, they are equally sceptical of the diplomat’s hitherto unquestioned claim – and somewhat snobbish assumption – to represent them.

In short, the good old days of an ambassador are over. Diplomats are going to have to work harder to be relevant and respected in this new world. In an anarchic world, influence in shaping events is going to go to those with the most convincing arguments and the most power, and they are not necessarily going to be working in government. Governments may still legislate the laws that govern their countries and, to a lesser extent, the globe, but these laws will reflect norms and values instituted and led by others, and only some of the time will these leaders be governments themselves.

I think this is a very exciting prospect, if slightly scary. A world without automatic deference to governments and their diplomats will be a better one. Forcing our traditional élites to get down and dirty on the ground with the people will improve their ideas, and will also make it more fun to be a diplomat. Foreign services will have to become much more eclectic and less hierarchical if they are to generate the kind of creativity needed to keep up with and even lead, an eclectic and non-hierarchical world. In some cases, this is beginning to happen. But in most, I fear it is not.

At root, a more fundamental reappraisal of the means of diplomacy is required. The traditional function of communicator and negotiator with other governments will remain, but should no longer be treated as the dominant or sole function to which all other functions are subordinated. Embassies and diplomats are going to have to work in partnership with (that means not patronise) a much wider range of actors if they are to understand what is going on around them, and influence this hectic circus. At the Bali climate change talks NGOs were an important and powerful presence, and their involvement in international deliberations of this kind will clearly become the norm rather than the exception.

So-called (and ill-named) “public diplomacy” has always been the poorer cousin of the self-regarding hard-core “real” diplomats who do the important stuff like negotiate treaties and start wars. For some reason, diplomats and governments have believed that somehow the message about the role of governments can be separated in the public’s mind from what they actually do. The Bush administration’s pathetic public diplomacy efforts during its global war on terror illustrates the dangers of believing that you can separate a country’s public messaging from perceptions of its actual behaviour. People in the Middle East found it difficult to accept the Bush Administration’s proclamations of its commitment to democracy and human rights while politically the US busily hopped into bed with virtually every non-democratic and human rights-abusing tyranny in the region. It is time to abandon the notion of public diplomacy altogether, and replace it with a more interactive and frankly humble approach.

I do not mean by this that foreign ministers or ambassadors should start blogging, but rather that if they are to shape public opinion in other countries or even globally they will need to take a much more sophisticated approach than paying for quasi-corporate PR. The internet brings with it the likelihood of an immediate chorus of voices to disprove overly extravagant claims or political hypocrisies. This means that governments will increasingly be judged by their actions and not by how they themselves describe them. This is a wholly positive development for those who want more accountability, but it requires governments and diplomats at last to realise that no one believes you unless you practice what you preach. As in all good theatre, showing is telling. Diplomats should by all means communicate their government’s message, but must be aware that thanks to generational changes and new technology the scepticism with which that message will be greeted has never been greater. To those who are smart, of course, this is as much an opportunity as a challenge. We the public must now beware of governments which just like commercial corporations infiltrate their messages into otherwise innocent soap operas, chat-rooms or movie scripts.

The world is increasingly complicated, fluidly dynamic and, to be honest, resistant to comprehensive analysis. As more and more people live away from countries of their birth, and more still assume multi-cultural identities, I find it less and less convincing that national governments, and thus national diplomats, can legitimately claim to speak for and act on behalf of such heterogeneity. This doubt is even greater in the case an aggregate of national governments like the EU. That diplomats are declining in importance is, I believe, inevitable. Acceptance of this truth is – paradoxically – the only path to relevance for modern diplomats: to be primary no longer but only one among many is an exciting challenge as much as a burden. Success will go to those who use mass networks effectively, build coalitions of states and concerned non-state actors, corporations and NGOs and can credibly lead opinion.