To keep up with the rapid pace of technological change and the huge market shifts we're experiencing, leaders have to be in a position to adapt or pursue new strategies. That requires a more purposeful business environment which fosters innovation and encourages people to work together more effectively.

To hone that environment requires us to take a much closer look at our business structures and cultures, and the impact these can have on our people and ultimately the future success of an organisation.

At The Zebra Project launch, Matthew Taylor, CEO from the RSA explored organisational and individual behaviour, the challenges for business leaders and how creative, people-focused communities can help organisations survive and thrive.

Here we summarise his insights:

Organisational drivers and challenges

In this business world, we have to make choices and decisions which can all be driven by different motivations. As a leader for example, we have to make decisions that ensure our companies hit their targets but at the same time, encourage our staff to perform to the best of their ability. However, individuals within our business will have different opinions on what needs to be done to achieve those outcomes based on different drivers. Some may say it's about better incentives, others may argue that better leadership and a clearer strategy is required. Those with their ear to the ground on the office floor may say that employees want to feel more supported rather than have stronger rules and incentives.

These different views and motivations mean that it can be hugely challenging to agree on the best proposal. Every idea is countered and although in some cases an action is agreed, this can often because those involved have given up. The outcome is a barrier to progress, and a sense of déjà vu with many people feeling that they've heard these conversations before and they have never made a difference.

What the theories tell us

Insights from sociology and anthropology help to explain why we are faced with these challenges and it can be so difficult to build and sustain successful organisations.

The social determination theory developed in the 70s identifies three basic psychological human needs. Competence, relatedness and autonomy. This thinking can also be applied to organisations and is the basis of the work of Dan Pink. Competence is related to the top-down hierarchical authority drive, relatedness as the group belonging, and autonomy, as the individualistic, 'what's in it for me?' drive.

There's also another framework which is less focused on individuals and derived from leading British anthropologist, Mary Douglas. She looked at everything from a tribe in a rainforest to modern organisations and families. From this analysis she designated two forces on cultures – Grid and Group.

Grid relates to how much rigidity is imposed on people and Group relates to the amount of shared belonging and what holds the people together. Societies with strong forces of Grid and Group are hierarchical. In the opposite corner, a weak Grid and weak Group would have a more individualistic culture.

In a nutshell these theories underline the fact that leaders have to manage a vast number of drives such as values, belonging and individual aspirations. To achieve success, these should be combined bringing both the hierarchical and group elements together.

Creative communities with a cause

Charlie Leadbeater carried out research into successful organisations and found a phrase to describe them – creative communities with a cause. They had struck the right balance between authority, shared values and individual incentives – their people all had a sense of belonging and were committed to excellence, but the organisation also gave them the flexibility to be creative and value innovation.

But that's not easy to achieve. The drivers that leaders have to manage antagonise each other leading to a very complex balancing act. For example, if there's too much hierarchy, there's a counter reaction. If there's too much individualism, there's conflict.

Our changing and increasingly complex world also adds to the challenge.

As levels of trust continue to decline in society, leaders will increasingly be unable to rely on willing obedience, which threatens hierarchy and organisations with a more traditional authority model.

Organisations have higher levels of accountability and transparency as well as stakeholders, including younger workers who not all leaders can relate to. The workforce is also more diverse and although that means that they may have less in common as individuals, they are more likely to share a strong interest in the values of the organisation they work for. They also care more about the purpose of the organisation and fairness as witnessed by the recent calls for greater action on pay disparities.

Young talent is vital for the future success of businesses, but their individualism must be carefully managed. The younger generation appear to be less obsessed with money and status, so individual incentives may no longer be effective. At the same time however, their sense of themselves can seem more fragile and less resilient.

The power of purpose

Although these changes will make it hard for businesses, especially those which want to create creative communities, there's an upside too.

We all have a purpose – something that drives us to succeed. Workers aren't just workers, they are individuals with different inner motivations. By acknowledging these drivers as well as each other's, we can more effectively incorporate them into the world of work.

This way of thinking will open up opportunities for new forms of leadership, which are more focused on people, adaptation and learning. In our ever-changing world, this could hold the key to ensuring organisations survive and thrive.

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.