Measuring Success
In business, there’s one metric that trumps all: profit. That’s not so for projects seeking a social impact. Different stakeholders value different outputs and outcomes. Measuring success becomes very subjective.
Furthermore, digital technology generates lots of data sets: clicks, views, shares, invites, impressions, tweets, downloads, fans, likes and users. Noise to some, signal to others – they are data nonetheless. What gets counted becomes what counts, encouraging “clicktivism” – easy actions that feel good but generate little impact.
Engagement, however, is about being “attached, committed, involved and productive,” according to summit participants. It requires multiple data sets measured from multiple vantage points over time. It also means paying attention to the offline world: How, for example, do we measure changes in real-world communities that come about because of participation online? One of the greatest challenges is to go beyond evaluating Web metrics to finding ways to measure behavorial changes on the ground. Many of these already exist in the social sciences – from measuring collective efficacy to social capital and neighborhood brand affiliation. Are they seeking to change public policy? Influence someone in a position of authority? Change cultural perceptions? Deliver services? Metrics must follow stated outcomes of programs.
There’s no silver bullet to define success. A small group of summit participants proposed an initial model for metrics here:
1. Who participated?
- Number of participants
- Demographics/diversity
- Prior level of engagement
2) Who was affected?
- Participants
- Targeted beneficiaries
- Other stakeholders
3) Did we do what we said?
- Stated goals
- Unintended consequences
- Effectiveness
4) What changed? Impact?
- Individual vs. collective value
- Short, medium, long-term
- Trust and efficacy
For each data set, it’s also important to get input from three different perspectives – the individual participants, the group or network and outsiders. Lastly, the data can be quantitative or qualitative. Impact and engagement may be captured by numbers as well as stories.
Yet, challenges begin to arise the minute you apply this framework.
One participant in the metrics discussions was Micki Krimmel, whose platform Favortree allows neighbors to exchange goods and services. Her goal is to increase social capital in each community and foster more civic engagement. The biggest challenge for Krimmel is knowing how to measure “social capital” – a common good without common metrics. Do you measure the number of acquaintances or relationships, the number of interactions with neighbors, the number of groups joined, or how often they work together to achieve common goals?
Not everything that counts can be counted, and not everything that can be counted counts.
– William Bruce Cameron
Favortree has been tracking new relationships being formed through the platform, in addition to how many exchanges participants make and how much money they save. But do new relationships formed around transactions constitute social capital or is engagement about something more?
There’s increasing pressure from funders and investors for concrete measures of success. They need to see the return on their investment. So do the innovators who create and manage these tools.
“If we can’t measure our work,” says Krimmel, “we’re not sure we’re putting resources in the right direction.”
Where to put resources isn’t just a question for innovators and their funders. It’s a question for the field as a whole, which is why sharing data and what you measure should be encouraged, if not required by funders. Ideally, there can be a common platform where engagement practitioners can share data with other practitioners.
Two summit participants, have started platforms to document instances of political and civic engagement.
1) Participedia – Archon Fung, professor at the Harvard Kennedy School, started this wiki to collect examples of democratic innovation and public engagement around the world. Hundreds of thousands of participatory processes occur each year. Participedia allows researchers and practitioners to compare them.
2) Cairns – Beth Simone Noveck, professor at New York Law School, started this website to navigate the “landscape of collaboration” and learn from the results of organizing efforts, citizen engagement projects – any kind of collaboration or collective action.
But the field is far from having a platform where common sets of data are identified and impartiality can be assured. Such a platform would need to provide the right incentives for participation and establish clear use cases for different audiences.
Recommendations: – Standardize metrics for assessing the impact of technology for engagement projects, including measures of online engagement, offline engagement and social capital. – Create metrics specific to the programmatic outcomes sought. – Adopt common survey tools and Web analytics to measure impact. – Explore approaches for transcending website stats by measuring the impact new platforms have on offline behavior. – Include qualitative data in the assessment to better understand the results. – To be bottom up, empower communities to codesign and contribute to impact assessments.