Monday, January 16, 2017

The Difficulty of Evaluating the Real Impact of Advocacy

An effective altruist asked if One Step had evaluated our current video against a more "standard" pro-veg video. This was our response:

While we’ll be testing a different pro-chicken video against our current one in February, we’ve not tested our message / video against a “standard” message, because we don’t know how to get data that would be meaningful. As you know, we would need to get impact data from everyone who sees the message, not just those who click through.

More importantly, and even more difficult is to get is the longitudinal data necessary to be able to make clear conclusions. Given that 80% of people who go veg go back to eating animals (and thus become spokespersons against ethical eating), it is important (but basically impossible for us at this time) to do long enough studies to see what everyone's full reaction is over time.

This is, of course, why the default for evaluators is to recommend corporate campaigns and other welfare reforms: those are easily measured. But if we want a truly better world, as opposed to one simply less bad, we need to change people's actual purchases, in a realistic and meaningful way.

Our intention, of course, is that by avoiding a veg message, we avoid creating four failed vegetarians for every lasting new vegetarian.


6 comments:

Brian said...

Never thought about the regressed veg as being a potential spokesperson against going Veg. Hmmmm

Jennifer Liepin said...

It's disappointing to read, "Given that 80% of people who go veg go back to eating animals (and thus become spokespersons against ethical eating), it is important to do long enough studies to see what everyone's full reaction is over time," but as someone involved heavily in an Vegan Outreach program, I'm glad to be aware of this stat.

Mon7que said...

Where is the evidence for the 80%??

Matt Ball said...

There have been a number of studies, in this country and the UK. This is the latest: https://faunalytics.org/how-many-former-vegetarians-and-vegans-are-there/
5x more former vegetarians than current.
Thanks for asking, and please consider supporting work that avoids this!

Unknown said...

So I wonder what the best target audience would be for OneStep? Can we really expect health concious omnivores to accept the message? People who don't eat much chicken anyway, like red meat eaters, might be willing to listen but would forget as soon as a friend offered to take them out for chicken wings (anecdotal observation).

Matt Ball said...

This is a great question. There are two answers:
1. Anyone who might be interested in changing their diet, but finds going full veg to be too hard. (See the Portland post for more on this.)
2. People who love animals but who, again, finds the idea of being veg too hard.
3. People who used to be veg because they found it too hard or didn't feel healthy, but they still want to make a difference.

The best thing about online outreach is we test different audiences via Facebook and find which one(s) are most receptive, in terms of click-throughs and conversions.