I at present lead a small knowledge group at a small tech firm. With all the pieces small, we’ve a whole lot of autonomy over what, when, and the way we run experiments. On this collection, I’m opening the vault from our years of experimenting, every story highlighting a key idea associated to experimentation.
And right here we’ll share a shocking end result from an early take a look at on our referral-bonus program and use it to debate the way you may slim your determination set for experiments (a minimum of after they contain people).
Background: It’s COVID and we have to rent a zillion nurses
IntelyCare helps healthcare services match with nursing expertise. We’re a glorified nurse-recruiting machine and so we’re all the time trying to recruit extra successfully. Nurses come to us from many sources, however those that come through referrals earn greater critiques and stick with us longer.
The 12 months was 2020. IntelyCare was a child firm (nonetheless is by most requirements). Our app was new and most options had been nonetheless primitive. Some examples…
- We had a manner for IntelyPros to share a referral hyperlink with mates however had no monetary incentives to take action.
- Our software course of was a significant slog. We required a small mountain of paperwork for overview along with a cellphone interview and references. Solely a small subset of candidates made it during to working.
Throughout a recruiting brainstorm, we latched onto the thought of referrals and agreed that including monetary incentives could be straightforward to check. One thing like, “Get $100 when your good friend begins working.” Zero creativity there, however an concept doesn’t must be novel to be good.
Figuring out that many individuals may refer time and again in the event that they earned a bonus, and realizing that our software course of was nothing wanting a gauntlet, we additionally questioned if it could be higher as a substitute to offer clinicians a small prize when their mates begin an software.
A small prize for one thing straightforward vs a giant prize for one thing tough? I imply, it depends upon many issues. There’s just one approach to know which is finest, and that’s to attempt them out.
The referral take a look at
We randomly assigned clinicians to one in all two experiences:
- The clinician earns an additional $1/hour on their subsequent shift when their referral begins a job software. (Tremendous straightforward. Beginning an software takes 1–2 minutes.)
- The clinician earns $100 when their referral completes their first shift. (Tremendous exhausting. Some nurses race by it, however most candidates take a number of weeks and even months in the event that they end in any respect).
We held out an equal third of clinicians as a management and let clinicians know the principles through a collection of emails. There’s all the time a threat of spillovers in a take a look at like this, however the considered one group stealing all of the referrals from the opposite group appeared like a stretch, so we felt good about randomizing throughout people.
Decidedly non-social: Many individuals hear these two choices and ask, “Did you consider attempting prosocial incentives?” (Instance: I refer you, you do one thing, we each get a prize). Studies show they’re often better than individual incentives they usually’re fairly widespread (instacart, airbnb, robinhood,…). We thought of these, however our finance group grew to become very unhappy on the concept of us sending $1 every to a whole bunch of people that might not ever grow to be staff.
I suppose Quickbooks doesn’t like that? In some unspecified time in the future, you simply settle for that it’s finest to not mess with the finance group.
Because the $1/hr reward couldn’t be prosocial with out changing into a significant headache, we restricted payouts in each packages to the referring particular person solely. This provides us two referral packages the place the important thing variations are timing and the payout quantity.
Seems timing and presentation of incentives matter. Loads. Social incentives additionally matter. Loads. If you happen to’re attempting to growth-hack your referral program, you’d be good to contemplate each of those dimensions earlier than growing the payout.
Nerdy apart: Fascinated with issues to check
Product knowledge science, with minimal exception, is fascinated by how people work together with issues. Typically it’s an internet site or an app, nevertheless it might be a bodily object like a pair of headphones, a thermostat, or an indication on the freeway.
The science comes from altering the product and watching how people change their habits in consequence. And you may’t do higher than watching your clients work together together with your product within the wild to study whether or not a change was useful or not.
However you’ll be able to’t take a look at all the pieces. There are infinite issues to check and any group tasked with experimenting should lower issues right down to a finite set of concepts. The place do you begin?
- Begin with the product itself. Ask people who find themselves accustomed to it how they prefer it, what they need was completely different, Sean Ellis, NPS, the Mom Test, and many others. That is the widespread place to begin for product groups and nearly everybody else.
- Begin with human nature. For a lot of a long time Behavioral Scientists have documented explicit patterns in human habits. These scientists go by completely different names (behavioral economists, behavioral psychologists, and many others.).
In my humble opinion, the 2nd of those beginning factors is severely underrated. Behavioral science has documented dozens of behavior patterns that may inform how your product may change most successfully.
A number of honorable mentions…
- Loss aversion: individuals hate dropping greater than they like successful
- Peak-End: individuals bear in mind issues extra favorably after they finish on a constructive be aware
- Social vs Market Norms: all the pieces adjustments when individuals pay for items and providers as a substitute of asking for favors
- Framing: individuals make selections based mostly on how data is offered
- Left-digit Bias: perceptions of a value are disproportionately influenced by the main digit ($0.99 = 🔥, $1.01 = 🥱)
- Present Bias: individuals hate waiting
These psychological shortcuts aren’t a silver bullet for product and Marketing. We’ve examined many of those in several settings to no impact, however some have labored. Maybe you’ll find $160M under your couch cushions.
Zooming in on current bias
Many people reveal a habits sample the place they like small, quick rewards over bigger rewards sooner or later. Social scientists name this present bias and measure it with robust questions like…
Individuals who select quick, smaller rewards have some measure of current bias.
Current bias can differ throughout individuals and circumstances. You could possibly be a affected person particular person in most conditions. However once you’re hungry or drained… not a lot. That is one cause why A/B exams are so helpful in comparison with, say, surveys or interviews.
For our experiment, the query just isn’t so completely different from the lab questions:
- Would you relatively have ~$8 someday this week or $100 someday within the coming weeks and possibly by no means?
Experiment outcomes
Many people (current firm included) thought the $100 provide would do finest. It sounds greater. It seems higher in an electronic mail. It’s straightforward to clarify.
I additionally thought we would get extra purposes from referrals with the $1/hr program, however didn’t assume that may carry by to extra referrals. I figured individuals would take the cash and run.
To my shock the $1/hr program delivered extra purposes through referrals and extra working clinicians, all at a a lot decrease value.
The $100 provide led to a 65% improve in referrals in comparison with our management group. That is large!. Many individuals like any such program as a result of we’re solely giving rewards when a referral is profitable. It seems like the cash is properly spent.
The $1/hr provide, nevertheless, is even higher. Referrals elevated by 81% in comparison with our management group. And despite the fact that the rewards had been smaller and paid out earlier, referrals from these IntelyPros nonetheless went on to begin working for us at about the identical fee because the $100 group.
Sure, many IntelyPros earned rewards for referrals that utilized after which didn’t cross the end line, however the math nonetheless labored out. Even with an imperfect conversion fee from software to working, the entire value per new working IntelyPro was lower than half of the $100 group’s value.
ABT: All the time be testing
The function that referrals play for a enterprise can change for a lot of causes. As an organization turns into greater and extra acquainted, referrals grow to be much less essential. You don’t want individuals to unfold the phrase as a result of the phrase is already out.
We ran this take a look at in 2020 throughout a world pandemic. We ran a smaller-scale take a look at a 12 months later and noticed related outcomes. Would we see the identical outcomes now, in 2025? Onerous to say. Issues are a lot completely different within the healthcare staffing house these days.
That is but one more reason why testing is so essential. One thing like referral bonuses might work wonders for firm Some time being a dud for firm B. The upside of product experiments is that the world is your laboratory. The draw back is that it’s exhausting to know what generalizes to different merchandise.
Key takeaways for many who made it this far
- If you’re working experiments to enhance your product and accumulate knowledge from the people utilizing your product, behavioral science is a fruitful place to search for take a look at concepts.
- Generally the concepts that appear much less nice prove to have the best affect!
- Timing and presentation matter for incentives. Earlier than you improve the scale of some payout, maybe put some thought into the effort and time required to earn that payout.
(This put up was refurbished from my own 2022 post on behavioraleconomics.com)
