Skip to content

Sample Matters (to Tracking Programs)

April 23, 2020

Best Practices for Sample Management

As Insights professionals who manage tracking programs, at some point we’ll be asked “Are those true changes in behaviors or perceptions?” That is when we want to be confident that changes in tracking results reflect shifts in the market, not sample inconsistencies! Start your project on the right path, and stay on track, by following these best practices for sample management.

1 – When the population is large, sample selection should be random; this is true even when the population is customers. Sometimes companies attempt to survey all customers, which can result in more surveys than needed, a lower response rate (due to repeated attempts), and less representative findings due to increased non-response.

2 – For an online general population sample, Big Village’s Census Balancer® tracks the demographics of respondents coming into the survey so that a representative audience is sampled for the survey screener. By doing so, any demographic differences found after any additional screener criteria is reflective of the market.

3 – For B2B studies, quotas are preferred to ensure that the sample mirrors the target audience. Weighting should be reserved for adjusting slight skews that arise from the inability to reach certain segments of the audience in proper proportions.

4 – For transaction-based surveys, sample procedures depend on frequency of the transaction. It is not necessary to get feedback on all re-occurring transactions (e.g., restaurant orders), but it is typically good to get feedback on all of the less common ones (e.g., a claim). If it is a call transfer or website intercept, immediately after a transaction, it’s not possible to control how frequently a person is invited to participate; however, if contact is not immediate, it is possible to control how often a person is contacted.

5 – When partnering with online sample providers it is best to remain consistent throughout a tracking period when trending is involved. To boost response rates, consider offering an incentive, extending the deadline, sending reminders, or attempting to reach via multiple modes.

When making changes, a parallel test is strongly recommended for one or more waves to determine the impact on results. If changes are drastic, the best approach might be to draw a line in the sand and not compare back to less representative samples. However, in less severe circumstances, calibration can be done on some key measures of the past research to align with current results.

The steps to align results from different samples or methods can differ based on the change (e.g, mode, reminders, new panel, etc.). Weighting is one option that is applicable in certain circumstances.

Taking appropriate actions to sample the population correctly will help to ensure changes in results are based on consumer or business opinions not sample miss-steps. Big Village has the expertise to guide sampling decisions, and other methodology considerations, so you can rest easy your tracking research is measuring what is intended.

Stay tuned or contact us for our future blog post that covers best practices in weighting!


Written by Sheilah Wagner, Research Director at Big Village Insights.