Skip to content
Insights

How Do Online Surveys Work?

March 22, 2021

The number of surveys completed over the internet has grown substantially in the past few years. The trend continues to grow as internet penetration becomes more prevalent and the cost of delivering surveys online decreases. There are some advantages with online surveys:

  • Convenience: Participants can take surveys anytime at their own pace
  • Cost effective: Compared to other methodologies, online surveys are the most economical data collection approach
  • Interview bias: The lack of an interviewer guiding a survey reduces the opportunity for bias
  • Multimedia: Insight organizations can host all sorts of multimedia such as graphics, audio and video which are not available in other survey modes

There are some drawbacks, though, with how sample representation works with online surveys. Online surveys can be subject to significant biases due to an absence of certain demographics relating to lack of internet access or interest in online survey participation. We have found that online surveys skew higher with younger and affluent demographics.

The method of how these participants are recruited to take surveys is also relevant as researchers strive to achieve representation with a properly balanced sample. There isn’t a cost effective or easily employed approach to achieve probability sampling of the general population using the internet (i.e. there is no national list of email addresses from which people could be sampled from); researchers must rely on panel sample companies for data collection. Prior to data collection, Big Village takes a consultative approach with clients and data scientists to choose an optimal data collection methodology.

 

Panel Sample

Survey participants can come from many different types of online sources. In one approach, survey takers are routed into surveys dynamically from an advertisement or website link, more commonly known as river sample. This method of sampling is fast and sometimes effective for taking short surveys; however, it is often not representative nor reliable as we really don’t know much about the identity of the participant. In contrast, panels are recruited through various mediums for the exclusive purpose of participating in online research and can be screened and profiled to achieve a stronger representation of the general population.

Ideally, survey panels are double opt-in, meaning they were invited from various channels such as loyalty programs (i.e. airline mileage reward members), social media, etc. and then registered and confirmed they are in control of the inbox with which they signed up. They’re made up of millions of members who can often be targeted using their profiling information. These members are usually preferred and represent the most common approach for survey data collection.

Big Village uses sample sources that use a double opt-in approach as a recruitment method, to ensure there is transparency in the identity of panelists throughout the sample design. In addition, Big Village uses proprietary technology to balance survey participants based on up-to-date census demographic data.

 

Online Sample Quality

 With the proliferation of AI bots and global survey farms, quality controls must be in place to verify the identity of the survey participant and that survey data is monitored to ensure it is legitimate. At Big Village, we use a 3-prong approach:

  1. Only using sample providers who have proprietary technologies and quality control programs that screen and monitor participants for fraudulent behavior and avoid duplicates amongst their members using fingerprint technology to oust those who are trying to cheat the system. For example, a person may try to complete the same survey from home and separately at another location using a different internet connection. Companies can track identity of these participants by cross matching device and IP address configurations.
  2. Employ third party fingerprint technology, licensed to monitor sample across all surveys in our data collection platforms. Third party fingerprint technology is independently managed and can become an essential tool for managing sample across different panel sources. Fingerprint technology provides other services to eliminate fraudulent responses including identification and removal of bots (autonomous programs), suspicious IPs addresses that clear their cookies and browsing histories and other similar hacker attacks.
  3. Every research agency should design and use their own independent system to track panelist behavior while surveys are being taken. Big Village uses a robust quality module that monitors performance across the following attributes:
    • Survey Duration: This tracks survey length to remove participants that speed through the survey
    • Red Herring: Participants are exposed to questions and specific instructions that must be answered in a truthful manner to proceed
    • Attentiveness: Participants answering questions in a certain pattern and then reversing their response at a later section
    • Open Ends: Open ends are monitored throughout data collection to remove rubbish responses
    • Bot Traps: Questions which can only be answered correctly by a human or seen by a bot
    • Active Monitoring: Researchers monitor fingerprint technology and proprietary quality module results on a periodic basis at a project level to proactively identify and remove fraud responses

The origins and inner workings of online sample sources are complex to understand, if certain precautions are in place the risk of poor-quality data collection can be reduced. Designing a strong quality system and partnering with proper technology solutions are critical in the process to attain accurate insights.

 

Written by Ricardo Ramos, Vice President at Big Village Insights.

 

References: https://www.pewresearch.org/politics/methodology/collecting-survey-data/internet-surveys/