The challenges of running social science experiments from home - and 14 tools that can help
This post originally appeared on SAGE Ocean
Never have the social and behavioral sciences been as critical as they have been during the COVID-19 pandemic and associated lockdown. More data was on demand and it was demanded instantly, along with the most robust analysis and policy recommendations, which meant that the classic research methods needed some creativity to transition to a socially distanced world. Data collection methods have been adapting not just to answer pressing questions about the impact of COVID on individuals and society, but also accelerating the ways in which they could be carried out digitally.
Digital methods are not new to the social sciences: surveys have gone digital since the 90s, with the recent Mechanical Turk, Prolific, Call for Participants and even social media marketing enhancing the digital recruiting of participants. The increase in the use of social media to cover 3.6B of the world’s population has allowed academics both to use the data from these platforms for research, as well as augment the sample snowballing methods. In the more recent years, trends around misinformation and fake news online have triggered teams of academics to develop online games such as GetBadNews and Hoaxy that teach the average internet user about misinformation, and also function as a way to test theories of learning and retention.
A growing number of social science researchers are shifting to digital methods, but it’s not an easy task, and this has been even more evident in lockdown. Some research methods are challenging (or even impossible) to run digitally, for example, experiments that can only be done in person or research that focuses on groups or regions that are hardly online. Even the fully digital VR experiments that we wrote about in another blog are difficult when the hardware is not available or can only be accessed in a lab. So what can you do if you want to continue doing social science experiments and scale them up while working from home?
We’ve selected 14 software tools that you can start using immediately to run your social or behavioral experiments online. Some come with integrated recruitment, others leave that bit to you; some are for asynchronous experiments, others let you run your experiment with multiple users at the same time; some work for small groups, others for thousands or more. As a bonus, and for those of you who want to go fully digital and forget about recruiting, we’ve included two bonus tools you can use to design and run simulated or computational experiments.
We spoke to some of these tools’ creators about what they think are the biggest challenges of running experiments remotely. Read their answers here or jump straight to the list of tools.
Claudia von Bastian of Tatool
The main challenges of conducting experiments remotely are portability (running experiments across platforms, browsers, and devices, usability (creating unambiguous instructions to ensure that participants are actually doing what you want them to do, and quality (having checks in place to determine whether the data quality of participant-generated data sets).
Chris Wickens of oTree
I think a big challenge of running online multiplayer experiments is dropouts. In a lab you can ensure that people play at the same time and that they all complete the experiment. Online this is much harder. With a multiplayer experiment, there is the risk that some participants will be stuck waiting for someone who dropped out (or lost their internet connection, etc).
Ting Qian of Finding Five
We are hearing from researchers that the biggest challenge in shifting to online experiments has been the lack of time and technical resources for (re)creating their experiments online. Further, since researchers can no longer invite participants to the lab and have them complete the task on a standardized lab computer, another concern is accounting for variability across participants’ devices. For instance, without adequate corrections, mouse tracking results can look vastly different when participants use a mouse as opposed to a laptop trackpad.
Jo Evershed of Gorilla
One concern for people new to online research is maintaining data quality when conducting online experiments. Thankfully, once you unpack all the different elements that can impact data quality, it’s not that much different to maintaining data quality in the lab. Dr Jenni Rodd gave an excellent lecture on this at our BeOnline2020 conference, the video can be viewed here.
Jason Radford of Volunteer Science
There are two big challenges to running online experiments: Choosing the right software and recruiting subjects. Different fields have different experimental traditions, so if you’re working across multiple areas then it can be hard to find the right software. When it comes to recruiting subjects, the issue is usually a lack of funding, or having to figure out where to find volunteers. It’s for these reasons that we designed Volunteer Science to be discipline agnostic, and to give you the choice of building your own participant pool, syncing with Mechanical Turk, or tapping into our existing pool of tens of thousands of volunteers.
PEBL is a free software specifically designed for use in psychology. It lets you design your own experiments or use any of the ready-made ones. It’s also a great tool to use in your teaching, since you can build and exchange experiments freely.
Tatool is an open-source easy to use tool for experts, as well as newbies. You can either download the software, or use the web version. You’d have to recruit your own participants, but they can access the tool from anywhere and any browser. It also has the option of running your experiments offline, but you’d probably need to be in a lab for that?
Experiment Factory is another open source tool that offers a collection of experiments and the ability to integrate the recruitment with Mechanical Turk. It’s still in beta, but you can sign up or reach out to the team behind it at the Poldrack Lab, Stanford University.
Gorilla is one of the more well-known commercial solutions for designing and running online experiments in behavioral sciences, integrates participant recruitment. Gorilla is great for research and for teaching!
E-Prime is a relatively comprehensive software for behavioral research. The company has some pretty sweet hardware tools and sensors as well, but you’d probably need to meet the person face to face and set them up.
OTree is an open source platform that is commonly used to design experiments in economics among other disciplines, with the ability to run multi-player strategy experiments. OTree also integrates with Mechanical Turk.
Lioness is a free web-based platform for designing and running your experiments that is super popular in economics as well. The team behind it works really hard to enable researchers to run experiments with simultaneous participants that are incentivized to hold their attention and not drop out!
PsyToolKit is a free-to-use toolkit for demonstrating, programming, and running cognitive-psychological experiments and surveys, including personality tests. It’s great for doing research and for teaching.
FindingFive is one of the newer tools for running your experiments online. It has some pretty cool features like blocking anyone who is trying to do the experiment again, and makes it easy to set up any exclusion criteria or prerequisites when you recruit your participants via Mechanical Turk.
Lab.js is also among the newer players in this space. It’s free and open source, you don’t need any coding skills, and has great resources for starting off doing an experiment or teaching about it.
nodeGame has a variety of features. It’s web-based, works on mobile too. It’s free and can scale to thousands of users participating in the experiment at the same time, but also lets you replace humans with simulated bots. I’d say pretty sweet!
Empirica is an open source tool, quite similar to nodeGame as both are developed in JavaScript, and lets you scale your experiments to thousands of users interacting simultaneously. Their mission is to help researchers easily iterate on sophisticated experimental designs.
Volunteer Science is an online experiments tool that enables researchers to run live and longitudinal experiments, with simultaneous or asynchronous participants. It’s excellent to use in the lab, from home or in the classroom. Read Jason’s recommendations on how to recruit your participants in a pandemic.
Breadboard is a software platform developed by a group of researchers at Yale that supports researchers in designing and running experiments with participants on networks. The platform also supports the recruitment.
Bonus: Tools for computational experiments
Wings is a semantic workflow system that assists scientists with the design of computational experiments.
CodaLab is an ecosystem for conducting computational research in a more efficient, reproducible, and collaborative manner. There are two aspects of CodaLab: worksheets and competitions. Worksheets lets you design and run your machine learning experiments in the cloud, plus, you can follow by publishing the experiment in a markdown file or executable paper. The competitions section is for hosting and participating in competitions in various areas requiring some computational tools and experiments.
