Advantages and challenges in using mobile apps for field experiments: a systematic review and a case study

review and a case study
Jingwen Zhang, Christopher Calabrese, 
Jieyu Ding, Mingxuan Liu
and Biying Zhang
University of California, Davis, USA
Abstract
As smartphone’s computing power continues to grow and as mobile applications 
(apps) continue to dominate digital engagement, apps have become a new frontier for 
advancing field experiment methodology. Using apps may help researchers to scale 
up the reach, precisely control randomization and experiment materials, collect a 
variety of objective and self-reported data over time, and more conveniently replicate 
and adapt an experiment. We performed a systematic review on field experiments 
involving apps published between 2007 and 2017. Seven databases were scanned 
using a predefined search strategy. The database search retrieved 4,810 citations; 
101 articles met the inclusion criteria. Our review suggests that scholars have only 
started to employ apps in field experiments in the last 4 years. Most studies only 
used apps as an experiment treatment instead of an experiment platform; therefore, 
researchers have yet to fully leverage the advantages. Almost all studies were from 
the health research domain and 77.2% used randomized controlled trial design. Only 
7 studies utilized smartphone sensors for collecting data. Only one study reported 
cost and ethical concerns regarding using apps for the experiment. Given these 
findings, we reported a case study that targeted a minority racial group and leveraged 
the advantages of apps as an experiment platform and as a data collection tool to 
illustrate practical challenges and lessons learned regarding time, financial cost, and 
technical support. In conclusion, we suggest apps provide new ways to study causal 
mechanisms with experiment big data. Limitations of generalizability, retention, and 
design quality were discussed as well.