15 Nov 2019

Replication Crisis



Replication crisis diminishes trust in empirical research (Open Science Collaboration 2015; Lupia 2018; Chopik et al. 2018)

slides: osf.io/b5vnh/ artzyatfailing2

 

(Fecher and Friesike 2014)

Replication Crisis



Replication crisis diminishes trust in empirical research (Open Science Collaboration 2015; Lupia 2018; Chopik et al. 2018)


Openness & science communication: (re)built trust (Fecher and Friesike 2014)


Particularly significant within professions that include reflections on evidence-based actions

slides: osf.io/b5vnh/ artzyatfailing2

Open Science & Perceived Trustworthiness

Current Research

(Wingen, Berkessel, and Englich 2019)

  • learning about replication crisis diminishes trust in psychological research
  • learning about reforms does not help to rebuilt trust


(Anvari and Lakens 2019)

  • learning about replication crisis (and QRP) diminishes trust in past psychological research
  • learning about reforms additionally reduces trust in future psychological research

slides: osf.io/b5vnh/ artzyatfailing2

Open Science & Perceived Trustworthiness

Shifting the Focus

descriptions of reforms visible OSP in journal articles (“badges”)

slides: osf.io/b5vnh/ artzyatfailing2

 

 

 

Current Study

Design

rotations

  • order of treatment
  • order of topic

slides: osf.io/b5vnh/ artzyatfailing2

Current Study

Measures

  • perceived trustworthiness (subscale integrity) (Hendriks, Kienhues, and Bromme 2015)
    • semantic differential: 7-point scale
    • sample item: honest - dishonest
  • treatment check
    • likert: 4-point scale
    • sample item: Used materials and collected data in this study are publicly available.


  • Preregistered Hypotheses (osf.io/2zypf):
    • perceived integrity: nonOSP nonvis visOSP
    • (topic specific) multiplistic epistemic beliefs perceived integrity

slides: osf.io/b5vnh/ artzyatfailing2

Current Study

Sample & Analyses

  • Sample
    • BFDA: N= 220 participants
    • pre-service teachers
  • Analyses: Informative Hypotheses Approach bain (Gu et al. 2019)

slides: osf.io/b5vnh/ artzyatfailing2

Dashboard

Thank you

tools used

Pictures

Anvari, Farid, and Daniel Lakens. 2019. “The Replicability Crisis and Public Trust in Psychological Science.” Preprint. PsyArXiv. https://doi.org/10.31234/osf.io/vtmpc.

Chopik, William J., Ryan H. Bremner, Andrew M. Defever, and Victor N. Keller. 2018. “How (and Whether) to Teach Undergraduates About the Replication Crisis in Psychological Science.” Teaching of Psychology 45 (2): 158–63. https://doi.org/10.1177/0098628318762900.

Fecher, Benedikt, and Sascha Friesike. 2014. “Open Science: One Term, Five Schools of Thought.” In Opening Science, edited by Sönke Bartling and Sascha Friesike, 17–47. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-00026-8_2.

Gu, Xin, Herbert Hoijtink, Joris Mulder, and Caspar J. van Lissa. 2019. Bain: Bayes Factors for Informative Hypotheses.

Hendriks, Friederike, Dorothe Kienhues, and Rainer Bromme. 2015. “Measuring Laypeople’s Trust in Experts in a Digital Age: The Muenster Epistemic Trustworthiness Inventory (METI).” Edited by Jelte M. Wicherts. PLOS ONE 10 (10): e0139309. https://doi.org/10.1371/journal.pone.0139309.

Lupia, Arthur. 2018. “The Role of Transparency in Maintaining the Legitimacy and Credibility of Survey Research.” In The Palgrave Handbook of Survey Research, edited by David L. Vannette and Jon A. Krosnick, 315–18. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-54395-6_41.

Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349 (6251): aac4716–aac4716. https://doi.org/10.1126/science.aac4716.

Wingen, Tobias, Jana Berkessel, and Birte Englich. 2019. “No Replication, No Trust? How Low Replicability Influences Trust in Psychology.” Preprint. Open Science Framework. https://doi.org/10.31219/osf.io/4ukq5.