21 Oct 2019

A (typical?) story

of a young researcher

slides: osf.io/64mbt

Incentive structures

in scholarly communication

novelty

positive results

(Nosek, Spies, and Motyl 2012)

slides: osf.io/64mbt

Degrees of freedom

in the research process

Degrees of freedom

in the research process

  • multitude of decisions to be made


Are football (soccer) referees more likely to give red cards to players with dark skin than to players with light skin?

  • 29 research teams
  • same data set
  • same results?

(Silberzahn and Uhlmann 2015)

slides: osf.io/64mbt

Degrees of freedom

in the research process

  • multitude of decisions to be made


Are football (soccer) referees more likely to give red cards to players with dark skin than to players with light skin?

  • 29 research teams
  • same data set
  • same results?

(Silberzahn and Uhlmann 2015)

slides: osf.io/64mbt

Nobody’s perfect - cognitive biases

Nobody’s perfect - cognitive biases

Nobody’s perfect - cognitive biases

confirmation bias

hindsight bias

(Munafò et al. 2017)

slides: osf.io/64mbt

QRP

Questionable Research Practices

outcome switching/ failing to report all DV

failing to report all conditions

selectively reporting studies that “worked”

(John, Loewenstein, and Prelec 2012)

selective
reporting

slides: osf.io/64mbt

QRP

Questionable Research Practices

peeking & optional stopping

excluding data (after looking at the impact)

HARKing

(John, Loewenstein, and Prelec 2012)

flexibility
in methods

slides: osf.io/64mbt

Open Science (Practices)

What is Open Knowledge?

Knowledge is open if anyone is free to access, use, modify, and share it - subject, at most, to measures that preserve provenance and openness.



Open Knowledge Foundation

slides: osf.io/64mbt

What is Open Science?

Open science is the movement to make scientific research, data and dissemination accessible to all levels of an inquiring society.



FOSTER Open Science
(Woelfle, Olliaro, and Todd 2011)

slides: osf.io/64mbt

Open Science Principles

  1. open access
  2. open data
  3. open source & materials
  4. open methodology
  5. open peer review
  6. open educational resoures (Piwowar and Vision 2013)

slides: osf.io/64mbt

 

(Fecher and Friesike 2014)

What’s in it for me?

What’s in it for me?

  • OA articles higher citation rate
      (Lewis 2018)
  • Preprint + Publication higher citation rate
      (Fraser et al. 2019)

slides: osf.io/64mbt

What’s in it for me?

  • OA articles higher citation rate
      (Lewis 2018)
  • Preprint + Publication higher citation rate
      (Fraser et al. 2019)
  • Open data higher citation rate
      (Piwowar and Vision 2013)
  • Data papers in Open Data Journals
      (incomplete list of journals)

slides: osf.io/64mbt

What’s in it for me?

  • OA articles higher citation rate
      (Lewis 2018)
  • Preprint + Publication higher citation rate
      (Fraser et al. 2019)
  • Open data higher citation rate
      (Piwowar and Vision 2013)
  • Data papers in Open Data Journals
      (incomplete list of journals)
  • Facilitate collaboration
     
  • Feedback improvement
     
  • …

slides: osf.io/64mbt

Hands-on part

  • storing & organizing materials on OSF
  • non-proprietary, reproducible Online Survey Tool
  • documentation of scales (to ensure usability)

 

projects on OSF

  • create new project
    • accessibility settings on private
  • create project component “survey structure & codebook”
    • accessibility settings on public
  • create data component “free data for free people”
    • accessibility settings on public

OSF

formr

Online Survey Tool

  • observe survey, compare with google-sheet
    • Which kind of items are the ones you’d use in future research?
  • create one survey page
    • with multiple and different items that seem interesting for future research
  • create run
    • “tmp_…” as first survey
    • “skip” as second survey
    • “skip” should be skipped, on the basis of a condition you define
    • try other options? Randomization, Loop, …

formr.org

slides: osf.io/64mbt

bit.ly/iwm-os

  • scroll down
  • click on “launch binder”

documentation of scales

3 different codebooks

  • compare codebooks
    • which elements are good for reproducibility, which less so?
  • look what’s “behind the scenes” (in the .Rmd files)

(Arslan 2019)

codebook

slides: osf.io/64mbt

how to stay at it

committing

how to stay at it

information & community

  • twitter (how-to)
    • @ TuebingenOSI
    • @ artzyatfailing2
  • streamed conferences
    • OSC
    • SIPS
  • Open Science MOOC
  • podcasts
    • Open Science Radio
    • ORION Open Science
    • Everything Hertz
    • The Black Goat
    • Circle of Willis

slides: osf.io/64mbt

Thank you

tools used

Pictures

Arslan, Ruben C. 2019. “How to Automatically Document Data with the Codebook Package to Facilitate Data Reuse.” Advances in Methods and Practices in Psychological Science 2 (2): 169–87. https://doi.org/10.1177/2515245919838783.

Fecher, Benedikt, and Sascha Friesike. 2014. “Open Science: One Term, Five Schools of Thought.” In Opening Science, edited by Sönke Bartling and Sascha Friesike, 17–47. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-00026-8_2.

Fraser, Nicholas, Fakhri Momeni, Philipp Mayr, and Isabella Peters. 2019. “The Effect of bioRxiv Preprints on Citations and Altmetrics.” Preprint. Scientific Communication; Education. https://doi.org/10.1101/673665.

John, Leslie K., George Loewenstein, and Drazen Prelec. 2012. “Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling.” Psychological Science 23 (5): 524–32. https://doi.org/10.1177/0956797611430953.

Lewis, Colby Lil. 2018. “The Open Access Citation Advantage: Does It Exist and What Does It Mean for Libraries?” Information Technology and Libraries 37 (3): 50–65. https://doi.org/10.6017/ital.v37i3.10604.

Munafò, Marcus R., Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Du Percie Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware, and John P. A. Ioannidis. 2017. “A Manifesto for Reproducible Science.” Nature Human Behaviour 1 (January): 0021. https://doi.org/10.1038/s41562-016-0021.

Nosek, Brian A., Jeffrey R. Spies, and Matt Motyl. 2012. “Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth over Publishability.” Perspectives on Psychological Science : A Journal of the Association for Psychological Science 7 (6): 615–31. https://doi.org/10.1177/1745691612459058.

Piwowar, Heather A., and Todd J. Vision. 2013. “Data Reuse and the Open Data Citation Advantage.” PeerJ 1 (October): e175. https://doi.org/10.7717/peerj.175.

Silberzahn, Raphael, and Eric L. Uhlmann. 2015. “Crowdsourced Research: Many Hands Make Tight Work.” Nature 526 (7572): 189–91. https://doi.org/10.1038/526189a.

Woelfle, Michael, Piero Olliaro, and Matthew H. Todd. 2011. “Open Science Is a Research Accelerator.” Nature Chemistry 3 (10): 745–48. https://doi.org/10.1038/nchem.1149.