The files in this supplement represent the code and data associated with Forecasting Forecaster Accuracy: Contributions of Past Performance and Individual Differences.
In total there are six datafiles: three for each HFC season, which are
These represent processed and cleaned data. The raw data can be found at https://dataverse.harvard.edu/dataverse/hfc-sage
The codebooks.xlsx file contains a table of contents tab, as well as one tab for each .csv datafile describing each variable. NOTE: the variable naming conventions were handled by different entities across the two seasons and as a result are not totally identical (mostly in the user info files).
There are also several files containing R and Stan code which was used to analyze the data. The results reported in the paper were analyzed on privately shared versions of the data prior to open access publication of the files. As a result there may be minor discrepancies between the results reported here and those based on running the public data through this code. Many analyses were re-tested on the public data to ensure consistency and discrepancies in the results were on the order of rounding errors. What follows is a brief description of each script and how they are to be used to reproduce the results reported in the paper
This script is based on data from both seasons. It subsets all questions which were at least 12 weeks in length, and generates a plot (using ggplot2) showing how normalized accuracy changes over time.
This is the Stan file used to fit IRT accuracy models. It is interfaced via RStan in Season 1 Stan Models.R and Season 2 Stan Models.R.
In Season 1, we used the intake scales to generate measurement models and obtain factor scores for intelligence and cognitive style. This script takes the individual difference data from participants who did NOT forecast to generate measurement model parameters, then uses those parameters to estimate factor scores for the participants who did forecast.
Because these models are so time consuming, they were estimated in a separate file on a remote server. They take approximately 15-90 minutes fore EACH model, depending on the amount of data. The file IRT accuracy.Stan should either be in your working directory to run this script, or itβs path should be referenced in the stan() function (more details on how the code works are included in annotations in both the .R and .Stan files).
The estimates for the IRT models were output to several .csv files, which are read back in using this file, with the relevant output isolated and merged. Note that because the result is first output to a .csv this works slightly differently than it would if it were done straight away based on the R object produced by the stan() function in the Season 1 Stan Models.R script (we did it this way because the previous script was run remotely while this on was not). This script also contains the plotly code used to produce the 3D figures in Figure 2, which can be run separately.
This file performs the remaining analyses. There are annotations indicating where/how which tables and figures are generated. Figures were generated using ggplot2. Tables were exported to .csv files for further formatting. All the previous Season 1 scripts should be run before this one so measurement model and IRT results can be loaded in separately.
These include repeated versions of the last three Season 1 scripts: Season 2 Stan Models.R, Merge IRT Results Season 2.R, and Season 2 Analysis Script.R. They function the same way, and again the IRT results should be obtained before running the analysis script so IRT results can be loaded in.