id,study,doi_or_url,country,sample_type,sample_desc,n,data_year,pub_year,year_used,language,items,mode,incentive,bb_correct_pct,bb_intuitive_pct,all3_pct,zero_pct,mean_score,where_in_source,read_from,quote,notes
frederick2005-boston-fireworks,Frederick 2005,10.1257/089533005775196732,United States,general_convenience,"People picnicking on the Charles River bank before the July 4th fireworks (ages 15-63, mean 24; many were students in the Boston/Cambridge area)",195,,2005,2004,English,original_3,,False,,,26.0,24.0,1.53,"Table 1, p. 29 (columns: mean CRT score; percentage scoring 0, 1, 2 or 3; N)",https://pdf.stafforini.com/frederick-2005-cognitive-reflection-and.pdf (copy of the Journal of Economic Perspectives 19(4):25-42 article),"CRT Scores, by Location | Boston fireworks display a 1.53 24% 24% 26% 26% 195","Table 1 gives mean and % scoring 0/1/2/3 per location; no item-level (bat-and-ball only) percentages are given by site. Part of 3,428 respondents in 35 studies collected over 26 months from January 2003; the paper gives no per-site collection date, so data_year is null. The 'Overall' row (N = 3428) is the sum of these 11 rows and is not entered separately. Most respondents were paid $8 to complete a 45-minute questionnaire (a flat fee for taking part, not stated to depend on correct CRT answers). Paid only a frozen ice cream bar. Completed in small groups of friends/family; paper notes say some may have discussed answers, which could raise scores."
frederick2005-bowling-green,Frederick 2005,10.1257/089533005775196732,United States,students,Bowling Green State University students (table says 'Bowling Green University'),52,,2005,2004,English,original_3,,False,,,12.0,50.0,0.87,"Table 1, p. 29 (columns: mean CRT score; percentage scoring 0, 1, 2 or 3; N)",https://pdf.stafforini.com/frederick-2005-cognitive-reflection-and.pdf (copy of the Journal of Economic Perspectives 19(4):25-42 article),"CRT Scores, by Location | Bowling Green University 0.87 50% 25% 13% 12% 52","Table 1 gives mean and % scoring 0/1/2/3 per location; no item-level (bat-and-ball only) percentages are given by site. Part of 3,428 respondents in 35 studies collected over 26 months from January 2003; the paper gives no per-site collection date, so data_year is null. The 'Overall' row (N = 3428) is the sum of these 11 rows and is not entered separately. Most respondents were paid $8 to complete a 45-minute questionnaire (a flat fee for taking part, not stated to depend on correct CRT answers)."
frederick2005-cmu,Frederick 2005,10.1257/089533005775196732,United States,students,Carnegie Mellon University students,746,,2005,2004,English,original_3,,False,,,25.0,25.0,1.51,"Table 1, p. 29 (columns: mean CRT score; percentage scoring 0, 1, 2 or 3; N)",https://pdf.stafforini.com/frederick-2005-cognitive-reflection-and.pdf (copy of the Journal of Economic Perspectives 19(4):25-42 article),"CRT Scores, by Location | Carnegie Mellon University 1.51 25% 25% 25% 25% 746","Table 1 gives mean and % scoring 0/1/2/3 per location; no item-level (bat-and-ball only) percentages are given by site. Part of 3,428 respondents in 35 studies collected over 26 months from January 2003; the paper gives no per-site collection date, so data_year is null. The 'Overall' row (N = 3428) is the sum of these 11 rows and is not entered separately. Most respondents were paid $8 to complete a 45-minute questionnaire (a flat fee for taking part, not stated to depend on correct CRT answers). Completed the survey as part of class (not paid $8)."
frederick2005-harvard,Frederick 2005,10.1257/089533005775196732,United States,students,"Harvard students, all members of a student choir group (42 of 51 women)",51,,2005,2004,English,original_3,,False,,,20.0,20.0,1.43,"Table 1, p. 29 (columns: mean CRT score; percentage scoring 0, 1, 2 or 3; N)",https://pdf.stafforini.com/frederick-2005-cognitive-reflection-and.pdf (copy of the Journal of Economic Perspectives 19(4):25-42 article),"CRT Scores, by Location | Harvard Universityb 1.43 20% 37% 24% 20% 51","Table 1 gives mean and % scoring 0/1/2/3 per location; no item-level (bat-and-ball only) percentages are given by site. Part of 3,428 respondents in 35 studies collected over 26 months from January 2003; the paper gives no per-site collection date, so data_year is null. The 'Overall' row (N = 3428) is the sum of these 11 rows and is not entered separately. Most respondents were paid $8 to complete a 45-minute questionnaire (a flat fee for taking part, not stated to depend on correct CRT answers). Note b in the table: all participants were members of a student choir, predominately female."
frederick2005-michigan-state,Frederick 2005,10.1257/089533005775196732,United States,students,Michigan State University students,118,,2005,2004,English,original_3,,False,,,6.0,49.0,0.79,"Table 1, p. 29 (columns: mean CRT score; percentage scoring 0, 1, 2 or 3; N)",https://pdf.stafforini.com/frederick-2005-cognitive-reflection-and.pdf (copy of the Journal of Economic Perspectives 19(4):25-42 article),"CRT Scores, by Location | Michigan State University 0.79 49% 29% 16% 6% 118","Table 1 gives mean and % scoring 0/1/2/3 per location; no item-level (bat-and-ball only) percentages are given by site. Part of 3,428 respondents in 35 studies collected over 26 months from January 2003; the paper gives no per-site collection date, so data_year is null. The 'Overall' row (N = 3428) is the sum of these 11 rows and is not entered separately. Most respondents were paid $8 to complete a 45-minute questionnaire (a flat fee for taking part, not stated to depend on correct CRT answers)."
frederick2005-mit,Frederick 2005,10.1257/089533005775196732,United States,students,MIT students (Frederick's lab/classroom studies),61,,2005,2004,English,original_3,,False,,,48.0,7.0,2.18,"Table 1, p. 29 (columns: mean CRT score; percentage scoring 0, 1, 2 or 3; N)",https://pdf.stafforini.com/frederick-2005-cognitive-reflection-and.pdf (copy of the Journal of Economic Perspectives 19(4):25-42 article),"CRT Scores, by Location | Massachusetts Institute of Technology 2.18 7% 16% 30% 48% 61","Table 1 gives mean and % scoring 0/1/2/3 per location; no item-level (bat-and-ball only) percentages are given by site. Part of 3,428 respondents in 35 studies collected over 26 months from January 2003; the paper gives no per-site collection date, so data_year is null. The 'Overall' row (N = 3428) is the sum of these 11 rows and is not entered separately. Most respondents were paid $8 to complete a 45-minute questionnaire (a flat fee for taking part, not stated to depend on correct CRT answers)."
frederick2005-princeton,Frederick 2005,10.1257/089533005775196732,United States,students,Princeton University students,121,,2005,2004,English,original_3,,False,,,26.0,18.0,1.63,"Table 1, p. 29 (columns: mean CRT score; percentage scoring 0, 1, 2 or 3; N)",https://pdf.stafforini.com/frederick-2005-cognitive-reflection-and.pdf (copy of the Journal of Economic Perspectives 19(4):25-42 article),"CRT Scores, by Location | Princeton University 1.63 18% 27% 28% 26% 121","Table 1 gives mean and % scoring 0/1/2/3 per location; no item-level (bat-and-ball only) percentages are given by site. Part of 3,428 respondents in 35 studies collected over 26 months from January 2003; the paper gives no per-site collection date, so data_year is null. The 'Overall' row (N = 3428) is the sum of these 11 rows and is not entered separately. Most respondents were paid $8 to complete a 45-minute questionnaire (a flat fee for taking part, not stated to depend on correct CRT answers)."
frederick2005-toledo,Frederick 2005,10.1257/089533005775196732,United States,students,University of Toledo students,138,,2005,2004,English,original_3,,False,,,5.0,64.0,0.57,"Table 1, p. 29 (columns: mean CRT score; percentage scoring 0, 1, 2 or 3; N)",https://pdf.stafforini.com/frederick-2005-cognitive-reflection-and.pdf (copy of the Journal of Economic Perspectives 19(4):25-42 article),"CRT Scores, by Location | University of Toledo 0.57 64% 21% 10% 5% 138","Table 1 gives mean and % scoring 0/1/2/3 per location; no item-level (bat-and-ball only) percentages are given by site. Part of 3,428 respondents in 35 studies collected over 26 months from January 2003; the paper gives no per-site collection date, so data_year is null. The 'Overall' row (N = 3428) is the sum of these 11 rows and is not entered separately. Most respondents were paid $8 to complete a 45-minute questionnaire (a flat fee for taking part, not stated to depend on correct CRT answers)."
frederick2005-umich-annarbor,Frederick 2005,10.1257/089533005775196732,United States,students,"University of Michigan, Ann Arbor students",1267,,2005,2004,English,original_3,,False,,,14.0,31.0,1.18,"Table 1, p. 29 (columns: mean CRT score; percentage scoring 0, 1, 2 or 3; N)",https://pdf.stafforini.com/frederick-2005-cognitive-reflection-and.pdf (copy of the Journal of Economic Perspectives 19(4):25-42 article),"CRT Scores, by Location | University of Michigan: Ann Arbor 1.18 31% 33% 23% 14% 1267","Table 1 gives mean and % scoring 0/1/2/3 per location; no item-level (bat-and-ball only) percentages are given by site. Part of 3,428 respondents in 35 studies collected over 26 months from January 2003; the paper gives no per-site collection date, so data_year is null. The 'Overall' row (N = 3428) is the sum of these 11 rows and is not entered separately. Most respondents were paid $8 to complete a 45-minute questionnaire (a flat fee for taking part, not stated to depend on correct CRT answers)."
frederick2005-umich-dearborn,Frederick 2005,10.1257/089533005775196732,United States,students,"University of Michigan, Dearborn students",154,,2005,2004,English,original_3,,False,,,6.0,51.0,0.83,"Table 1, p. 29 (columns: mean CRT score; percentage scoring 0, 1, 2 or 3; N)",https://pdf.stafforini.com/frederick-2005-cognitive-reflection-and.pdf (copy of the Journal of Economic Perspectives 19(4):25-42 article),"CRT Scores, by Location | University of Michigan: Dearborn 0.83 51% 22% 21% 6% 154","Table 1 gives mean and % scoring 0/1/2/3 per location; no item-level (bat-and-ball only) percentages are given by site. Part of 3,428 respondents in 35 studies collected over 26 months from January 2003; the paper gives no per-site collection date, so data_year is null. The 'Overall' row (N = 3428) is the sum of these 11 rows and is not entered separately. Most respondents were paid $8 to complete a 45-minute questionnaire (a flat fee for taking part, not stated to depend on correct CRT answers)."
frederick2005-web,Frederick 2005,10.1257/089533005775196732,United States,online_panel,Participants in two online studies: college students and others whose e-mail addresses were obtained from online retailers,525,,2005,2004,English,original_3,online,False,,,13.0,39.0,1.1,"Table 1, p. 29 (columns: mean CRT score; percentage scoring 0, 1, 2 or 3; N)",https://pdf.stafforini.com/frederick-2005-cognitive-reflection-and.pdf (copy of the Journal of Economic Perspectives 19(4):25-42 article),"CRT Scores, by Location | Web-based studiesc 1.10 39% 25% 22% 13% 525","Table 1 gives mean and % scoring 0/1/2/3 per location; no item-level (bat-and-ball only) percentages are given by site. Part of 3,428 respondents in 35 studies collected over 26 months from January 2003; the paper gives no per-site collection date, so data_year is null. The 'Overall' row (N = 3428) is the sum of these 11 rows and is not entered separately. Most respondents were paid $8 to complete a 45-minute questionnaire (a flat fee for taking part, not stated to depend on correct CRT answers). Unpaid, entered into a lottery for iPods and other prizes."
oechssler2009-germany-online,"Oechssler, Roider & Schmitz 2009 (J Economic Behavior & Organization)",10.1016/j.jebo.2009.04.018,Germany,students,"564 participants of a web-based experiment who answered the questionnaire; recruited by e-mail through the experimental-economics labs of Bonn, Cologne and Mannheim (90% university students, 46% female, mean age 24)",564,,2009,2008,German,original_3,online,True,54.8,40.2,41.5,10.1,2.05,"Section 2 Experimental design (2.1 Cognitive reflection test, sample description); Table 1 (distributions of answers: Correct / Impulsive / Other)",https://epub.uni-regensburg.de/21701/2/roder2.pdf (Regensburg repository copy of the published article),"In total, 1250 subjects participated in our online, web-based experiment. // A subset of 564 subjects also answered a questionnaire with several decision problems related to the well-known behavioral biases discussed above // Bat and ball 54.8% 40.2% 5.0% // The average CRT score (i.e., the average number of correct responses) in our sample was 2.05 // Of our subjects, 41.5% answered all three questions correctly, 30.7% answered two questions correctly, 17.7% answered one question correctly, and the remaining 10.1% answered none of the questions correctly. // Of the participants, 90% were university students, 25% studied economics or business, and 46% were female.","n = 564 is the subset that answered the questionnaire containing the CRT (1,250 played the web experiment). Table 1 row for the bat-and-ball item: correct 54.8%, impulsive 40.2%, other 5.0%. The CRT was worded in cents (110 cents / 100 cents) and given in German (translation of the instructions in the paper's Appendix A). Incentive: 6 of the 600 participants were drawn at random and paid EUR 5 for each correct answer to questions 1, 4, 6, 7 and 9 (question 1 is the bat-and-ball item), so payment was probabilistic. Data year not stated (paper received May 2008)."
campitelli2010-buenosaires,Campitelli & Labollita 2010 (Judgment and Decision Making),10.1017/S1930297500001066,Argentina,general_convenience,"157 volunteers from the Buenos Aires metropolitan area (mean age 24.4, range 16-43, 110 female) who filled in a paper booklet",157,,2010,2009,not stated,original_3,paper,,,,6.0,59.0,0.656,Section 2.1 Participants; Section 3 Results (first paragraph),https://www.cambridge.org/core/services/aop-cambridge-core/content/view/2EA5389B4D2E8840B485E55BD3ED0B47/S1930297500001066a.pdf/div-class-title-correlations-of-cognitive-reflection-with-judgments-and-choices-div.pdf (open-access PDF),"One hundred and fifty seven volunteers from Buenos Aires metropolitan area participated in the study. | The average number of correct CRT answers was 0.656 (S.D.= 0.9). 6% of the participants solved the three problems correctly, 11% of the participants solve 2 out of 3 problems correctly, 24% of the participants solve 1 problem correctly, and 59% of the participants solve no problems correctly.",Booklet = paper-and-pencil (the participants 'filled in a 4-section booklet'). No bat-and-ball item breakdown printed. Percentages rounded to whole numbers by the authors. Language of the booklet is not stated in the paper.
koehler2010-campus-canada-e1,"Koehler & James 2010 (Memory & Cognition, Experiment 1)",10.3758/MC.38.6.667,Canada,students,"84 students (49 male, 29 female, 6 not recorded) recruited from a campus student center and paid up to Can$10 in a probability-learning game, Experiment 1",84,,2010,2009,English,original_3,,False,,,27.4,21.4,1.67,"Experiment 1, Method (Participants); Results; Figure 3 caption (numbers of participants scoring 0, 1, 2 and 3 on the CRT)",https://zenodo.org/api/records/1067017 (author-deposited PDF of the article),"Eighty-four students (49 male, 29 female, 6 not recorded) were recruited from a campus student center and were paid up to Can$10 for their participation, depending on the accuracy of their performance. // numbers of participants scoring 0, 1, 2, and 3 on the Crt were 18, 15, 28, and 23, respectively.","Computed from the counts in the Figure 3 caption: 18/15/28/23 of 84; zero = 18/84 = 21.4%, three correct = 23/84 = 27.4%, mean = (15 + 2x28 + 3x23)/84 = 140/84 = 1.67. The payment (up to Can$10) depended on the accuracy of the choice task, not on CRT answers. Country follows the Canadian-dollar payment; the university is not named in the text read. Data year not stated; CRT mode not described."
deneys2011-leuven,"De Neys, Novitskiy, Geeraerts, Ramautar & Wagemans 2011 (PLoS ONE)",10.1371/journal.pone.0027107,Belgium,students,403 psychology undergraduates at the University of Leuven (screening sample of an ultimatum-game study),403,,2011,2010,Dutch,original_3,,,,,,,1.12,"Ultimatum Screening, Participants; Materials (Cognitive Reflection Test); Results (Ultimatum Screening)","https://www.ebi.ac.uk/europepmc/webservices/rest/PMC3212542/fullTextXML (Europe PMC open-access full text, PMC article page https://pmc.ncbi.nlm.nih.gov/articles/PMC3212542/)","A total of 403 psychology undergraduates participated in return for course credit. // The brief test was translated in Dutch // The average CRT score in the sample was 1.12 (out of 3, SE = .05).",Course credit for taking part; ten ultimatum participants were paid by game earnings (not by CRT answers). The CRT came after the ultimatum game and some unrelated tasks. Data year not stated. Mode not stated.
iyer2012-yourmorals-us,"Iyer, Koleva, Graham, Ditto & Haidt 2012 (PLoS ONE)",10.1371/journal.pone.0042366,United States,general_convenience,"9,721 visitors to the YourMorals.org research website who completed the CRT (7,384 self-identified liberals, 1,267 conservatives, 1,070 libertarians; 4,971 men), raised in the US until at least age 14",9721,,2012,2011,English,original_3,online,,,,,,1.75,"Methods (Participants and Sampling Considerations); Cognitive Reflection Task section; Table 3 (means: libertarians, liberals, conservatives)",https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0042366&type=printable (open-access PDF),"The measure was completed by 9,721 participants (4,971 men; 7,384 liberals, 1,267 conservatives, and 1,070 libertarians). // Cognitive Reflection Task 2.06 1.73 1.57 // Only participants who were raised in the United States until at least the age of 14 were included in these analyses. // Participants usually find YourMorals.org through publicity about psychological research or by typing keywords related to morality into a search engine.","Table 3 prints CRT means only by political group: libertarians 2.06, liberals 1.73, conservatives 1.57. mean_score 1.75 is the computed pooled mean: (1,070 x 2.06 + 7,384 x 1.73 + 1,267 x 1.57)/9,721 = 1.745 (rounded 1.75; group means rounded to 2 decimals so +/-0.005). The sample is self-selected volunteers (the authors say it is not representative of the general population) and excludes people who chose another ideology label. Data collected June 2007 to January 2011 (for the site as a whole); no exact CRT collection year. No payment mentioned."
liberali2012-brazil,"Liberali, Reyna, Furlan, Stein & Pardo 2012 (J Behavioral Decision Making, Study 1)",10.1002/bdm.752,Brazil,students,"259 undergraduates (management, engineering, accounting) at three Brazilian universities; snowball convenience sample (mean age 24.0)",259,,2012,2011,not stated,original_3,,,30.9,64.1,19.0,44.0,1.1,"Study 1 Participants; Results (CRT paragraph below Table 2); Table 6 (percentage of responses for the CRT, Study 1 column N = 259, item 1)",https://pmc.ncbi.nlm.nih.gov/articles/PMC3716015/ (author manuscript in PubMed Central),"Participants were 259 undergraduate students (mean age= 24.04years) from three different courses | It was a convenience sample, gathered through a snowball technique applied to professors and lecturers. | the percentages of the participants who gave zero, one, two, or three correct responses, respectively, were 44%, 20%, 17%, and 19%; the mean number of correct responses on the CRT was 1.10. | Correct response (5) | 30.9 | 47.9 | Mindless answer (10) | 64.1 | 38.4","Table 6 gives % per response for the bat-and-ball item: correct (5) 30.9, mindless answer (10) 64.1, other errors 2.7, no response 2.3 (Study 1). Language and mode of administration not stated in the text read. Incentive not stated."
liberali2012-cornell,"Liberali, Reyna, Furlan, Stein & Pardo 2012 (J Behavioral Decision Making, Study 2)",10.1002/bdm.752,United States,students,190 Cornell University students recruited in psychology courses and via campus postings (mean age 21.1),190,,2012,2011,English,original_3,online,False,47.9,38.4,26.0,24.0,1.5,"Study 2 Participants and Procedure; Results (CRT paragraph); Table 6 (Study 2 column N = 190, item 1)",https://pmc.ncbi.nlm.nih.gov/articles/PMC3716015/ (author manuscript in PubMed Central),"The participants were 190 Cornell University students (116 women, 74 men) | The respondents participated in the experiment online (and received credit toward course requirements). | the percentages of the participants who gave zero, one, two, or three correct responses, respectively, were 24%, 27%, 23%, and 26%; the mean number of correct responses to the CRT was 1.50. | Correct response (5) | 30.9 | 47.9 | Mindless answer (10) | 64.1 | 38.4","Table 6, Study 2 column, bat-and-ball item: correct (5) 47.9, mindless answer (10) 38.4, other errors 10.0, no response 3.7. Course credit, not paid per correct answer."
nieuwenstein2012-groningen-e1,"Nieuwenstein & van Rijn 2012 (Judgment and Decision Making, Experiment 1)",10.1017/S1930297500003338,Netherlands,students,"University of Groningen undergraduate Psychology students (8 male, 40 female, mean age 20.1), Experiment 1",48,,2012,2011,not stated,original_3,,False,,,,,1.27,"Section 2.1 Method (Subjects); Section 2.2 Results, last paragraph (CRT)",https://www.cambridge.org/core/services/aop-cambridge-core/content/view/876C6081F70DCDBABAF789080FBEFE2C/S1930297500003338a.pdf/div-class-title-the-unconscious-thought-advantage-further-replication-failures-from-a-search-for-confirmatory-evidence-div.pdf (open-access PDF),"Forty-eight undergraduate Psychology students (8 male, 40 female, mean age: 20.1, SD = 1.7) of the University of Groningen | The results obtained with the CRT showed that subjects on average answered 1.27 items correctly.",Students took part for course credit or EUR 4. The CRT was a side measure after an unconscious-thought task; only the mean is printed. The CRT sentence sits in the Experiment 1 results section (n = 48 per the t(46) tests there). Language of the test not stated. Data year not stated.
baldi2013-milan,"Baldi, Iannello, Riva & Antonietti 2013 (Studia Psychologica)",10.21909/sp.2013.04.641,Italy,students,"168 undergraduates (84 men, mean age 22.3) of Arts, Humanities, Sociology and Philosophy courses (no Economics or Psychology students), tested on a university campus (Catholic University of the Sacred Heart, Milan)",168,,2013,2012,Italian,original_3,paper,,,,,,1.29,"Method (Participants, Procedure); Table 2 (CRT means and SDs; CRT-total 1.29, SD 1.15)",https://doi.org/10.21909/sp.2013.04.641 (open-access PDF; saved as n_studia.pdf),"168 undergraduate students (84 men, M age = 22.3; SD = 2.1) participated in the study. // Students who agreed were randomly given one of two booklets with the decision-making task and then the CRT. // CRT-total 1.29 1.15 // The Italian version of the CRT appeared to be adequate for the purposes of this study.","Table 2 also gives the item means (CRT1 0.34, CRT2 0.46, CRT3 0.49 correct); the paper does not say which item is the bat-and-ball one, so bb_correct_pct is left null. Booklet (paper) administration inferred from 'booklets'. The decision-task chi-square tables use N = 84 and N = 83, i.e. one participant fewer (167) in one group. Payment not mentioned. Data year not stated."
fehr2013-wzb-berlin,Fehr & Huck 2013/2016 (WZB discussion paper SP II 2013-306r),10.1007/s10683-015-9461-0,Germany,students,"Subjects of the WZB-TU laboratory in Berlin, recruited through ORSEE (mostly students), who also played a beauty-contest game",240,,2013,2012,German,original_3,,True,32.0,60.0,,,1.49,"Section 2 (Setup); Table 1 'Distribution of answers in the CRT' (correct 32%, intuitive 60%, other 8%); footnote 10 (average score)",https://discovery.ucl.ac.uk/1573569/1/ii13-306r.pdf (author version of the working paper),We have data from 240 subjects playing BCGs | which were conducted at the WZB TU laboratory in Berlin between May 2010 and June 2013 | Bat and Ball (BBQ) 32% 60% 8% 65% | The average score in the 3 item CRT is 1.49 | Subjects were rewarded with 1 Euro for each correct answer in the CRT,"Table 1 columns: correct / intuitive / other answers / expected correct (65% = what subjects thought others would get right). Data collected May 2010 to June 2013 (spread over years, so data_year left null). n = 240 is the number of subjects in the study; the paper does not give a separate CRT response count. Instructions were translated from German; amounts in the CRT given in Euro (1.10 Euro / 1.00 Euro)."
kahan2013-yougov-us,Kahan 2013 (Judgment and Decision Making),10.1017/S1930297500005271,United States,national_representative,"1,750 US adults recruited online by YouGov, stratified to reflect US demographics (54% female, mean age 52)",1750,,2013,2012,English,original_3,online,,13.0,,,,0.65,"Section 3.1 Sample; Section 3.2 (CRT mean); Table 1 (item % correct: bat and ball 13%, widgets 26%, lilypad 25%)",https://www.cambridge.org/core/services/aop-cambridge-core/content/view/F8A6A74C9022363D672B0FD14DD8B89F/S1930297500005271a.pdf/div-class-title-ideology-motivated-reasoning-and-cognitive-reflection-div.pdf (open-access PDF),Subjects for the study consisted of a nationally diverse sample of 1750 U.S. adults. | The mean score for subjects in this study was 0.65 (SD = 0.95). | BA TBALL A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? 13%,"Table 1 prints '% correct' per item: widgets 26%, bat and ball 13%, lilypad 25%. Sample is stratified YouGov online panel, 'representative of the United States general population' per the paper. Data year not stated in the paper text read."
stupple2013-derby-e1,"Stupple, Gale & Richmond 2013 (Proceedings of the Cognitive Science Society, Experiment 1)",https://escholarship.org/uc/item/36989187,United Kingdom,students,"65 undergraduates of the University of Derby (aged 18-45), recruited by opportunity sampling; none had met the CRT before; pen-and-paper CRT",65,,2013,2012,English,original_3,paper,False,,,,,1.32,Experiment 1 Participants; General method (CRT pen and paper); Results Experiment 1 (first paragraph),https://escholarship.org/content/qt36989187/qt36989187.pdf (open-access PDF),"Sixty-five undergraduates from the University of Derby , aged 18-45, were recruited via opportunity sampling. // Each received a voucher (value £5) for participating. // The CRT was a pen and paper task. // The Mean CRT score for the sample in Experiment 2 was 1.32 (SD= 1.11)","Label problem in the source: the sentence 'The Mean CRT score for the sample in Experiment 2 was 1.32 (SD= 1.11)' sits under the heading 'Results Experiment 1' (the real Experiment 2 result, 1.12, is printed under 'Results Experiment 2'); it is entered here as Experiment 1 (65 students). GBP 5 voucher for taking part (not per correct answer). Data year not stated."
stupple2013-derby-e2,"Stupple, Gale & Richmond 2013 (Proceedings of the Cognitive Science Society, Experiment 2)",https://escholarship.org/uc/item/36989187,United Kingdom,students,"49 undergraduates of the University of Derby (aged 18-45), recruited by opportunity sampling; none had met the CRT before; pen-and-paper CRT",49,,2013,2012,English,original_3,paper,False,,,,,1.12,Experiment 2 Participants; Results Experiment 2 (first paragraph),https://escholarship.org/content/qt36989187/qt36989187.pdf (open-access PDF),Forty-nine undergraduates from the University of Derby aged 18-45 were recruited via opportunity sampling. // The CRT was a pen and paper task. // The Mean CRT score for the sample in Experiment 2 was 1.12 (SD= 1.14),GBP 5 voucher for taking part (not per correct answer). The paper notes the mean is below the 1.24 average reported by Frederick (2005). Data year not stated.
weller2013-us-merged,"Weller, Dieckmann, Tusler, Mertz, Burns & Peters 2013 (J Behavioral Decision Making, Study 1)",10.1002/bdm.1751,United States,other,"Three samples merged: 302 community members (online and newspaper adverts), 163 introductory-psychology undergraduates, and 1,505 adults of the RAND American Life Panel (online); median age 48",1970,,2013,2012,English,original_3,,,,,,,0.83,"Study 1 Method (Participants); Table 4 (descriptive statistics for numeracy measures: CRT mean 0.83, SD .99, median 0, mode 0)",https://pmc.ncbi.nlm.nih.gov/articles/PMC7161838/ (full text in PubMed Central),Participants were 1970 subjects collected from three separate samples. // The first sample consisted of 302 community members // The second sample consisted of 163 undergraduates in an introductory psychology class. // the third sample was an online study of adults using the American Life Panel ( n = 1505). // CRT (three items) | 0.83 (.99) | 0 | 0 | .88,"The paper merges the three samples and reports CRT only for the merged dataset (Table 4: mean 0.83, SD .99, median 0, mode 0). Table 5 gives the 0/1/2/3 distribution only by education (high school/trade: 64.1% scored 0 and 4.2% scored 3; some college: 55.8% and 5.8%; college grad: 36.1% and 15.3%), so zero_pct and all3_pct are left null. Community members and the panel were paid; students got course credit. Country is taken from the American Life Panel (a US panel); the locations of the community and student samples are not stated in the text read. Data year not stated."
albaity2014-malaysia,"Albaity, Rahman & Shahidul 2014 (Judgment and Decision Making)",10.1017/S1930297500005507,Malaysia,students,"Undergraduate and postgraduate students at two public universities in Malaysia (58.3% female; 61.7% Malay, 30.9% Chinese); 880 usable of 1,000 questionnaires",880,,2014,2013,English,original_3,,,30.2,,,,0.88,Section 2 Data and methodology; Table 2 (mean scores for CRT questions); Section 3 Results (text under Table 2),https://www.cambridge.org/core/services/aop-cambridge-core/content/view/77A9F51C7824F8795C46A24BD07D679C/S1930297500005507a.pdf/div-class-title-cognitive-reflection-test-and-behavioral-biases-in-malaysia-div.pdf (open-access PDF),"The sample consists of undergraduate as well as postgraduate students from two public universities in Malaysia. | The total number of questionnaire distributed was 1000; 898 questionnaires were returned, and 880 were usable for the analysis. | Bat & ball 30.2 69.8 0.30 | The sum of total mean correct answer was 0.88","Table 2 gives % correct 30.2 and % wrong 69.8 for bat and ball (the paper does not split wrong answers into intuitive vs other). Questionnaire in English (medium of instruction). The paper says the CRT questions were 'adapted from Frederick (2005) and Oechssler et al. (2009)'; the appendix with the exact wording was not in the text read, so items = original_3 is the paper's label (Bat & ball, Widgets, Lily pads). Data year not stated."
boschdomenech2014-granada,"Bosch-Domenech, Branas-Garza & Espin 2014 (Psychoneuroendocrinology)",10.1016/j.psyneuen.2014.01.023,Spain,students,"First-year business and economics students, University of Granada (623 ethnically homogeneous subjects, 260 male)",623,2011,2014,2011,Spanish,original_3,paper,,,,7.9,53.9,0.74,"Methods (sample, paper-and-pencil CRT); Table 2 (distribution of correct answers by sex); text under Table 2 (means by sex)",https://bw.bse.eu/wp-content/uploads/2015/09/698-file.pdf (author copy of the accepted paper),"In October 2011, 927 firstyear students at the College of Business and Econ omics of the University of Granada (Spain) were asked to participate | The resulting sample was composed of 623 ethnically homogeneous subjects | answered the CRT's three questions using paper and pencil | 0 correct answers 43.46 61.43 | 3 correct answers 11.54 5.23 | The mean (±sem) number of correct responses in the CRT was 0.958 ± 0.064 for males and 0.584 ± 0.045 for females","The paper reports results only by sex (260 men, so 363 women). Pooled values are computed from the printed by-sex percentages applied to those counts, and each divides out to a whole number of people: zero correct = 113 men + 223 women = 336/623 = 53.9%; all three = 30 + 19 = 49/623 = 7.9%; mean = (249 + 212)/623 = 0.74, matching the printed sex means 0.958 and 0.584. Item-level columns (CRT item 1/2/3) are not labelled as bat-and-ball in the text, so bb_correct_pct is left null. Incentive for CRT answers not stated. Spanish language of the sample is an inference from the setting (University of Granada); the paper does not name the language."
pennycook2014-waterloo-e2,"Pennycook et al. 2014 (Memory & Cognition, Experiment 2)",10.3758/s13421-013-0340-7,Canada,students,"University of Waterloo undergraduates (136 female, 64 male; mean age 19.9), Experiment 2",200,,2014,2013,English,original_3,,False,,,,,1.08,"Experiment 2 Participants paragraph; Table 1 (descriptive statistics, CRT row: mean proportion correct .36, SD .37)",https://zenodo.org/api/records/1067036/files/article.pdf/content (author-deposited PDF of the article),"A group of 200 University of Waterloo undergraduate students (136 females, 64 males; average age = 19.9 years) participated in a session lasting approximately 30 min. | CRT .36 .37 | Performance for CRT and WordSum represents proportions of correct responses","The paper prints the CRT as the mean PROPORTION of correct responses (.36), not a 0-3 count. mean_score 1.08 = .36 x 3; because .36 is printed to two decimals the true mean lies between 1.07 and 1.10. Participants received course credit. Only Experiment 2 contained the CRT. Data year not stated."
royzman2014-mturk-us,"Royzman, Landy & Goodwin 2014 (Judgment and Decision Making, Study 1)",10.1017/S1930297500005738,United States,online_panel,"US-based Amazon Mechanical Turk workers, Study 1 (572 assigned to Study 1, paid $1.00 for a ten-minute survey)",570,,2014,2013,English,original_3,online,False,,,25.3,36.0,1.34,"Study 1 Method (Participants); Table 2 (CRT score frequency); Table 1a (descriptive statistics, CRT mean)",https://www.cambridge.org/core/services/aop-cambridge-core/content/view/E2DA43EAC074793C7E0062ABC9F7C05D/S1930297500005738a.pdf/div-class-title-are-good-reasoners-more-incest-friendly-trait-cognitive-reflection-predicts-selective-moralization-in-a-sample-of-american-adults-div.pdf (open-access PDF),"Seven hundred and twenty participants located in the United States were recruited, paid, and debriefed through Amazon's Mechanical Turk website. | CRT Score Frequency Percentage 0 206 36.0 1 108 18.9 2 111 19.4 3 145 25.3 No response 2 0.3 Total 527 100.0 | Mean 1.34 21.43 5.77 31.77 93.08 57.31 82.33 76.48","Table 2 prints 'Total 527' but its rows sum to 570 (206+108+111+145) plus 2 non-responders = 572, which matches the 572 participants assigned to Study 1 and the printed percentages (206/572 = 36.0%). So 527 looks like a typo for 572; n = 570 answered. Mean 1.34 is the Table 1a value and checks against the counts: 765/570 = 1.34. Flat $1.00 payment, not tied to correct answers."
browne2015-queensland,"Browne, Thomson, Rockloff & Pennycook 2015 (PLoS ONE)",10.1371/journal.pone.0132562,Australia,other,"1,256 adults in the Australian state of Queensland, random-digit phone sample from the 2012 Queensland Social Survey (computer-assisted telephone interview)",1256,2012,2015,2012,English,original_3,,,,,,50.2,0.45,Materials and Methods (sampling); Table 1 (CRT score > 0 / > 1); Table 2 (CRT M[SD]),https://pmc.ncbi.nlm.nih.gov/articles/PMC4556675/ (open-access full text in PubMed Central),"Data was collected in the Australian state of Queensland, as part of the 2012 Queensland Social Survey // A twostage stratified sampling strategy was used to select both households and individuals randomly in Queensland. // CRT score > 0 | 49.8% | (626/1256) // CRT score > 1 | 14.8% | (186/1256) // CRT | .45[.76]","Telephone interview (CATI), so mode is neither paper nor online. zero_pct 50.2 is computed: 100% - 49.8% scoring above 0 (626/1256 scored 1+, so 630 scored 0 = 50.16%). 14.8% (186/1256) scored 2 or 3. Mean 0.45 (SD 0.76) is the M[SD] printed for CRT in the regression table. Random sample of Queensland (state-level, not national), slightly over-representing people aged 55+. The first CRT item read was the widgets question. Payment not mentioned."
corgnet2015-western-us-millennials,"Corgnet, Hernan-Gonzalez, Kujal & Porter 2015 (PLoS ONE, 'Cognitive Reflection and the Diligent Worker')",10.1371/journal.pone.0141243,United States,students,"264 undergraduates (52% male, mean age 20.1) in a participant pool at a Western US university, all millennials (born 1981-2000)",264,,2015,2014,English,original_3,online,,,,,,1.22,Methods (Participants); Cognitive measures paragraph,https://pmc.ncbi.nlm.nih.gov/articles/PMC4636387/ (open-access full text in PubMed Central),"Participants were 264 undergraduates (52.27% male; average age 20.08) enrolled in a pool of potential participants at a Western U.S. university. // We measured cognitive reflection using the CRT developed by [ 21 ] ( M = 1.22, SD = 1.10). // The distribution of CRT scores in our sample does not significantly differ from the distribution reported by [ 21 ] which was based on a sample of 3,428 respondents","Participants were paid $15 for a one-hour survey. The article says the CRT was the survey measure; whether CRT answers were paid is not stated for the CRT itself (another incentivised task in the same survey paid a lottery ticket per correct answer). The authors report no significant difference from the Frederick (2005) distribution (chi-square(3) = 3.606, p = 0.307). Data year not stated."
finley2015-texas,"Finley, Tang & Schmeichel 2015 (PLoS ONE)",10.1371/journal.pone.0138922,United States,students,"410 undergraduates from an introductory psychology class (318 female, mean age 18.6) tested online; authors at Texas A&M University",410,,2015,2014,English,original_3,online,,,,,,0.74,Methods (Participants; Analytic task),https://pmc.ncbi.nlm.nih.gov/articles/PMC4581709/ (open-access full text in PubMed Central),"454 undergraduate students enrolled in an introductory psychology class accessed the study online. // The remaining 410 participants (318 female, 275 Caucasian, M = 18.55 years, SD = 1.02 years) // The number of correct answers formed the measure of analytic thinking ( M = 0.74, SD = 0.92).","Mean 0.74 (SD 0.92) is for all 410 participants (the CRT was given before or after a religion measure depending on random condition; the paper reports no difference in CRT by condition, t(408) = 0.69). Country is taken from the authors' affiliation (Texas A&M University), the methods text does not name it. Payment/credit for taking part not stated in the text read. Data year not stated."
graffeo2015-trento,"Graffeo, Polonio & Bonini 2015 (Frontiers in Psychology)",10.3389/fpsyg.2015.00844,Italy,students,"35 participants (14 male, 21 female, mean age 24.5) of an eye-tracking experiment at the University of Trento, after excluding 9 of 44 (3 of them because they already knew some CRT answers)",35,,2015,2014,not stated,original_3,,,,,,,1.4,Method (participants); descriptive-statistics table (Average and SD of CRT number of correct responses),"https://www.ebi.ac.uk/europepmc/webservices/rest/PMC4469827/fullTextXML (Europe PMC open-access full text, PMC article page https://pmc.ncbi.nlm.nih.gov/articles/PMC4469827/)","Forty-four persons participated in the experiment, but nine of them were excluded (five due to poor calibration of the eye-tracking system and excessive number of trial losses, one because of the unusually high mean number of fixations across problems, and three because they reported, after the end of the experiment, that they were previously aware of some of the answers of the CRT). // The mean age of the remaining 35 participants (14 males and 21 females) is 24.5 (SD = 3.47). // Cognitive Reflection Test (CRT) number of correct responses | 1.40 | 1.31","Small sample (n = 35). Mean 1.40, SD 1.31; 17 participants scored 2-3 ('high CRT') and 18 scored 0-1 ('low CRT'). Country follows the authors' affiliation (University of Trento, Italy); the participants' recruitment place is not named. Payment and administration mode not stated in the text read."
grehl2015-leipzig,Grehl & Tutic 2015 (PLoS ONE),10.1371/journal.pone.0136524,Germany,general_convenience,"269 subjects in 17 laboratory sessions run for the Institute of Sociology, Leipzig University, recruited through the online tool hroot (student status not stated)",269,,2015,2014,German,original_3,,,,,,,1.45,Section 3 (Participants and procedure); Section 4.1 Cognitive reflection test,https://pmc.ncbi.nlm.nih.gov/articles/PMC4552469/ (open-access full text in PubMed Central),"Participants were recruited via the internet recruitment tool hroot // In total 269 subjects attended, earning an average of EUR 14.55 in 17 sessions lasting about 1.5 hours. // The numbers of correctly solved CRT questions (CRT score) are relatively evenly distributed, ranging from 21 to 30% (see Fig 1 (a) ), which results in a mean score of 1.45 correct answers.","The paper prints only that each of the four score levels (0-3) holds between 21 and 30% of subjects, and the mean 1.45; the per-level shares are in Fig 1(a) only. The CRT was the first task in each session. Language 'German': the instructions were translated from German (supplement S1); the wording of the CRT itself is not shown. Whether CRT answers were paid is not stated. Data year not stated."
pennycook2015-waterloo-naive,Pennycook et al. 2015 (Behavior Research Methods),10.3758/s13428-015-0576-1,Canada,students,"University of Waterloo undergraduates who said they had not seen the CRT before (125 of 497 who had seen it, or did not answer, were excluded)",373,,2015,2014,English,original_3,online,False,30.3,64.9,12.1,41.0,1.04,"Table 2 (item responses: Bat & Ball column), Table 3 (scores 0-3 correct), Table 4 (mean CRT-Reflective)",https://zenodo.org/api/records/1067045/files/article.pdf/content (author-deposited PDF of the article),Correct 113 (30.3 %) 113 (30.3 %) 161 (43.2 %) | Intuitive incorrect 242 (64.9 %) 218 (58.4 %) 152 (40.8 %) | Correct 153 (41 %) 98 (26.3 %) 77 (20.6 %) 45 (12.1 %) | 1. CRT–Reflective 1.05 1.04,"Only the naive subsample is reported. The text says 372 participants (268 female, 104 male) but the counts in Tables 2-3 sum to 373 (153+98+77+45); n = 373 follows the tables. Participants got course credit (not paid per correct answer). Mean 1.04 is the Table 4 'Means' column for CRT-Reflective; checks as (98+2x77+3x45)/373 = 1.04. Data year not stated."
ponti2015-luiss-rome,Ponti & Rodriguez-Lara 2015 (Frontiers in Behavioral Neuroscience),10.3389/fnbeh.2015.00146,Italy,students,"96 Dictators (students of LUISS Guido Carli, Rome, recruited by ORSEE) who answered the CRT in a dictator-game experiment run in Italian",96,,2015,2014,Italian,original_3,,,,,22.9,36.5,1.28,"Materials and methods (Participants); Results, Descriptive statistics, Table 1 (CRT scores of Dictators)","https://www.ebi.ac.uk/europepmc/webservices/rest/PMC4473592/fullTextXML (Europe PMC open-access full text, PMC article page https://pmc.ncbi.nlm.nih.gov/articles/PMC4473592/)",A total of 196 students were recruited among the undergraduate population of LUISS Guido Carli in Rome using the ORSEE recruiting system // The experiment was run in Italian // CRT | 35 | 21 | 18 | 22 | 40 (41.7%) | 49 (51.0%) | 7 (7.3%) | (36.5%) | (21.9%) | (18.8%) | (22.9%),Table 1 gives counts of Dictators scoring 0/1/2/3: 35/21/18/22 (total 96 = 36.5%/21.9%/18.8%/22.9%). mean_score 1.28 is computed from these counts: (21 + 2x18 + 3x22)/96 = 123/96. Only the 96 dictators answered the CRT (the 98 in the sample description vs 96 in Table 1 are as printed). Payment for CRT answers not stated. Data year not stated.
thoma2015-uk-traders,"Thoma, White, Panigrahi, Strowger & Anderson 2015 (PLoS ONE, Study 1)",10.1371/journal.pone.0123202,United Kingdom,professionals,"44 traders from the trading desks of four UK investment banks and trading houses in London (88% male, mean age 34.3), Study 1",44,,2015,2014,English,original_3,online,,,,,,2.73,Study 1 Method (Participants); Study 1 Results (CRT paragraph),"https://www.ebi.ac.uk/europepmc/webservices/rest/PMC4395391/fullTextXML (Europe PMC open-access full text, PMC article page https://pmc.ncbi.nlm.nih.gov/articles/PMC4395391/)","A total of 44 participants (M = 34.3 years, SD = 7.6), from trading desks of four UK investment banks and trading houses (in London), were recruited for Study 1. // All participants completed an online questionnaire // most traders scored highly or very highly (M = 2.73, SD = 0.61, of a maximum score of 3)",Mean score only. The paper links the high score to a ceiling effect. Recruited by e-mail via desk managers; no payment mentioned. Data year not stated.
thoma2015-finance-workers,"Thoma, White, Panigrahi, Strowger & Anderson 2015 (PLoS ONE, Study 2, finance group)",10.1371/journal.pone.0123202,Not stated,professionals,"53 banking and financial-services workers (14 said they were traders), online survey, Study 2",53,,2015,2014,English,original_3,online,,,,,,1.62,"Study 2 Method (Participants); Table 2 (CRT score row, 'Finance workers' column)","https://www.ebi.ac.uk/europepmc/webservices/rest/PMC4395391/fullTextXML (Europe PMC open-access full text, PMC article page https://pmc.ncbi.nlm.nih.gov/articles/PMC4395391/)","A total of 110 participants took part in Study 2, with 53 (14 of whom were females) participants coming from a range of jobs in banking and financial services // CRT score | 1.47 (1.1) | 1.58 (1.1) | 1.31 (1.04) | 1.62 (1.1) | 1.33 (1.06) | .20 // Overall (N = 110) | Male (n = 55) | Female (n = 45) | Finance workers (n = 53) | Nonbanking (n = 57)","Table 2 columns: Overall (N = 110) 1.47, Male 1.58, Female 1.31, Finance workers 1.62 (SD 1.1), Nonbanking 1.33. The Study 2 text does not name the country of the respondents (Study 1 was UK traders; the authors are UK-based), so country is 'Not stated'. Role claims were not verified by the authors. Data year not stated."
thoma2015-nonbanking,"Thoma, White, Panigrahi, Strowger & Anderson 2015 (PLoS ONE, Study 2, non-finance group)",10.1371/journal.pone.0123202,Not stated,general_convenience,"57 participants with jobs unrelated to finance, opportunity sample recruited by e-mail and Facebook messages, online survey, Study 2",57,,2015,2014,English,original_3,online,,,,,,1.33,"Study 2 Method (Participants); Table 2 (CRT score row, 'Nonbanking' column)","https://www.ebi.ac.uk/europepmc/webservices/rest/PMC4395391/fullTextXML (Europe PMC open-access full text, PMC article page https://pmc.ncbi.nlm.nih.gov/articles/PMC4395391/)",another 57 participants from jobs not related to finance were recruited (using an opportunity sample based on emails and facebook messages) // CRT score | 1.47 (1.1) | 1.58 (1.1) | 1.31 (1.04) | 1.62 (1.1) | 1.33 (1.06) | .20,Table 2 prints 1.33 (SD 1.06) for the 'Nonbanking' column (n = 57); 89% of the 110 Study 2 respondents had at least a first university degree. Country is not stated for this group in the text read. Data year not stated.
barcellos2016-brazil-accountants,"Barcellos, Cardoso & Aquino 2016 (Advances in Scientific and Applied Accounting)",10.14392/asaa.2016090206,Brazil,professionals,"4,902 Brazilian professional accountants (preparers, auditors, analysts, managers, professors; mean age about 40) who voluntarily answered an online survey run by the Federal Accounting Council (CFC)",4902,,2016,2015,not stated,original_3,online,False,,,21.71,13.61,1.55,"Abstract; Section 3.1 (data collection and sample description, sample-selection table); Section 4.1 and Table 3 (row 'Total')",http://asaa.anpcont.org.br/index.php/asaa/article/download/211/164 (open-access PDF),"Our sample consists of 4,902 Brazilian accountants who voluntarily answered an online survey. // Total 1.55 0.98 13.61 39.11 25.58 21.71 4,902 // the mean CRT score of Brazilian professional accountants (1.55)","Table 3 'Total' row: mean 1.55, SD 0.98, percentage scoring 0/1/2/3 = 13.61 / 39.11 / 25.58 / 21.71, N = 4,902 (1.6% of the 302,697 registered accountants). Selection: of 9,355 CRT answers, 516 inconsistent answers were dropped (including bat-and-ball answers below 10 or above 110 cents: 228), plus those who skipped the income question or fitted no professional segment, so the sample is filtered. Data year not stated (about 2013-2015). Not paid. Language of the questionnaire not stated."
haigh2016-online-volunteers,Haigh 2016 (Advances in Cognitive Psychology),10.5709/acp-0193-5,Not stated,general_convenience,"142 online volunteers recruited through social-media research forums (Reddit, Facebook, Twitter) and participant pools (Call for Participants, Prolific Academic); mean age 28",142,,2016,2015,English,original_3,online,False,,,45.8,19.0,1.93,"Method (Design and Participants); Table 1 (percentage scoring 0, 1, 2 or 3 on the CRT, row 'Overall')",https://pmc.ncbi.nlm.nih.gov/articles/PMC5225989/ (open-access full text in PubMed Central),"Volunteers responded to online adverts seeking participants to take part in a short problem solving task. // A total of 142 volunteers completed the task (65 male, 72 female, 5 preferred not to disclose their gender; M age = 28 years). // Overall ( N = 142) | 1.93 | 19.0% | 14.8% | 20.4% | 45.8% // Participants who completed the study via Prolific Academic ( n = 44) were paid","Table 1 also splits the sample: prior exposure to any CRT problem (n = 73, 51.4%): mean 2.36, 8.2% scored 0, 61.6% scored 3; no prior exposure (n = 69): mean 1.48, 30.4% scored 0, 29.0% scored 3. The overall row is entered as the sample. Only 44 Prolific participants got GBP 0.5 (flat fee); the rest were unpaid. The country of the volunteers is not stated (the paper says only that adverts were posted online). Data year not stated."
hanaki2016-paris-leep,"Hanaki, Jacquemet, Luchini & Zylbersztejn 2016 (Frontiers in Psychology)",10.3389/fpsyg.2016.01188,France,students,"200 participants (155 students, 94 males, mean age 25.6) in sessions at the LEEP laboratory, Paris School of Economics; recruited via Orsee",200,2014,2016,2014,not stated,original_3,,False,,,6.0,70.0,0.479,Section 2 (Experimental design and procedure); Section 3 Results (cognitive skills and strategic behavior),"https://www.ebi.ac.uk/europepmc/webservices/rest/PMC4978737/fullTextXML (Europe PMC open-access full text, PMC article page https://pmc.ncbi.nlm.nih.gov/articles/PMC4978737/)","All the sessions were conducted in February and March 2014. Out of the 200 participants (94 males), 155 were students with various fields of specialization. // All sessions took place at the Laboratoire d'Economie Experimentale de Paris (LEEP) at Paris School of Economics. // We also observe that many subjects (70%) of our 200 participants fail to provide at least one correct answer in our standard CRT. 16% provide exactly one, 8% two, and 6% three correct answers. // (0.479 out of 3 with SD 0.852)","The paper writes 'The average score in Raven's test (CRT) is 8.679 out of 16 ... (0.479 out of 3 with SD 0.852)'; the figure 0.479 out of 3 is the CRT mean (it agrees with the printed 70/16/8/6% split: 0.50). Payment: a fixed five euros for completing the post-experimental tasks, not per correct CRT answer. Language of the test not stated."
juanchich2016-uk-panel,"Juanchich, Dewberry, Sirota & Narendran 2016 (J Behavioral Decision Making)",10.1002/bdm.1875,United Kingdom,online_panel,"384 British adults recruited via a marketing agency (aged 18-85, mean 44; 53% male, 82% White British), online questionnaire",384,,2016,2015,English,original_3,online,False,,,,,0.97,Method (Participants; Materials and Procedure; Predictors: Cognitive reflection),"https://eprints.bbk.ac.uk/id/eprint/15592/1/15592.pdf (Birkbeck repository, author version)",A total of 401 British participants took part in the study. The sample was recruited via a marketing agency and participants were rewarded with shopping vouchers for participating in the study. | The remaining 384 participants were aged between 18 and 85 years | Participants completed an online questionnaire measuring cognitive reflection | The mean CRT score was 0.97 (SD = 1.11).,"401 took part; 17 were excluded for speeding or very long completion times, leaving 384. Vouchers were for taking part, not per correct answer. The paper says slightly more than 50% of participants had at least one answer correct. Printed in the author version dated 2015; journal issue 2016 (JBDM 29(1))."
noori2016-iran-allameh,Noori 2016 (Judgment and Decision Making),10.1017/S1930297500007634,Iran,students,"Students of Allameh Tabataba'i University (Iran) who completed a paper questionnaire; part of 395 students from three universities (mean age 22.4, 63% male)",146,2015,2016,2015,Persian,original_3,,True,,,19.9,38.4,1.18,"Section 2 Method; Table 1 (CRT scores by location: mean, share scoring 3/2/1/0, N); Table 2 (distribution of answers, pooled)",https://www.cambridge.org/core/services/aop-cambridge-core/content/view/C70FFD62D8DA697D6FA89D8460075CF7/S1930297500007634a.pdf/div-class-title-cognitive-reflection-as-a-predictor-of-susceptibility-to-behavioral-anomalies-div.pdf (open-access PDF),"Four hundred and ten university students from three universities (Allameh Tabataba'i University, Amirkabir University of Technology, and Sharif University of Technology) participated in the research and completed a questionnaire (15 students did not complete the questionnaire, so the results are based on 395 subjects). // The data were collected during February to June, 2015. // Subjects who correctly answered 2 or 3 CRT questions were paid 30 thousand rials // Table 1: CRT scores, by location. // Bat and balls 68.9% 26.1% 5.1% // Allameh Tabataba'i University 1.18 19.9% 18.5% 23.3% 38.4% 146","Table 2 (all 395 pooled, not split by university) gives the bat-and-ball item as 68.9% reflective (correct), 26.1% impulsive, 5.1% other; this pooled figure is not available per university, so the bat-and-ball fields are left null in the three university rows. Data collected February to June 2015; subjects who answered 2 or 3 CRT questions correctly were paid 30 thousand rials (259 of 395 received it), so the payment depended on CRT answers. The questionnaire was in Persian (an English translation is in the paper's appendix) and the CRT was worded in cents (110 cents / 100 cents). This row: Table 1 line for Allameh Tabataba'i University."
noori2016-iran-amirkabir,Noori 2016 (Judgment and Decision Making),10.1017/S1930297500007634,Iran,students,"Students of Amirkabir University of Technology (Iran) who completed a paper questionnaire; part of 395 students from three universities (mean age 22.4, 63% male)",115,2015,2016,2015,Persian,original_3,,True,,,46.1,8.7,2.17,"Section 2 Method; Table 1 (CRT scores by location: mean, share scoring 3/2/1/0, N); Table 2 (distribution of answers, pooled)",https://www.cambridge.org/core/services/aop-cambridge-core/content/view/C70FFD62D8DA697D6FA89D8460075CF7/S1930297500007634a.pdf/div-class-title-cognitive-reflection-as-a-predictor-of-susceptibility-to-behavioral-anomalies-div.pdf (open-access PDF),"Four hundred and ten university students from three universities (Allameh Tabataba'i University, Amirkabir University of Technology, and Sharif University of Technology) participated in the research and completed a questionnaire (15 students did not complete the questionnaire, so the results are based on 395 subjects). // The data were collected during February to June, 2015. // Subjects who correctly answered 2 or 3 CRT questions were paid 30 thousand rials // Table 1: CRT scores, by location. // Bat and balls 68.9% 26.1% 5.1% // Amirkabir University of Technology 2.17 46.1% 33.9% 11.3% 8.7% 115","Table 2 (all 395 pooled, not split by university) gives the bat-and-ball item as 68.9% reflective (correct), 26.1% impulsive, 5.1% other; this pooled figure is not available per university, so the bat-and-ball fields are left null in the three university rows. Data collected February to June 2015; subjects who answered 2 or 3 CRT questions correctly were paid 30 thousand rials (259 of 395 received it), so the payment depended on CRT answers. The questionnaire was in Persian (an English translation is in the paper's appendix) and the CRT was worded in cents (110 cents / 100 cents). This row: Table 1 line for Amirkabir University of Technology."
noori2016-iran-sharif,Noori 2016 (Judgment and Decision Making),10.1017/S1930297500007634,Iran,students,"Students of Sharif University of Technology (Iran) who completed a paper questionnaire; part of 395 students from three universities (mean age 22.4, 63% male)",134,2015,2016,2015,Persian,original_3,,True,,,60.4,0.7,2.42,"Section 2 Method; Table 1 (CRT scores by location: mean, share scoring 3/2/1/0, N); Table 2 (distribution of answers, pooled)",https://www.cambridge.org/core/services/aop-cambridge-core/content/view/C70FFD62D8DA697D6FA89D8460075CF7/S1930297500007634a.pdf/div-class-title-cognitive-reflection-as-a-predictor-of-susceptibility-to-behavioral-anomalies-div.pdf (open-access PDF),"Four hundred and ten university students from three universities (Allameh Tabataba'i University, Amirkabir University of Technology, and Sharif University of Technology) participated in the research and completed a questionnaire (15 students did not complete the questionnaire, so the results are based on 395 subjects). // The data were collected during February to June, 2015. // Subjects who correctly answered 2 or 3 CRT questions were paid 30 thousand rials // Table 1: CRT scores, by location. // Bat and balls 68.9% 26.1% 5.1% // Sharif University of Technology 2.42 60.4% 22.4% 16.4% 0.7% 134","Table 2 (all 395 pooled, not split by university) gives the bat-and-ball item as 68.9% reflective (correct), 26.1% impulsive, 5.1% other; this pooled figure is not available per university, so the bat-and-ball fields are left null in the three university rows. Data collected February to June 2015; subjects who answered 2 or 3 CRT questions correctly were paid 30 thousand rials (259 of 395 received it), so the payment depended on CRT answers. The questionnaire was in Persian (an English translation is in the paper's appendix) and the CRT was worded in cents (110 cents / 100 cents). This row: Table 1 line for Sharif University of Technology."
pineros2016-us-clinicians,"Pineros, Doctor, Friedberg, Meeker & Linder 2016 (Family Practice)",10.1093/fampra/cmw015,United States,professionals,"187 primary-care clinicians (81% physicians, rest nurse practitioners or physician assistants; mean age 50; 64% women) from 50 practices in Boston and Los Angeles",187,,2016,2015,English,original_3,online,False,,,22.0,31.0,1.41,Abstract (Results); Methods; Results (first paragraph),https://pmc.ncbi.nlm.nih.gov/articles/PMC4931815/ (author manuscript in PubMed Central),"As part of a larger study, we invited 245 clinicians from 50 primary care practices in Boston and Los Angeles by email to complete a webbased survey in June 2013 and January 2014, respectively // The survey response rate was 76% (187/245). // A total of 57 clinicians (31%) scored 0 points on the CRT; 38 (20%) scored 1; 51 (27%) scored 2; and 41 (22%) scored 3. // For completing the survey, clinicians or their practices received $100.","Counts 57/38/51/41 sum to 187 (30.5%/20.3%/27.3%/21.9%, printed as 31/20/27/22). mean_score 1.41 is computed from these counts: (38 + 2x51 + 3x41)/187 = 263/187. Data were collected in Boston (June 2013) and Los Angeles (January 2014), so data_year is left null. The $100 was a flat payment for the whole 20-30 minute survey, not per correct answer. Full text read from the PubMed Central author manuscript."
ring2016-kiel,"Ring, Neyse, David-Barett, Schmidt 2016 (Frontiers in Psychology)",10.3389/fpsyg.2016.01680,Germany,students,"131 undergraduate students at Kiel University (72 women, mean age 24.7); 7-item CRT on paper, then performance predictions",131,,2016,2015,not stated,original_3_plus_more,paper,True,,,,,1.76,Materials and Methods (Participants; procedure); footnote 3,"https://www.ebi.ac.uk/europepmc/webservices/rest/PMC5089055/fullTextXML (Europe PMC open-access full text, PMC article page https://pmc.ncbi.nlm.nih.gov/articles/PMC5089055/)","Participants of the study were undergraduate students at Kiel University ( N = 131; 72 women; mean age = 24.7). // If we consider only the first three items as in Frederick (2005) , the average number of correct answers is 1.76 ( SD = 1.068). // For each correct answer in the CRT, the participants were paid €0.5. // Participants used pen and paper to answer both the CRT questions and the prediction task.",The study used a 7-item CRT (the original 3 plus 4 added items; mean 4.44 of 7). The 1.76 is the paper's own mean for the first three items (footnote 3). Language of the questionnaire is not stated (the first item is printed in English in the paper). Paid EUR 0.5 per correct CRT answer. Data year not stated.
sartarelli2016-alicante,Sartarelli 2016 (PLoS ONE),10.1371/journal.pone.0164412,Spain,students,"432 undergraduate students of the Universidad de Alicante in lab sessions (LaTEx) in 2013-2015, 48% female",432,,2016,2015,not stated,original_3,,,,,7.6,60.4,0.699,"Materials and methods (sample); Table 1 (summary statistics: CRT score, I(CRT>0)); Table 2 panel B text","https://www.ebi.ac.uk/europepmc/webservices/rest/PMC5082911/fullTextXML (Europe PMC open-access full text, PMC article page https://pmc.ncbi.nlm.nih.gov/articles/PMC5082911/)","Students from the Universidad de Alicante were recruited to participate in the experiment by using ORSEE // Experimental sessions were carried out at the Laboratory of Theoretical and Experimental Economics (LaTEx) // in 2013, 2014 and 2015 // CRT score | 0.699 | 0.000 | 0.986 | 0.000 | 3.000 | 432 // I ( CRT > 0) | 0.396 // 22.6% of subjects answer correctly 2 or more questions in the CRT and only 7.6% all 3 questions","zero_pct 60.4 is computed as 100% - 39.6% (the mean of the dummy I(CRT>0) = 0.396 is the share with at least one correct answer). The paper says about 40% have at least one right answer and 22.6% two or more. The medium of the CRT (paper or screen) is not described in the text read. Payment for CRT answers not stated. The data (2013, 2014, 2015) cover three years."
stieger2016-dach-convenience,Stieger & Reips 2016 (PeerJ),10.7717/peerj.2395,Multiple,general_convenience,"2,272 German-speaking volunteers (predominantly Austria and Germany), age-stratified convenience sample recruited by word of mouth; 56.6% women, mean age 39.8",2272,,2016,2015,German,original_3,paper,False,28.3,68.1,,,1.41,"Method (Participants, Procedure); Table 1 (success rate of the CRT items, row Item 1); Table 2 (CRT sum score, experienced vs naive)","https://kops.uni-konstanz.de/server/api/core/bitstreams/f4db011e-306c-4326-941d-eb347adb1556/content (open-access PDF, University of Konstanz repository copy of the PeerJ article)","Participants (N = 2,272) were recruited by word-of-mouth through friends, relatives, and friends-of-friends of several research assistants following a convenience sampling approach. | The final sample was age-stratified and constitutes German-speaking volunteers (predominantly Austria and Germany) | CRT Item 1 5 10 28.3 68.1 2.3 1.3 100.0 | CRT sum score 1.65 (1.11) 1.21 (1.06) | leading to a final sample size of N = 2,137 | were not financially remunerated for participation","Table 1 row for the bat-and-ball item: correct (5) 28.3%, wrong intuitive (10) 68.1%, wrong but not intuitive 2.3%, missing 1.3% (base = all 2,272). The paper prints no overall mean: mean_score 1.41 is computed from Table 2 = (959 experienced x 1.65 + 1,178 naive x 1.21) / 2,137 complete cases (1,178 = 2,137 - 959); Table 3 group means re-weighted give the same naive mean 1.21. 44.9% (n = 959) said they had done the CRT or a similar task before; naive-only mean is 1.21. Questionnaires were put in an envelope and a box (paper). Currency in the item is Euro. Data year not stated in the text read."
tay2016-cork-medstudents,"Tay, Ryan & Ryan 2016 (Canadian Medical Education Journal)",10.36834/cmej.36777,Ireland,students,"128 medical students (61 preclinical, 67 clinical; 79 female) of the School of Medicine, University College Cork, tested at the end of a lecture",128,,2016,2015,English,original_3,paper,False,56.0,40.0,44.0,10.0,2.02,Methods; Results (first paragraph); Table 1 (responses by preclinical and clinical students),"https://www.ebi.ac.uk/europepmc/webservices/rest/PMC5344059/fullTextXML (Europe PMC open-access full text, PMC article page https://pmc.ncbi.nlm.nih.gov/articles/PMC5344059/)","Of the remaining 128 students, 49 were male and 79 were female, while 61 were preclinical and 67 were clinical students. // Ten percent of students (13/128) answered none of the CRT questions correctly, 21% (27/128) answered one correctly, 25% (32/128) answered two questions correctly while 44% (56/128) answered all three correctly. // Over half of respondents (56%) obtained the correct (analytical) answer to the first question, with 40% giving the intuitive answer. // The mean number of questions answered correctly was 2.02. // The CRT was distributed to students at the end of a lecture","Question 1 is the bat-and-ball item. Table 1 gives it by group (preclinical 65% correct / 30% intuitive; clinical 49% / 49%); the overall 56% / 40% is the paper's text. Voluntary completion at the end of a lecture, no payment mentioned. 13/128 = 10.2%, 56/128 = 43.8% (printed as 10% and 44%). Data year not stated."
thomson2016-mturk200,"Thomson & Oppenheimer 2016 (Judgment and Decision Making, Introduction sample)",10.1017/S1930297500007622,Not stated,online_panel,200 workers on Amazon Mechanical Turk given the CRT with a rational-thinking battery; 72% had seen at least one CRT question before,200,,2016,2015,English,original_3,online,,,,,,1.73,"Introduction, paragraph on prior exposure (a separate MTurk sample from Study 1 and Study 2)",https://www.cambridge.org/core/services/aop-cambridge-core/content/view/42F4F70E998C63952D26DF2F4EBEA59E/S1930297500007622a.pdf/div-class-title-investigating-an-alternate-form-of-the-cognitive-reflection-test-div.pdf (open-access PDF),"we administered the CRT to 200 subjects on MTurk along with a rational thinking battery | In our sample, 72% of subjects reported prior exposure to at least one of the CRT questions. | Subjects who reported prior exposure answered significantly more CRT questions correctly (M = 1.90, N = 144) than those who did not (M = 1.29, N = 56)","No overall mean is printed. mean_score 1.73 is computed: (144 x 1.90 + 56 x 1.29)/200 = 1.73 (rounding of the two group means leaves +/-0.01). Prior-exposure mean 1.90 vs no-exposure 1.29 is itself the finding: scores rise with familiarity. The paper does not say MTurk workers were restricted to one country, so country is 'Not stated'. A later passage gives 1.86 vs 1.29 for the 120 high-exposure vs 56 low-exposure subjects of the same sample."
thomson2016-ucla,"Thomson & Oppenheimer 2016 (Judgment and Decision Making, Study 2)",10.1017/S1930297500007622,United States,students,"UCLA undergraduate psychology subject pool (mean age 20.5, 68% female), Study 2; original CRT given together with the new CRT2",143,,2016,2015,English,original_3,,False,48.2,44.0,,,1.39,Study 2 Method (Subjects); Study 2 Results; Appendix G (CRT Q1 = bat and ball),https://www.cambridge.org/core/services/aop-cambridge-core/content/view/42F4F70E998C63952D26DF2F4EBEA59E/S1930297500007622a.pdf/div-class-title-investigating-an-alternate-form-of-the-cognitive-reflection-test-div.pdf (open-access PDF),143 subjects from the UCLA undergraduate subject pool completed the study | Subjects on average got 46.3% of the original CRT items correct and 56.2% of the CRT2 items correct. | Appendix G: Percentages of responses by type for CRT and CRT2 questions Question Correct Intuitive errors Other CRT Q1 48.2% 44.0% 7.8%,"n = 143 who completed; the paper says item-level sample sizes varied (nine did not finish, some cells missing). mean_score 1.39 = 46.3% x 3 (the paper gives the share of original CRT items correct). Appendix G labels the item only 'CRT Q1'; it is the first original CRT item (bat and ball) in the order of the paper's Table 1 correlations. Credit through the psychology subject pool (course requirement or extra credit), not paid per correct answer."
wood2016-edinburgh-physics,"Wood, Galloway & Hardy 2016 (Physical Review Physics Education Research)",10.1103/PhysRevPhysEducRes.12.023101,United Kingdom,students,"148 students of a first-year physics course (physics majors and non-majors) at the University of Edinburgh, answering by smartphone/laptop in the second lecture",148,,2016,2015,English,original_3,online,,74.0,,57.0,5.0,2.31,Section I (CRT questions); Section II (methods); Section III Results; Table I (percentage of students with each CRT score),http://link.aps.org/pdf/10.1103/PhysRevPhysEducRes.12.023101 (open-access PDF),"In total 148 students answered every question on the CRT and also completed both the FCI precourse and FCI postcourse tests. // Students scored higher on the CRT than is typical for the general population [23], with 74% answering Q1 correctly, 71% answering Q2 correctly, and 86% answering Q3 correctly. // University of Edinburgh (148) 5 (8) 16 (23) 22 (32) 57 (85) 2.31 // (1) A bat and a ball cost £1.10 in total. The bat costs £1.00 more than the ball. // For Q1, 90% of students who answered incorrectly did so with the heuristic answer.","Q1 is the bat-and-ball item, worded in pounds and pence (GBP 1.10 / GBP 1.00; intuitive answer 10p). Table I counts 8/23/32/85 of 148 (printed 5/16/22/57%). Only students who answered all three questions are included. The CRT was delivered through the Top Hat online response system (numerical answers) and introduced as a test of the new system. 90% of the wrong Q1 answers were the intuitive 10p (so about 23% of all answered 10p, not printed). Data year not stated (2015-16 academic year implied); no payment mentioned."
gervais2018-australia,Gervais et al. 2018,10.1017/S1930297500007701,Australia,students,"University students, 13-country study of cognitive reflection and belief in God",143,,2018,2017,English,original_3,,,,,25.0,36.1,1.28,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: Australia 143 20.11 [5.34] 1.28 [1.20] 54.82 [38.83] students (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=144, 3 correct=25.0%, 0 correct=36.1%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=144 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: students. Payment: course credit. English version: yes (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
gervais2018-china,Gervais et al. 2018,10.1017/S1930297500007701,China,general_convenience,"Community sample, 13-country study of cognitive reflection and belief in God",207,,2018,2017,,original_3,,,,,33.3,12.1,1.8,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: China 207 29.82 [5.95] 1.80 [1.04] 28.74 [35.71] community (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=207, 3 correct=33.3%, 0 correct=12.1%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=207 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: community. Payment: money. English version: no (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
gervais2018-finland,Gervais et al. 2018,10.1017/S1930297500007701,Finland,general_convenience,"Mixed (students and community) sample, 13-country study of cognitive reflection and belief in God",992,,2018,2017,,original_3,,,,,33.2,17.7,1.76,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: Finland 992 28.15 [8.23] 1.76 [1.10] 31.28 [35.32] mixed (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=1003, 3 correct=33.2%, 0 correct=17.7%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=1003 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: mixed. Payment: none. English version: no (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
gervais2018-hong-kong,Gervais et al. 2018,10.1017/S1930297500007701,Hong Kong,students,"University students, 13-country study of cognitive reflection and belief in God",128,,2018,2017,,original_3,,,,,20.9,30.2,1.37,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: Hong Kong 128 21.01 [2.97] 1.37 [1.13] 62.94 [36.20] students (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=129, 3 correct=20.9%, 0 correct=30.2%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=129 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: students. Payment: lottery. English version: no (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
gervais2018-india,Gervais et al. 2018,10.1017/S1930297500007701,India,online_panel,"Mechanical Turk workers (community sample) screened to people who had not done the CRT before, 13-country study of cognitive reflection and belief in God",224,,2018,2017,English,original_3,online,,,,24.4,32.0,1.42,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: India 224 30.79 [8.38] 1.42 [1.17] 85.69 [25.70] community (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=225, 3 correct=24.4%, 0 correct=32.0%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=225 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: community. Payment: money. English version: yes (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. Indian participants were recruited from Mechanical Turk and screened to include only people who reported NOT having done the CRT before, so this sample is lower-familiarity than a typical MTurk sample. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
gervais2018-mauritius,Gervais et al. 2018,10.1017/S1930297500007701,Mauritius,general_convenience,"Community sample, 13-country study of cognitive reflection and belief in God",161,,2018,2017,,original_3,,,,,11.2,29.8,1.13,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: Mauritius 161 21.73 [1.33] 1.13 [0.97] 76.46 [39.21] community (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=161, 3 correct=11.2%, 0 correct=29.8%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=161 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: community. Payment: none. English version: no (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
gervais2018-netherlands,Gervais et al. 2018,10.1017/S1930297500007701,Netherlands,students,"University students, 13-country study of cognitive reflection and belief in God",212,,2018,2017,,original_3,,,,,11.8,35.8,1.12,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: Netherlands 212 19.49 [2.14] 1.12 [1.03] 21.23 [29.96] students (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=212, 3 correct=11.8%, 0 correct=35.8%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=212 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: students. Payment: course credit. English version: no (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
gervais2018-new-zealand,Gervais et al. 2018,10.1017/S1930297500007701,New Zealand,students,"University students, 13-country study of cognitive reflection and belief in God",160,,2018,2017,English,original_3,,,,,20.2,33.1,1.24,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: New Zealand 160 23.06 [7.98] 1.24 [1.11] 41.48 [39.84] students (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=163, 3 correct=20.2%, 0 correct=33.1%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=163 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: students. Payment: lottery. English version: yes (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
gervais2018-singapore,Gervais et al. 2018,10.1017/S1930297500007701,Singapore,students,"University students, 13-country study of cognitive reflection and belief in God",161,,2018,2017,English,original_3,,,,,31.7,12.4,1.83,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: Singapore 161 20.83 [1.69] 1.83 [1.01] 69.75 [30.39] students (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=161, 3 correct=31.7%, 0 correct=12.4%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=161 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: students. Payment: course credit. English version: yes (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
gervais2018-united-arab-emirates,Gervais et al. 2018,10.1017/S1930297500007701,United Arab Emirates,students,"University students, 13-country study of cognitive reflection and belief in God",145,,2018,2017,English,original_3,,,,,6.1,61.9,0.63,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: UAE 145 19.92 [1.57] 0.63 [0.93] 94.24 [19.04] students (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=147, 3 correct=6.1%, 0 correct=61.9%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=147 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: students. Payment: course credit. English version: yes (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
gervais2018-united-kingdom,Gervais et al. 2018,10.1017/S1930297500007701,United Kingdom,students,"University students, 13-country study of cognitive reflection and belief in God",150,,2018,2017,English,original_3,,,,,24.2,37.6,1.3,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: UK 150 25.33 [9.60] 1.30 [1.20] 34.26 [36.90] students (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=149, 3 correct=24.2%, 0 correct=37.6%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=149 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: students. Payment: lottery. English version: yes (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
gervais2018-united-states,Gervais et al. 2018,10.1017/S1930297500007701,United States,students,"University students, 13-country study of cognitive reflection and belief in God",591,,2018,2017,English,original_3,,,,,10.9,57.0,0.77,Table 1 (main paper); dataset data.csv column CRT,"http://journal.sjdm.org/18/18228/jdm18228.pdf (Table 1, supplement http://journal.sjdm.org/18/18228/supp.pdf, dataset http://journal.sjdm.org/18/18228/data.csv)","Table 1: Brief demographics of samples in 13 countries. CRT scores reflects the number of correct answers provided on the CRT out of 3 possible. Row: USA 591 19.36 [3.07] 0.77 [1.04] 79.55 [32.04] students (columns Country, N, Mean Age [SD], Mean CRT [SD], Mean Belief [SD], Source). All-3 and zero shares computed from dataset data.csv: n=596, 3 correct=10.9%, 0 correct=57.0%.","Original three CRT items (bat and ball, 5 machines/widgets, lily pads; items printed in the OSF/JDM supplement). Table 1 n and mean are after exclusions; all-3 and zero percentages are computed from the posted JDM dataset data.csv (n=596 incl. a few participants later excluded; its mean is within 0.02 of Table 1). Source type in Table 1: students. Payment: course credit. English version: yes (other countries used translated materials; language not named in the source). Data collected in an earlier unrelated project (Gervais et al. 2017), year not stated. US data come from a larger university sample used for precision. Countries were convenience sampled (study states: 'basically countries where Gervais had contacts')."
meyer2018-mturk-fall-2013,"Meyer, Zhou & Frederick 2018",10.1017/S1930297500007683,Not stated,online_panel,"Amazon Mechanical Turk workers, first appearance in the dataset (Fall 2013 series: 11/19/2013 to 11/21/2013; CRT only (2 surveys))",982,2013,2018,2013,English,original_3,online,,42.9,50.8,29.2,34.5,1.433,"Table 1; dataset data.csv columns series, nPrev, batAndBall, crtCor",http://journal.sjdm.org/18/18228a/jdm18228a.pdf (Tables 1-2) and dataset http://journal.sjdm.org/18/18228a/data.csv; variables http://journal.sjdm.org/18/18228a/vars.pdf,"Table 1 (Data overview): Fall 2013, 11/19/2013 to 11/21/2013, subjects 982. Computed from dataset data.csv, rows with nPrev=1 (subject's first appearance in the whole dataset) and series='Fall 2013': n=982, batAndBall answer 5 or 0.05 = 42.9%, 10 or 0.1 = 50.8%, crtCor mean=1.433, 3 correct=29.2%, 0 correct=34.5%.","Computed by us from the authors' posted dataset (JDM, CC BY 3.0 article licence; dataset licence not separately stated). Uses only each person's FIRST appearance in the 14,053-person dataset (nPrev=1), so repeat-exposure inflation inside the study is excluded, but people could still have met the CRT elsewhere. Bat-and-ball coded correct if the typed answer was 5 or 0.05, intuitive if 10 or 0.1; blanks/NA and other answers count as neither (denominator = all first appearances). Country of the MTurk workers is not stated in the paper. Original three items (bat and ball, 5 machines/widgets, lily pads). Winter 2015 series (modified CRT) not used."
meyer2018-mturk-fall-2014,"Meyer, Zhou & Frederick 2018",10.1017/S1930297500007683,Not stated,online_panel,"Amazon Mechanical Turk workers, first appearance in the dataset (Fall 2014 series: 9/3/2014 to 1/12/2015; CRT, SAT, self-reported exposure (17 surveys))",5518,2014,2018,2014,English,original_3,online,,36.6,58.5,22.6,34.7,1.318,"Table 1; dataset data.csv columns series, nPrev, batAndBall, crtCor",http://journal.sjdm.org/18/18228a/jdm18228a.pdf (Tables 1-2) and dataset http://journal.sjdm.org/18/18228a/data.csv; variables http://journal.sjdm.org/18/18228a/vars.pdf,"Table 1 (Data overview): Fall 2014, 9/3/2014 to 1/12/2015, subjects 6,910 in the series. Computed from dataset data.csv, rows with nPrev=1 (subject's first appearance in the whole dataset) and series='Fall 2014': n=5518, batAndBall answer 5 or 0.05 = 36.6%, 10 or 0.1 = 58.5%, crtCor mean=1.318, 3 correct=22.6%, 0 correct=34.7%.","Computed by us from the authors' posted dataset (JDM, CC BY 3.0 article licence; dataset licence not separately stated). Uses only each person's FIRST appearance in the 14,053-person dataset (nPrev=1), so repeat-exposure inflation inside the study is excluded, but people could still have met the CRT elsewhere. Bat-and-ball coded correct if the typed answer was 5 or 0.05, intuitive if 10 or 0.1; blanks/NA and other answers count as neither (denominator = all first appearances). Country of the MTurk workers is not stated in the paper. Original three items (bat and ball, 5 machines/widgets, lily pads). Winter 2015 series (modified CRT) not used."
meyer2018-mturk-spring-2014,"Meyer, Zhou & Frederick 2018",10.1017/S1930297500007683,Not stated,online_panel,"Amazon Mechanical Turk workers, first appearance in the dataset (Spring 2014 series: 3/14/2014 to 6/6/2014; CRT with feedback, Raven, Linda (6 surveys))",4864,2014,2018,2014,English,original_3,online,,37.8,55.3,26.2,35.1,1.367,"Table 1; dataset data.csv columns series, nPrev, batAndBall, crtCor",http://journal.sjdm.org/18/18228a/jdm18228a.pdf (Tables 1-2) and dataset http://journal.sjdm.org/18/18228a/data.csv; variables http://journal.sjdm.org/18/18228a/vars.pdf,"Table 1 (Data overview): Spring 2014, 3/14/2014 to 6/6/2014, subjects 5,191 in the series. Computed from dataset data.csv, rows with nPrev=1 (subject's first appearance in the whole dataset) and series='Spring 2014': n=4864, batAndBall answer 5 or 0.05 = 37.8%, 10 or 0.1 = 55.3%, crtCor mean=1.367, 3 correct=26.2%, 0 correct=35.1%.","Computed by us from the authors' posted dataset (JDM, CC BY 3.0 article licence; dataset licence not separately stated). Uses only each person's FIRST appearance in the 14,053-person dataset (nPrev=1), so repeat-exposure inflation inside the study is excluded, but people could still have met the CRT elsewhere. Bat-and-ball coded correct if the typed answer was 5 or 0.05, intuitive if 10 or 0.1; blanks/NA and other answers count as neither (denominator = all first appearances). Country of the MTurk workers is not stated in the paper. Original three items (bat and ball, 5 machines/widgets, lily pads). Winter 2015 series (modified CRT) not used."
sirota2018-uk-open,Sirota & Juanchich 2018,10.3758/s13428-018-1029-4,United Kingdom,online_panel,"UK nationals living in the UK from the Prolific Academic online panel (approval rate above 90%, not previously tested with the CRT in the authors' lab); open-ended answer-format condition only (n=147 of 452 randomised)",147,,2018,2017,English,original_3,online,False,39.5,56.5,27.9,29.9,1.503,"Method > Participants and Design; Materials; dataset MCQCRTdata.csv columns XP, CRT1, CRTALL3_R","OSF preprint https://osf.io/download/5a8d525191b689000c9d7b45/ (PsyArXiv rvh3y) and open dataset https://osf.io/mzhyc/ (MCQCRTdata.csv https://osf.io/download/d43sh/, codebook https://osf.io/download/gxfye/)","Preprint: 'A total of 452 participants (with ages ranging from 18 to 72 years ... 60.2% of whom were female) completed the questionnaire'; 'the participants solved the extended seven-item Cognitive Reflection Test, comprised of the original three items (Frederick, 2005) and four additional items (Toplak et al., 2014) ... presented in one of three test formats: (i) the original open-ended version'. Computed from MCQCRTdata.csv rows with XP=0 ('CRT open'), n=147: CRT1 (bat and ball) coded 1 correct = 58 (39.5%), 2 intuitive = 83 (56.5%); CRTALL3_R (original three items) mean = 1.503, 3 correct = 27.9%, 0 correct = 29.9%.",Only the open-ended condition is used (the 2-option and 4-option multiple-choice conditions give different results: bat-and-ball 29.0% and 38.0% correct). Seven CRT items were given in random order; the three original items are scored separately in the file (CRTALL3_R). Flat payment of GBP 1.40. Data year not stated (preprint posted Feb 2018). Panel: Prolific Academic ('The participants were recruited from an online panel (Prolific Academic)'). OSF project mzhyc is licensed CC BY 4.0.
woike2019-mturk-2016,Woike 2019,10.3389/fpsyg.2019.02646,United States,online_panel,"MTurk workers (US only), random assignment to the original bat-and-ball wording (one of four item variants), 2016",457,2016,2019,2016,English,bat_ball_only,online,,43.5,53.2,,,,"Study 1, section 2.3.1 'Correct and False Answers'; section 2.2.1-2.2.2 (sample, n)",https://pmc.ncbi.nlm.nih.gov/articles/PMC6909056/ (Europe PMC XML PMC6909056),"Responses to the standard version fell into the expected categories (see Figure 1A), with 43.5% correct answers (5 cents) and 53.2% false and intuitive answers (10 cents; only 3.3% of answers were different from 5 and 10 cents). [Group size: 'the original (n1.1 = 457)'.]","Only the bat-and-ball item was asked (original wording). MTurk HITs in 2016 paid a fixed $1.00-1.10 plus bonus (not tied to correct answers for this item as far as stated). Participants were restricted to the US, median 1,000 previous HITs. Everyone was also asked whether they had met the item before (see key facts)."
woike2019-mturk-2018,Woike 2019,10.3389/fpsyg.2019.02646,United States,online_panel,"MTurk workers restricted to US (min 98% approval, 101+ HITs), August 2018, within-subject comparison of original CRT and a transformed CRT",700,2018,2019,2018,English,original_3,online,,57.6,39.9,47.0,20.1,1.93,"Study 2, sections 3.2.1 and 3.3.1; dataset STUDY 2 file columns I1, I1binary, CRT3Score (Harvard Dataverse doi:10.7910/DVN/OWF4UY, CC0 1.0)",https://pmc.ncbi.nlm.nih.gov/articles/PMC6909056/ (Europe PMC XML PMC6909056); dataset https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/OWF4UY (CC0 1.0),"The mean CRT score (M = 1.93, SD = 1.19) was higher than the mean CRTt score [M = 1.60, SD = 1.13, t(699) = 13.05, p < 0.001 ...]. [Sample: 'a final sample size of n3 = 700'; 'conducted ... on MTurk in August 2018'.] Bat-and-ball, all-3 and zero shares computed by us from the CC0 replication dataset (Harvard Dataverse doi:10.7910/DVN/OWF4UY, file STUDY 2 DEIDENTIFIED_22nov19SHARED.tab, n=700): item I1 answered 5 cents by 403 (57.6%) and 10 cents by 279 (39.9%); CRT3Score 3 = 329 (47.0%), 0 = 141 (20.1%).","Original three CRT items were shown before the three transformed items (so no order effect on the original scores). Median 5,528 prior HITs; this is a very experienced panel. Fixed payment $3.00 plus bonus up to $1.00. The author notes data-quality concerns (attempts from Venezuela filtered out)."
woike2019-qualtrics-2018,Woike 2019,10.3389/fpsyg.2019.02646,Not stated,online_panel,"Commercial online panel (Qualtrics Panels), collected in 2018 for another project (Lewandowsky et al., in preparation); less CRT exposure than MTurk",1238,2018,2019,2018,,original_3,online,,11.7,,4.8,72.1,0.45,"Study 1, section 2.3.5 'Comparison With Data From a Different Online Panel'",https://pmc.ncbi.nlm.nih.gov/articles/PMC6909056/ (Europe PMC XML PMC6909056),"I re-analyzed the CRT data (n = 1, 238) collected using Qualtrics Panels in 2018 ... The percentage of correct solutions to the bat-and-ball question was clearly much lower (11.7%, see Figure 4A) ... The average score across the three items was 0.45, with 72.1% of the sample not solving a single item correctly and only 4.8% solving all.","Country is not stated in the main text (details in the Supplementary Material sections 1.2 and 2.1.3, which we did not read); do not assume US. The paper contrasts this with MTurk (43.5% in 2016)."
cresswell2020-academic,Cresswell & Speelman 2020,10.1371/journal.pone.0236153,Australia,professionals,"Research academics in the mathematical sciences, tested at their workplace (University mathematics-training study, paper questionnaire, one 'high-ranking Australian university')",30,,2020,2019,English,original_3,paper,False,,,77.0,0.0,2.77,"Results, Table 2 (Cognitive Reflection Test as a function of group)",https://pmc.ncbi.nlm.nih.gov/articles/PMC7390332/ (Europe PMC XML PMC7390332),Table 2 'Performance on the Cognitive Reflection Test as a function of group' (columns Introductory | Standard | Advanced1 | Advanced2 | Academic): N 62 | 27 | 34 | 44 | 30; Number correct 0: 21% | 0% | 0% | 0% | 0%; 1: 21% | 30% | 3% | 14% | 0%; 2: 32% | 0% | 26% | 16% | 23%; 3: 26% | 70% | 71% | 70% | 77%. Column used: Academic.,"mean_score computed by us from the rounded percentages in Table 2 (so +-0.03). The three CRT problems (Lily Pads, Widgets, Bat and Ball) were embedded in a longer 10-problem paper questionnaire; untimed, no calculators, supervised, voluntary, not incentivised. Data year not stated (ethics protocol 2016/647). Groups are selected by mathematics training, so they are NOT representative of students in general; they show how much maths training lifts CRT scores. N in Table 2 (62 introductory) is smaller than the 123 first-year volunteers mentioned in the Participants section; the paper does not explain the difference."
cresswell2020-advanced1,Cresswell & Speelman 2020,10.1371/journal.pone.0236153,Australia,students,"First-year second-semester mathematics/physics students with very strong mathematics background (University mathematics-training study, paper questionnaire, one 'high-ranking Australian university')",34,,2020,2019,English,original_3,paper,False,,,71.0,0.0,2.68,"Results, Table 2 (Cognitive Reflection Test as a function of group)",https://pmc.ncbi.nlm.nih.gov/articles/PMC7390332/ (Europe PMC XML PMC7390332),Table 2 'Performance on the Cognitive Reflection Test as a function of group' (columns Introductory | Standard | Advanced1 | Advanced2 | Academic): N 62 | 27 | 34 | 44 | 30; Number correct 0: 21% | 0% | 0% | 0% | 0%; 1: 21% | 30% | 3% | 14% | 0%; 2: 32% | 0% | 26% | 16% | 23%; 3: 26% | 70% | 71% | 70% | 77%. Column used: Advanced1.,"mean_score computed by us from the rounded percentages in Table 2 (so +-0.03). The three CRT problems (Lily Pads, Widgets, Bat and Ball) were embedded in a longer 10-problem paper questionnaire; untimed, no calculators, supervised, voluntary, not incentivised. Data year not stated (ethics protocol 2016/647). Groups are selected by mathematics training, so they are NOT representative of students in general; they show how much maths training lifts CRT scores. N in Table 2 (62 introductory) is smaller than the 123 first-year volunteers mentioned in the Participants section; the paper does not explain the difference."
cresswell2020-advanced2,Cresswell & Speelman 2020,10.1371/journal.pone.0236153,Australia,students,"Second-year second-semester university mathematics students with strong mathematics background (University mathematics-training study, paper questionnaire, one 'high-ranking Australian university')",44,,2020,2019,English,original_3,paper,False,,,70.0,0.0,2.56,"Results, Table 2 (Cognitive Reflection Test as a function of group)",https://pmc.ncbi.nlm.nih.gov/articles/PMC7390332/ (Europe PMC XML PMC7390332),Table 2 'Performance on the Cognitive Reflection Test as a function of group' (columns Introductory | Standard | Advanced1 | Advanced2 | Academic): N 62 | 27 | 34 | 44 | 30; Number correct 0: 21% | 0% | 0% | 0% | 0%; 1: 21% | 30% | 3% | 14% | 0%; 2: 32% | 0% | 26% | 16% | 23%; 3: 26% | 70% | 71% | 70% | 77%. Column used: Advanced2.,"mean_score computed by us from the rounded percentages in Table 2 (so +-0.03). The three CRT problems (Lily Pads, Widgets, Bat and Ball) were embedded in a longer 10-problem paper questionnaire; untimed, no calculators, supervised, voluntary, not incentivised. Data year not stated (ethics protocol 2016/647). Groups are selected by mathematics training, so they are NOT representative of students in general; they show how much maths training lifts CRT scores. N in Table 2 (62 introductory) is smaller than the 123 first-year volunteers mentioned in the Participants section; the paper does not explain the difference."
cresswell2020-introductory,Cresswell & Speelman 2020,10.1371/journal.pone.0236153,Australia,students,"First-year second-semester university students with weak high-school mathematics results, taking a compulsory unit (e.g. Biology or Geography majors) (University mathematics-training study, paper questionnaire, one 'high-ranking Australian university')",62,,2020,2019,English,original_3,paper,False,,,26.0,21.0,1.63,"Results, Table 2 (Cognitive Reflection Test as a function of group)",https://pmc.ncbi.nlm.nih.gov/articles/PMC7390332/ (Europe PMC XML PMC7390332),Table 2 'Performance on the Cognitive Reflection Test as a function of group' (columns Introductory | Standard | Advanced1 | Advanced2 | Academic): N 62 | 27 | 34 | 44 | 30; Number correct 0: 21% | 0% | 0% | 0% | 0%; 1: 21% | 30% | 3% | 14% | 0%; 2: 32% | 0% | 26% | 16% | 23%; 3: 26% | 70% | 71% | 70% | 77%. Column used: Introductory.,"mean_score computed by us from the rounded percentages in Table 2 (so +-0.03). The three CRT problems (Lily Pads, Widgets, Bat and Ball) were embedded in a longer 10-problem paper questionnaire; untimed, no calculators, supervised, voluntary, not incentivised. Data year not stated (ethics protocol 2016/647). Groups are selected by mathematics training, so they are NOT representative of students in general; they show how much maths training lifts CRT scores. N in Table 2 (62 introductory) is smaller than the 123 first-year volunteers mentioned in the Participants section; the paper does not explain the difference."
cresswell2020-standard,Cresswell & Speelman 2020,10.1371/journal.pone.0236153,Australia,students,"First-year second-semester university students with fair to good high-school mathematics results (e.g. IT or Computer Science majors) (University mathematics-training study, paper questionnaire, one 'high-ranking Australian university')",27,,2020,2019,English,original_3,paper,False,,,70.0,0.0,2.4,"Results, Table 2 (Cognitive Reflection Test as a function of group)",https://pmc.ncbi.nlm.nih.gov/articles/PMC7390332/ (Europe PMC XML PMC7390332),Table 2 'Performance on the Cognitive Reflection Test as a function of group' (columns Introductory | Standard | Advanced1 | Advanced2 | Academic): N 62 | 27 | 34 | 44 | 30; Number correct 0: 21% | 0% | 0% | 0% | 0%; 1: 21% | 30% | 3% | 14% | 0%; 2: 32% | 0% | 26% | 16% | 23%; 3: 26% | 70% | 71% | 70% | 77%. Column used: Standard.,"mean_score computed by us from the rounded percentages in Table 2 (so +-0.03). The three CRT problems (Lily Pads, Widgets, Bat and Ball) were embedded in a longer 10-problem paper questionnaire; untimed, no calculators, supervised, voluntary, not incentivised. Data year not stated (ethics protocol 2016/647). Groups are selected by mathematics training, so they are NOT representative of students in general; they show how much maths training lifts CRT scores. N in Table 2 (62 introductory) is smaller than the 123 first-year volunteers mentioned in the Participants section; the paper does not explain the difference."
byrd2024-australia,"Byrd, Stich & Sytsma 2024 (preprint)",10.31234/osf.io/qrz9g,Australia,general_convenience,Volunteers who clicked a Google ad inviting them to take online personality/philosophy surveys and get personality-test results (collected 2009-2018 in several batches); Study 1 of an 8-country religiosity-and-reflection study,243,,2024,2023,English,original_3,online,False,29.6,64.6,19.3,30.9,1.3,Study 1 Method/Participants and Table 1; dataset 'Study 1 cleaned.xlsx',"Preprint PDF https://osf.io/download/664f7b8ed835c40b3b4ce0f5/ (Table 1, Study 1); open dataset https://osf.io/8wf43/ file 'Study 1 cleaned.xlsx' (https://osf.io/download/t7vfe/), CC BY 4.0","Paper: 'From 2009 to 2018, a Google ad invited people to take surveys in return for personality test results ... Participants completed the original, 3-item Cognitive Reflection Test (Frederick, 2005).' Table 1 (country counts after exclusions): Australia n=243. Computed from dataset Study 1 cleaned.xlsx (columns COUNTRY, CRT1-CRT3): n=243, ball answered 5 (or 0.05) = 29.6%, 10 (or 0.1) = 64.6%, 3 of 3 = 19.3%, 0 of 3 = 30.9%, mean correct = 1.300.","Preprint (PsyArXiv, not peer reviewed; v2 cited as 2025 elsewhere). Data pooled over 2009-2018 collection batches A-K; batch dates are not given, so no year is assigned. Self-selected volunteers via Google ads, 71% women overall, 97% of the final sample is from the US, so non-US rows are small convenience samples of English-speaking web users, not representative of their countries. Exclusions reconstructed by us (age 100+, missing/out-of-range answers: ball <0, widgets >500, lily pad >48 days, country with <100 respondents): our country counts equal the paper's Table 1 for UK 508, India 560, Canada 366, Australia 243, Philippines 196, South Africa 176, Nigeria 117, and are 15 below the paper's 63,663 for the US. Scoring: ball 5 cents (0.05), widgets 5 minutes, lily pad 47 days; our total score equals the file's CRT_SCORE column for 65,765 of 65,814 rows. Intuitive ball answer = 10 cents (0.1). Survey was in English for all countries."
byrd2024-canada,"Byrd, Stich & Sytsma 2024 (preprint)",10.31234/osf.io/qrz9g,Canada,general_convenience,Volunteers who clicked a Google ad inviting them to take online personality/philosophy surveys and get personality-test results (collected 2009-2018 in several batches); Study 1 of an 8-country religiosity-and-reflection study,366,,2024,2023,English,original_3,online,False,27.0,70.2,17.5,40.7,1.139,Study 1 Method/Participants and Table 1; dataset 'Study 1 cleaned.xlsx',"Preprint PDF https://osf.io/download/664f7b8ed835c40b3b4ce0f5/ (Table 1, Study 1); open dataset https://osf.io/8wf43/ file 'Study 1 cleaned.xlsx' (https://osf.io/download/t7vfe/), CC BY 4.0","Paper: 'From 2009 to 2018, a Google ad invited people to take surveys in return for personality test results ... Participants completed the original, 3-item Cognitive Reflection Test (Frederick, 2005).' Table 1 (country counts after exclusions): Canada n=366. Computed from dataset Study 1 cleaned.xlsx (columns COUNTRY, CRT1-CRT3): n=366, ball answered 5 (or 0.05) = 27.0%, 10 (or 0.1) = 70.2%, 3 of 3 = 17.5%, 0 of 3 = 40.7%, mean correct = 1.139.","Preprint (PsyArXiv, not peer reviewed; v2 cited as 2025 elsewhere). Data pooled over 2009-2018 collection batches A-K; batch dates are not given, so no year is assigned. Self-selected volunteers via Google ads, 71% women overall, 97% of the final sample is from the US, so non-US rows are small convenience samples of English-speaking web users, not representative of their countries. Exclusions reconstructed by us (age 100+, missing/out-of-range answers: ball <0, widgets >500, lily pad >48 days, country with <100 respondents): our country counts equal the paper's Table 1 for UK 508, India 560, Canada 366, Australia 243, Philippines 196, South Africa 176, Nigeria 117, and are 15 below the paper's 63,663 for the US. Scoring: ball 5 cents (0.05), widgets 5 minutes, lily pad 47 days; our total score equals the file's CRT_SCORE column for 65,765 of 65,814 rows. Intuitive ball answer = 10 cents (0.1). Survey was in English for all countries."
byrd2024-india,"Byrd, Stich & Sytsma 2024 (preprint)",10.31234/osf.io/qrz9g,India,general_convenience,Volunteers who clicked a Google ad inviting them to take online personality/philosophy surveys and get personality-test results (collected 2009-2018 in several batches); Study 1 of an 8-country religiosity-and-reflection study,560,,2024,2023,English,original_3,online,False,39.3,53.8,30.5,29.5,1.534,Study 1 Method/Participants and Table 1; dataset 'Study 1 cleaned.xlsx',"Preprint PDF https://osf.io/download/664f7b8ed835c40b3b4ce0f5/ (Table 1, Study 1); open dataset https://osf.io/8wf43/ file 'Study 1 cleaned.xlsx' (https://osf.io/download/t7vfe/), CC BY 4.0","Paper: 'From 2009 to 2018, a Google ad invited people to take surveys in return for personality test results ... Participants completed the original, 3-item Cognitive Reflection Test (Frederick, 2005).' Table 1 (country counts after exclusions): India n=560. Computed from dataset Study 1 cleaned.xlsx (columns COUNTRY, CRT1-CRT3): n=560, ball answered 5 (or 0.05) = 39.3%, 10 (or 0.1) = 53.8%, 3 of 3 = 30.5%, 0 of 3 = 29.5%, mean correct = 1.534.","Preprint (PsyArXiv, not peer reviewed; v2 cited as 2025 elsewhere). Data pooled over 2009-2018 collection batches A-K; batch dates are not given, so no year is assigned. Self-selected volunteers via Google ads, 71% women overall, 97% of the final sample is from the US, so non-US rows are small convenience samples of English-speaking web users, not representative of their countries. Exclusions reconstructed by us (age 100+, missing/out-of-range answers: ball <0, widgets >500, lily pad >48 days, country with <100 respondents): our country counts equal the paper's Table 1 for UK 508, India 560, Canada 366, Australia 243, Philippines 196, South Africa 176, Nigeria 117, and are 15 below the paper's 63,663 for the US. Scoring: ball 5 cents (0.05), widgets 5 minutes, lily pad 47 days; our total score equals the file's CRT_SCORE column for 65,765 of 65,814 rows. Intuitive ball answer = 10 cents (0.1). Survey was in English for all countries."
byrd2024-nigeria,"Byrd, Stich & Sytsma 2024 (preprint)",10.31234/osf.io/qrz9g,Nigeria,general_convenience,Volunteers who clicked a Google ad inviting them to take online personality/philosophy surveys and get personality-test results (collected 2009-2018 in several batches); Study 1 of an 8-country religiosity-and-reflection study,117,,2024,2023,English,original_3,online,False,9.4,83.8,1.7,72.6,0.368,Study 1 Method/Participants and Table 1; dataset 'Study 1 cleaned.xlsx',"Preprint PDF https://osf.io/download/664f7b8ed835c40b3b4ce0f5/ (Table 1, Study 1); open dataset https://osf.io/8wf43/ file 'Study 1 cleaned.xlsx' (https://osf.io/download/t7vfe/), CC BY 4.0","Paper: 'From 2009 to 2018, a Google ad invited people to take surveys in return for personality test results ... Participants completed the original, 3-item Cognitive Reflection Test (Frederick, 2005).' Table 1 (country counts after exclusions): Nigeria n=117. Computed from dataset Study 1 cleaned.xlsx (columns COUNTRY, CRT1-CRT3): n=117, ball answered 5 (or 0.05) = 9.4%, 10 (or 0.1) = 83.8%, 3 of 3 = 1.7%, 0 of 3 = 72.6%, mean correct = 0.368.","Preprint (PsyArXiv, not peer reviewed; v2 cited as 2025 elsewhere). Data pooled over 2009-2018 collection batches A-K; batch dates are not given, so no year is assigned. Self-selected volunteers via Google ads, 71% women overall, 97% of the final sample is from the US, so non-US rows are small convenience samples of English-speaking web users, not representative of their countries. Exclusions reconstructed by us (age 100+, missing/out-of-range answers: ball <0, widgets >500, lily pad >48 days, country with <100 respondents): our country counts equal the paper's Table 1 for UK 508, India 560, Canada 366, Australia 243, Philippines 196, South Africa 176, Nigeria 117, and are 15 below the paper's 63,663 for the US. Scoring: ball 5 cents (0.05), widgets 5 minutes, lily pad 47 days; our total score equals the file's CRT_SCORE column for 65,765 of 65,814 rows. Intuitive ball answer = 10 cents (0.1). Survey was in English for all countries."
byrd2024-philippines,"Byrd, Stich & Sytsma 2024 (preprint)",10.31234/osf.io/qrz9g,Philippines,general_convenience,Volunteers who clicked a Google ad inviting them to take online personality/philosophy surveys and get personality-test results (collected 2009-2018 in several batches); Study 1 of an 8-country religiosity-and-reflection study,196,,2024,2023,English,original_3,online,False,8.2,83.7,2.6,75.0,0.367,Study 1 Method/Participants and Table 1; dataset 'Study 1 cleaned.xlsx',"Preprint PDF https://osf.io/download/664f7b8ed835c40b3b4ce0f5/ (Table 1, Study 1); open dataset https://osf.io/8wf43/ file 'Study 1 cleaned.xlsx' (https://osf.io/download/t7vfe/), CC BY 4.0","Paper: 'From 2009 to 2018, a Google ad invited people to take surveys in return for personality test results ... Participants completed the original, 3-item Cognitive Reflection Test (Frederick, 2005).' Table 1 (country counts after exclusions): Philippines n=196. Computed from dataset Study 1 cleaned.xlsx (columns COUNTRY, CRT1-CRT3): n=196, ball answered 5 (or 0.05) = 8.2%, 10 (or 0.1) = 83.7%, 3 of 3 = 2.6%, 0 of 3 = 75.0%, mean correct = 0.367.","Preprint (PsyArXiv, not peer reviewed; v2 cited as 2025 elsewhere). Data pooled over 2009-2018 collection batches A-K; batch dates are not given, so no year is assigned. Self-selected volunteers via Google ads, 71% women overall, 97% of the final sample is from the US, so non-US rows are small convenience samples of English-speaking web users, not representative of their countries. Exclusions reconstructed by us (age 100+, missing/out-of-range answers: ball <0, widgets >500, lily pad >48 days, country with <100 respondents): our country counts equal the paper's Table 1 for UK 508, India 560, Canada 366, Australia 243, Philippines 196, South Africa 176, Nigeria 117, and are 15 below the paper's 63,663 for the US. Scoring: ball 5 cents (0.05), widgets 5 minutes, lily pad 47 days; our total score equals the file's CRT_SCORE column for 65,765 of 65,814 rows. Intuitive ball answer = 10 cents (0.1). Survey was in English for all countries."
byrd2024-south-africa,"Byrd, Stich & Sytsma 2024 (preprint)",10.31234/osf.io/qrz9g,South Africa,general_convenience,Volunteers who clicked a Google ad inviting them to take online personality/philosophy surveys and get personality-test results (collected 2009-2018 in several batches); Study 1 of an 8-country religiosity-and-reflection study,176,,2024,2023,English,original_3,online,False,10.8,83.0,2.3,72.2,0.415,Study 1 Method/Participants and Table 1; dataset 'Study 1 cleaned.xlsx',"Preprint PDF https://osf.io/download/664f7b8ed835c40b3b4ce0f5/ (Table 1, Study 1); open dataset https://osf.io/8wf43/ file 'Study 1 cleaned.xlsx' (https://osf.io/download/t7vfe/), CC BY 4.0","Paper: 'From 2009 to 2018, a Google ad invited people to take surveys in return for personality test results ... Participants completed the original, 3-item Cognitive Reflection Test (Frederick, 2005).' Table 1 (country counts after exclusions): South Africa n=176. Computed from dataset Study 1 cleaned.xlsx (columns COUNTRY, CRT1-CRT3): n=176, ball answered 5 (or 0.05) = 10.8%, 10 (or 0.1) = 83.0%, 3 of 3 = 2.3%, 0 of 3 = 72.2%, mean correct = 0.415.","Preprint (PsyArXiv, not peer reviewed; v2 cited as 2025 elsewhere). Data pooled over 2009-2018 collection batches A-K; batch dates are not given, so no year is assigned. Self-selected volunteers via Google ads, 71% women overall, 97% of the final sample is from the US, so non-US rows are small convenience samples of English-speaking web users, not representative of their countries. Exclusions reconstructed by us (age 100+, missing/out-of-range answers: ball <0, widgets >500, lily pad >48 days, country with <100 respondents): our country counts equal the paper's Table 1 for UK 508, India 560, Canada 366, Australia 243, Philippines 196, South Africa 176, Nigeria 117, and are 15 below the paper's 63,663 for the US. Scoring: ball 5 cents (0.05), widgets 5 minutes, lily pad 47 days; our total score equals the file's CRT_SCORE column for 65,765 of 65,814 rows. Intuitive ball answer = 10 cents (0.1). Survey was in English for all countries."
byrd2024-united-kingdom,"Byrd, Stich & Sytsma 2024 (preprint)",10.31234/osf.io/qrz9g,United Kingdom,general_convenience,Volunteers who clicked a Google ad inviting them to take online personality/philosophy surveys and get personality-test results (collected 2009-2018 in several batches); Study 1 of an 8-country religiosity-and-reflection study,508,,2024,2023,English,original_3,online,False,31.1,63.2,19.5,41.1,1.148,Study 1 Method/Participants and Table 1; dataset 'Study 1 cleaned.xlsx',"Preprint PDF https://osf.io/download/664f7b8ed835c40b3b4ce0f5/ (Table 1, Study 1); open dataset https://osf.io/8wf43/ file 'Study 1 cleaned.xlsx' (https://osf.io/download/t7vfe/), CC BY 4.0","Paper: 'From 2009 to 2018, a Google ad invited people to take surveys in return for personality test results ... Participants completed the original, 3-item Cognitive Reflection Test (Frederick, 2005).' Table 1 (country counts after exclusions): United Kingdom n=508. Computed from dataset Study 1 cleaned.xlsx (columns COUNTRY, CRT1-CRT3): n=508, ball answered 5 (or 0.05) = 31.1%, 10 (or 0.1) = 63.2%, 3 of 3 = 19.5%, 0 of 3 = 41.1%, mean correct = 1.148.","Preprint (PsyArXiv, not peer reviewed; v2 cited as 2025 elsewhere). Data pooled over 2009-2018 collection batches A-K; batch dates are not given, so no year is assigned. Self-selected volunteers via Google ads, 71% women overall, 97% of the final sample is from the US, so non-US rows are small convenience samples of English-speaking web users, not representative of their countries. Exclusions reconstructed by us (age 100+, missing/out-of-range answers: ball <0, widgets >500, lily pad >48 days, country with <100 respondents): our country counts equal the paper's Table 1 for UK 508, India 560, Canada 366, Australia 243, Philippines 196, South Africa 176, Nigeria 117, and are 15 below the paper's 63,663 for the US. Scoring: ball 5 cents (0.05), widgets 5 minutes, lily pad 47 days; our total score equals the file's CRT_SCORE column for 65,765 of 65,814 rows. Intuitive ball answer = 10 cents (0.1). Survey was in English for all countries."
byrd2024-united-states,"Byrd, Stich & Sytsma 2024 (preprint)",10.31234/osf.io/qrz9g,United States,general_convenience,Volunteers who clicked a Google ad inviting them to take online personality/philosophy surveys and get personality-test results (collected 2009-2018 in several batches); Study 1 of an 8-country religiosity-and-reflection study,63648,,2024,2023,English,original_3,online,False,10.3,84.7,4.4,69.5,0.478,Study 1 Method/Participants and Table 1; dataset 'Study 1 cleaned.xlsx',"Preprint PDF https://osf.io/download/664f7b8ed835c40b3b4ce0f5/ (Table 1, Study 1); open dataset https://osf.io/8wf43/ file 'Study 1 cleaned.xlsx' (https://osf.io/download/t7vfe/), CC BY 4.0","Paper: 'From 2009 to 2018, a Google ad invited people to take surveys in return for personality test results ... Participants completed the original, 3-item Cognitive Reflection Test (Frederick, 2005).' Table 1 (country counts after exclusions): United States n=63663. Computed from dataset Study 1 cleaned.xlsx (columns COUNTRY, CRT1-CRT3): n=63648, ball answered 5 (or 0.05) = 10.3%, 10 (or 0.1) = 84.7%, 3 of 3 = 4.4%, 0 of 3 = 69.5%, mean correct = 0.478.","Preprint (PsyArXiv, not peer reviewed; v2 cited as 2025 elsewhere). Data pooled over 2009-2018 collection batches A-K; batch dates are not given, so no year is assigned. Self-selected volunteers via Google ads, 71% women overall, 97% of the final sample is from the US, so non-US rows are small convenience samples of English-speaking web users, not representative of their countries. Exclusions reconstructed by us (age 100+, missing/out-of-range answers: ball <0, widgets >500, lily pad >48 days, country with <100 respondents): our country counts equal the paper's Table 1 for UK 508, India 560, Canada 366, Australia 243, Philippines 196, South Africa 176, Nigeria 117, and are 15 below the paper's 63,663 for the US. Scoring: ball 5 cents (0.05), widgets 5 minutes, lily pad 47 days; our total score equals the file's CRT_SCORE column for 65,765 of 65,814 rows. Intuitive ball answer = 10 cents (0.1). Survey was in English for all countries."
li2024-china-lab,Li et al. 2024,10.3390/bs14040348,China,students,"Students of Shanghai Normal University (first-year college to second-year graduate), tested in the laboratory with a paper-and-pencil CRT, Experiment 2",120,,2024,2023,Chinese,original_3,paper,False,87.2,,,,2.572,Table 2 (Experiment 2) and section 3.1.1-3.1.2; general discussion ('87%'),https://pmc.ncbi.nlm.nih.gov/articles/PMC11047395/ (Europe PMC XML PMC11047395),"Table 2: Experiment 2: ACC | Bat-and-Ball 0.872 (0.349) | Lily Pads 0.846 (0.622) | Machine 0.854 (0.353). [Experiment 2: 'A total of 120 participants (48 males, mean age = 21.225 ± 2.833 years) were recruited from Shanghai Normal University.']","Same items as Exp 1 (currency changed to CNY). mean_score = sum of the three item accuracies (0.872+0.846+0.854) = 2.572 (computed by us). Participants wrote answers on a printed sheet and received a flat 10 CNY. The paper's Table 2 SDs for lily pads (0.622) look inconsistent with a 0/1 item, so treat SDs with care; proportions are used here."
li2024-china-online,Li et al. 2024,10.3390/bs14040348,China,students,"Chinese college students (84.5% bachelor, 11.4% master, 4.1% high school) recruited online via a questionnaire platform (wjx.cn), Experiment 1; asked to answer within a limited time",438,,2024,2023,Chinese,original_3,online,False,68.5,,,,1.713,Table 2 (Experiment 1) and section 2.1.1,https://pmc.ncbi.nlm.nih.gov/articles/PMC11047395/ (Europe PMC XML PMC11047395),"Table 2 'The accuracy of each question in CRT task [M (SD)]': Experiment 1: ACC | Bat-and-Ball 0.685 (0.465) | Lily Pads 0.418 (0.498) | Machine 0.610 (0.515). [Experiment 1: 'A total of 438 participants (126 males, mean age = 21.40 ± 2.44 years)']","Items taken from Frederick 2005 with the currency changed from USD to CNY (translation, counted as original). mean_score = sum of the three item accuracies (0.685+0.418+0.610) = 1.713 (computed by us). Participants were paid a random 2-4 CNY after the task (flat payment). Participants were told to answer in a limited time (time limit not stated for Exp 1). Data year not stated (platform accessed Dec 2021)."
campitelli2025-argentina-chess,"Campitelli, Labollita & Bilalic 2025",10.3390/jintelligence13030040,Argentina,professionals,"Twenty-five highly rated Argentine chess players (mean international Elo 2405; 7 grandmasters, 9 international masters); 21 male, mean age 27.8",25,,2025,2024,,original_3,,,,,,,2.082,"Results, Table A1 (Appendix C) and Section 2.1","https://pmc.ncbi.nlm.nih.gov/articles/PMC11942737/ (Europe PMC XML PMC11942737; MDPI, CC BY)","Appendix C, Table A1: CRT Chess Players M = 0.694, SD = 0.346, 95% CI [0.551, 0.837]; text: 'chess players (M = .694, SD = .34, [.551, .837]) outperformed non-chess players (M = .252, SD = .34, [.113, .392])'; 'Twenty-five Argentine chess players (21 male; Mage = 27.8, SD = 9.7, range = 15-55) ... The chess players sample was highly qualified (M international rating (Elo) = 2405, SD = 122)'.","CRT is a PROPORTION of the three problems correct (0.694); mean_score = 3 x 0.694 = 2.082 (computed by us). Elite chess players: a high-ability specialist group, not a general population. Original CRT (Frederick 2005). Tiny sample (25)."
campitelli2025-argentina-students,"Campitelli, Labollita & Bilalic 2025",10.3390/jintelligence13030040,Argentina,students,"Second-year undergraduate psychology students at a university in Buenos Aires, the non-chess comparison group matched to the chess players on age and gender",25,,2025,2024,,original_3,,,,,,,0.756,"Results, Table A1 (Appendix C) and Section 2.1-2.2","https://pmc.ncbi.nlm.nih.gov/articles/PMC11942737/ (Europe PMC XML PMC11942737; MDPI, CC BY)","Appendix C, Table A1 (Descriptive Statistics): CRT Non-Chess Players M = 0.252, SD = 0.337, 95% CI [0.113, 0.392]; text: 'chess players (M = .694, SD = .34, [.551, .837]) outperformed non-chess players (M = .252, SD = .34, [.113, .392])'; 'For CRT and TOL the y-axis indicates the proportion of correct answers'; 'We calculated the average number of correct problems solved.' Sample: 'twenty-five non-chess players ... second-year undergraduate psychology students in a university in Buenos Aires'.","CRT is reported as a PROPORTION of the three problems correct (0.252); mean_score = 3 x 0.252 = 0.756 (computed by us). Original CRT (Frederick 2005; bat-and-ball quoted with $1.10 and 'cents'), language of administration not stated (Spanish presumably, not stated). Tiny sample (25). Data year not stated."
kumari2026-india-jharkhand,Kumari et al. 2026,10.1177/09727531261417818,India,general_convenience,Young voters aged 18-35 living in Jharkhand who use at least one digital platform for political/news content; online questionnaire circulated via social media and student networks after the state election campaign,300,,2026,2025,,original_3,online,,,,,,1.06,Materials and Methods; Results (descriptive statistics),https://pmc.ncbi.nlm.nih.gov/articles/PMC12920153/ (Europe PMC XML PMC12920153; SAGE open access),'The three classic items (bat-and-ball; widgets; lily-pads) were used. Each item was scored 0/1 and summed to a 0–3 total' and 'Mean CRT performance was 1.06 (SD 0.85) on a 0–3 scale.' Sample: 'The sample comprised 300 young voters aged 18–35 years residing in Jharkhand.' (Table 1: Descriptive Statistics for Study Variables (N = 300)),"Social-media/student-network snowball sample, not representative of Indian youth. Language of the questionnaire is not stated in the text we read; data collection year not stated (election-period survey). Only the mean is reported (no item-level percentages)."