Week 7 laboratory · 21 October

A Methods section is a recipe. Cook from one, then reverse one.

Part 1: write the Methods for a Stroop task, then build exactly that task from the Week 2 reaction-time page and run it on yourself. Part 2: start from a published result, guess the Methods that produced it, check against the paper, and rerun the authors' own data.

The task

Stroop: name the ink, not the word

BLUE in red ink is slower to name than RED in red ink. The slowdown (incongruent minus congruent RT) is the Stroop effect, one of the most replicated results in psychology. The task is a small edit of the page you built in Week 2.

The paper

Hedge, Powell & Sumner (2018)

“The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences.” Behavior Research Methods, 50(3), 1166–1186. doi:10.3758/s13428-017-0935-1. Open data on OSF. Everyone shows a Stroop effect, yet the same person's effect three weeks later is only partly predictable. Do not open the PDF until Part 2.

Get the files

Your week07-methods/ folder

Pull your private repo in VS Code (Source Control → Sync). If week07-methods/ is there, you are set. If not, make the folder and save these three files into it:

  • rt.html, the Week 2 reaction-time template, unchanged. You turn it into the Stroop task.
  • README.md, your report, with the Methods headings.
  • analyze_stroop.py, for Part 2.

Right-click → Save Link As. Keep the file names.

Part 1 · forward · about 80 minutes

Write the Methods, then build what it says

  1. Before touching code, fill in Participants, Materials, Procedure, and Data handling in the README. Numbers, not adjectives: which words and ink colours, which keys, how long the fixation and the gap between trials last, how many trials per condition, how the order is shuffled, which trials are thrown out. Commit. The commit time shows the Methods came first.
  2. Open week07-methods/rt.html and turn it into the task your Methods describes. The pieces you need:
    // (a) colours, words, and keys at the top
    const colours = [
      { name: "RED", key: "z", rgb: [230, 70, 60] },
      { name: "BLUE", key: "x", rgb: [80, 140, 255] },
      // ...
    ];
    
    // (b) a shuffled list of trials, half congruent, half incongruent
    trials.push({ condition: "congruent", word: ink.name, ink: ink });
    trials.push({ condition: "incongruent", word: other.name, ink: ink });
    trials = shuffle(trials);
    
    // (c) in draw(): show the word in the ink colour instead of the square
    fill(current.ink.rgb[0], current.ink.rgb[1], current.ink.rgb[2]);
    text(current.word, width / 2, height / 2);
    
    // (d) in keyPressed(): send z/x/n/m to handleResponse(k),
    //     and work out correct = (k === current.ink.key) ? 1 : 0
    
    // (e) add condition, word, ink, key, correct to results.push({...}),
    //     to the CSV header, and to the .join(",") list
  3. Every time the code makes you decide something the Methods did not say (does the word stay until a key press? what happens after a wrong key? is a practice block counted?), write it under Decisions I had to invent, then add it to the Methods.
  4. Run a practice block, then the scored block on yourself. Export stroop-week07.csv into week07-methods/. It needs the columns condition, rt, and correct. Write your congruent mean, incongruent mean, and Stroop effect under My result.

The test: could a stranger rebuild your page from your Methods alone, without seeing the code?

Part 2 · backward · about 60 minutes

From Hedge's result back to Hedge's Methods

Here is what the paper reports for the Stroop task. Every participant was tested twice. Reliability is an intraclass correlation (ICC): 1 means people kept exactly the same rank order from session 1 to session 2, 0 means no relation.

Study 1Study 2
Participants analysed4756
Test-retest ICC, Stroop RT cost.60.66
Test-retest ICC, Stroop error cost.48.44
  1. Guess first. Under My guess in the README, write the Methods you think produced this: who, how many trials, which colours and keys, how long between sessions, which trials and which people were excluded. Commit.
  2. Reveal. Open the open-access PDF and read the Stroop Methods (or open the box below). Under What I missed, list three details you got wrong or never thought of, and whether each could change the result.
  3. Reanalyse. In a terminal in week07-methods/:
    python3 analyze_stroop.py --mine stroop-week07.csv
    It downloads the tidied Hedge data (every trial, both studies, both sessions), applies their exclusion rules, and prints the group effect, the session 1 vs 2 correlation, and where your own effect falls. Paste output/numbers.md into the README.
  4. Run it again with --rule course (our 150–2500 ms rule instead of theirs). What changed? One sentence on why exclusion rules belong in a Methods section, written in numbers.
Reveal: Hedge et al.'s Stroop Methods, in brief
  • Participants. Undergraduates. Study 1: 50 (ages 18–21); Study 2: 62 (ages 18–47). Two 90-minute sessions, three weeks apart, same time of day. People who did not return were dropped, as was anyone below 60% accuracy in either session.
  • Materials. PsychoPy. Words in Arial, size 70, centred. Ink red, blue, green, or yellow, answered with Z, X, N, M.
  • Conditions. Congruent (word matches ink), incongruent (word names another response colour), and neutral (lot, ship, cross, advice, matched for length and frequency).
  • Procedure. Word stays until a response; 750 ms between trials. 240 trials per condition, 720 in all, randomly intermixed, in five blocks of about 4 minutes with feedback on mean RT and errors after each block.
  • Data handling. RT cost = incongruent − congruent. RTs under 100 ms, or more than three median absolute deviations from the person's median, were left out.

Notice what is still ambiguous. Is that median taken per condition or across all trials? The script picks one reading and says so. And the script keeps 105 people where the paper analysed 103 (47 + 56): the 60% rule alone does not catch every person they dropped. What else would the Methods need to say?

Tonight

Agenda

18:00 The paradox
A famous effect everyone shows, that still fails to rank people. What would you need to know to check that claim?
18:15 Part 1: write
Stroop Methods in the README, with numbers, committed before any code.
18:35 Part 1: build and run
Turn rt.html into your Stroop. Log every decision the code forced on you. Run yourself, export the CSV.
19:35 Part 2: guess and reveal
Write Hedge's Methods from their result, then read the real ones.
20:00 Part 2: reanalyse
Run analyze_stroop.py on their data and yours. Swap the exclusion rule and watch the numbers move.
20:30 Wrap
Which missing details changed a number? Next week: one dataset, many analysts.
Weekly report · due Tuesday 27 October, 17:00

What to submit

Push week07-methods/ with your Stroop rt.html, stroop-week07.csv, the filled README.md (Part 1 Methods, invented decisions, your result; Part 2 guess, what you missed, reanalysis), and output/numbers.md. Paste the path week07-methods/README.md on Canvas. Individual work; you are the only participant in your CSV.