在线数据收集

推荐实验方案:

  • Please ensure your IRB protocol covers your experimental procedures
  • Use Qualtrics to code the survey/experiment and then post that link to Amazon Mechanical Turk (MTurk, 有关MTurk的一般操作方法:请参阅 here and here)
  • For a great how-to on Qualtrics and programming a psychological experiment: see here
  • 有关在HTML中编码的技巧:请参阅此 document
  • 在人类神经科学实验室, 我们有使用文字和图片刺激来测试新旧识别记忆的直接经验, recall, 源内存(e).g., for spatial location), as well as a variety of creativity tasks (e.g.、交替使用任务和远程关联任务). 有关模板，请联系 Preston. We also have experience with counterbalancing and looping in Qualtrics (e.g., see this 一个关于循环的简短视频), which is necessary for appropriate and efficient experimental designs!
  • 如果使用Qualtrics，则有 FOUR 重要的事情:
    • 1a) Ensuring that you monitor off-task performance (for related publication, Permut et al., 2019)
      • a)参见这里的代码添加到您的 Header (see the 'Look and Feel' and then 'General' section of your survey)
      • b)参见这里的代码添加到您的 Footer (see the 'Look and Feel' and then 'General' section of your survey)
      • c) Make sure to add the Embedded Data fields as detailed in Permut et al. (2019)
    • 1b)由于Qualtrics中的兼容性问题，1a中列出的代码在某些情况下可能无法工作
    • 2)在线实验结束时, 每个MTurk员工都有一个唯一的代码，他们被要求输入MTurk的调查. They must do this before submitting the Human Intelligence Task (HIT) to receive payment. If participants do not enter this code, payment is not given.
      • For example, build a final question that looks like this
    • 3)确保你在MTurk创建一个资格，以确保MTurk不能重做你的实验
      • 你为什么需要这个? See here and here
      • 要获得您唯一的MTurk代码: http://uniqueturker.myleott.com/
      • 在发布您的电子游戏正规平台时，您可以在您的MTurk请求者帐户中设置其他资格/限制, 请看这里的人类神经科学实验室 settings (正如你在图片中看到的那样, I have set a restriction such that the study has not been participated in before, here called, 记住列表)
        • E.g., We like recruiting participants with a HIT approval rate of >95% (meaning that those who participate are those that are likely to finish it as that is their prior history), 并且参与者至少有50个hit被批准，这意味着他们已经完成了相当数量的MTurk电子游戏正规平台(注意，这些设置遵循其他已发表的电子游戏正规平台).
        • 还要注意一些限制, like age, 花费额外的钱(所谓的, “溢价限制”)
    • 如何保存/提取您的数据(e).g., as a .csv file)
      • Turn on variable naming using the gear button next to the question. 复选框称为“变量命名”:这将保存实际选择的响应，而不是点击或选择的位置.
      • 在导出部分中，确保导出 .csv with the 'choice text' selected (see also additional options like removing blanks)
      • Point is, do not assume all your data/responses will be saved. 确保输出捕获它!

质量的替代方案

  • 您可以直接在MTurk中编写Java/HTML代码, however that is not GUI-based and can be time consuming to learn and do
  • If interested in this approach, see these very helpful 视频和教程
  • 此外，您还可以使用 REDCap or PsyToolkit
    • REDCap 是基于调查的. 电子游戏正规平台服务@BC可以与电子游戏软件社区成员会面，讨论个别REDCap项目. 个人咨询或定制类咨询可通过电子邮件researchservices@bc.Edu或dalgin@ginxian.com.
    • PsyToolkit has an amazing library 经典的认知心理学实验模板有些你认为不适合在线数据收集(例如.g., Stroop Task, 心理旋转, 波斯纳提示任务, 威斯康星车分拣任务, Go/NoGo task, 返回抑制, Multitasking, etc.)

(Not Comprehensive) Checklist for 在线数据收集

这些是建议，不是规定有关更多信息，请参阅此处)

1) Hiding sensitive content in your client-side code

  • 检查客户端代码中的敏感内容. Open your experimental procedure in a browser window. If you are using Chrome, click View > Developer > View Source. In Firefox, click Tools > Web Developer > Page Source. (Google “how to view page source in [browser name]” to find instructions for other browsers.)将打开一个新的浏览器窗口，其中包含您正在查看的实验特定页面的源代码. 滚动这段代码, and make sure there isn’t anything in there you wouldn’t want participants to see (e.g. 隐性收益值, 逻辑揭示了你的实验操作, 不同条件的名称, etc.).
  • 混淆您发送给客户端的任何JavaScript代码，这样参与者就不能通过查看源代码来查看隐藏的值或过程的逻辑. 开源工具 javascript-obfuscator created by Timofey Kachalov can be used to obfuscate your JavaScript code. javascript-obfuscator@0的Web UI.18.可以在 obfuscator.io. (更好的选择是将所有敏感内容保存在服务器端，但这并不总是可行的.)

2)测试你的程序

  • 创建一个Mechanical Turk工作人员帐户，以便您可以在公开启动之前作为参与者测试您的程序
  • Set a custom qualification and assign it only to your own worker account
  • Launch your experiment with the custom qualification assigned to your worker account, 阻止其他工作人员查看HIT
  • 登录你的员工账户, 完成你的HIT至少8次, each time using one of the following 8 configurations (Browser + Operating System):
    • Chrome + Mac
    • Chrome + Windows
    • Firefox + Mac
    • Firefox + Windows
    • Edge + Mac
    • Edge + Windows
    • Safari + Mac
    • Safari + Windows
  • 阻塞移动设备, or test your HIT on mobile to ensure your procedure functions as intended. (You can simulate a mobile request in Chrome by selecting View > Developer Tools, 单击“切换设备工具栏”图标, 选择“中间层移动”，在弹出的设备工具栏中选择移动设备型号. 更多信息可在 这个页面的谷歌开发人员帮助中心.)
  • 检查程序中每个页面的url，以确保它们不包含任何可能泄露您的实验操作的信息
  • Check your logged data to make sure all activity is being tracked appropriately
  • 让几个朋友完成你的调查，并记录他们对每个问题的回答时间, 这样你就对平均参与者完成这个过程需要多长时间有了一些感觉

3)保护自己免受低质量数据的侵害

  • 总是包含一个中立的答案(如.g. “选择这里”)作为下拉列表等表单元素中的预选或默认响应
  • 测试多种回应方式(例如.g. radial, slider, drop down) to determine whether response type may bias results
  • Do not advertise eligibility requirements as part of the study sign up process, instead use qualification restrictions (such as country, 完成的电子游戏正规平台数目, etc.) or a pre-survey to restrict eligibility (Siegel et al., 2015)
  • Prevent retakes by assigning custom qualifications through the Mechanical Turk API, 或者使用Qualtrics的内置系统
  • Collect IP addresses and location data, as these can be used to detect bot activity
  • 在实验开始时使用抄写任务，要求参与者抄写一张手写文字的照片
  • Use a comprehension check with a non-obvious pattern (i.e. 多项选择理解题的答案不应该都被分配到每个选择题列表的顶部。
  • 在你的调查结束时包括一个强制性的自由形式的回答(如果他们没有回应/评论，你可以指示参与者简单地键入“na”)
  • Use custom JavaScript to count the number of times a subject leaves the page, to keep track of how long the subject spends away from the page, or even to display a pop up message when the participant leaves the page, 警告她，如果她在完成程序之前离开，HIT将不会被批准. (You can find some sample code that implements this sort of tracking in Qualtrics, see above)

4)减少人员流失的负面影响

  • 询问个人人口统计问题, and announce chances for additional financial incentives at the start of your procedure, as both of these measures have been shown to reduce dropouts (Reips, 2002a)
  • 在程序开始时，通过集中动机上的不利因素来引入一个高障碍任务
  • 在实验开始前，请参与者估计他们完成整个实验的可能性
  • 用呼吁良心的方式发出警告.g. “我们的电子游戏正规平台依赖于高质量的数据, and that quality is compromised whenever participants fail to complete the full procedure.”) (Zhou & Fishbach, 2016)
  • 使用练习试验, 试行新的实验方案, 或者要求参与者在开始焦点实验程序之前完成一项无聊的任务

5)分析数据

  • 等待，直到所有数据都已发布到服务器, 或停用您的质量调查, 在开始分析之前-一些平台(例如.g. 在参与者发起响应后的一周内，Qualtrics不会将未完成的响应发布到服务器, and you will not be able to measure attrition rates until these data have been posted
  • Check every free-form response you have collected, looking for suspicious patterns (e.g. one script I’ve detected in my data copies text from the surrounding page, 混淆了单词, then pastes the result into the free-form response box)
  • Check for repeated IP addresses and latitude/longitude coordinates, as significant numbers of such repeated information may be a sign of bot activity (Bai, 2018)
  • Include browser and operating system as factors in your analysis, as system configurations may impact participants’ interaction with your procedure, 并被发现与可能影响结果的重要个体差异相关(布坎南) & Reips, 2001)
Back To Top