Talk: oTree for social scientists

At the 2017 TIBER conference I gave a talk about oTree, an open-source platform for conducting web, field, and lab experiments.

oTree for social scientists

oTree is an open-source platform for conducting web, field, and lab experiments. This is particularly useful for social scientists. Below you can find a link to a talk I gave at the 2017 TIBER conference in which I argue and demonstrate the benefits of oTree for social scientists. You can also find a transcript of the talk below (I like writing out what I’m going to say for a talk).

My talk was mostly a demonstration of oTree, so it contains links to oTree experiments. Unfortunately I cannot promise that these links will always remain active, so check it out while you can!

View slides


Slide 1: Welcome everyone. I’m going to give a bit of a different talk. I’m not going to talk about theories or data. Instead, my talk is more of a demonstration. Specifically, a demonstration of oTree.

Slide 2: oTree is an open-source platform for conducting lab, web, and field experiments. It’s software to both create and administer experiments, like Qualtrics or Inquisit.

The point of this presentation is to introduce this platform, demonstrate it, and explain why it might be a useful platform for you.

Now, I’m probably going to sound like an oTree spokesperson, but I am in no way affiliated with them; I’m merely a user. I’m simply very excited about oTree and the reason is that I think that oTree addresses certain challenges we face as social scientists.

Slide 3: The first I want to mention is that a lot of our theories rely on social interaction. Think of trust decisions, social exclusion, bargaining, coalition formation, etc. Unfortunately, doing interaction research is quite difficult. You need to code an interactive study and invite groups of people into your lab, and hope they show. This is difficult and effortful work, which is probably also a reason why, at least in social psychology, we tend to investigate interaction a bit differently. We either use deception to make the participant think they are interacting with another person, while in fact they are not. Or we use such tricks such as telling people that they will play with computers but the responses of that computer is based on real behavior from prior participants. Anyway, these are creative solutions to the issue that doing true interaction research is hard.

Which brings me sort of to the second challenge, which is that we’re creative folk. We do many different kinds of experimental studies. There are dozens of economic games that tap into different psychological constructs, such as for example the prisoner’s dilemma or public goods games. We also use a variety of measures, such as self-report, reaction speeds, mouse tracking, etc. So we do a lot of different types of research.

The final challenge I want to mention is probably the most important one, and that’s sample size. Lately it’s been made clear that larger sample sizes are better, and in some cases desperately needed. I’m hopeful that many researchers see the need for this, but it’s of course easier said than done. We need the opportunity to actually recruit more participants.

Slide 4: In response to these challenges, we are in the process of setting up a new lab in our department, called EXPLORE. Explore stands for Experimental Psychology Lab for Online Research and, as this name hopefully conveys, its focus is to conduct experiments online.

We managed to get a startup grant to get this lab off the ground, …and, I suppose, into the cloud.

The team consists of Ilja, Tony, me, Bastian, and Joeri, and together we decided to use oTree to actually pull this of. So in the next few slides I want to talk about what oTree is, how it works, and show you the features it has that made us decide on using this platform to address the challenges we face as social scientists.

Slide 5: Let’s begin with some background information.

Slide 6: oTree started in 2013, from Daniel Chen’s group at the Swiss Federal Institute of Technology in Zurich. It is funded by Chen’s ERC grant. And it is a homage to z-Tree, which you may know, I don’t, as an application to run interaction studies offline.

They have a published paper, which you can read to get a more extensive background and a bit of the philosophy behind oTree. You also have to cite this paper if you use their work for yours.

Slide 7: oTree can be used to create controlled behavioral experiments like you see in economics, psychology, and related fields. This included multiplayer games such as the prisoner’s dilemma and the public goods game, but also simple surveys and quizzes.

Slide 8: In very broad terms, oTree works as follows. You code your experiment. This is mostly done in Python and Django. Python is a high level general-purpose programming language. High level actually means it’s quite easy to use, and in the case of oTree it serves as the underlying mechanics of your study. So if you want to randomize conditions or something like that, you would code that in Python. Django is a web framework, which makes it possible that oTree can actually run online. You use this mostly to access variables that you’ve created using Python and display them to the participant.

Once you’ve coded it, you put oTree and your experiment on a server. This can be any server you rent, or probably your university’s.

Then you simply have to give a link to participants, and you can watch the data come in. Literally.

I say literally because it’s actually a feature of oTree to watch the progress of your studies in real-time.

Slide 9: And it’s the features of oTree that I want to discuss next. There are a lot of them, so I’ll briefly mention a few, and go a bit more in depth about some others.

Slide 10: First, and this may be very appealing to some, it’s free. This I find generally worrisome because platforms tend not to be sustainable when they’re free, but in this case it’s not too bad because there are few costs associated with oTree. In fact, the only thing that really costs money is renting the server, which you have to pay for yourself, or the university, so you can rely on oTree sticking around for a while.

Next, it’s open-source. This means the code is fully transparent, you can see how everything works, if you were so inclined, and it means that it’s also completely extensible. If you want to change something, you can. Personally, however, I think most of us aren’t programmers, so we’re not going to do that. But it’s still a nice sentiment! It also means that those who are programmers can work on this project as well and make it better.

The third feature is that’s it beautiful. I just want to mention that it’s pretty. Without a lot of work it already looks good.

Next is that oTree comes with an admin pane to manage your experiments. This I can best directly show you.

So here you see a list of demo studies that are actually also full fledged studies you can run. oTree comes with more than 20 sample games, but most of what you see here are studies we made. We could click on one to demo it, but instead let’s make an actual session. You do that by going to Session, and there you see that some sessions are already in place, which I put there in preparation of this talk. But let’s make a new one. We select one of the studies we created, and select how many participants we want. We then create it, after which we see the links we can give to our participants. There’s 1 session wide link, which is probably the one you want since anyone who clicks on that link gets assigned a slot and can participate in the study. There are also individual links, but these are less useful, unless you have specific roles for specific participants.

I clicked on the session wide link and you see the first question. Let me just answer all of these questions very quickly. -Answer questions- And here we see how well we did. I nailed it. Then it says finished. Normally you would have this link to a page, like an Mturk code or instructions to go to the test leader. Let’s go back to the slides.

Slide 11: I can tell you oTree also works on any device, such as a phone or tablet.

Another feature, which I already briefly mentioned, is that you can monitor the studies in real time.

I can show this too, because let’s take my iPad and do the same study there as well. Before I do, I’ll go to the Monitor page in the Admin Pane, to wait for my next participant. Then, when on my iPad I tap on the link, I can see that I’ve take up a slot in the study. Like I said before, you can actually also see the data come in, because if you go to the Data page, there you will see all the data collected so far, and the new data that is coming in. Let me show you that by answering some questions. As you see, when a new response is recorded, the cell lights up green.

Mostly this is not super useful, because if you coded everything right, you don’t need to keep track of it. But 1) it’s a lot of fun, and 2) if you have a somewhat more complicated design, with interactions, then you do want to keep track of things. Which brings me to the next feature.

Slide 12: I want to talk about 3 key features next, because these next features address the challenges that I previously mentioned.

Slide 13: The first is interaction. oTree enables true interactive studies between participants. This works by having participants go through a sequence of pages, until they get to a waiting page. There they have to wait for all other participants in their group to get there too, and then the study continues. Let me show you this with an example.

Let’s do a Trust Game. We begin with some instructions, and then I have to decide. How many points do I want to give. I’m nice, so I’m gonna give 100. Now I have to wait.

So here on my iPad I’m player 2, and I get to give somewhere between 0 and 300 points. I’m a jackass, so I’m going to give 0.

And here we see the results. 0 points. In the words of one of our participants that also participated in a trust game and didn’t get any money: Well that sucks.

So here we had a true interactive experiment. There were two people, who were actually in interaction with each other. It’s done in a manner of wait pages, where all participants gather so that the responses are collected, and then something is done with the responses. This could be part of a trust game, where it’s 1 one 1, but it could also be with larger groups, such as a coalition formation game or whatnot. Technically you could have up to a 100 or so people grouped together.

This addresses the first challenge: the need for interactive tasks.

Slide 14: The next feature addresses the second challenge: a need for flexible experimental designs. Like I said before, oTree is open source, so it’s completely under your control and you can extend it in any way you want. However, this is quite hard to do, so I’m actually not expecting a lot of people to do that. Fortunately, there are several other ways to extend the functionality. For example, you can use Javascript libraries to add functionality to your study. In our lab we already created a few tasks that can be useful in experimental design. I’ve listed a few examples here, such as a mouse tracking task, a venn task (which is a name I made up for a task in which you drag and resize circles to indicate how much overlap you see between the concepts described in the circles), or you can use charts to visualize data, also to participants, and there’s webcam support. Let me demonstrate one of these. Like… let’s do mouse tracking.

The task here is to categorize pictures by indicating whether you see a woman or a man. You then see the results of your mouse movements. These can be used to see how easy it was for someone to categorize the stimulus. You instantly see the results, which I added for demo purposes, because it shows the results of the mouse tracking, and it demos the use of charts.

Maybe another example, like the webcam one. I’ve gotten this to work on Chrome, not Safari. So this is a nice example of a limitation that you should discover while testing. This is just a demo of webcam functionality, not a real task, but this could be used to investigate self presentation, perhaps in a dating setting. In fact, that’s how we are planning on using it.

So, as you can see, we already added a few relatively complex tasks to oTree, thus demonstrating that it can be used in flexible ways, and letting us scientists do what we do best: creating cool studies.

Slide 15: The final feature I want to talk about is what should be clear by now, and that’s that oTree works online. So all of the tasks I showed were done online, including the complex ones. This enables us to run them on MTurk, or any other recruitment platform, so that we can get the participants we need. In fact, oTree has Mturk support built in, so it’s actually even easier to run online studies. Admittedly, we haven’t made use of this yet. We have simply provided Mturkers with a link, just like you would do with Qualtrics or so. But it’s possible!

Regarding examples, Bastian’s presentation was based on a study we did via oTree. You may recall that that wasn’t an interactive study, but it was a study that consisted of over a 1000 photos, which we did not want to upload to Qualtrics manually, because that would be effortful and effort is the worst, but also because we probably would have made some mistakes, linking the wrong URLs to the wrong host_ids, or other errors like that. Using oTree, we could simply load in a list of file names linking to the photo’s and the host_id, and code the logic needed to present the right participants with the right set of photos.

If time permits, I can also demonstrate another cool study we did. This one is interactive, and consists of a participant filling in a personality questionnaire, which is then shown to the participant they are partnered up with. So, each participants sees the personality scores of the other participants, and then they play a trust game. That way we can test what aspects of someone’s personality is used to form trustworthiness impressions.

I can quickly show you how that worked. I fill in some scores. And then I see the results. First my own, so that the participant gets to understand what each score means, and then that of the other participant, while deciding whether to trust that person or not. On this page you can also hover over the bars to see what they mean again, and we actually record also this behavior. When the participants make a decision, they fill in another few questions, wait again, and then they see the results. Like I said, I’m not going to talk about data here, so if you’re interested in this, go talk to Tony!

So, this last feature, that we can run all these studies online, is very powerful. This enables us to run complex, interactive studies online, on platforms such as MTurk, or anywhere else with an internet connection. Although technically you don’t need an internet connection, you just need a server, which could be a laptop, so you could also do this in the field. Either way, with this design it’s possible to gather more data, thus addressing our need for larger sample sizes.

Slide 16: After all this positive talk, it’s time for some limitations. When it comes to interactions, dropout becomes a significant problem. There is not an elegant way to deal with that. If a participant drops out, like in the personality study, then the participant he or she is partnered with is also out. There are some potential technical solutions to this, like providing a link to the still remaining participant to re-start the experiment, but usually this means the participant has to go through everything again, and spend a longer time doing your study.

We have done a few studies online now, and although there is some dropout, it’s not dramatic, I think it’s around 5% or so.

When you do online research, you have to be mindful of different screen sizes. So, you have to code your experiments in such a way that it allows for a variety of screen sizes. You can do this by making responsive designs. oTree uses Twitter’s bootstrap functionality, which means many of the default things you can do in oTree will resize well. It also works quite well on tablets because everything is touch supported. However, there are exceptions. For example, the mouse tracking task does not work on small screens, and of course, only with a mouse. Finally, a downside is that you need to learn more programming than you may be used to.

Slide 17: Actually, there’s a set of skills required for this, but it’s not too bad. I think anyone can learn this. I’ve provided a list of required skills here, and some links to places where you can learn them. In most cases, you don’t need to learn everything. You simply need to learn enough to get by. Also, if you master the art of Google, you’ll get a looong way.

Slide 18: In conclusion, oTree is a very well-designed platform that can be used to address at least three key challenges faced by social scientists: the use true interactions between participants, flexible experimental designs, and running online studies to acquire large samples sizes.

Thank you for your attention.