A perl script to make granting Mechanical Turk bonuses a little easier

Posted on by

One of the problems I quickly encountered when noodling around with Mechanical Turk is the limited and clunky web interface. Amazon has a handy comparison table which shows you what I mean by “limited”. Below is a look at the web interface for managing submitted HITs which will show you what I mean by “clunky” (which you can click for bigger.) None of it is JavaScript enabled — so every button-click requires a page reload. And there’s no logging for who’s been paid, and who hasn’t. Aargh!

mechanical turk management interface

After my first foray into using bonuses to engineer better results, I found that I needed to pay over a hundred bonuses. It rapidly became clear that paying these using the web interface would be nearly impossible, forcing me to look at the command line interface tools a little faster than I’d been planning.

Amazon provides a pretty full toolkit — but right now I’m only really interested in the grantBonus.sh tool. This meant I could (fairly quickly) knock up an improved workflow.

  1. Download all completed HITs as a CSV using Mechanical Turk’s Export Results button;
  2. Use a spreadsheet to approve and reject each HIT manually (need to improve upon this);
  3. Export the sheet as a CSV and use Mechanical Turk’s Review Offline button to load it back into the system and automatically process the basic payments;
  4. Append bonus values in a new column;
  5. Export the sheet as a tab-delimited file;
  6. Use a perl script to build a list of shell commands (I’m guessing if I were more comfortable with shell scripts I could skip this step); and
  7. Run that list of shell commands

Not – of course – as simple as it could be. Each stage could be automated more; and at some point will come the moment when I don’t need to use the web interface at all.

Anyway — here’s that perl script.

#!/usr/bin/perl

# A quick & dirty script to make paying bonuses on
# Amazon's Mechanical Turk less of a nightmare
# Requires a tab-delimited export of Workers'
# results edited to include a new "Bonus" column
# appended at far right
# Also requires MTurk CLI tools from http://bit.ly/cUhscK

# author: Mat Morrison
# date: Tuesday March 9, 2010

my $count;

# load data from the file arguments specified at the command line

open(INFILE,  $ARGV[0]) or die "can't open file to read: $!";
print "\tReading from $ARGV[0]. ";

if ($ARGV[1]) { 	#has an output filename has been specified?
	$outfile = $ARGV[1];
	} else {		#if not...
	print "No outfile specified - reverting to default\n";
	$outfile = "multiplebonuses.sh"; #write to a default filename
	}

print "\tReading from $ARGV[0] parsing to $outfile\n";

open(OUTFILE, ">$outfile") or die "can't open file to write: $!";

print OUTFILE "#!/usr/bin/env sh";

while (<INFILE>) {
	chomp ($_);
	# read the relevant variables out of each line
	# there are lots of columns, but we only need three of them
	# easier to use tab-delimited -- otherwise random commas get in the way...
	my (undef,undef,undef,undef,
		undef,undef,undef,undef,
		undef,undef,undef,undef,
		undef,undef,$AssignmentId,$WorkerId,
		undef,undef,undef,undef,
		undef,undef,undef,undef,
		undef,undef,undef,undef,
		undef,undef,undef,$Bonus) = split("\t",$_);

	# if there's a value in $Bonus (i.e. a payable bonus)
	if ($Bonus > 0) {
		# print the command to the instructions list script
		print OUTFILE "./grantBonus.sh -workerid $WorkerId -amount $Bonus -assignment $AssignmentId -reason \"altruism or overperformance bonus as per http://bit.ly/97c8z5\"\n";
		#increment the counter
		$count++
	}
}

print "\tProcess complete: $count bonuses to grant in $outfile\n";
END

Leave a Reply