Comments on: Generalizability coefficient for Mechanical Turk annotations

By: Heather Piwowar

Heather Piwowar — Sun, 04 Jan 2009 20:50:17 +0000

Yes, unfortunately that is all of the data I collected. As I was writing up the pilot study, I ran across a recent survey with 400 datapoints annotated by experts, so it was no longer necessary to derive such a dataset via MTurk.

PMCID is the PubMed Central ID (related to but not the same as the PubMed ID).

If you do have resources in search of a similar problem, I certainly have some I could suggest. For example, I’d access to have an annotated set of studies that REUSE datasets. A start is the list maintained by GEO… but it doesn’t have any true negatives. A reuse list would be interesting in and of itself as prevalence of data reuse (= benefits of data sharing), and could be used to evalutate an NLP engine for identifying reuse (which could then be used for a more systematic analysis of reuse patterns).

That said, I’m sure everybody on the bioNLP mailing list would have their own ideas about annotation datasets they’d love to see ;)

Anyway, let me know if I can be of more help.

Heather

By: lingpipe

lingpipe — Fri, 02 Jan 2009 22:19:15 +0000

Was the data at the end of the R code all you collected? I’ll probably need a bit more data than that to reliably infer annotator accuracies.

Is PMCID the article (PubMed?) ID?

We’re going to be running lots of mechanical Turk jobs in the next few months (we have an intern coming who’ll spend most of her time on it). We could collect more of this kind of data — the problem’s very interesting because it’s so directly relevant to a researcher.

By: Heather Piwowar

Heather Piwowar — Wed, 31 Dec 2008 14:02:14 +0000

Absolutely you may include it. Have at it. It sounds like an interesting problem!
Let me know if you have any questions about the data, such as it is.
Heather

By: lingpipe

lingpipe — Tue, 30 Dec 2008 15:50:10 +0000

Could you share your annotation data and gold standard?

I’d like to add your experiment to my inter-annotator data set, because I’m trying to establish the robustness of Bayesian approaches to inferring gold standards, problem difficulties, and coder accuracies. I’m also releasing all my data along with the R and BUGS code on the LingPipe sandbox.

Here’s a link to the blog entry linking to the paper about the models I’ve been using:

http://lingpipe-blog.com/2008/11/17/white-paper-multilevel-bayesian-models-of-categorical-data-annotation/