Research Remix

October 31, 2011

The promise of another open: Open impact tracking

Filed under: Uncategorized — Heather Piwowar @ 11:28 am

I’ve been thinking a lot recently about a scholarly Open that hasn’t gotten much attention yet:  Open impact tracking.

Impact tracking:  Usage data from publishers + websites + apps on the objects they host.  Downloads and views, but also bookmarks, discussions, posts… indications that people have interacted with the objects in some way.

We all know that companies value this information when the digital objects are pointers to consumer products: who is talking about the product?  How many people are talking about it?  What are they saying?  What does it mean?

Now imagine that the digital objects are scholarly products.  Papers, preprints, datasets, slidedecks, software.  Don’t we still want to know who is interested?  How many people are interested?  What they think, what they are doing with it, whether it is making a difference in their own related work?

Yup, as scholars and people who fund and reward scholars, we certainly do want to know those things.

We want to know the numbers, and we want to know the context of the numbers.  Not so we can overinterpret them as the end-all-and-be-all of an assessment scheme, but as insight into dimensions of impact that are totally hidden when we focus on pre-publication metrics (particularly the totally-inappropriate-for-article-level-assessment Journal Impact Factor) or even just the single dimension citation tracking.

PLoS has led the way: since 2009 PLoS has been collecting and displaying Article-Level Metrics for its articles.  Jason Priem and others have articulated the promise of altmetrics and begun digging into what these metrics mean.

Over the last few months I’ve been having a great time hacking on an app that reveals open altmetrics stats (and their context) for diverse research products.  total-Impact started in a 24-hour hackathon at the Beyond Impact workshop funded by the Open Society Foundations.  Since then a few of us have been unable to put it down.  I’ll talk about it a bit more in a future blog post [added link, also see here], but you are welcome to read more and play around with the alpha release now!

The time is clearly right for this sort of app… several similar ones are emerging now too.

In this post I want to highlight one thing about this space:

Impact information should be Open

The source data for scholarly research impact metrics should be Open.  Open facilitates mashups.  Open enables unexpected use, from unexpected places.  Open lets the little players in and brings the innovation.  Open permits transparency to detect problems.

Total-Impact got going in large part because PLoS and Mendeley have APIs which make their impact data freely and openly available.   Some publishers and websites do the same (or at least display their usage data on webpages and permit scraping) — but most don’t.  Why?

  1. It costs money, a rep from a Very Big Publisher told me last week.  Yup.  But these days not that much money.  This isn’t the beginning of Citation Counting when it was all manual and the only choice was to charge money.  This is routine web stuff.  Consider it one of your publishing costs, as PLoS does.
  2. It can be gamed, we don’t know what it means, it might send the wrong message.  Ok, yes.  But we are using it right now anyway, with all of those “Highly accessed” badges and monthly emails to authors.  The difference?  The data isn’t openly available for analysis and critique and deep understanding and improvement.  I say: open up your data, say what it means and what its limitations are, and work toward standards.
  3. Privacy.  For sure, don’t do things that would make your service users mad.  But that leaves a lot of room for sharing some useful data.  Aggregate download stats, maybe some breakdowns by date or geography or return visitors.  Drill-down to reviews or publicly-available details.  Here are a few of the sources doing it… you can do it too.
Note that I’m not advocating that all *uses* of impact information should be Open.  That has advantages, sure, but so does making money.  Making money is important: people who add value through interpretation should be able to be rewarded for that.  But the raw data that backs them up?  Open.
This means:
  • open usage stats.  Views and downloads of scholarly research products over time, number of bookmarkers, etc.  This means publishers and institutional repositories and data hosts and blogging platforms and value-add services.
  • open full text queries.  This doesn’t require OA: Google Scholar allows full text queries into research articles.  Unfortunately Google SCholar doesn’t allow using its information in an automated fashion.  Publisher websites could allow this, ideally through an API.  PubMed Central is a leader here, with eUtils (though its 3 queries/second limit prohibits a lot of useful applications).
  • open reference lists.  You know how abstracts are “open”… or at least free?  If reference lists were also in front of the pay wall and available for aggregation we could have a lot more players in the citation aggregation space, and more agile innovation than Web of Science+ Scopus + Google Scholar alone can provide.  Again PubMed Central is a leader here in making citation information Open through its eUtils api.
Let’s make it clear that we expect Open access to data demonstrating our impact.
Toll-access to articles limits what we scholars can do with aggregated scholarly work. So too, hidden and toll-access to impact information has implications for how we as scholars can filter, navigate, understand, and interpret scholarly work.  It matters.
ETA: link to related blog post


  1. […] I mentioned in a previous post, I’ve been one of several people working on an app called total-Impact.  Total-Impact is in […]

    Pingback by What total-Impact brings to the party « Research Remix — October 31, 2011 @ 11:31 am

  2. Harnad, S. & Carr, L. (2000) Integrating, Navigating and Analyzing Eprint Archives Through Open Citation Linking (the OpCit Project). Current Science 79(5): 629-638.

    Harnad, S. (2001) Research access, impact and assessment. Times Higher Education Supplement 1487: p. 16.

    Hitchcock, Steve, Tim Brody, Christopher Gutteridge, Les Carr, Wendy Hall, Stevan Harnad, Donna Bergmark, Carl Lagoze (2002) Open Citation Linking: The Way Forward. D-Lib Magazine. Volume 8 Number 10. October 2002.

    Hitchcock, Steve; Woukeu, Arouna; Brody, Tim; Carr, Les; Hall, Wendy and Harnad, Stevan. (2003)
    Evaluating Citebase, an open access Web-based citation-ranked search and impact discovery service

    Harnad, S. (2003) Measuring and Maximising UK Research Impact. Times Higher Education Supplement. Friday, June 6 2003

    Harnad, S. (2004) Enrich Impact Measures Through Open Access Analysis. British Medical Journal BMJ 2004; 329:

    Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open Research Web: A Preview of the Optimal and the Inevitable, in Jacobs, N., Eds. Open Access: Key Strategic, Technical and Economic Aspects, chapter 21. Chandos.

    Brody, T., Carr, L., Gingras, Y., Hajjem, C., Harnad, S. and Swan, A. (2007) Incentivizing the Open Access Research Web: Publication-Archiving, Data-Archiving and Scientometrics. CTWatch Quarterly 3(3).

    Brody, T., Carr, L., Harnad, S. and Swan, A. (2007) Time to Convert to Metrics. Research Fortnight pp. 17-18.

    Harnad, S. (2008) Validating Research Performance Metrics Against Peer Rankings. Ethics in Science and Environmental Politics 8 (11) doi:10.3354/esep00088 The Use And Misuse Of Bibliometric Indices In Evaluating Scholarly Performance

    Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1) Also in Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds. (2007)

    Harnad, S. (2009) Multiple metrics required to measure research performance. Nature (Correspondence) 457 (785) (12 February 2009)

    Comment by Stevan Harnad (@AmSciForum) — November 2, 2011 @ 3:39 pm

  3. […] The promise of another open: Open impact tracking […]

    Pingback by Around the Web: Irreverant scientists, Bookstores & choices, Myths about women in tech and more : Confessions of a Science Librarian — November 3, 2011 @ 9:53 am

  4. […] papers and data, full-text search because of course a paper is the best metadata for a dataset, and article open metrics of use are getting off the ground. It is crucial these are available for both humans and machines (apis) […]

    Pingback by thoughts on where journals are now, what to do next « Research Remix — December 3, 2011 @ 2:59 pm

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Blog at

%d bloggers like this: