31 Flavors of Research Impact through #altmetrics

The impact of a research paper has a flavour.  It might be champagne: a titillating discussion piece of the week.  Or maybe it is a dark chocolate mainstay of the field.  Strawberry: a great methods contribution.  Licorice: controversial.  Bubblegum: a hit in the classrooms.  Low-fat vanilla: not very creamy, but it fills a need.

CC-BY-NC by maniacyak on flickr

There probably aren’t 31 clear flavours of research impact.  How many are there?  Maybe 5 or 7 or 12?  We don’t know.  But it would be a safe bet that, just like ice cream, our society needs them all.  It depends whether we have a cone or a piece of apple pie.  The goal isn’t to compare flavours: one flavour isn’t objectively better than another.  They each have to be appreciated on their own merits for the needs they meet.

To do this we have to be able to tell the flavours apart.  Imagine that for ice cream all you had to go by was a sweetness metric.  Not happening, right?  So too, citations alone can’t fully inform what kind of difference a research paper has made on the world.  Important, but not enough.

We need more dimensions to distinguish the flavour clusters from each other.  This is where #altmetrics comes in.  By analyzing patterns in what people are reading, bookmarking, sharing, discussing, AND citing online we can figure out what kind — what flavour — of impact a research output is making.

Unfortunately we can’t accurately derive the meaning of these activities by just thinking about them.  What kind of impact *is* it if someone tweets about a paper a lot?  Is it a titilating champagne giggle because the title was amusing, or a strawberry indication they were thrilled because someone just solved their method struggle?  We need to do research to figure this out.

Flavours are important for research outputs other than just papers, too.  Some publicly available research datasets are used all the time in education but rarely research, others are used once or twice by really impactful projects, others across a field for calibration, etc.  Understanding and recognizing these usage scenarios will be key in recognizing and rewarding the contributions of dataset creators.

Below is a concrete example of impact flavour, based on analysis that Jason Priem (@jasonpriem), Brad Hemminger, and I are in the midst of writing up for the soon-to-be-launched altmetrics Collection at PLoS ONE.  (Edited April 3 to add: manuscript is now up on arXiv)  We have clustered all PLoS ONE papers published before 2010 using five metrics that are fairly distinct from one another: HTML article page views, number of Mendeley reader bookmarks, Faculty of 1000 score, Web of Science citation counts as of 2011, and a combo count of twitter, Facebook, delicious, and blog discussion.

We normalized the metrics to account for differences due to publication date and service popularity, transformed them, and standardized to a common scale.  We tried lots of cluster possibilities; it seems that five clusters fit this particular sample the best.

Here is a taste of the clusters we found.  Bright blue in the figure below means that the metric has high values in that cluster, dark grey means the metric doesn’t have much activity.  For example, papers in “flavour E” in the first column have fairly low scores on all five metrics, whereas papers in “flavour C” on the far right have a lot of HTML page views and Sharing (blog posts, tweeting, facebook clicking, etc) activity.

Since this is a blog post I’ll take the liberty of indulging in a bit of unsupported extrapolation and speculation and give these flavours some names.  I also include the titles of three exemplar papers from each cluster:

flavour E: Not much attention using these metrics  (53% of the papers in this sample)

Remember these papers may be impactful in ways we aren’t measuring yet!

[1] “Synaptic Vesicle Docking: Sphingosine Regulates Syntaxin1 Interaction with Munc18”
[2] “Sperm from Hyh Mice Carrying a Point Mutation in αSNAP Have a Defect in Acrosome Reaction”
[3] “Role of CCL3L1-CCR5 Genotypes in the Epidemic Spread of HIV-1 and Evaluation of Vaccine Efficacy”

flavour B: Read, bookmarked, and shared (21%)

[1] “Vision and Foraging in Cormorants: More like Herons than Hawks?”
[2] “Tissue Compartment Analysis for Biomarker Discovery by Gene Expression Profiling”
[3] “Protein Solubility and Folding Enhancement by Interaction with RNA”

flavour A: Read and cited (20%)

[1] “Roles of ES Cell-Derived Gliogenic Neural Stem/Progenitor Cells in Functional Recovery after Spinal Cord Injury”
[2] “Bone Marrow Stem Cells Expressing Keratinocyte Growth Factor via an Inducible Lentivirus Protects against Bleomycin-Induced Pulmonary Fibrosis”
[3] “Immune Regulatory Neural Stem/Precursor Cells Protect from Central Nervous System Autoimmunity by Restraining Dendritic Cell Function”

flavour D: Expert pick (3%)

[1] “Hemispheric Specialization in Dogs for Processing Different Acoustic Stimuli”
[2] “The Oncogenic EWS-FLI1 Protein Binds In Vivo GGAA Microsatellite Sequences with Potential Transcriptional Activation Function”
[3] “Retinal Pathology of Pediatric Cerebral Malaria in Malawi”

flavour C: Popular hit (3%)

[1] “Genetic Evidence of Geographical Groups among Neanderthals”
[2] “Perceptual Other-Race Training Reduces Implicit Racial Bias”
[3] “Symmetry Is Related to Sexual Dimorphism in Faces: Data Across Culture and Species”

What do you think, do they look like they might be meaningful clusters to you?  They are certainly interesting, uncovering impact made by papers we keep in our personal libraries but never cite, and demonstrating we do indeed share papers that aren’t just “popular hits” for example.

It is worth noting: Flavour E, D, and C are quite stable in this dataset, whereas the center of the clusters for Flavours B and A change a bit depending on clustering algorithm etc.  The cluster analysis needs more altmetric components to tease out the more subtle patterns.  We don’t even touch the crucial step of correlating the clusters with observed behaviour to validate whether they do in fact have real-life meaning.

The goal of our analysis here is not to report the quintessential impact clusters — a lot more research is needed!  Instead, we hope it serves as an illustration of what it might look like to begin describing research impact with a full flavour palette…. and one of the reasons we are so excited about altmetrics.

Want to learn more about altmetrics?  Yesterday’s article in The Chronicle of Higher Education is a great place to start! :)

Edited to add links to Jason Priem (@jasonpriem) and Brad Hemminger, so you can easily follow them if you aren’t already!

Also updated the graphic to a nicer version, April 3, 2012.