Research Remix

May 13, 2013

My champions of Open Science

Filed under: Uncategorized — Heather Piwowar @ 10:56 am

We have a huge and valuable opportunity to honor hard work and dedication in our community:  the White House is calling for nominations for “Open Science” Champions of Change.

Awards matter.  They feel good, they help people get taken seriously, and they make it easier to get funding.  Let’s run with this opportunity!

Nominations must be in by May 14 2013 (tomorrow!) and you can nominate as many people and organizations as you like.  It isn’t clear, but it seems like multiple people will be honored.  The nominator and nominated must be both be affiliated with a US address.  Under “Theme of Service,” choose “Open Science”.

Here are my personal open science champions of change.  I hesitated to post this, because of course the list isn’t complete: I’ve got a bad memory so I’ve no doubt forgotten many, it is likely skewed to people whose tweets and blog posts I’ve recently read, and it emphasizes my biomed-centric, research data, coding, textmining view of the world.  That said, just because I can’t do it all is no reason not to make a start, right?  (what is the real quote for that?  open science words to live by if there ever were any), so here’s a go:

  • peter suber
  • vitek tracz
  • bmc
  • mike eisen
  • plos
  • john wilbanks
  • creative commons
  • heather joseph
  • sparc
  • mike carroll
  • jason hoyt
  • pete binfield
  • peerj
  • mendeley
  • alf eaton
  • iain hrynaszkiewicz
  • cameron neylon
  • f1000research
  • jean-claude bradley
  • peter murray-rust
  • john ioannidis
  • victoria stodden
  • @gavialib library loon
  • barbara fister
  • jenica rogers
  • dorothea salo
  • amy buckland
  • john dupuis
  • joe kraus
  • abigail goben
  • rosie redfield
  • mike taylor
  • timothy gower
  • arxiv
  • jdap
  • todd vision
  • dryad
  • figshare
  • icpsr
  • github, twitter, friendfeed, RSS, google reader
  • wikipedia
  • R, hadley wickam, knitr, ropensci
  • scienceexchange reproducability initiative
  • sherpa romeo
  • bill hooker
  • geoff bilder
  • mark patterson
  • ian mulvany
  • bjoern brembs
  • martin fenner
  • ojs
  • orcid
  • catherine ball
  • mit open courseware
  • campus OA mandates
  • max haussler
  • david shotton
  • ross mounce
  • scott edmunds
  • egon willighagen
  • dcc
  • jisc
  • ris
  • okfn
  • ands
  • pmc, ncbi, eutils
  • california digital library
  • @openscience
  • @oatp
  • trials, and others with open peer review
  • ben goldacre
  • carl boettiger, jon eisen, antony williams, steven roberts, ethan white, titus brown
  • wellcome trust
  • sloan foundation
  • jason priem
  • impactstory :)
  • @fakeelsevier
  • aaron swartz
  • and all the unnamed people who push behind the scenes on open initiatives everywhere

ok, I’d better stop there.  I have some nominating to do.

Who are your champions of open science?  Do add in comments… and if they have a US affiliation, don’t forget to nominate them!


May 11, 2013

OA options for a society journal

Filed under: Uncategorized — Heather Piwowar @ 10:52 pm

Does your society want to embrace Open Access but not know where to start?  Maybe this will help.

I was invited to be part of an OA Exploration Taskforce last fall: AMIA is/was considering OA for its society journal JAMIA and deciding what it wanted to do, if anything.  As the person on the taskforce most familiar with current OA trends I was in charge of reporting back on possible OA options.  Others on the taskforce surveyed current membership, did financial modelling, and so on.

I was surprised how confusing and time-consuming it was to gather and sift information on options, so hopefully blogging the report here (belatedly) will save others time and spur conversation.  Thanks very much to all the publishers I contacted for their information, and for permission to post this information, especially the specifics provided by BMJ and Ubiquity Press.

Here is my initial post with notes, thoughts, and contact details, and the organized list of options is below.  Much has changed in the OA landscape since last fall when this was written, so some of this might already be out of date…. please check with the publishers for current information.  Sorry it has taken me months to post this; it has been busy.

I’m not actually sure of the current status of JAMIA and OA, since my participation on the task force has ended.  I know they took a subset of these options to the Board and there was active consideration of options.  Nancy Roderer summarized the situation to me this way, a few months ago:

“I believe they are committed now to OA, but worry about the finances, and what you have given helps in convincing them that if they say what the want and put out a request, publishers will make some good offers and the best ones will come from the newer publishers.”

I’m very glad to have contributed to the task force.  I felt honoured to be invited, and it was empowering to have an opportunity to help make AMIA the open society I think it can and should be.

Overview of options (details follow)

  1. Continue with current OA offering

  2. Increase openness within current framework

    1. change Unlocked license to CC-BY

    2. make all papers CC-BY when they become free in PMC

    3. shorten time until papers are free in PMC

    4. lower Unlocked fee

    5. offer discount Unlocked fee to AMIA members

    6. offer “coupons” for Unlock fees to organizations that subscribe to JAMIA

  3. Introduce new OA journal possibly to complement JAMIA

    1. through BMJ

    2. through F1000Research and F1000Posters

    3. others

  4. Transition JAMIA to 100% OA with a publisher

    1. through BMJ

    2. through BioMed Central

    3. through Ubiquity Press

    4. others

  5. Publish JAMIA within AMIA as 100% OA

    1. hosted internally

    2. hosted by a journal-management system hosting service

    3. others

  6. Other options

    1. probably out of scope: publishing cooperative, overlay journal, etc.

    2. related revenue and cost-saving possibilities


OA Options for a society journal: details

Edited to add:  a number of people have said my original post didn’t highlight the financial positives from openness.  I’ll try to add this.  In the mean time, I’ve added a “positives from openness” placeholder to possible financial implications below.  The potential financial positives include increased membership numbers due to more exposure and future-embracing leadership, more interest in publishing in the journal due to wider readership and openness, and so on.

1. Continue with current OA offering

The current offering:

  • JAMIA published by BMJ
  • All articles are free after 12 months in PubMed Central

  • “Unlocked” articles are Open under a CC-BY-NC license immediately

  • Articles become “Unlocked” either by authors paying a $2000 article processing charge (APC) or by nomination by the editors

  • JAMIA authors may self-archive after 6 months on their personal website or through their institution’s repository, but may not post the publisher’s version or PDF


2. Increase openness within current framework

Some small changes could increase the Openness of JAMIA content.

– Change Unlocked license to CC-BY

“Unlocked” articles could be made Open under a CC-BY license rather than a CC-BY-NC license.  CC-BY is generally considered the best OA practice because it facilitates most reuse. NC means “non-commercial”: it prohibits reuse by companies without further negotiation.  Startup and established commercial companies often want to reuse article text for indexing, building research tools, text-mining for research, and disseminating reprints.  Restrictions can be surprising: images and text can not be used in Wikipedia when licensed CC-BY-NC, for example.  For more discussion on NC clause, see:

BMJ has confirmed they would support this change.

Financial implication: positives from openness; eliminate reprint revenue.


– Make all articles CC-BY when they are available for free in PMC

JAMIA could choose to make articles available under a CC-BY license once they are available for free to PMC.  “Free in PMC” is not the same as Open Access because it does not allow Open reuse and redistribution; moving to a delayed OA license would facilitate this reuse.  RUP uses this model: subscriptions are needed to view the content for the first N months, then the articles are OA.

Financial implication: positives from openness; decreased reprint revenue, possible drop in subscriptions.


– Shorten time before papers are free in PMC

All JAMIA articles currently become free in PMC after 12 months.  This could be shortened to 6 months.

Financial implication: positives from openness; possible drop in subscriptions (though no evidence of this to date)


– lower Unlocked fee

JAMIA could decrease the Unlocked fee.  Many hybrid journals have Unlocked fees similar (or higher) than JAMIA’s, but the article-processing-fee for most OA journals is lower.

Financial implication: positives from openness; decreased revenue per APC.


– offer discount Unlocked fee to AMIA members

JAMIA could offer an Unlocked discount to AMIA members.

Financial implication: positives from openness; decreased revenue per APC.


– offer “coupons” for Unlock fees to organizations that subscribe to JAMIA

JAMIA could offer coupons to subscribers to offset Unlock fees.  This would make it easier for authors of subscribing institutions to publish their work as Unlocked articles.  The Royal Society of Chemistry has recently proposed a similar idea to facilitate  community transition to OA.

Financial implication: positives from openness; decreased  revenue per APC.


3. Introduce new OA journal, possibly to complement JAMIA

– through BMJ

AMIA could launch a companion journal through BMJ to complement the existing journal.  It could take the same form as BMJ Open: 100% open access, peer-reviewed on scientific rigor but not anticipated impact or novelty, online-only, and agile with respect to emerging publishing practices.  Authors could submit directly to “JAMIA Open”, and also editors of JAMIA could suggest JAMIA Open to authors of papers submitted to JAMIA that are sound but not appropriate for JAMIA.  JAMIA Open would be a standalone journal which would be indexed.

Financial implication: positives from openness; possibly increased revenue from new Journal article charges.

– through F1000Research and F1000Posters

JAMIA could launch a novel publishing collection through F1000Research and F1000Posters.  Authors could submit papers, data papers, posters, and presentations to F1000Research and F1000Posters: these could then be consolidated into a “JAMIA page”, similar to the emerging institution pages.  This would be a diverse collection. Many of the research papers would be indexed in traditional indexes (based on post-publication peer review status), but the collection itself would not be a standalone journal.

Financial implication: positives from openness; more discussion would be necessary with F1000.  Submission to F1000Research will have a fee.

– others

There may be other possibilities to partner with PLOS Currents, PeerJ, etc.  More research would be needed.

4. Transition JAMIA to 100% OA with a publisher

– through BMJ

AMIA could transition JAMIA to a 100% OA journal with BMJ.  This would retain the existing infrastructure with BMJ, but eliminate the subscription base and require that all articles (except those with waivers) pay an Unlock fee.  BMJ does not do this with any of its other society journals at the moment — its only 100% OA journals are BMJ and BMJ Open — but could be open to this. The Unlock fee could change, depending on how much revenue JAMIA wanted.

Financial implications: positives from openness; Costs and Ad revenue would continue as before.  Eliminates subscription revenue.  Would receive more Unlock revenue.

– through BioMed Central

AMIA could transition JAMIA to a 100% OA journal with a publisher who publishes many 100% OA society journals.  BioMed Central (BMC) is such a publisher (OA society journals it publishes).  See feature list BMC has provided to the JAMIA OA explorations taskforce.

Detailed financial and operational terms would require a conversation about AMIA’s goals and requirements, but BMC has offered example terms based on its experiences with other societies.

Financial implications (rough estimates: bespoke agreement would be based on detailed discussion):

  • ‘standard’ article processing charge (APC) is currently £1075.

  • This increases by £175 if editorial office support is required and by £195 if copy editing, provided by the publisher, is required. If both services are required the there is a discount, and the APC increases by £360

  • The Standard APC level provides a small honorarium to the society. The APC can be increased, by a surcharge, to raise the potential revenue available to the society

  • Typically the society receives a 50% share of advertising revenue where they have brought the advertiser/contact to BMC.

  • No reprint revenue because OA

  • AMIA could offer discounts to members, or prepay APCs for other reason.  Discounts are available when APCs are prepaid.  For example, a society depositing $100,000USD would automatically qualify for at least a 15% discount on the APCs it covers for its members

  • Print can be a major cost and revenue stream so knowing about JAMIA’s plans for print will also help us tailor our proposal more appropriately. BMC does not print any journals on a monthly or issue basis but we often prepare special and anniversary printed collections for a number of our titles, such as Arthritis Research & Therapy. If regular printing was an absolute requirement, we could discuss providing it but have found that the limitations of print – in terms of article length, publication delays etc – are challenging for all concerned (authors, the society as well as the publisher).


– through Ubiquity Press

AMIA could transition JAMIA to a 100% OA journal with an emerging publisher of OA society journals.  Ubiquity Press is such a publisher.  See feature list Ubiquity Press has provided to the  JAMIA OA exploration taskforce.

Detailed financial and operational terms would require a conversation about AMIA’s goals and requirements, but Ubiquity Press has offered example terms.

Financial implications (rough estimates: bespoke agreement would be based on detailed discussion):

  • A suggested article-processing charge (APC) of $200

  • At an APC of $200, UP production and hosting would cost between $1,000 per year (production and hosting only) to $15,000 per year (production and hosting plus peer review management, copyediting and marketing)

  • print-on-demand is available and could be a source of revenue for the journal

  • back issue conversion is available, with cost TBD


– others

Wiley Open, Elsevier, Copernicus, and MedKnow would also be interested publishing JAMIA as an OA journal.  They require a detailed conversation on JAMIA’s needs before suggesting financial terms.  Wiley Open and Elsevier would both be willing to publish with a CC-BY license (Copernicus and MedKnow: unknown).

  • Wiley details: fully gold OA journals pricing ranges between $1650 and $3500 at the moment.  APC’s are set on a range of factors including field of research and level of funding, IF, brand and positioning of the journal, pricing of other titles in the discipline, etc.  Could be flexible in terms of model based on discussion of needs and goals.

  • Copernicus details: We finance our Open Access publication services by charging Article Processing Charges (APCs) on page basis from the authors. Therefore, these APCs depend on the length of the article as well as on the journal format (traditional vs. interactive). In most of the cases this results in the APCs range between €500 and €1000 per article.   If a journal is already established, and has an Impact Factor, as JAIMIA has, we suggest to start with reduced APCs and then by stages raise them to the full level. However, the journal owners have to decide upon the range of the APCs because they would need to subsidise any reduction.


5. Publish JAMIA within AMIA

– hosted internally

AMIA could run an online-only JAMIA journal itself, using self-hosted open source journal management software.

Financial implications can be assessed based on expenses and time estimates in


– hosted by external sources

AMIA could run an online-only JAMIA journal itself, using externally-hosted open source journal management software.

Financial implications can be assessed based on expenses and time estimates above, and also these estimates for external hosting:


6. Other

Other ideas that are probably out of scope:


– related revenue and cost-saving possibilities

  • could move to print-on-demand, or eliminate print

  • could release openly, have HTML available for free, but charge per-article or membership fee for PDF access (ie JMIR)

  • could charge for expedited peer-review (ie JMIR)

  • could charge submission fees in general

  • could charge for iphone apps, etc


Misc Resources:

May 6, 2013

resources to argue for strong funder data archiving policies

Filed under: Uncategorized — Heather Piwowar @ 7:44 am

I was recently asked for a quick list of resources to make the case that government funders should have strong data archiving policies.

Here’s my quick response.  Since I’m certainly missing things, and the list is likely to be dated quickly, please add more links in the comments!

The main arguments I’d made are in the White House OSTP response Todd and I put together two years ago, advocating that the US impose stronger data policies.  It has some references at the end.

The OSTP received 118 comments: most are carefully written, thoughtful, and worth reading.

Federal policies of particular note, and a few recent statements that came out after we wrote the above letter:

There’s strong stuff to talk about with respect to medical data in the current UK AllTrials initiative.

That’s what I’ve got off the top of my head.  Hope it helps!

March 13, 2013

Why may Google textmine but Scientists may not?

Filed under: Uncategorized — Heather Piwowar @ 1:50 pm

I recently posted about why Google is not a good enough solution for searching the academic literature (because can’t build on the results! and read the comments on that post for more).

It is sad indeed, then, that PMC and Publishers forbid scientists and others from spidering/indexing/mining their content…. while giving Google privilege to do exactly this.

Check out the robot.txt files for PMC  for /pmc/articles/  and notice that GoogleBot is allowed, Bing and a few others are allowed, but User-Agent:* (the rest of us) are not.  The same is true for ScienceDirect robots.txt:  Google may textmine everything, experimenting scientists, nothing.  (hat tip to Alf Eaton on twitter)

Is this defensible on the grounds that Google knows what it is doing but The Rest Of Us Can Not Be Trusted?  I sure hope not.  Scientists are routinely trusted with a lot more than writing a script that won’t bring down a server.  There are other ways to ensure someone won’t bring down a server than a global robots.txt ban.

Perhaps a ban is the only way to prevent unauthorized redistribution of large numbers of papers gathered via spidering?  Nope.  Require people to register.  Monitor use.  Clearly state what may be redistributed, what may not, and what actions will be taken if people behave badly.

Maybe they are just waiting till Scientist-initiated indexing projects gets Big and Important and Ask Nicely and then they will write them in as an allowed user.  Maybe.  But restricting play and experimentation is a pretty poor way to bring about that future and we should not accept this as the default behaviour of the keepers of our scientific literature.

PMC calls its prohibition against bulk downloading a “copyright” issue.  That doesn’t make any sense to me.  Sounds much more like a Terms of Use issue than a copyright issue.  Am I wrong?  If so, educate me in the comments.  If I’m right, then I think we should ask PMC to change its wording because calling this a copyright issue just muddies already muddy waters.

It does appear to be, at least in part, a contract issue.  In the contract between publishers and PMC ( point 16, ht again to Alf Eaton), PMC volunteers in its terms that PMC will prohibit bulk downloading.  Why does PMC include this sentence?  Is it part of the NIH Public Access law that PMC has to include this sentence?  If not, isn’t it capitulating an awful lot to publishers… basically undermining the ability for scientists to build enhanced searching tools, etc?

(and, how, given this, does Google get access?  Don’t get me wrong.  I think Google is fantastic!  I want Google to keep having access!  I just want all responsible systems to have the possibility of the same access to our publicly funded and hosted research, so that someone will build infrastructure that properly supports research and research tools.)

Anyway, these spidering policies strike me as unfair, and something that people should be talking about and complaining about and doing something about, especially as we start to craft new policies for how people and computers can access our Public Access research output under the new OSTP policy.

March 5, 2013

Why Google isn’t good enough for academic search

Filed under: Uncategorized — Heather Piwowar @ 6:49 am

People often ask: why all the fuss about Search for academic papers?  Google does a fine job, we can find everything we need, what’s the problem?

I gave an answer to this in a comment on Mike Taylor’s blog and it got a bit of twitter pickup, so reposting my comment here for this audience.  Summary:  no one can build on the results!  

Google isn’t an acceptable answer to Searching across academic papers (toll access, green OA, gold OA, whatever) because it doesn’t support a way for people to digest the search results, add value, and apply the results in new and innovative ways. Google search results can only be used on Google’s website manually, or embedded as-is in other websites.

Neither Google nor Google Scholar offer an API — for love nor money, as far as I can tell, point me to it if I am wrong — that would let us do a Google Search and then sort/filter/enhance the results to add value and use in research and in scholarly tools.

Totally unacceptable as a search solution for the scholarly literature.  Think of the opportunity cost to research and research tools, and all the things that better research tools facilitate.

It doesn’t have to be this way.  Search results can be openly available for reuse (see the search APIs and API terms of use for PLOS, PMC, etc).

February 28, 2013

Do your review instructions ask if data+software are available?

Filed under: Uncategorized — Heather Piwowar @ 5:33 pm

It looks like PLOS Biology doesn’t ask reviewers to help uphold their data availability policies…. and I’m sure they aren’t the only journal missing this step.

I just send this email to PLOS Biology.  When you review a paper, check the material you are sent to see if you are asked to assess appropriate availability of materials, and if not (or not with sufficient emphasis) please make your voice heard.  You are welcome to use my dashed-off email as a template if it helps, needless to say.

Hi PLOS Biology,

I’m reviewing a paper for you now.  I’ve just realized that your email to reviewers contains several important prompt questions, but no prompts asking us whether data+software have been made appropriate publicly available, as per PLOS guidelines or community norms, whichever are stricter.

Sections 5 and 6 in your reviewer guidelines don’t cover this either… actually it doesn’t seem covered by your reviewer guidelines at all.

Your author instructions say:  “All appropriate datasets, images, and information should be deposited in public resources”… but there does not appear to be any reviewer check?

Seems a pretty big lost opportunity: reviewers are very well placed to make recommendations about what data should be made available.  The “detailed protocols” mentioned in your reviewer guidelines are unlikely to suggest datasets or software to most people.


a big fan of data
Update:  a few days later, PLOS Biology responded as such (and gave me permission to post their response), with a CC to four internal employees:

Dear Heather

Many thanks for raising this issue with us. We are actually working on two fronts that will, we hope address your concerns in the near future (although not absolutely immediately). One is a general review of our policies, instructions and guidelines for the PLOS journals around data issues, and the other is improvements to the instructions and forms we use with reviewers. Both of these give us good opportunities to improve what we ask for and stipulate around data, which as you suggest is not yet optimal. It would be great if you let us know of other opportunities you think we’re missing, or any other suggestions you have in this area.



and I replied:

Thanks for the response, Theo!

This all sounds good, though I hope you don’t hold off on easy small improvements (adding a sentence or two to reviewer instructions to ask whether existing author instructions on data have been followed) until large changes are thoroughly designed and implemented.  [..]

Summary:  journals want to hear from us.  It is definitely worth the time to raise these issues. Please write to your journals too!

January 16, 2013

ResearchFish: CVs with alternative products

Filed under: Uncategorized — Heather Piwowar @ 9:14 am

I received an email today and have been given permission to post it here to help spread the word.   See below (emphasis is mine).

Looks like ResearchFish is useful for funders and universities, and free for researchers to generate CVs that include alternative products.  Cool!  I do think the generated CV line items need some ImpactStory badges, what do you think?  :)

Dear Heather,

Please find enclosed a letter from Frances Buck, Director of Researchfish, in response to your article published in Nature on 10th January. We have submitted this to the Correspondence team.

With best wishes, Rebecca


Nature’s article, Altmetrics: Value all research products (493,159, 10 January 2013), by Heather Piwowar, suggests that funders are mainly interested in research papers when assessing grant applications.

While publications undoubtedly help demonstrate the impact and significance of research projects, it would be wrong to suggest that this is the only factor funders consider when reviewing a researcher’s contributions — indeed the MRC has been pivotal in promoting a broader approach.

Working in collaboration with Researchfish and five other major medical research charities in the UK, the MRC have developed a new online facility that enables researchers to comprehensively record the outcomes of their work. Funders can then review and easily evaluate this information. Researchfish’s portal is currently being used by 16 funding agencies, and over 6,500 PIs have already signed up, recording a wide range of products, from publications, to intellectual property and patient outcomes.

Frances Buck

Director Researchfish St John’s Innovation Centre Cowley Road Cambridge CB4 0WS, UK

January 11, 2013

Process behind a Nature Comment

Filed under: Uncategorized — Heather Piwowar @ 8:55 am

Publishing a Comment in Nature involved a process unlike any I’ve experienced to date, so I figured I’d document it (Comment itself is here).   I wish more people would document the story behind their papers (and #OverlyHonestMethods :) ), and also the process behind their scientific communication to help us all peek behind the curtains.  Or, yknow, take down the curtains.


I received an email from a Nature editor on November 1:

[..] I’m an editor in the Comment section at Nature, which features opinions by scientists. [..] I’m writing because a few issues have popped up that we thought you might have some insights on [..]

We’re interested in exploring a piece about the NSF’s decision to change “papers” to “projects” in scientists’ list of achievements. [..]

Does this topic spark any interest? If so, let’s chat – we’d want to time something to the first of the year, when the NSF change goes into effect. [..]

(I won’t name the editor because I don’t want to catch her unaware…. I’m not sure if it is appropriate to name her, so I’ll err on not.  She was very skilled and pleasant to work with, fwiw!).

Needless to say, the proposed topic is of interest to me and a Comment in Nature seemed like a great way to reach a broad and “traditional” audience with my thoughts on where this is going.  We set up a phone call for later in November.

Writing and Editing

The editor and I had a 15 minute call about my thoughts on the topic, and also about how Comments work.

I mentioned that I’d recently given a brief talk about implications of the NSF Biosketch policy change.  She suggested I send that along to her, and she’d reply with a paragraph-by-paragraph suggestion on how I compose a first draft of the comment.

The editor sent me a reply that had a surprisingly detailed outline:

Starting with the text you sent about your talk is great – it’s a good tone and level for our readership. We can just build on that. [..]

First paragraph: “hook” the reader. Like feature and news stories, or even editorials in a newspaper (which is really our model here), we need something that will “grab” the reader, make them want to [..]

Next 1-2 paragraphs: Describe the NSF change in policy, for readers who aren’t familiar with it [..]

Third paragraph: Present the crux of your argument: I think this change in NSF policy, along with other examples mentioned, indicate X [..]

Background, 2-3 paragraphs: Present examples of the changes [..]

Next 2-3 paragraphs: Explain more why these changes are so significant for science. Here is where you’ll put [..]

Final paragraphs: Here, we present “solutions.” How should things change further? What direction would [..]

Wow!  ok, sure, if that is how it works, I can do that.  So I pulled together a first draft, which I’ve posted here.  That’s when it got intense.  I’ve never had anything so heavily edited.  In addition to emailing drafts back and forth, we had two (or three? I forget) quick phone calls where the editor asked me clarification questions, then she’d send me another draft.  It took five revisions till it was time for her to pass it to her boss and the subeditors.

The subeditor was also great.  The subeditor sent me a revised version, and at this point it was layed out as a PDF.  I had a list of changes to maintain accuracy given the new edits.  There were about 3-4 more versions after this, with small changes.

Overall, I’d say this whole process made the resulting paper much more readable than it started.  It also changed the focus a bit, to having a stronger altmetrics focus, rather than being primarily about the alternative products.  I’m ok with that, though I do mourn some of the details in the original draft that didn’t make it into the final version.  I do kinda feel like the editor should be a coauthor, for what it is worth…. I think we’ve all had coauthors who did less than she did!  Feels a little strange that there is so much behind-the-scenes help in crafting these articles and that isn’t transparent at all.

One area I had no clear say was the title and subheading.  It went through 2-3 titles and 4-5 subheading phrases and locations in the versions I saw.  I did object to one of the versions (“creeping changes”), but in general it wasn’t clear that the title was my decision.  I didn’t know that the title in the HTML version and CrossRef was going to be prefixed with “Altmetrics:” because the title on the PDF copy I saw was simply titled “Value all research products”.  I’m a little unhappy about the leading “Altmetrics:” because I think it complicates the main thrust of the piece and makes it easy for people to get tangled, for example about whether blog posts are alt-products or sources of altmetrics (answer:both).  oh well, that’s ok: altmetrics is sexy, it makes sense to lead with it, and I’m certainly a big believer!


Because the article was due out just after the holiday break, with a fixed publication date to coincide with the new policy implementation on Jan 14, the turn-around time I had for many of these revisions was very short (10 days for the initial draft, a few days for revisions, near the end less than a day for final revisions).  This was fine with me, I just note it so that others will know what you are getting into.

Copyright and Paywall

The other point I want to mention here is how Copyright works with Comments.   I admire Nature’s policy for copyright for research articles, given that they are a non open-access journal:  they do not require that authors sign away their copyright, instead they ask that authors grant Nature an exclusive license to publish.

Nature has a different policy for Comments.  You have to sign away your Copyright to Nature.  As a huge proponent of Open Access, I thought long and hard about whether I was ok with this.  I decided for this editorial content I was.  Happy to discuss :)

Here is the form that I signed. UK Comment CA  I upload it because I did not sign an NDA, and I know that I would have liked to find it online when I was first contacted by them to help me understand details of the agreement I’d be entering.

The first editor who contacted me knew that I am a strong supporter of OA.  Though she said that it would not be possible to make this comment OA, she said that we could nominate it to be one of the “free” articles.  I held fast to this, and in mid December requested with her that we do indeed make this request if she hadn’t already done so.  She was happy to do so, and asked me for a once sentence justification for why this paper should be freely available, because that is used by the group who makes the decisions.  Not sure I knocked this out of the park , but fwiw here’s what I sent:

People will likely circulate this article outside academia, since altmetrics is about valuing broad contributions to science, and broad interactions with science — high school viewers of wetlab YouTube videos, silicon valley dotcom contributors to science source code repositories, etc.

The good news is that they did decide to make my article free for “at least a week.”  It wasn’t free when it first went up, interestingly, but the paywall page stopped appearing within 12 hours.

One more thing for completeness.  I’ve heard some people are paid small amount for Comments?  I’m not sure if that is true or not.  In any event:  money was never mentioned to me, I wasn’t paid anything.

So there ya go.  Now you know everything I know about how Nature Comments work.

January 10, 2013

First draft of just-published Value all Research Products

Filed under: Uncategorized — Heather Piwowar @ 9:00 am

The copyright transfer agreement (arg) I signed for the Comment in Nature included restrictions on where I may post a copy of the article:

Although ownership of all Rights in the Contribution is transferred to NPG, NPG hereby grants to the Authors a licence […]
c) To post a copy of the Contribution as accepted for publication after peer review (in Word or Tex format) on the Authors’ own web site, or the Authors’ institutional repository, or the Authors’ funding body’s archive, six months after publication of the printed or online edition of the Journal, provided that they also link to the Journal article on NPG’s web site (eg through the DOI).

The article is available for free for a week or two on Nature’s site, and I’ll post the text here as soon as I can, six months from now.

In the meantime, as per contract lingo above, I may post the first draft that I sent the Nature editors.  So here is the first draft, for the benefit of those who are looking for a free version in the first half of 2013, and for anyone who cares to compare the first draft to the final draft :)   [Hint: there were MANY rounds of editing.  more on that in next post…. ]

NSF policy welcomes alt-products, increases need for altmetrics

(or perhaps NSF welcomes bragging about software, datasets in proposals)

Research datasets and software no longer have to masquerade as research papers to get respect.  Thanks to an imminent policy change at the NSF, non-traditional research products will soon be considered first-class scholarly products in their own right, and worth bragging about.  This policy change will prove a key incentive to produce and disseminate alternative products, and have far-reaching consequences in how we assess research impact.

Starting January 14th, the NSF will begin to ask Principal Investigators to list their research Products rather than Publications in the Biosketch section of funding proposals.  Datasets and software are explicitly mentioned as acceptable products in the new policy, on par with research articles.

The policy update reflects a general increase in attention to alternative forms of scholarly communication.  Policies, repositories, tools, and best practices are emerging to support an anticipated increase in dataset publication, spurred, in part, by now-required NSF data management plans.  Tools for literate programming, reproducible research, and workflow documentation continue to improve, highlighting the need for shared software.  Open peer review, online lab notebooks, post-publication discussion — as it gets easier to “publish” a wide variety of material online it becomes easy to recognize the breadth of our intellectual contributions.

I believe in the long run this policy change from Publications to Products will do much more than just reward an investigator who has authored a popular statistics package.  It is going to change the game, because it is going to change how we assess research impact.

The change starts by welcoming alternative products.  The new policy welcomes datasets, software, and other research output types in the same breath as publications: “Acceptable products must be citable and accessible including but not limited to publications, data sets, software, patents, and copyrights. Unacceptable products are unpublished documents not yet submitted for publication, invited lectures, and additional lists of products.”  In contrast, previous versions of the Biosketch instructions policy allowed fewer types of acceptable products (“Patents, copyrights and software systems”) and considered their inclusion to be a “substitution” of the main task of listing research paper publications.

The next step will become apparent when we consider what peer reviewers will want to know when they see these alternative products in a Biosketch.  What is this research product?  Is it any good?  What is the size and type of its contribution?  We often assess the quality and impact of a traditional research paper based on the reputation of the journal that published it.  In fact the UK Engineering and Physical Sciences Research Council makes this clear in its fellowship application instructions: “You should include a paragraph at the beginning of your publication list to indicate … Which journals and conferences are highly rated in your field, highlighting where they occur in your own list.”

Including alternative products will change this: it necessitates a move away from assessment based on journal title and impact factor ranking.  Data and software can’t be evaluated with a journal impact factor — repositories seldom select entries based on anticipated impact, they don’t have an impact factor, and we surely we don’t want to calculate one to propagate the poor practice of judging the impact of an item by the impact of its container.  For alternative products, Item level metrics are going to be key evidence for convincing grant reviewers that a product has made a difference.  The appropriate metrics will be more than just citations in research articles: because alternative products often make impact ways that aren’t fully captured by established attribution mechanisms, alternative metrics (altmetrics) will be useful to get a full picture of how research products have influenced conversation, thought, and behaviour.

The ball will bounce further.  Once altmetrics and item level metrics become expected evidence to help assess the impact of alternative products, the use of item-level altmetrics will bounce back to empower innovations in the publication of traditional research articles.  Starting a new or innovative journal is risky: many authors are hesitant to publish their best work somewhere unusual, somewhere without a sky-high impact factor.  When research is evaluated based on its individual post-publication reception, innovative journals become attractive, perhaps competitively more attractive than staid established run-of-the-mill alternatives.  Reward for innovative journals will result in more innovations in publishing.  Heady stuff!

A few large leaps are needed to realize this future, of course.  First, this one policy change hardly represents a consistent message across the NSF.  Accomplishment-Based Renewals are still based on “six reprints of publications”, with no mention of alternative products.  Even in the Grant Proposal Guide, the same document that houses the new Products policy, the instructions for the References Citations section are written as if only research articles would be cited in a grant proposal.  What about preliminary data on figshare, or supporting software on RunMyCode, or a BioStar Q&A solution, or a patent, or a blog post, or, for that matter, an insightful tweet?  If we think these products are potentially valuable, the NSF should welcome and encourage their citation anywhere it might be relevant.

The second hurdle is that a policy welcoming the recognition of alternative products is not yet common outside the NSF.  A brief investigation suggests that many other funders — including the NIH, HMMI, Sloan, and UK MRC– still explicitly ask for a list of research papers rather than products.  A few, like the Wellcome Trust and UK BBSRC just seem to ask broadly for a CV, leaving the decision about its contents to the investigator.  This could be good, but because investigators are not used to considering alternative products to be first-class citizens, explicit welcoming is important to drive change.

The third challenge between us and a new future brings us to an exciting area under active development.  When products without journal title touchpoints start appearing in BioSketches, how will reviewers know if they should be impressed?  Reviewers can (and should!) investigate each research product itself and evaluate it with their own domain expertise.  But what if an object is in an area outside their expertise?  They need a way to tap into the opinion of expert in that domain.  Furthermore, beyond the intrinsic quality of the work, how will reviewers know if the Intellectual Merit has indeed been impactful on scholarship and the world, and thus should lend credence to the proposal under consideration?

Many data and software repositories keep track of citations and download statistics.  Some repositories, like ICPSR, go a step further and provide anonymous demographic breakdowns of usage to help us move beyond “more is better” to an understanding of the flavour of the attention.  This context will become richer as more types of engagement are added:  is the dataset being bookmarked for future use?  Who is cloning and building on the open software code?  Are blog posts be written about the contribution?  Who is writing them and what do they say?

Tools are available today to collect and display this evidence of impact.  Thomson Reuter’s Data Citation Index aggregates citations to datasets that have been identified by data repositories. identifies blog posts, tweets, and mainstream media attention for datasets with a DOI or handle: try it out using their bookmarklet.  The nonprofit organization ImpactStory tracks the impact of datasets, software, and other products, including blog and twitter commentary, download statistics, and attribution in the full text of articles: give it a try.  I’m a cofounder of ImpactStory: we as scientists need to go beyond writing editorials on evaluation and actually start building the next generation of scholarly communication infrastructure.  We need to create business models for infrastructure that support open dissemination of actionable, accessible and auditable metrics for research and reuse.

Finally, the practice shift to value broad impact will be more rapid and smooth if funders and institutions explicitly welcome broad evidence of impact.  Principal investigators should be tasked with making the case that their research has been impactful.  Most funders, including the NSF, do not currently ask for evidence of impact.  This may be changing: the NIH issued an RFI earlier this year on BioSketch changes that would include documenting significance.  In the meantime, the lack of an explicit welcome hasn’t stopped cutting-edge investigators from augmenting their free-form CVs and annual reviews to mention that their work has been “highly accessed” or received a F1000 review.  This — and next generation evidence with context — should be explicitly welcomed.

Despite these hurdles, the future is not far away.  You and I can start now.  Create research products, publish them in their natural form without shoehorning everything to look like an article, make citation information clear, track impact, and highlight diverse contributions when we brag about our research.  We’re on our way to a more useful and nimble scholarly communication system.

Just published: Value all research products

Filed under: Uncategorized — Heather Piwowar @ 8:05 am

A Nature editor contacted me in November, asking if I’d like to write a Comment about the upcoming NSF policy change in Biosketch instructions.  It sounded like a great chance to talk about the value of alternative research products with a wide audience, so I agreed.  The comment was published yesterday and is now available here:

Piwowar H. (2013). Value all research products, Nature, 493 (7431) 159-159. DOI:

Because of Nature’s policies about copyright assignments for Comments, the comment is not open access and it is behind a paywall.  Arg.  That said, I requested that it be one of their “free” articles and they agreed, so it will be freely available at the above link for a week or two.  I will post the text up on my website as soon as I am able, 6 months from now.

Working on a blog post about the process behind the scenes, because it was certainly unlike anything else I’ve published to date!

Questions about the piece, or thoughts or opinions?  Welcome below, or on twitter to @researchremix.

« Newer PostsOlder Posts »

Blog at