Research Remix

A view of the rights and responsibilities of the NSF wrt data

Heather Piwowar — Thu, 12 Jan 2012 04:04:17 +0000

The US government has asked for our thoughts on how the NSF and other federal agencies should disseminate research results.

I come to the question with a passion for the effective use of research results, a commitment to practical solutions, and naivete about how the NSF actually works. From this perspective I offer my current rough thoughts on what the NSF — and all public science funders — should do with regard to dissemination of digital research results (= datasets, code, and publications, as we know them and in their future incarnations). I wrote them down to help me think through my responses to the RFIs.

So far the number of online responses to the RFI for data has been very low. Answering open-ended questions is difficult. Critiquing a straw man is easier, a lot more fun, and often just as revealing. Where does the vision below differ from yours? It is interesting, for example, to compare it to the vision articulated in this response — also from proponents of open research data.

Do you have strong opinions? You can respond to the White House till Jan 12th and the National Science Board till Jan 18th. Feel free to reuse text and ideas directly from here for agreement or critique.

A vision for NSF infrastructure for digital research results
(considering digital research results to include datasets, code, and publications — as we know them and in their future incarnations)

Principles

A public science funder has both a right and a responsibility to communicate its findings in the most generative form it can. Projects funded with public money must be conducted under this premise.
Effective communication of research results requires strong statements of principles, enforceable policies, and useful infrastructure.
Individual disciplines and communities can opt-out of funder-wide approaches if they make a strong public case that the principles and goals are not applicable to their area, or that they plan to achieve the same goals in a different but equally-effective way.
Costs for disseminating research results in accordance with a funder’s requirements should be included and funded as part of the cost of doing a research project. This money should be used by investigators to pay for publication services — including editing, registering, reviewing, certifying, hosting, publicising, and preserving — in a competitive marketplace.
Anticipated benefits from disseminating a project’s research results should be included in proposal evaluation. Disseminating research results is the responsibility of the PI. Research results that have not been disseminated in accordance with policy will not be acknowledged as output of the grant for the purposes of evaluation. Measured impact of a research results, interpreted broadly, will be used for evaluation of the project and its investigators.
The intellectual property rights of researchers and institutions must be respected, but can not infringe on the rights of the public and the scientific community to replicate and build on funded research findings in a timely manner.
A science funder and its infrastructure for reporting research results must be nimble to stay abreast of changing norms, needs, and technologies.
The effects of adopted policies and infrastructure tools must be systematically monitored and adjusted accordingly. Decisions should be informed by evidence whenever possible. A funder should fund collection of the relevant, actionable evidence it needs to make decisions on policy and tools: research for more effective research.
Infrastructure should use existing commodity software when it meets the needs at a competitive price, with a preference for open source software. When funders develop their own infrastructure software it should be open source to allow outside contributions, customizations, and tailored solutions.
Science communication infrastructure should be open to findings from all funders whenever incremental costs can be recovered.

Policies (to be enacted after 2 years notice)

Immediate online open access to the published article-of-record and the data and code that support its findings. Embargoes or exceptions may be granted at the discretion of the program officer, especially for sensitive information such as human subject data or the location of endangered species.
The openly available article-of-record, data, and code must be registered with the funder to be considered research findings of the grant.
Research results must be made openly available to any one, for any purpose, with the sole condition of appropriate attribution.
Articles, data, and code must be available for 50 (?) years after completion of the research project.
Publicly funded websites and software must report use of research results through funder-compatible impact-tracking infrastructure.

Infrastructure
This is infrastructure that funders ought to obtain and directly and continuously fund.

Unique identifiers. Unique IDs for each investigator, grant, institution, and licence variety. Also unique IDs for each research result (see below). Some initiatives are already underway. There should be inter-funder and international coordination.
An open registry of research results. Investigators (or their publishers) must register research results here immediately upon article publication. Data and code that support the article findings must be registered at the same time (or data and code may also be registered without of an associated article).
At a minimum, a record includes fields for: research object type (article, data, code), research object ID, license, grant ID, investigator IDs, institution IDs, publisherID, an abstract, keywords about the topic and methods, metadata about embargos or exemptions, and links to other directly associated research object IDs. The research object ID must resolve:
1. for articles, to the full text of the article-of-record (in XML format or similar)
2. for data, to the full dataset and associated metadata
3. for code, to the snaphot version of the code
4. The registry must have a read/write API, and would ideally be extensible through third-party client-based add-ons to support discipline-based customization.
A mechanism to gather and report raw data on the impact of research findings. This could be done by adopting and hosting an open-source web analytics solution like Piwik (http://piwik.org) and requiring that all funded initiatives report usage statistics for activity (reading, commenting, remixing, etc) involving research results.
Impact metrics would be also be populated by machine extraction of citations from the article text linked to through the open registry of research results, thereby providing Open access to citation data that the commercial sphere has not facilitated. Funder customizations to the source should be remain open source to facilitate additional contributions and reuse. The raw impact data should be made widely available to fuel innovative products in discovery and filtering of research reports, next generation bibliometrics assessment, and policy evaluation.

Also:

Dissemination and preservation of research results. Although successful discipline-wide models already exist for papers (journals and conferences) and some types of data, many disciplines lack appropriate data repositories and few disciplines have determined appropriate hosting and archiving solutions for code. Research funders should provide seed funding for such infrastructure until it can survive as a sustainable service like those discussed below.

It is assumed that scholarly societies, non-profits, and the commercial sector will continue to offer value-add services such as editing, mark-up, layout, organizing pre-publication peer-review, certification, dissemination, and archiving. These are envisioned as services that would be paied for with line-item funds from the project research budget for publications, data, and code. It is unlikely that funders will need to fund infrastructure in this services sector, other than occasional seed funding for innovative approaches.

Money
The NSF should compute the subscription fees paid by institutional libraries to gain access to NSF-funded research today (for example, total subscription fees X (number of articles/number of articles reporting on NSF funded research projects)).

Starting two years from now (synchronized with the requirement that all final research products be openly accessible, and the expectation that all research projects include budget items for publication services) it should decrease the indirect costs by this amount.

Not included in the articulation of this vision, mostly due to time:

Is it worth it to archive Very Large Data? Dryad can archive up to 10Gb per paper…. this covers the data that supports findings for most investigator-driven research, but not all. I don’t know what to do with the really big stuff. GigaScience model?
How this ought to sync up with availability of research proposals.
Training.
Advocating for challenges and prizes.

Other notes:

goal isn’t availability, it is use
focus on both replication *and* reuse. Neither is sufficient alone.
focus here on data associated with final research findings. Much to suggest that other datasets should be made available for reuse as well!
lots more that needs to be done about annotating the kind of contribution a citation makes to a research result.
is 2 years the right amount of time before goes into effect? 50 for preservation? I don’t know, but they are the right order of magnitude.
embargos should be used to permit exclusive use by investigators when appropriate during this transition period (particularly for longitudinal datasets that take years to collect), active IP explorations, etc. The length of the embargo should be as short as is reasonable and must be specified at the time of research reporting.

Research Works Act attacks data dissemination too

Heather Piwowar — Sun, 08 Jan 2012 06:23:42 +0000

Sponsors and supporters of the Research Works Act keep claiming that it doesn’t cover “the raw data generated by government-funded research” [1] or “raw data outputs” [2]. That’s not what I get from a direct reading of the bill. The relevant sentence reads:

“Such term does not include progress reports or raw data outputs routinely required to be created for and submitted directly to a funding agency in the course of research.”

This doesn’t say it excludes raw data. It says some complicated thing about it excluding raw data that is routinely required to be submitted directly to the funding agency. There aren’t many NIH or NSF program officers I know who want me to routinely sent them all my raw data, and many data repositories are not hosted at funding agencies.

This means that practically all “published” research datasets (including those in tables, supplementary information, and presumably non-federal data archives) are subject to the Research Works Act. On purpose or because of poor sentence structure? I don’t know, but the effect is the same.

Read the bill again with data in mind [comments added]:

No Federal agency may adopt, implement, maintain, continue, or otherwise engage in any policy, program, or other activity that–

(1) causes, permits, or authorizes network dissemination of any private-sector research work [government-funded investigator dataset] without the prior consent of the publisher of such work [who would be considered publishers? publishers of the articles that describe the data collection? data archives too?]; or

(2) requires that any actual or prospective author, or the employer of such an actual or prospective author, assent to network dissemination ["distributing, making available, or otherwise offering or disseminating a private-sector research work through the Internet or by a closed, limited, or other digital or electronic network or arrangement"] of a private-sector research work [government-funded investigator dataset].

Insane. The government can’t *permit* (let alone require) the *making available* of data *on a closed digital network* (let alone the internet) without the publisher’s approval? Data they funded collecting?

ok I think writing that down has disgusted me so much that I’m done thinking about the Research Works Act for now. It is just insulting.

Stand up against the AAP and everyone who supports this.

[1] http://www.michaeleisen.org/blog/?p=807#comment-52048

[2] https://twitter.com/#!/DarrellIssa/status/155023586040627200

What should the publishers lobby for?

Heather Piwowar — Sun, 08 Jan 2012 04:41:16 +0000

The Research Works Act is a very poor move by traditional publishers.

Publishers will come out strongest if they side with the future: the future is immediately open, reusable, all-value-added versions of research results and peer reviewed publications.

Sure, they should lobby to shape government policy to their favour. That’s their right and maybe even their responsibility to their stakeholders.

What *should* traditional publishers fight for, to stay on the right side of both history and their balance sheets?

Time. They should insist that any federal mandate that requires the article-of-record be made openly, immediately available does not take effect for a year or two, to give themselves time to change their business models (to author/funder pay-on-publish or pay-on-submit or some other method, thereby saving their companies and jobs).
Access to publication funds for federally-funded authors. Publication costs are already available to NSF and NIH awardees as budget line items in grants. I don’t know if all other federally-funded investigators have access to author-pays grant money… if not, publishers should argue that access to these resources must be a condition of a mandate. There must be a creative way to redirect money which payed for university library subscriptions into university OA funds or federally-disseminated research distribution reimbursement (has anyone proposed such an approach yet?)…. publishers should lobby for this.
Measurement of the impact of the papers they publish. When research papers are openly distributed, redistributed, deconstructed, and mashed up it becomes much harder for publishers to understand (and therefore brag about, and then capitalize on with higher publication charges) the impact their publications have had vis-a-vis their competitors. Publishers could insist that all federal hosting services report back usage stats (as PubMed Central does), and lobby for requiring a manner of attribution that facilitates easy and robust impact tracking (beyond just mention or citation).

I’m not saying I think they should get all of these things, necessarily… but this would be a much more constructive, productive and supportable stance.

What do you think they should fight for?

Threat of job loss as motivation for Research Works Act: real or fear-mongering?

Heather Piwowar — Sat, 07 Jan 2012 22:21:41 +0000

I find it difficult to silently read the AAP endorsement of the Research Works Act. Multiple readings later I still find myself yelling aloud at its twisted positions and statements.

Most of my rebuttals take the form of expletive + “no, that isn’t true” or “give me a break” + a silent prayer that policymakers won’t take the positions seriously.

But what about the AAP threat that the industry and 30,000 jobs are in danger if the government supports, requests, or requires more open scholarship? I suspect many politicians in the US government will take that very seriously. Threatening job losses in the US right now is a Great Card, and publishers keep bringing it to the table.

The Jobs Card makes me furious because *if* scholarship can be done better it *should* be done better, full stop. But with the government really considering about these issues and soliciting feedback, it is time to get beyond that and see if, you know, it happens to be true that 30,000 US jobs are in jeopardy. If so, we need to figure out how to address that in our discussions, because we can be darn sure the government will balance that against the benefits of openness.

First, what is at risk? Scholarly publishing as an industry isn’t going away any time soon: physicists have arXiv and they still publish in journals. What is at risk is the traditional subscription-based business model. If the scholarly article-of-record is made immediately available with no restrictions, there is no reason for anyone to pay subscription fees. Luckily, BMC and PLoS have already demonstrated that an author/author’s funder-pays model can work (at least for fields funded by federal money, with access to publication budget line items… the same fields that would be subject to federal mandates on openness). Traditional publishers could move to this model if they wanted to. So what is at risk is the business model, not the industry.

Second, what about the jobs? Well, the publishing industry would still need to employ people, even if it changes to a PLoS business model. Maybe fewer people, though, since PLoS has been designed from the ground up to be a streamlined operation?

Queue the Saturday-morning kitchen-table back of envelope analysis (ok, post-its).
*** btw if others have more refined or robust estimates of the job situation, please share!

In 2010, the first year PLoS reported covering its costs with revenue, PLoS published 8662 papers. PLoS has about 120 employees. That’s 72 papers/employee.

Let’s compare that to traditional publishing models. Elsevier publishes 250,000 papers per year. Elsevier has 7000 employees worldwide. That’s 35 papers/employee. However, Elsevier also works on non-article products and services, like book publishing, SciVerse software, Gray’s Anatomy, MD Consult, and conference organizing. I have no idea what proportion of their employees work on these things. If 20%, the number of employees working on articles is 5600. So 250,000/5600 = 45 papers/employee.

What does this mean?

At 45 papers/employee, publishing 250,000 papers at a PLoS-level of efficiency would take 250,000/72=~3500 employees. We estimated that Elsevier has 5600 worldwide employees working on research articles now, so 2100 more than would be necessary at PLoS levels of efficiency. Would these people have to be let go? Heck no! Publishers often argue that they offer a better product than the bare-bones PLoS offering — those 2100 people are doing something useful — so they could charge more money for their product than PLoS! Compete! Tout your high impactness and careful copyediting! $500 more per article would do it (250000*500/2100=$60k). Totally reasonable, and much less than many existing OA fees.

Let’s think about how many jobs it really is in the USA. Björk et al estimates there were about 1.35 million articles are published per year in 2008: let’s call it 1.5 for 2010. 1.5 million papers/72 employees per paper=20,100 scholarly communication jobs would be required to publish that many papers, at a PLoS levels of efficiency. 1.5 million papers/45=33,333 jobs would be required at an Elsevier level of efficiency. The difference is 13k jobs worldwide. Worldwide. AAP claims that 45% of worldwide scholarly publishing is by North American science publishers. Let’s say US science+non-science publications is 50%, maybe? 50% of 13k jobs =~ 7,000 jobs. I reiterate that these jobs do NOT have to be lost. Presumably these employees are indeed creating value (through more thorough copyediting or peer-review routing or paper-article-publishing or whathaveyou) then the publishers could (and should) CHARGE EXTRA and pay for these jobs that create this added value.

But let’s give a bit of perspective to 7,000 jobs. Google “jobs lost USA.” This was the top hit when I did it: nearly 1000 people out of work because a hospital is closing.

Enough said.

Publishers are fear mongering with talk of loss of jobs. As far as I can tell, there is no big risk, if publishers are willing to move with the times and embrace new business models.

The US government has no business sacrificing the progress of science and taxpayer access to maintain a particular business model.

The opportunity cost, however, of continuing with the current pathetic dissemination of research results? Don’t get me started.

Go set your government official straight about the Research Works Act and Public Access in general (due Jan 12)! Feel free to use analysis in this post with or without attribution.

If you know of other sources of evidence and analysis around this issue, on either side, please let me know.

Proposal inviting Citizen Scientists to enrich the scientific literature

Heather Piwowar — Fri, 16 Dec 2011 14:00:25 +0000

I submitted a proposal to the Citizen Science Alliance yesterday. It is really exciting! I’d briefly discussed the idea of classify-citations-to-find-data-reuse with members of Zooniverse previously: they suggested that I keep the scope small to start and focus on an area of interest to the public, like cancer. Thanks to Todd Vision for great feedback on an last-minute draft.

Here are the main bits of the proposal. Feedback welcome!

Project title
Tracking the building blocks of cancer research

Abstract
Funders and charities spend a lot of money on cancer research. Are we getting as much research progress we can get from this funding? How is cancer research used to make additional discoveries? Are there lessons we can learn from current behaviour to improve research efficiency in the future?

These questions are currently hard to answer because it difficult to identify which resources have contributed to new research. This project will invite volunteers to enrich the scientific literature by classifying literature citations to previous work. Researchers cite previous work for many reasons: only through examining citation context can we differentiate citations made to reference background information from citations that attribute the reuse of materials, methods, software, and datasets.

We propose to begin attribution tracking in a specific domain: cancer-related gene expression microarray datasets. By categorizing citations to papers that describe such datasets we will begin to understand how often these datasets are used by others, how scientists attribute reuse of these research building blocks, and what contribution the data has made to research progress. We hope these findings will help make similar information more easily discoverable in the future.

This is a great chance for citizen scientists to begin enriching the scientific literature. Subscription and licencing restrictions make most of the scientific literature off limits to automated markup. Human-aided text mining of freely available full text has the potential to extract a coherent and incredibly useful body of knowledge from literature that is otherwise unavailable for large-scale markup.

Describe the tasks you envisage citizen science participants carrying out – including (a) any required specialist knowledge, (b) minimum requirements for success, (c) desired outcomes.

Volunteers will be given a link to an article in PubMed Central and text that identifies a specific item in that article’s References section. They will be asked to find all mentions of the Reference item in the article full text, record the section, extract the sentences surrounding the citation, and classify the citation context into one of a few broad categories (cited for background information, to attribute use of method, to attribute use of data, etc). Finally, the volunteers will be asked to make an assessment about whether the cited resource played a significant role in the conduct of the reported research project.

We think questions can be phrased so as not to require specialist knowledge.

Minimum requirements for success would be annotation of 1000 citations since about 10% are likely in context of data reuse. Annotation of all citations would allow a more through analysis of patterns.

Describe the nature of the data that would be used in the proposed project – include (a) its format (filetype, size, number of files) (b) any restrictions (including copyright) on its use (c) its availability (is it archive data or still being collected).

The input data is simply a list of pairs: a URL that points to a paper in PubMed Central paired with a text string that identifies (authors, title, year, journal) a reference item known to be cited within the PubMed Central paper.

The URLs and citation text would be derived from a dataset currently hosted on the Dryad repository:
Piwowar HA (2011) Data from: Who shares? Who doesn’t? Factors associated with openly archiving raw research data. Dryad Digital Repository. doi:10.5061/dryad.mf1sd

Specifically, PubMed IDs for cancer gene expression microarray studies published in 2005 (n=792) would be extracted from the abve dataset. An equal number of studies that made their data publicly available (n=129) and studies without publicly available datasets would be retained (total n=258, http://tinyurl.com/7mkbpyq). PubMed Central reports hosting 3540 papers that cite these 258 studies. The 3540 citing papers would be randomized and a subset of 1000 would be extracted to achieve our initial analysis goals.

Because it may be against PubMed Central terms of use to embed the PMC article in a web page, we envision Zooniverse supplying a link that opens in a new tab and the corresponding Reference-Entry-To-Classify text to each volunteer.

What automatic processing routines exist which attempt to solve the problem being addressed? Why can’t they be used instead of humans?

Automated processing schemes have been developed to classify citation context (e.g. Teufel et al. Automatic classification of citation function. EMNLP 2006). It is not known how accurate these algorithms are for the specific task of identifying data reuse attribution.

The primary hurdle to automated processing is legal: publishers rarely allow full text to be harvested or used for text mining.

This proposal suggests an approach to work around these access and use limitations: leverage citizen scientists and publicly available research papers to gain large scale access to the scientific literature.

PubMed lists 804184 publications from 2009 with links to full text. Of these, 247421 (31%) have free full text, available for public view. Only a small subset of these, about 67000 (8%), are open access with full text that can be systematically downloaded and used for text mining. [http://researchremix.wordpress.com/2011/12/15/computing-availability-of-full-text-for-reuse/]

If possible, estimate the minimum number of times a task must be performed on a given element of data to be useful for science (assuming all tasks are performed by competent citizen scientists; once might be enough for exceptionally clear tasks, more times could be required for fuzzier tasks or lots may be necessary if accurate estimates of uncertainties are needed). How many total tasks must be completed before your research goals are achievable?

We believe three replicates would be sufficient, but a bit of experimentation may be needed to understand how many classifications are needed to achieve sufficient accuracy. A master’s student with a bachelor degree’s in forestry was able to complete the task accurately with little training. Five replicates achieved the necessary generalizability when we asked people on Mechanical Turk to complete a more complex task based on the same sort of papers in the biomedical literature (details: http://researchremix.wordpress.com/2008/12/29/generalizability-coefficient-for-mechanical-turk-annotations/).

Assuming three replicates would be sufficient, 1000 citations would require completion of 3000 tasks.

All data from Zooniverse projects must be eventually made public.

Are there potential extensions to the project that you have in mind?

Yes! We are excited about extensions to this project in at least three dimensions:
1. Data reuse estimates. Investigating instances of data reuse for additional years, domains and datatypes, to understand how patterns differ.
2. Open citations and citation context. The proposed project could be the first step in creating a repository of openly available citation information, ideally with semantic metadata. This would be of broad interest. As one concrete example, citation slices and dices could be included as measures of impact in http://total-impact.org.
3. Toll-access literature. Experiment with applying this approach to subscription-based literature, either by negotiating with publishers or by leveraging the subset of volunteers with university affiliations and subscriptions.

Even more profoundly, this mechanism for enriching the scientific literature will be of deep interest to a wide variety of researchers for all sorts of additional purposes.

And fun for a lot of citizen scientists too, I think! :)

Computing availability of full text for reuse

Heather Piwowar — Thu, 15 Dec 2011 15:10:56 +0000

Rough estimate:

PubMed lists 804184 publications from 2009 with links to full text. Of these, 247421 (31%) have free full text, available for public view. Only a small subset, about 67000 (8% of all publications), are open access with full text that can be systematically downloaded and used for text mining.

I’ll show how I got these numbers for future reference:

First, get all publications from 2009 with links to full text using this query in PubMed:

“loattrfull text”[sb] AND (“2009″[PDAT] : “2009″[PDAT])

(direct url) returns 804184 results

Next limit these to publications with links to *free* full text using this query in PubMed:

“loattrfree full text”[sb] (“2009″[PDAT] : “2009″[PDAT])

(direct url) returns 247421 results

Finally we want to identify which of these are open access. This is a bit tricky because as far as I know this filter is not available in PubMed. It is, however, available in PubMed Central. So:

Start with this query in PubMed as above:
“loattrfull text”[sb] AND (“2009″[PDAT] : “2009″[PDAT])
(direct url)
On the right is a menu that says “Filter your results” Under that one of the options is
Links to PMC (158729)
Click this. This will show, in PubMed, all 158729 articles that have records in PMC. (Note there are quite a few papers will free full text that aren’t in PubMed Central, comparing this number to 247421)
Now we want to see these articles within the PMC interface rather than the PubMed interface. To do this, have a look at the “Find related data” menu a bit lower down on the right.
For Database, select PMC.
In Option, select Free in PMC
then click Find Items.
This will show the same articles but within the PMC interface. Or rather, it shows the first 10000 of the articles.
Have a look at the right menu now. It has a link that says
Open Access (4209)
That is how many of the 10000 articles are available as Open Access articles, as far as PubMed Central knows.
To finish, we need to extrapolate 4209 back to the full set, because it only represents the first 10000 articles. Assuming that the 158729 articles have the same breakdown of OA/non-OA (a safe assumption? could definitely do a bit more digging to be sure), we estimate that (158729/10000)*4209=66809
of the articles are available as Open Access.

Other filters can obviously be ANDed to each step to see this ratio in specific topic area. (There were a bunch of such calculations done by others a few years ago but I can’t easily find them on the web now. Anyone have related links?)

a future where data attribution Counts #idcc11

Heather Piwowar — Wed, 07 Dec 2011 09:40:42 +0000

Below is the rough text of my #idcc11 (International Data Curation Conference) talk. Slides at slideshare, updated now to now include (very tiny) text in speaker notes on each slide. [Anyone know how to increase size of font of presenter notes and/or extract them in to text document from Keynote?]

A future where data attribution Counts.

Sharing data makes our shoulders broader.

This is a great story, right? And why where are all here.

But it is also a great illustration of the problem

What exactly do broad shoulders get the individual researcher?

Pain!

Nobody looks at the supporting structure of an impressive tower. We are all busy oggling the top. That means these people? These ones with the shoulders? They’ve got nothing.

Ok, maybe they have some citations. But do we think the promise of citation is enough?

Nope.

Don’t get me wrong, I’m a fan of studies that show a citation benefit for sharing data :) . But it won’t be enough.

If it were, we’d have researchers knocking down the doors of our IR for the 10 minute job of sending in their preprints. They aren’t doing that. Because a few citations, as much as we’d like to think otherwise, aren’t enough to offset the Fear Uncertainty and Doubt that accompanies the costs of uploading a dataset in the current culture.

So.

What to do about it? How to change the culture?

We need to facilitate deep recognition of the labour of dataset creation.

Ok let me say that again because it is so important

We need to facilitate deep recognition of the labour of dataset creation.

And while we are at it, we need to value the contributions of funders, the people who pay for all the gym equipment to help us build to the shoulders, and data repositories, who we might like to view as perhaps personal trainers.

Let’s dig in to how these groups do impact tracking now, and how they’d like to do it in the future.

Investigators, today, can list research products on CV

A CV is sort of bland, don’t you think? It has no context of use.

We can see one version of a more useful future comes from a tool called total-Impact. Continuing a project that started as a hackathon at the Open Society Foundation workshop Beyond Impact organized by Cameron Neylon here in the UK last spring, Jason Priem, me, and a few other people have been working on a tool called total-impact. http://total-impact.org

It aggregates metrics for papers and also non-traditional research metrics, like datasets. The metrics are citations, but also altmetrics, or article-level citations…. various indications that others have found your research worth bookmarking, or blogging, or referencing on Wikipedia. It doesn’t currently look for dataset identifiers in public R packages, but it could, for example, as indication of use.

This makes a “live CV” if you will, giving post-publication context to research output. (Could also be applicable as a CV for a department, or a grant, or a grant portfolio….)

To do this really well we need to be able to list all metrics. Right now, many, are unavailable for this sort of mashup due to licending terms. This includes citations identified by Google Scholar, Thomson, and Scopus.

Repositories, today, can look at graphs of their deposit counts.

Many know their own download statistics, some share this with their authors or the public.

As a result of intenstive manual digging, some have metrics about how mamy times their datasets have been mentioned in the literature. I’ll splash by a few graphs of preliminary research findings…. come find me or my blog if you want more info. We are starting to be able to estimate third party reuse. Tools that support data citation will help this.

This is all a nice start.

What repositories really want, though, though — correct me if I’m wrong — is to show that they are indispensable. That they generate new, profound science not otherwise possible. That they are a great financial investment in scientific progress. This requires knowing more than just a citation count, it requires knowing the context of reuse. This means we need access to the full text of the paper that cites the data.

What about funders?

They want to know the impact the data had on society. Did it facilitate innovation, reduce discrimination, create jobs, save the rainforest, increase our GDP.

That kind of tracking is beyond what I know how to do :)

We’re going to need digital tracking technology that as far as I know isn’t available yet but I’m sure people are working on. Google analytics meets digital RF-ID tags…. I dunno… but I do know we need it. Furthermore, we need these digital tracking mechanisms to be affordable and open, to facilitate mashups.

Ok, so with that sort of future vision for tracking, what do we need as a scholarly ecosystem need to power this future world?

We need innovation and experimentation.

We need 1000 flowers blooming.

We need solutions that are open and generative.

I don’t have all the answers, but here is part of it:

open access to citation data. We can’t just rely on Scopus, Thomson, and Google Scholar. Those are only three players, They good at what they do and have been invaluable, but they can’t possibly be as nimble as a whole bunch of startups. It is taking them a long time to come out with a data tracking tool. Why? Probably because they have an ambitious vision and need time to fit it into their other product offerings. Some of the rest of us would be happy with iterating on a quick and dirty solution. We need more competition in this space. The barrier to entry is extrodinarily high because of course reference lists are almost all behind copyright and paywalls…. but open access gives us a toehold.
open access to full text. Open access also gives us a toehold into citation context information. A citation to a dataset tells us that the dataset played some role in that new research paper. What role? Was it used to validate a new method? Detect errors? Was it combined with other datasets to solve a problem that was otherwise intractable? The answers to these questions are fundamental to what funders and others need to know about impact. It won’t be easy to derive them from the text of the paper, but I strongly believe it is possible. We need this to be true open access — can be used by anyone for any purpose — none of this Non Commercial nonsense… we must allow startups to use this information if we are going to get the innovation we need.
open access to other metrics of use. We need broad-based metrics… not just citations, but blog posts about data, slides, tutorials that include R data, bookmarks to data on bookmarking sites. Altmetrics. If you run a data repository, make your download stats publicly available. We frankly don’t know what all of this info means yet, but we didn’t know what citations to papers meant 50 years ago either. We’ll all figure it out, the more data the better.

Here’s what each of us need to do:

raise our expectations.

what we could and should be able to mash up. open citation lists, open access (true open access, none of this NC stuff that renders it useless for startups), open metrics, open data, the impact you can make

raise our voices.

http://www.whitehouse.gov/blog/2011/11/07/request-information-public-access-digital-data-and-scientific-publications

Get excited and make something

http://www.flickr.com/photos/blackbeltjones/3365682994/

The future where data attribution Counts.

The future is about what kind of impact a dataset makes,
not just a citation number.

The future is open.

Open data.

Open data about our data.

thoughts on where journals are now, what to do next

Heather Piwowar — Sat, 03 Dec 2011 22:59:08 +0000

With my Dryad hat on I was recently invited to participate in a “Future of Research Dissemination” day at BMJ. Invitees were asked to give brief introductory remarks on what journals are doing now to enhance the experience of research and readers, what researchers and readers want, and then at the end what publishers ought to be doing now to future-proof themselves.

My take, fwiw, with some links to previous blog posts with more detail:

What are journals doing now wrt data?

Journals are increasingly recognizing that the datasets which support the findings in their articles are a crucial resource. They are working to make them more available and more useful.

There is increasing recognition that datasets are different than articles:

how they are peer-reviewed (or not)
how they are licensed
how they are discovered
how they are preserved
how they are financed

For all of these reasons, supplementary information is probably not the right place for datasets. Journals, along with others in the scholarly ecosystem (spurred on by recent requirements by funders for increased data availability, and evidence that researchers often don’t makedata available, including for example in cancer) are trying to figure out how best to move forward.

Data repositories are gaining traction as a best practice solution.

Some journals, like BMJ Open, integrate data submission with data repositories like Dryad to make things as easy as possible for authors and in some cases also peer-reviewers.

Journals are also reconsidering their own policies with respect to data, becoming more and more explicit about what is expected from authors.

There is repeated evidence (1, 2, 3, others) of a strong correlation between data archiving policies and impact factor; high IF journals are more likely to expect data to be publicly available, and indeed have been measured to have the highest rate of data availability.

Requiring data archiving in the current culture can feel daunting. One approach taken by the top-tier journals in evolutionary biology was to adopt a coordinated Joint Policy for Data Archiving. Starting in this last January all of those journals began requiring data archiving as a condition of publication simultaniously.

Finally, journals just in the beginning of ways to support synergistic discovery. Links between papers and data, full-text search because of course a paper is the best metadata for a dataset, and article open metrics of use are getting off the ground. It is crucial these are available for both humans and machines (apis) to enable innovative and meaningful use. Also it is key they are open. Google Scholar etc isn’t, can’t spider, can’t reuse, can’t mashup.

What do researchers and readers want?

Lots of things. To pick one: Recognition for the labour they’ve put in to creating data, and meaningful credit for anything built on top of it.

This is primarily a function for funders and institutions, but journals can play a unique role in making the appropriate credit explicit.

Citations to datasets is a start, but we must go further than that, because citations are too minor. For example, we could ask authors what resources were essential to the research they are reporting and then revealing those debts in structured and open ways for remixing.

What steps should journals take now (in the next year or two) to future-proof themselves?

(I was one of the last ones in the room to chip in. OA (and in particular proper OA without NC), article level metrics, collaborating with other publishers, extending open peer review, experimenting in general, adopting stronger data policies, etc had already been mentioned.

open computer programming interfaces to full text search, open impact metrics, and deep metadata to facilitate external innovation
software challenges for innovative applications, because those are the relationships you want to build
signal the way to best practices, by asking reviewers if “all resources have been made appropriately available” and by leaving space on submission form for “dataset IDs” (hat tip to John Wilbanks). When in doubt with these tactics and policies be brave not conservative.
experiment with new and more profound forms of acknowledgement for essential scholarly building blocks
start practicing living within lower profit margins. (!).

Doing data archiving well

Heather Piwowar — Fri, 18 Nov 2011 14:30:00 +0000

It is easy to think that archiving data is easy: just put the data files up on a website. To do it well, though, isn’t that easy. The Dryad digital repository has been thinking hard about these issues for years, working toward a practical, simple, and rewarding solution. For Dryad’s website and promotional material we’ve articulated some of the issues we feel are important; see Why Should I Choose Dryad for the up-to-date version.

I copy the current text here to inspire a conversation about “selling points” for a data archive, and even more importantly illustrate how involved it is to make a data archive great.

From http://datadryad.org/depositing

Dryad aims to make data archiving as simple and as rewarding as possible:

Simple

Dryad welcomes data files associated with any published article in the biosciences, as well as software scripts and other files important to the article.
There is no restriction regarding data formats.
Dryad works with journals to integrate article and data submission, streamlining the deposit process. Once the files are prepared, deposition typically takes less than 15 minutes (2-minute video here).
Data destined for more specialized repositories can, in some cases, be submitted through Dryad, reducing the time and complexity of data submission yet further.
Dryad provides a single clear and best-practice option for terms of reuse.
A curator will check your files for technical problems before they are released.
By default, data are embargoed until journal article publication. Dryad makes sure this happens so you do not need to.
If it is supported by the policy of the journal, you may, during the submission process, select a‘no-questions-asked’ embargo on data downloads for one year post-publication. Dryad will support a longer embargo if directed by a journal editor.
You are free to provide additional keywords that make the data easier to discover and additional documentation (in the form of ReadMe files) to help ensure proper data reuse.
You have the ability to add new versions of data files in order to make updates or corrections.
Dryad can make data securely available for peer review at the request of the journal.

Rewarding

Dryad works to ensure that you get credit for reuse of your data by promoting adoption of best-practice data citation policy and the trackability of data citations.
Data files receive persistent, resolvable Digital Object Identifiers (DOIs) that can be used in a citation as well as listed on your CV.
Dryad’s terms of reuse for data facilitate the maximum impact for your work.
Data in Dryad are independently discoverable, providing a new route by which others may learn about your work.
Discovery is supported through the indexing of Dryad’s contents by services such as Google Scholar, Web of Science, and others.
Usage statistics are available for you to highlight when your datasets are frequently downloaded.
Since Dryad does not reject data for being of the wrong type or in the wrong format, all the data files associated with an article can be archived together.
Dryad can host files that are larger than those accepted by most journal websites (up to 1 GB per file and 10 GB per package).
Your data are preserved and made available for the long-term, even beyond the lifespan of Dryad, through continuous backup and replication services.
Dryad is community-led, with priorities and policies shaped by the members of the Dryad Consortium, including scientific societies, publishers, and other stakeholder organizations.
Dryad is a nonprofit, but takes sustainability seriously, ensuring that funds are available for long-term preservation.
Dryad is an active participant in organizations developing best-practices for data management such as Biosharing, DataCite and DataONE. You as a researcher benefit from, and contribute to, the work of these organizations by depositing and using Dryad.

Have we left out any characteristics that matter to you? Or do you have a wishlist of things you’d like to see in a data archive like Dryad? Let us know in the comments, or send us an email. Thanks!

designing an awesome total-impact api

Heather Piwowar — Wed, 09 Nov 2011 20:54:58 +0000

APIs are awesome. They let other people leverage your product and make unexpected things. Total-impact exists because it could be built quickly on the APIs of others.

So total-impact itself should have an awesome and easy to use api.

We’ve made huge strides in this regard in the last few days. We now have an api roadmap and have implemented the first part of it. The total-impact web app will soon be doing all its data accesses through this api. We really like saying we are built on our own api : )

Play around with the examples below and see what you think (please don’t use the api heavily or in production yet: it isn’t doing good caching, etc). Suggestions for improvements in the design are very welcome!

Initial implementation includes:

GET /items/ID1,ID2,ID3 or GET /items/ID1,ID2,ID3.html

returns html for those IDs, as it would appear on the total-impact website.

GET /items/ID1,ID2,ID3.json

all metrics info in json format

GET /items/ID1,ID2,ID3.xml

all metrics info in xml format

GET /items/ID1,ID2,ID3.json?fields=biblio,aliases,metrics,debug

allows subsetting the metrics info returned

Examples: (to try other IDs replace / in IDs with %252F)

html: http://total-impact.org/api/v1/items/18428094,10.1371%252Fjournal.pmed.0020124,http:%252F%252Fopensciencesummit.com%252Fprogram%252F,10.5061%252Fdryad.8048
html, just metrics (good for easy embedding) http://total-impact.org/api/v1/items/18428094,10.1371%252Fjournal.pmed.0020124,http:%252F%252Fopensciencesummit.com%252Fprogram%252F,10.5061%252Fdryad.8048?fields=metrics
json: http://total-impact.org/api/v1/items/18428094,10.1371%252Fjournal.pmed.0020124,http:%252F%252Fopensciencesummit.com%252Fprogram%252F,10.5061%252Fdryad.8048.json?fields=metrics
just biblio (api supports returning just subsets of data elements) http://total-impact.org/api/v1/items/18428094,10.1371%252Fjournal.pmed.0020124,http:%252F%252Fopensciencesummit.com%252Fprogram%252F,10.5061%252Fdryad.8048.json?fields=biblio

Full roadmap in the works (feedback encouraged!)