Research Remix

November 15, 2010

Other NSF Directorates, where are your data sharing guidelines?

Filed under: Policies — Heather Piwowar @ 1:03 pm

The NSF has released its revised policy on dissemination and sharing of research results.

I’m quite impressed with the policy.  The high level information is rather vague, but the detailed guidances provided by the directorates and divisions have given it real substance.  As you can see from this summary, the guidelines definitely vary across fields!

Unfortunately, only a few of all NSF directorates and divisions have issued guidances, as per this official inventory.

Other directorates, where are your data sharing guideances?  I know that writing them is difficult, but it is important.  Now is the time when they are needed.  Even if the best you can say right now is that you are playing it by ear, as the Physics division has done in the MSP guidance, stating it explicitly has value as a starting place:

The Physics Division is not in a position to recommend a Division-specific single data sharing and archiving approach applicable to the disparate communities supported through the Division.  The Division will rely on the process of peer review to allow each of these communities to identify best practices.

Below is a list of all the directorates and divisions in the NSF.  The bolded ones have data sharing plan guidances here.  If you belong to one of the directorates or divisions that does not yet have a data sharing plan guidance, speak up and work towards the beginning of a written expression of your community’s expectations. This sort of process institutionalization is key to a fair funding process for new investigators and to an efficient review process for everyone.

Directorate for Biological Sciences

  • Division of Molecular & Cellular Biosciences
  • Division of Biological Infrastructure
  • Division of Integrative Organismal Systems
  • Division of Environmental Biology

Directorate for Computer & Information Science & Engineering

  • Division of Computing and Communication Foundations
  • Division of Computer and Network Systems
  • Division of Information and Intelligent Systems

Directorate for Education & Human Resources

  • Division of Research on Learning in Formal and Informal Settings
  • Division of Graduate Education
  • Division of Human Resource Development
  • Division of Undergraduate Education

Directorate for Engineering

  • Division of Chemical, Bioengineering, Environmental, and Transport Systems
  • Division of Civil, Mechanical & Manufacturing Innovation
  • Division of Electrical, Communications & Cyber Systems
  • Division of Engineering Education & Centers
  • Division of Industrial Innovation & Partnerships

Directorate for Geosciences

  • Division of Atmospheric and Geospace Sciences
  • Division of Earth Sciences
  • Division of Ocean Sciences
  • Integrated Ocean Drilling Program

Directorate for Mathematical & Physical Sciences

  • Division of Astronomical Sciences
  • Division of Chemistry
  • Division of Materials Research
  • Division of Mathematical Sciences
  • Division of Physics

Directorate for Social, Behavioral & Economic Sciences

  • Division of Social and Economic Sciences
  • Division of Behavioral and Cognitive Sciences
  • Division of Science Resources Statistics

October 12, 2010

Prelim proposal: impact of journal data sharing mandates

Filed under: Policies, proposal — Heather Piwowar @ 6:27 am

Do you want to know the impact of journal policies that mandate data sharing?  Me too.  I’ve been working on a study design to correlate policy adoption by journals with changes in scientists’ attitudes, experiences, and practices around publicly archiving research datasets.

I’d love some community input because writing a good survey and design is hard.  Summary and a link to the draft questionnaire are below.  I’m still finishing up the document to be submitted to my research ethics board ASAP.  Thoughts or suggestions?  I want it to be a useful study, one that focuses on just a few questions and studies them well.

Study overview

Survey preview link (note: this link allows perusal only… survey responses are not being collected at this time)

Research questions:

  • How do authors’ attitudes, experiences, and practices around public data archiving change when the journals they publish in adopt mandatory data archiving policies?
  • Are changes specific to the journals that implement the policies, or do they extend to other journals in the same subfield?
  • Do authors believe that adoption of a mandatory data archiving policy would dissuade them from submitting to a journal?  Does that opinion change after they publish in a journal with such a policy?

Participants: Participants will be recruited from corresponding authors who publish in specific journals between November 2010 and November 2013.

Method: An invitation to the survey will be issued once a month to all corresponding authors who published in specific biology journals the prior month.  Participants will be asked to answer a brief online web questionnaire designed to elicit their practices, experiences, and opinions around public archiving of research datasets.  The answers will be compared over time, to understand if answers of authors in journals that adopt the “Joint Data Archiving Policy” (JDAP) begin to differ from those of authors in comparable within-community and outside-community journals.

Details below.

Update Oct 15 2010:  I’ve updated the proposal details with several changes, including a longer data collection timeline and an estimate of study size.

October 8, 2010

A look into the revised NSF data sharing policy

Filed under: Notes, Policies — Tags: , , , , — Heather Piwowar @ 6:09 am

Curious about details on the NSF’s revised policy on Dissemination and Sharing of Research Results?  I’ve been digging into the documents released by the NSF and its Directorates.  Here are my notes, in case they are useful for someone:  excerpts from the docs, grouped by topic.


  • [SES] Division of Social and Economic Sciences
  • [EAR] Division of Earth Sciences
  • [ENG] Engineering Directorate
  • [OCE] Division of Ocean Sciences
  • [IODP] Integrated Ocean Drilling Program
  • [MPS]  Mathematical and Physical Sciences Directorate

What is considered “data”/research results covered by this policy ?

  • “may include, but is not limited to: data, publications, samples, physical collections, software and models”  FAQ
  • “Investigators are expected to share with other researchers, at no more than incremental cost and within a reasonable time, the primary data, samples, physical collections and other supporting materials created or gathered in the course of work under NSF grants”  AAG
  • “Investigators and grantees are encouraged to share software and inventions created under the grant or otherwise make them or their products widely available and usable.”  AAG
  • “Qualitative resources.   If it is appropriate for other researchers to have access to them, the investigators should specify a time at which they will be made generally available, in an appropriate form and at a reasonable cost.”  SES
  • “In addition, complete information on how an experiment was conducted and any unusual stimulus materials should be made available, so that failures to replicate will not turn out to depend on one scientist’s incomplete understanding of another’s procedure.”  SES
  • “Mathematical and computer models.  Investigators should plan to make these models available to others wanting to apply them to other data sets or experimental situations. In some cases, the descriptions in published articles are sufficient; more often, it will be necessary for investigators to prepare fully documented and robust versions of these models, typically on disk, so that they can be provided to others.”  SES
  • “Preservation of all data, samples, physical collections and other supporting materials needed for long- term earth science research and education”  EAR
  • “Experimental Research: In experimental research, individuals, be they people, animals, or objects, are subjected to preplanned conditions and their responses tabulated in some fashion. Investigators should plan to make these tabulated data available to other investigators requesting them” SES
  • “Data archives must include easily accessible information about the data holdings, including quality assessments”  EAR
  • “Archiving of both physical and digital data must be addressed in the plan”  ENG
  • “Under the following definitions, all data must be included in the DMP that result fully or in part from activities supported by ENG.”  ENG
  • “Research data are formally defined as “the recorded factual material commonly accepted in the scientific community as necessary to validate research findings” by the U.S. Office of Management and Budget (1999).”  ENG
  • “The basic level of digital data to be archived and made available includes (1) analyzed data and (2) the metadata that define how these data were generated…. Analyzed data are (but are not restricted to) digital information that would be published, including digital images, published tables, and tables of the numbers used for making published graphs.  Necessary metadata are (but are not restricted to) descriptions or suitable citations of experiments, apparatuses, raw materials, computational calculation input conditions”  ENG
  • “These are data that are or that should be published in theses, dissertations, referred journal articles, supplemental data attachments for manuscripts, books and book chapters, and other print or electronic publication formats.”  ENG
  • “What data are not included at the basic level? The Office of Management and Budget statement (1999) specifies that this definition does not include “preliminary analyses, drafts of scientific papers, plans for future research, peer reviews, or communications with colleagues.” Raw data fall into this category as “preliminary analyses.””  ENG
  • “Describe the types of data and products that will be generated in the research, such as images of astronomical objects, spectra, data tables, time series, theoretical formalisms, computational strategies, software, and curriculum materials.”  MPS astonomy
  • Particular attentionshould be paid to data sets that are products of well-defined surveys.”  MPS

Where?  Public repository?  other?

  • “There is no public database for my type of data. What can I do to provide data access? Contact the cognizant NSF Program Officer for assistance in this situation.”  FAQ
  • “Quantitative Social and Economic Data Sets.  This may be the Inter-University Consortium for Political and Social Research (ICPSR) at the University of Michigan, but other public archives are also available.”  SES
  • “institutional archives that are standard for a particular discipline (e.g. IRIS for seismological data, UNAVCO for GPS data)” EAR
  • “Experimental research. SES will work with the research community to identify and resolve problems with developing and establishing centralized archives.”  SES
  • “to other investigators requesting them” SES
  • “Where no data or sample repository exists for the collected data or samples, metadata must be prepared and made available. The Principal Investigator (PI) is required to address alternative strategies for complying with the general philosophy of sharing research products and data as described above”  OCE
  • “The PI is invited to discuss this issue with NSF Program Officers in advance of submitting proposals.”  OCE
  • “for most ocean data there are designated National Data Centers where data must be deposited… Appendix I. National Data Centers”  OCE
  • “For some special programs and focused community initiatives, alternative database activities exist… Principal Investigators are encouraged to submit their data to these databases when appropriate. Since such databases may not provide long-term archival capabilities, such submission will satisfy the Principal Investigator’s obligations only if the database submits the data to one of the National Data Centers….  Appendix III: Other Database Activities…. Appendix IV. Sample Repositories”  OCE
  • Experimental Research:  “at a minimum along the lines suggested by Geoffrey Loftus in his editorial in the January, 1993, issue of Memory and Cognition”  SES  [Loftus, G.R. (1993). Editorial Comment. Memory & Cognition, 21(1), 1-3.  pdf]
  • “Describe your plans, if any, for providing such general access to data,including websites maintained by your research group, and direct contributions to publicdatabases (e.g., the Protein Data Bank, Cambridge Crystallographic Data Centre,Inorganic Crystal Structure Database in Karlsruhe, Zeolite Structure Database).”  MPS
  • “Finally, note as well any anticipated inclusionof your data into databases that mine the published literature (e.g., PubChem, NISTChemistry WebBook).”

Who needs access:  researchers, educators, public?

  • “The National Science Foundation is committed to the principle that the various forms of data collected with public funds belong in the public domain.”  SES
  • But it is a bit confused.  Even within one paragraph:  “The National Science Foundation is committed to the principle that the various forms of data collected with public funds belong in the public domain. Therefore, the Division of Social and Economic Sciences has formulated a policy to facilitate the process of making data that has been collected with NSF support available to other researchers.”  SES
  • “for research and education” EAR
  • “Data inventories should be published or entered into a public database periodically and when there is a significant change in type, location or frequency of such observations.” EAR
  • “Policies for public access and sharing should be described”  ENG
  • “samples and data to research scientists (Science Party members and postmoratorium researchers), educators, museums, and outreach institutions” IODP
  • “interested parties” MPS


  • “The expectation is that all data will be made available after a reasonable length of time.” FAQ
  • “One standard of timeliness is to make the data or samples accessible immediately after publication.” FAQ
  • “However, what constitutes a reasonable length of time will be determined by the community of interest through the process of peer review and program management” FAQ
  • “Quantitative Social and Economic Data Sets: For appropriate data sets, researchers should be prepared to place their data in fully cleaned and documented form in a data archive or library within one year after the expiration of an award.”  SES
  • “For those programs in which selected principle investigators have initial periods of exclusive data use, data should be made openly available as soon as possible, but no later than two (2) years after the data were collected. This period may be extended under exceptional circumstances, but only by agreement between the Principal Investigator and the National Science Foundation. For continuing observations or for long-term (multi-year) projects, data are to be made public annually.”  EAR
  • “Publication delay policies (if applicable) must be clearly stated.  Investigators are expected to submit significant findings for publication quickly that are consistent with the publication delay obligations of key partners, such as industrial members of a research center.”  ENG
  • “Public release of data should be at the earliest reasonable time. A reasonable standard of timeliness is to make the data accessible immediately after publication, where submission for publication is also expected to be timely.”  ENG
  • “Principal Investigators are required to submit all environmental data collected to the designated National Data Centers (Appendix I) as soon as possible, but no later than two (2) years after the data are collected. Inventories (metadata) of all marine environmental data collected should be submitted to the designated National Data Centers within sixty (60) days after the observational period/cruise. For continuing observations, data inventories should be submitted periodically if there is a significant change in location, type or frequency of such observations.”  OCE
  • “Also describe your practiceor policies regarding the release of data for access, for example whether data are posted before or after formal publication.”  MPS-AST

Data retention and preservation

  • “Minimum data retention of research data is three years after conclusion of the award or three years after public release, whichever is later.”  ENG
  • “Exceptions requiring longer retention periods may occur when data supports patents, when questions arise from inquiries or investigations with respect to research, or when a student is involved, requiring data to be retained a timely period after the degree is awarded.”  ENG
  • “Research data that support patents should be retained for the entire term of the patent”  ENG
  • “Longer retention periods may also be necessary when data represents a large collection that is widely useful to the research community. For example, special circumstances arise from the collection and analysis of large, longitudinal data sets that may require retention for more than three years. Project data-retention and data-sharing policies should account for these needs”  ENG
  • “If maintenance of a web site ordatabase is the direct responsibility of your group, provide information about the period of timethe web site or data base is expected to be maintained.”  MPS-AST
  • “Describe how data will be archived and how preservation of access will be handled. Forexample, will hardcopy notebooks, instrument outputs, and physical samples be stored ina location where there are safeguards against fire or water damage? Is there a plan totransfer digitized information to new storage media or devices as technological standardsor practices change? Will there be an easily accessible index that documents where allarchived data are stored and how they can be accessed?”  MPS-CHE

Program-specific additional requirements

  • several noted some programs, institutions, communities may have more stringent requirements.  A few (OCE) go into some specifics.

Reporting, review, and consequences

  • “The Data Management Plan will be reviewed as an integral part of the proposal, coming under Intellectual Merit or Broader Impacts or both, as appropriate for the scientific community of relevance.”  GPG, MPS
  • MPS Divisions will rely heavily on the merit review process in this initial phase to determinethose types of plan that best serve each community and update the information accordingly.  MPS
  • “NSF program management will implement these policies for dissemination and sharing of research results, in ways appropriate to field and circumstances, through the proposal review process; through award negotiations and conditions; and through appropriate support and incentives for data cleanup, documentation, dissemination, storage and the like.”  AAG
  • “Within the proposal review process, compliance with these data guidelines will be considered in the Program Officer’s overall evaluation of a Principal Investigator’s record of prior support.” EAR
  • “Efficiency and effectiveness of the DMP will be considered by NSF and its reviewers during the proposal review process.”  ENG
  • “After an award is made, data management will be monitored primarily through the normal Annual and Final Report process and through evaluation of subsequent proposals.  Subsequent proposals. Data management must be reported in subsequent proposals by the PI and Co-PIs under “Results of prior NSF support.””  ENG
  • “Strategies and eventual compliance with this policy will be evaluated not only by proposal peer review but also through project monitoring by NSF program officers, by division and directorate Committees of Visitors, and by the National Science Board.”  ENG
  • “Plans for the handling of data and other products will be considered in the review process.”  OCE
  • “Annual reports, required for all projects, should address progress on data and research product sharing. The Division of Ocean Sciences requires that final reports document compliance or explain why it did not occur. In cases where the final report is due before the required data or sample submission, the PI must report submission of metadata and plans for final submission. The PI should notify the cognizant Program Officer by e-mail after final data and/or sample submission.”  OCE
  • “Within the proposal review process, compliance with these data guidelines will be considered in the Program Officer’s overall evaluation of a Principal Investigator’s record of prior support.”  OCE
  • “Many of the proposals to DMS that require significant data management plans will beinterdisciplinary submissions… DMS expects principal investigators to address the customary data practices of partner disciplines in their proposals’data management plans, and reviewers are likely to be asked to comment on the suitability of those plans from the perspectives of the relevant disciplines.”  MPS-DMS


  • All documents recognize the special needs of sensitive (eg human subjects) data and the need to protect IP rights.
  • “A valid Data Management Plan may include only the statement that no detailed plan is needed, as long as the statement is accompanied by a clear justification. “ GPG
  • “It is acceptable to state in the Data Management Plan that the project is not anticipated to generate data or samples that require management and/or sharing.  PIs should note that the statement will be subject to peer review.”  FAQ
  • “legal rights to intellectual property [..] Such incentives do not, however, reduce the responsibility that investigators and organizations have as members of the scientific and engineering community, to make results, data and collections available to other researchers.”  AAG
  • “General adjustments and, where essential, exceptions to this sharing expectation may be specified by the funding NSF Program or Division/Office for a particular field or discipline to safeguard the rights of individuals and subjects, the validity of results, or the integrity of collections or to accommodate the legitimate interest of investigators. “  AAG
  • “For example, human subjects protection requires removing identifiers, which may be prohibitively expensive or render the data meaningless in research that relies heavily on extensive in-depth interviews.”  SES
  • “These guidelines are considered to be a binding condition on all EAR-supported projects” EAR
  • not peer review?  “Exceptions to these data guidelines require agreement between the Principal Investigator and the NSF Program Officer.”  EAR
  • “Some proposals may involve proprietary or other restricted data. For example, projects having proprietary information that will eventually lead to commercialization, such as [..].  In addition, membership agreements, contracts, involvement with other agencies, and similar obligations may place some restrictions on data sharing.  The proposal’s DMP would address the distinction between released and restricted data and how they would be managed.”  ENG
  • “Exceptions to the basic data-management policy should be discussed with the cognizant program officer before submission of such proposals.” ENG
  • “if you plan to provide data and images on your website, will the website contain disclaimers, or conditions regarding the use of the data in other publications or products? If the data or products (e.g., images) are copyrighted (by a journal, for example), how will this be noted on the website?”  MPS-AST


  • “Should the budget and its justification specifically address the costs of implementing the Data Management Plan?  As long as the costs are allowable in accordance with the applicable cost principles, and necessary to implement the Data Management Plan, such costs may be included (typically on Line G2) of the proposal budget, and justified in the budget justification.”  FAQ
  • “It is NSF’s strong expectation that investigators will share with other researchers, at no more than incremental cost”  FAQ  “no more than incremental cost” means that they can charge researchers to recover costs.
  • “These plans should cover how and where these materials will be stored at reasonable cost, and how access will be provided to other researchers, generally at their cost.”  SES

What to put in the data management plan?

  • “This supplement may include types of research output expected to be created, standards to be used, policies for sharing, provisions for reuse, and plans for preservation.”  GPG with emphasis added
  • “The DMP should clearly articulate how “sharing of primary data” is to be implemented…. The DMP should describe the types of data, samples, physical collections, software, curriculum materials, and other materials to be produced in the course of the project. It should then describe the expected types of data to be retained…. The DMP should describe the period of data retention… The DMP should describe the specific data formats, media, and dissemination approaches that will be used to make data available to others, including any metadata.”  ENG
  • “It should outline the rights and obligations of all parties as to their roles and responsibilities in the dissertations, refereed journal articles, supplemental data management and retention of research data. It must also consider changes to roles and  responsibilities that will occur should a principal investigator or co-PI leave the institution. “ENG
  • “Any costs should be explained in the Budget Justification pages. “ ENG
  • “requires that proposal Project Descriptions outline plans for preservation, documentation, and sharing of data, samples, physical collections, curriculum materials and other related research and education products”  OCE
  • “DMR PIs should include in their Data Management Plan those aspects of data retention andsharing that would allow them to respond to a question about a published result.”  MPS-DMR
  • “Due to the diverse communities supported by DMR, the Division is not in a position to recommend a Division-specific single data sharing and archiving approach.”  MPS-DMR
  • “The Physics Division is not in a position to recommend a Division-specific single data sharing and archiving approach applicable to the disparate communities supported through the Division.The Division will rely on the process of peer review to allow each of these communities toidentify best practices.”  MPS-PHY

Other notes:

  • I didn’t go into detail extracting info from the IODP doc.  Useful, clear, lengthy doc!
  • Looking forward to hearing from the Biology Directorate.  Others?
  • “Goal: Provide for clear, effective, and transparent implementation of NSF policy for data management and dissemination”  ENG.  Awesome.
  • “Where data are stored in unusual or not generally accessible formats, explain how the data may be converted to a more accessible format or otherwise made available to interested parties. In general, solutions and remedies should be provided.”  MPS
  • “Ensure that dissemination of the scientific findings of all IODP drilling projects/expeditions are planned so as to gain maximum scientific and public exposure”  IODP.
  • A lot of emphasis on “other researchers.”  Obligations to share data with commercial researchers are not clear, except where the language emphasizes “public”
  • Overall, I’m pretty impressed by all of this.  I was hesitant about the new NSF policy based on preliminary info:  it felt like too small a step.  But the Directorates have stepped up and given it meat and a backbone.  Nice work.  NIH, your turn again.

Reference docs on current policy

  • [SES] Division of Social and Economic Sciences
  • [EAR] Division of Earth Sciences
  • [ENG] Engineering Directorate
  • [OCE] Division of Ocean Sciences
  • [IODP] Integrated Ocean Drilling Program

Related documents

  • Committee on Strategy and Budget Task Force on Data Policies Charge and timeline (Draft final report expected first half of 2011)
  • NSF Press Release 10-077:  Scientists Seeking NSF Funding Will Soon Be Required to Submit Data Management Plans (May 10 2010)

ETA:  added MPS guidelines

August 13, 2010

Supplementary materials is a stopgap for data archiving

Filed under: Policies — Tags: , , , , — Heather Piwowar @ 11:23 am

The Journal of Neuroscience has issued a new policy on supplementary materials:

Beginning November 1, 2010, The Journal of Neuroscience will no longer allow authors to include supplemental material when they submit new manuscripts and will no longer host supplemental material on its web site for those articles

I think this will benefit the reporting of methods and exploratory analyses. I am thrilled that citations will no longer be lost in supplementary materials, assuming the additional citations make it into the main references list rather than being omitted.

But what about data?

A journal’s supplementary material section is not a great place for data. Limitations include:

  • not good for data formatting and reporting standards
  • not good for discoverability
  • not good for truly permanent storage
  • not good for machine retrievability
  • not good for journals sticking to core competencies
  • not good for journal planning, efficiency
  • not good for free access (in subscription journals)
  • not good for open access (or at least conveying openness clearly)
  • not good for lots of other things that I don’t know about and publishers don’t know about but repository professionals do know about

Most people would agree that well-designed, well-supported data repositories are the best place for data. The problem is, such repositories are few and far between. All is well and good if an experiment is in a discipline or produces a datatype for which a best-practice repository exists: the data should go there. All may be good if the authors are in an institution with an institutional repository that is well-equipped to handle scientific data, though these are uncommon. Otherwise where can investigators put their datasets?

Supplementary information is not a perfect home, it is not even very good, but it is better than hosting data on a lab websites or email-on-demand. It is a useful stopgap while more discipline-based repositories and institutional repositories rise to fill the need.

By removing this stopgap, in my opinion (and with the important caveat that I know very little about the journal or its discipline), The Journal of Neuroscience has sent three messages with its new policy:

1. They don’t consider archiving data to be their responsibility

This was already clear from their lackluster policy on data archiving:

Policy on Concerning Availability of Materials
It is understood that by publishing a paper in The Journal of Neuroscience the author(s) agree to make freely available to colleagues in academic research any clones of cells, nucleic acids, antibodies, etc. that were used in the research reported and that are not available from commercial suppliers.

Policy on DNA Sequences
[…] By the time a paper is sent to press, sequences must be deposited in a database generally accessible to the neuroscience community; the sequence accession number should be provided. Exceptions to this policy may be considered on an individual basis.

That’s it. Compare this to the comprehensive policies of other journals, particularly their statements of motivation. For example, in Science:

After publication, all data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science.

And in Stem Cells (similar in Cell):

Stem Cells supports the efforts of the National Academy of Sciences (NAS) to encourage the open sharing of publication-related data. Stem Cells adheres to the beliefs that authors should include in their publications the data, algorithms, or other information that is central or integral to the publication, or make it freely and readily accessible; use public repositories for data whenever possible; and make patented material available under a license for research use.

The Journal of Neuroscience has said that it wants to “maintain its leading position.” For what it is worth, evidence suggests that the highest impact journals have the strongest data sharing policies.

2. They don’t consider archiving data important

Based on the policy and the wording of its announcement, I was left with the impression that the Journal doesn’t consider data archiving important. In particular, stating that “supplemental material is inherently inessential” and “We should remember that neuroscience thrived for generations without any online supplemental material” belittles data sharing, given that much data is currently shared in supplementary materials for lack of a better place to put it.

The policy has left investigators with fewer better-than-nothing places to share data. I hope the next journal that is tempted to eliminate supplementary material will consider these alternative approaches to address its problems while supporting data archiving:

  • Fix rather than eliminate supplemental material policies: clearly specify that supplemental info is not peer-reviewed, specify that suppl info is only for data (for example), remind reviewers and authors that suppl info is not for defensive material, etc.

    One example is the thoughtful response by Cell to its problems with supplemental material, a solution of defining what should and shouldn’t be included:

    “One of the first issues we confronted in thinking about structuring supplemental material was one of setting limits. Limits of course have both positives and negatives. On the plus side, it seems in the best interest of everyone in the scientific community that the concept of a ‘‘publishable story’’ be at least roughly defined. […] strict overall length limits struck us as somewhat arbitrary, and we instead focused on a more conceptual organization.”

  • Or, if you do indeed want to eliminate supplementary materials, recommend and in fact require that links to supplementary information elsewhere are either to established repositories or to resources archived through one of the many mechanisms for url permanence.
  • Or, engage with Dryad or another discipline-based repository to find a win-win solution
  • And please commit to participating with the community to find solutions, rather vaguely suggesting, “It is conceivable that removing supplemental material from articles might motivate more scientific communities to create repositories for specific types of structured data, which are vastly superior to supplemental material as a mechanism for disseminating data.”

3. Change is needed

I completely agree with them here. Change is needed. I also applaud the Journal for taking a bold step, even if I disagree with its particulars. I think it will motivate, inspire, and induce change. Bring on the market disruption… although it is a real shame if we lose a bunch of (expensive) (irreplaceable) data (forever) in the process.

A follow-up post with references on supplementary material.

Other blogosphere commentary:

ETA: link to followup post

March 24, 2008

Preprints of conference submissions?

Filed under: Open Notebook Science, Policies — Tags: , , , , — Heather Piwowar @ 12:45 pm

It continues to surprise me how hard [1, 2] it is to figure out the rules for publishing in most journals, particularly with respect to preprints. I’m finding the policies for publishing in conferences are just as ill-defined. This is a shame because it surely decreases sharing.

I’ve been putting my work up on Nature Precedings. I submitted a poster a few days before the ISMB conference. Nature Precedings asked me if I knew whether ISMB would permit this. I emailed to ask: they quickly responded that it was no problem. I’ve since asked PSB and ELPUB: again, no problem. I think that ELPUB did make state this during in the submission process, but in all three cases it would be great if they wrote this explictly on their websites so that people could feel free to post and discuss draft submissions… I’ll send a few emails to suggest it.

I recently asked AMIA. I was pointed to the contributor license agreement [below], which left me guessing. What does “publish” mean? For most publishers I’ve talked to it means “published in an official capacity” and they don’t count blogs or pre-print servers. Yet clearly the journals who disallow preprints disagree with this definition [quick list for Clinical Medicine, or Sherpa/RoMEO for an extensive list]. JAMIA, the official journal of AMIA, does not allow preprints. So I guess I should email AMIA again and ask for further clarification.

ETA:  I emailed again and quickly got a very clear response: Preposting is fine so long as the other sites don’t formally publish the work.”   Great news, thanks AMIA.

I’d like to “ask for forgiveness rather than permission” but the potential down-side is too great in this case. Student publication in AMIA is highly-valued in my department.

I wonder if author copyright addenda can apply to conference submissions??? I don’t see why not. Will have to keep that in mind next time.

AMIA Annual Symposium 2008 license agreement:


Blog at