To Post or Not to Post: Publishing to the ArXiv Before Acceptance

by Kelle on December 12, 2011

Inspired by a discussion elsewhere, I’d like start an open thread about the pros and cons of posting a paper to the arXiv before it’s accepted by a refereed journal. To get the convo going, here’s my summary of what came out of the previous discussion:

Pros to posting before acceptance:

  • Problems and omissions get caught before publication.
  • More people have a chance to “referee” and give feedback and the published paper might be better and richer as a result.
  • Results get out to the community faster.

Cons to posting before acceptance:

  • Wrong results get circulated and could possibly never be corrected or retracted.
  • Could end up with several very different versions of same paper in circulation resulting in confusion.
  • It’s possible, but not confirmed, that NASA HQ will not issue a press release about a paper that has been put on arXiv and later accepted by the journal. The reasoning is that since the the paper is in the public domain, the story is already out there. This essentially results in an official policy that precludes one from posting before acceptance. (Can anyone confirm this?)
  • Some people will not referee a paper if they see that it’s already been posted to the arXiv.

I personally would like to post to the arXiv at the same time as submitting in order to get the additional scrutiny and feedback from my colleagues. However, maybe an alternative is to simply circulate the draft to the ~10 people who are most likely to have something to say?

What do you think? When do you think we should post papers to the arXiv and why?

(As a sidenote, I have noticed a near universal switch to writing “arXiv”, but saying “astro-ph”. Neither here nor there, just something I noticed that might confuse the young’ns.)

{ 59 comments… read them below or add one }

1 Tigran Khanzadyan December 12, 2011 at 9:03 am

Provided that you are also planning to submit your article to some of the refereed publications, you should post in astro-ph ONLY after the acceptance of it in the refereed journals. Alternatively it’s kind of cheating to increase your paper count. If you want to circulate you paper to some people just put it in some “dropbox” and send the link to your loved ones.

Ideally I would opt for abandoning all the journals and make astro-ph submission coupled with a refereeing process. In that way we submit to only one place, get evaluated say with bunch of “nice” scientist and get published electronically.


2 Benne Holwerda December 12, 2011 at 9:15 am

Circulating to people you consider interested was the way “preprints” were done in the pre-internet days. All in all not a bad system. I am pretty certain that it is not done to post if you want any kind of media blitz to go with it since most journalists a) get your blurb in advance but with b) an embargo deadline and a disclaimer as to when they can publish this.
So if any kind of media attention is required, it is poor form to post. Otherwise I’m starting to lean towards posting before acceptance as hopefully, one will be able to incorporate more feedback from the community.


*still posting only after acceptance. Was told as a grad student it was bad form to do so before*


3 Mike Peel December 12, 2011 at 9:27 am

I agree completely with the ‘pros’ here – and I sometimes find it frustrating to spot (small) errors in people’s papers, but know that it’s futile to point it out to them since the paper’s already been accepted and/or published by the journal.

A lot of papers nowadays have large coauthor lists – and have most likely gone through an initial set of internal peer-reviewing – so it’s generally a lot less likely that problems will be raised with them in the journal’s peer review process. Single- or few-author papers are a bit different, in that they won’t have gone through this process, so might be a bit more suspect. That’s the sort of information you can take into account when looking at the paper and deciding whether it’s trustworthy, though.

It’s a shame that the arXiv doesn’t have an option so that you can choose whether to read papers only after acceptance, or when they are posted on the arXiv – or at least make it so that you can clearly mark the paper as pre-acceptance – such an option would make the whole argument null and void.

I don’t get Tigran’s point about “cheating to increase your paper count” – since people should either look at the number of peer-reviewed papers you have, or take into account “in prep” papers (as are so often listed on people’s CVs when applying for jobs..) analogously to those available on arXiv…


4 Eilat December 12, 2011 at 10:15 am

There are many reasons we read papers. Sometimes we want to learn about a new subject in which we are not experts, other times we want to place our work in context with a larger field, and often, we read papers directly relating to our area on which we can comment expertly*.

I have no problem with papers in the last category being posted to arXiv. I can read those papers with the most critical eye, spot mistakes, contact the authors and alert them that they have not cited me ;-) etc.

But for all the other categories, I want my papers nicely vetted, thank you very much.

Unfortunately, not everyone works in my specific area so my selfish desires cannot be applied to everyone.

That is why I firmly believe in the peer review process — however flawed. And it *is* flawed. Perhaps we can add to this discussion (or discuss in parallel) ways to improve the peer review process so that we do not feel the need to bypass it via wiki-referee on arXiv.

*Not an exhaustive list for why we read papers, but I think those are the main categories.


5 Kelle December 12, 2011 at 11:00 am

let’s hold off on a discussion of ways to improve the peer review process. I’ll put up a new post next week about it. Someone actually just sent me a nice article to help motivate that discussion…

6 Brian Hayden December 12, 2011 at 10:50 am

I think the cons in this situation vastly outweigh the pros. I think it is very important that the paper that comes out on arXiv is free from incorrect data / plots / tables and free from biases that should be removed by the referee.

If you’re collaborating with several people you should have already gone through a fairly rigorous peer review process. Especially if the paper is part of a larger collaboration, that has to ‘sign off’ on it. Regardless, using arXiv as a means of peer review is an awful idea because the cons you listed are very detrimental in this day and age of google searching and ‘plot propagation’ across talks and presentations (by this I mean people re-using plots over and over without checking for updated versions).

Most people only read the arXiv mailing lists and such; and rarely look at updated versions. So let’s say I find an interesting paper, print it off, read it, and leave it sitting on my desk. 5 months later I need a paper on this topic; it’s up to me to check for an updated version, but is it necessarily true that everyone will do this?

I think the real issue is a lack of responsibility by authors, and editors and reviewers. Authors can simply ask for a different referee, and be granted one, without anything but the discretion of the editor. I know I personally wouldn’t want to post a potentially flawed version of my paper up for people to use against my work later. And it WILL happen.

Sure, we all understand the research process. But how long until a competitor says ‘So and So et al. now claim the same results, though a previous version of their paper claimed something else. So this essentially discredits their work.’

If we want better papers, we need to improve the official peer review process itself, not use arXiv as a substitute for professional responsibility.


7 Lee Anne Willson December 12, 2011 at 11:12 am

Circulating to targeted individuals is better, particularly as it helps distinguish pre- from post-refereed versions.

Even there, one can run into problems. I once sent a paper to someone for pre-referee comments; the response I got was a preprint, six months later, criticizing the pre-referee version for content that wasn’t in the published version. Not good! That was in the days of paper pre-prints and perhaps the recipient didn’t realize this was a pre-referee copy. By handling pre- and post-referee versions differently you can perhaps avoid such confusion.


8 Eric December 12, 2011 at 11:32 am

The older I get, the more the pros outweigh the cons (after having been preached the cons for many years).

One point made me laugh:
“Wrong results get circulated and could possibly never be corrected or retracted.”

This is almost a joke because the refereed literature is *full* of wrong results. One could spend an entire career rebutting wrong results in the literature, and do nothing more than be cleaning up after sloppy scientists. The refereeing process as is, relies on an overtaxed editor doing the best they can, often acting as editor for papers that they are not experts on, and relies on their judgement of who constitutes a proper referee (which works most of the time, but not always), and then of course relies on the quality of the referee, their judgement, and their ability to write a useful review. We’ve all had really good/useful referee reports and really bad/unhelpful referee reports. One argument for putting the papers on the arxiv *first* is to help open up the review process to the “cloud” of scientists – and those that are really interested in the paper might be able to send some feedback before the paper gets published (in principle, this is similar to emailing a pre-submission draft to ~10 of your colleagues and asking for input, which also is helpful). The author can obviously filter out what comments they deem helpful or not helpful. But overall, this process should lead to higher quality *final* publications, which is a good thing. There are so many *bad* papers out there (moreso in certAAin journals than others), where there has been a failure by the coauthors, the referee, and the editor to question the assumptions and analysis that went into the work. The classical view is that work can be easily rebutted by a response in the form of another refereed publication – but we don’t have infinite time for doing this for every instance of interstellar baloney that we detect (so usually only the most important papers have any serious challenges in the refereed literature).

[snipped off-topic bit about citation. That's a discussion for another day, probably Friday. --kelle]


9 Ian Crossfield December 12, 2011 at 3:42 pm

“[snipped off-topic bit about citation. That's a discussion for another day, probably Friday. --kelle]”

As much as I’m a fan of this site, I do not support this site’s editors when they decide which comments are off-topic and selectively edit them.

10 Marshall Perrin December 13, 2011 at 12:10 am

“[snipped off-topic bit about citation. That's a discussion for another day, probably Friday. --kelle]”

Ditto to what Ian said. This is a really surprising edit to see.

11 Michael Schneider December 12, 2011 at 1:34 pm

In my sub-sub-field (theoretical cosmology) I think it is the “norm” to post manuscripts on the arXiv at the same time as submitting the paper to the journal. In fact, a popular code for computing the CMB and matter power spectra ( has a requirement to submit drafts to the arXiv as part of its terms of use. I’ve never had any issue with this as I have actually never experienced any of the cons listed above. I can see the point about press releases, but really most papers do not have an associated press release. And all the “pros” seem in favor doing better science via more communication and transparency.


12 Ben December 12, 2011 at 3:15 pm

Many people post to the arXiv at submission because they’re afraid, if they wait, that a competitor will “scoop” them or claim precedence by earlier posting (then they have to cite the competitor in the final version and the competitor doesn’t, even if the refereed versions of the papers actually appear at the same time). Ultimately, scooping rarely matters unless it’s an extremely high profile result, because nothing we do is really so important that it matters whether one set of photons emitted millions or billions of years ago are published three months before another.

However, it’s a classic Prisoner’s Dilemma problem. You have to post before submission because your competitors might, and vice versa. This is one thing I don’t like about posting at submission. You’re making the choice not only for yourself, but forcing the same choice on the others in your field, even if they would prefer to post on acceptance.


13 David W. Hogg December 12, 2011 at 3:16 pm

If we have learned anything from open science, we have learned that the most helpful commenters and readers are invariably *not* among the 10 people you hand-pick. So send it to the whole community via arXiv and have them understand it is submitted not accepted! I think we are all adult enough to use that information responsibly.

And as Eric and others say, since when does the single referee’s eyes catch all the mistakes? If you want your published papers to be correct, you are *much* better off putting them on arXiv early and often! Like Eric, I laugh when I hear the idea that waiting until refereeing makes sure your results are “more correct”. Actually, it makes sure they are less correct, because fewer people have vetted them.


14 Eric Perlman December 12, 2011 at 3:59 pm

I think the cons here far, far outweigh the positives in most situations. While I agree with Eric and David that the literature is full of wrong results, and that refereeing is not a panacea, I think the chance of a wrong result coming out and/or being cited is much greater if papers are posted on arXiv before they are accepted. At least if your paper is accepted someone (sometimes several people) has been specifically charged with going through your paper with a fine tooth comb, which — if they’ve done their job right — is more than 98% of people reading a paper on arXiv will do. That’s not to say you don’t get helpful comments from people reading your paper on arXiv — you do. But I’ve found those happen much less often than helpful comments from a referee, and moreover, I feel strongly that once a journal article-to-be is posted on arXiv it should be good enough to cite.

Unfortunately, contrary to what certain people on this blog have said, one runs into seriously flawed papers on arXiv reasonably often — and much more often when they are posted before acceptance. Yet they get cited in the literature… is that right? And then they are used as priority claims when a discovery is made. That’s not right either. Yes, certain large collaborations have strict internal refereeing rules. But many of us do not work in large collaborations. And many of those same large collaborations have the rule that papers are not to be posted on arXiv before they are accepted, for exactly these reasons.


15 August Muench December 12, 2011 at 11:26 pm

hi eric,

isn’t this about culture?

- the flawed papers garnering citations are more likely garnering self-cites because astronomy citation practices are pretty darn insular.

- you assert that you get useful feedback from a referee but not from arXiv readers because… that is what our current culture deems normative. if we valued feedback along the lines of a formal post-publication peer review model then I think you would be experiencing things differently.

as for your rather strong assertion in post #13 that:

contrary to what certain people on this blog have said, one runs into seriously flawed papers on arXiv reasonably often — and much more often when they are posted before acceptance.

i have to say that as some one very interested in the role arXiv plays in “scholarly communication” i would appreciate working with someone (yourself?) in quantifying these examples. even a few examples would probably give great insight into the cultural antics of these authors and help inform those of us who want to see arXiv used in other ways.


16 Kyle Willett December 12, 2011 at 5:32 pm

Mostly for the first two reasons in Kelle’s list, I believe that the cons outweigh the pros of submitting before acceptance. I’m proud and grateful that the physics/astrophysics community has a free, widely-used service like arXiv to circulate our work. Even in my (relatively) young career, though, I’ve had papers change by enormously significant amounts between the first submission and eventual publication (including feedback from referees, editors, co-authors, and my own revisions). For me, the benefits of getting a paper out a month early don’t outweigh the serious danger of incorrect results being circulated. I agree that the peer-review system isn’t perfect, but it’s far better than making our results completely open access.

Like the good scientists we all are, it would be very useful to have some additional data to support out various opinions. Is it possible to study the number of astro-ph papers that have been retracted/revised and see whether there’s any relation to posting before acceptance? If anyone has specific examples — eg, “I read an astro-ph paper and cited it, and later found their results changed” — it would be useful to know, even in a qualitative sense.


17 Eva December 12, 2011 at 6:10 pm

Given that anybody can submit absolutely anything they want to the arXiv, I much rather rely on a flawed refereeing system than on no filter whatsoever. The only way I will cite an astro-ph paper is if it has been accepted and waiting for a volume number. For me, if a paper has not been accepted, it can go from random gossip to great work, while accepted material tends to be more serious on average (with certAAin exceptions, of course).

In my personal experience, more than half of my papers have changed a lot (always improved) from submission to acceptance. I don’t want the worse version of my paper to be the first one to see the light and be remembered by the community. Also, the one paper that ended up as a press release, was chosen as such way after submission. I’m sure that if I had put it on the arXiv, there would be no press release.


18 Sriganesh November 24, 2012 at 4:20 am

I absolutely agree with this comment.

Most of serious scientists would always try sending their works to journals that are reputed. These journals are reputed only because of their rigorous peer-review system and they publish only high-quality outcomes. If I am submitting to a new journal, I always check its editorial board and review system to be sure that it has the good people in the field on board and a review system that ensures valuable feedback about my work. As in the cases mentioned here, even my papers have undergone significant changes (improvements) from submission to final acceptance. This is certainly because of the rigorous peer-review system of journals. And therefore, I would rely more on submitting to arXiv after acceptance rather than upon submission – I will not like the community to remember my work that has flaws and has not been peer-reviewed.

ArXiv might be an option to get feedback about your work prior submission. But, such feedback can be obtained by sending your work to ~10 of your friends working in the same field.

19 Adam Ginsburg December 12, 2011 at 6:32 pm

I post only after acceptance. I’ve had a paper go through a 1.5 year reviewing process, which was largely my fault but I still wouldn’t have wanted the community citing the version that had very different results and emphasis than the final.

An important reason I don’t like to see arXiv papers pre-acceptance is that this is an easy way to say “We have released the data…” when in fact the attached data will not be available from the publisher until (long after?) acceptance. This is currently an implementation problem more than a philosophical one – if there was a reliable place to post data concurrently with arXiv posting, it would be less of an issue. I know folks are working on such a solution, but until it’s available, I don’t want to see papers on arXiv claiming to make data available when it is not.

Also, Kelle, thanks for the arXiv – astro-ph comment. I would certainly have been confused a few years ago.


20 Mike Peel December 12, 2011 at 6:40 pm

An easy way to solve that problem is to upload the data to your institution’s website (or the collaboration’s) – there’s no need for the journal to be the only ones hosting it (and it’s often easier to access on a non-journal website too, depending on the type of data).

21 Adam Ginsburg December 12, 2011 at 6:59 pm

re:Mike – I agree, to a limited degree. I don’t post data on my institution’s website because I know that my site will be deleted within a few years since I’m not tenured faculty. Putting that sort of a link in a paper is unreliable. I haven’t encountered any collaborations with permanent websites; the main collaboration I’m on has all of its data hosted at IPAC.

More important, though, is electronic-only data tables that really deserve to be on, say, Vizier or some other catalog service. I’ve seen many papers with large data tables saying “the full version is available in the electronic version….” which doesn’t become available until months later.

22 August Muench December 12, 2011 at 9:12 pm

adam is definitely on to something here. this drives me crazy. do people know that you can post all kinds of ancillary “data” into your paper on arXiv? not that i’ve ever found a single paper example in astronomy where someone has (and i’ve trolled the sources looking).

there is no reason for the data to live on a soon to die home page via a soon to break link or for you to wait for the post print version to push it out there just because its a long table. unless there is a reason and it has nothing to do with access.

23 Ben December 12, 2011 at 11:58 pm

This is not that hard. When I make a machine readable table for a paper, I include it in the astro-ph tarball. I also have put the tables on my web page so that they are available (and findable with a Google search) after acceptance and astro-ph submission, but before the paper appears on the ApJ website. Yes, my webpage will die eventually, but by that time the table will long since be on the ApJ site and ingested into Vizier. If you really need a webpage that will be around longer, there are ways to do that (

Data is released when it is freely downloadable. If you want to make it available, there is always SOME way to do it. Most of the roadblocks are in people’s heads.

24 Adam Ginsburg December 13, 2011 at 12:25 am

re: August – I think I encountered the arXiv ancillary data thing once, but was off-put by the size limitations (I wanted to upload a data cube… not a chance). Still, I doubt many folks go searching through arXiv’s capabilities when posting.

re: Ben – I agree, it’s not hard to find sites to host data, and I like the idea of finding a holdover storage site while the journals are busy formatting the data. If we could make early-posting common practice, it would dampen my objection.

As for, I don’t think google is supporting it any more? And that’s the problem with nearly anything on the web… not much lasts as long as academic careers. The most impressive example of longevity I’ve encountered so far is Bruce Draine’s website, which is cited in books at least a decade old (older?) but every piece of code and data is still available at the same url. I would like to see that kind of longevity available to all astronomers.

25 August Muench December 13, 2011 at 12:27 am


these are good points; i think that we were responding to observation that often, when a paper contains an extended table, the extension is not in the arXiv version.

i guess your point is that it might well be there but only if you download the original source tar ball and unpack it.

and my ancillary data in arXiv example was that one can force arXiv to expose your unlisted .tbl file in a unique way. Like on this page: on the right hand side under “ancillary files”. these do not count against the PDF size limit people complain about.

just like you said though — the arguments are often in people’s heads. the examples adam and I are thinking of might be cases where the authors excluded the tables because they didn’t know arXiv had this option or that they thought it would count against their upload limit.

26 Jeremy December 13, 2011 at 3:16 pm

My solution in that case is to leave the text in the paper as is (which mentions that the full data are available in the electronic version), but in the Comment section of the arXiv submission to say “The full data tables are available at“.

27 Tom December 12, 2011 at 6:33 pm

I don’t understand the issue – ultimately, arXiv papers almost always have an indication of whether the paper is accepted or just submitted, and whether it is to a refereed journal, or to a conference proceedings, so readers can just read papers with the appropriate pinch of salt, right? I choose not to upload papers to arXiv until they have been accepted, but I have no issues with reading papers I know have not been accepted yet.

In any case, as several people have posted above, there are many terrible ‘accepted’ papers out there!


28 August Muench December 12, 2011 at 9:00 pm

this is pretty awesome — so the summary is that posting a paper to arXiv at submission is bad idea for you because a half anonymous read over by a so called referee the peer review process is necessary to catch your mistakes, bad plots, tables, data etc.


because what you should do wrt arXiv was the original question, not one about everyone else’s retching of papers on to arXiv or in to the journals or the ratio of refuse between them.

personally, the only “con” i consider to be serious is the one about versions. posting to arXiv at all might be not a good idea given that some journals prevent you (sic) from syncing their version with the open version; not that I’ve responsibly tried to keep them in sync. ever.


29 Eric Perlman December 12, 2011 at 10:13 pm

Do you take the referee’s role this lightly? I assure you that when I referee, I do not. The referee’s job is not to read over the paper, it’s to go over it with a fine tooth comb. That role is essential to astronomy, and in my view it is important that this process is complete *before* a paper (as distinct from a proceedings article) is posted to arXiv.

The cons have nothing to do with ‘retching’ papers to arXiv, what they have to do with is the fact that once a paper is posted to arXiv most regard it as being published. A citation to arXiv is perfectly legitimate in our field, as it should be — but we both know that a citation is far stronger when the paper is accepted than it is when the paper is merely submitted.

Let me ask you a hypothetical… paper X makes a claim to a discovery, and is posted to arXiv on submission by its authors. It is found to have serious problems by the referee. Meanwhile two months later a much better paper is submitted by a different team that makes the same claim, but this time with much better support and it sails through. Which paper should be credited with the discovery? I think the only defensible answer is that the second paper gets priority — they did the job right, had the better data and their paper got ACCEPTED first. But far too many times I have seen the first paper’s authors claim priority and then that gets spread throughout the field.

30 August Muench December 12, 2011 at 11:06 pm

hi eric,

i do not take my role as a referee this lightly; i assure you that i do not remember commenting on my own actions as a referee. yet my characterization of the peer review process is the only valid prior i can think of to explain the quality control that published materials receive.

in fact i am squarely on your side about our job as referees. it troubles me that i take days upon days fine tooth combing a paper for bad plots, tables and simple errors because it wasn’t treated to the level of respect it should have been before it was submitted to whichever publisher is not named arXiv (bc I don’t review arXiv papers. is there an editor for that?).

this discussion is really about culture — whether or not we would or wouldn’t cite arXiv at all, about how well (or not) we do our jobs as peer reviewers sans any meaningful measure of peer pressure or attribution/blame. if our culture accounts for ‘publishing to arXiv’ and if we really can value load citation with something more than a counter increment or an unclicked URL, then we can deal responsibly with arXiv publication in advance of or maybe in place of acceptance.

as for your hypothetical, I repeat my earlier admonition that the question is about what you or I do as authors with our papers. your hypothetical treatment of bad papers written by bad authors caught by a good, responsible referee foiled by arXiv is precisely that — a hypothetical. i understand the problem you are trying to illustrate — the circumstance of misattribution of value to whoever published whatever first — but your hypothetical forgoes the slightly more probable reality of the first paper’s referee who is also the second paper’s first author.

what is worse, much worse, is that we actually do not need an arXiv proxy for the circumstance of misattribution of value to whoever published whatever first; it happens all the time… in the journals… precipitated by a substandard refereeing process.


31 Marshall Perrin December 13, 2011 at 12:09 am

One benefit of the traditional peer review process is that authors actually have to respond to the criticism of their readers, the referee(s), one way or another instead of just ignoring them. I would be much more in favor of papers posted on arxiv without prior refereeing if there were some mechanism where, say, readers could post comments or criticism of the paper and then the authors *had* to update the paper in response. That would prevent people from just posting half-baked work and then leaving it sit. And likewise, there might need to be some incentive for readers to do the really nitty-gritty detailed reading and commenting that a good referee can. (“you must post detailed critiques of at least two other papers before you can submit your next paper”?)

Like them or not, deadlines are a tool for focusing the mind, and likewise having formal structures around the refereeing process can be a motivating factor to make your paper better (rather than your just going on to the next project sooner, say).

On a semi-related note, I wonder what the community would think about making the referee process much more open. In this day of Wikipedia and open-source software development, why not approach academic writing with the same level of openness? Give every paper on arxiv a ‘comments’ box, and have the comments provided appear right there with the paper, and the authors’ responses along with ‘em… If done right, this might also help provide checks and balances on overzealous (or conversely lazy and/or just plain bad) referees, too.


32 Scott Ransom December 13, 2011 at 12:14 am

I have pretty strong opinions on this topic: I think that posting your paper to the arXiv at the same time you submit it to a journal is an _excellent_ idea. In fact, I see almost no downsides to it.

Note that I’m pretty specific about exactly what I’m advocating: posting only when you and your team are confident enough in your results so that you submit to a journal at the same time. I agree that the arXiv should not be where crappy draft manuscripts are incrementally improved version by version before submission to a journal or (worse) otherwise abandoned.

If you do this, I guarantee that your papers will be better for it.

Here is an example that happened to me that sealed my opinion.

When I was a postdoc, I was working on a very hot topic that had just exploded on the scene: the Double Pulsar. I managed to take some excellent very high quality data on the system and I found a result which was shocking. My collaborators included some of the foremost pulsar people in the world and we could find nothing wrong with the data or the analysis. So I wrote a quick ApJ Letter and posted it to the arXiv at the same time I submitted it to ApJ. Within 24 hours, I received two emails from other pulsar experts who suggested that what I was seeing was a particular data analysis issue that you would only see with such a relativistic system. They were correct. I immediately posted a letter to the arXiv (v2 of the paper!) explaining what had happened and the true reason for what we saw (which no one has seen in any other pulsar before or since), I thanked those who pointed out the issue to me, and then I retracted my submission from ApJ.

This was, of course, very embarrassing to me at the time: heck, lots of members of the pulsar community still jokingly call the data analysis issue the “Ransom Effect”. However, it was _highly_ valuable both to me and the community. Given that almost no one in the world knew about that potential issue, a random referee would almost certainly have accepted the paper (especially given the expert co-authors I had), and I would have had a major mistake in the literature. In addition, the whole pulsar community became instantly aware of the result and learned about the potential issues involved. So while my pride was hurt, things turned out much better this way in the end. You can still see both versions on the arXiv (it never forgets!).

This anecdote (and yes, it is only anecdote) exemplifies, though, everything that I think is good about early scientific discourse. We correct each others mistakes quickly and science improves as a result.

I don’t see why we should be worried about the arXiv clogging up with papers because of this behavior. Since most journal-submitted papers end up getting accepted eventually, as long as you submit to both arXiv and a journal and then update your paper to the accepted version when it happens, we shouldn’t get extra bogus papers because of this practice. In addition, if we hold ourselves and our collaborators to high standards for our papers in the first place, there would be even fewer “bogus” papers to contend with.

As for the argument that doing this can cause you not to get a NASA press release, that could be true in some circumstances (I’m not positive). However, I know for a fact that I’ve had a NASA (Fermi) press conference on pre-accepted or even pre-submitted results before. In addition, some journals have embargoes that can sometimes apply to posting to arXiv (but note that they often do _not_ apply to communications to colleagues but only to communications with the press). So you do need to carefully take these things in consideration. (We should remember that we scientists hold the power here in the long term, though: we are the ones on the NASA committees who can change these policies if they don’t make sense.)

Finally, I completely agree that real, careful refereeing is an crucial thing for science, but it’s not a panacea. I’m not arguing against it at all — if anything, I’m arguing for more of it from the arXiv. Have you ever sat in an Ohio State – style astro-ph coffee and heard how critical they are towards papers??

Bottom line is that I argue that for _your_ papers, the world will get a better product if you post to arXiv early. People _will_ read it. They _will_ be critical and let you know. And you _will_ make improvements because of it, including those that a single referee would miss. Finally, you’ll get more citations for doing it.


33 Tanya December 13, 2011 at 3:45 am

I believe this is a double edged sword in which the people have all good intentions, but it can easily go off to wanting to game the system so one can move quicker to the top. I mean, just look at the effect of “the importance of being first” and the number of articles that are posted each day within the first 5 minutes of 20:00 GMT.

Yes, in the end the good work will shine through, but for the short term gain, “the importance of being first” (publishing an article a few months before acceptance) might just get you that grant, that observing proposal, that prof come to talk to you at a conference to offer you a job. You are posting about submitting to ArXiv after you submitted to the journal, but there’s an even more worrying trend of people posting “draft versions”. What’s next? Post up a successful observing proposal, the data is coming in, results are mine?

I was listening to a podcast with Gary Taubes, a science journalist the other day and it is scary how he summarized the positions of scientists from his point of view nowadays. I do not claim I agree with his view, but it was interesting to hear it. Here is an excerpt:

“Now there’s room for hundreds because of the blogsphere, because of the explosion in information. And the competition is to be loud and sure of yourself. That’s how you get to the top. And of course, you could still be right. I don’t want to suggest that being loud and sure of yourself proves you are wrong. It doesn’t. But it does change the culture of the profession. The very first book I wrote about physics, Nobel Dreams, the physicist I was writing about who ran this experiment was a very Machiavellian Italian physicist who–he taught at Harvard and he worked in Geneva and he commuted between the two weekly; and he had been on on virtually everything he had done. But he’d just about demonstrated how he could move up to the very top of his field by claiming a discovery and then kind of moving on to the next experiment and leaving better scientists to clean up the mess after him. And this has been a common theme in everything I’ve written about. In making declarative pronouncements based on preliminary data, and fighting viciously to get people to believe in you, you can do very well for yourself in these careers. Nobody moves forward by spending their career checking other people’s work to see if it was right or not. And actually one of my favorite lines from physics was from this Nobel Prize winner Sam Ting at MIT, who said to me, if I can get this right: To be first and right is good; to be first and wrong is not so good; and to be second and right is meaningless. That’s deep.”

I admit, I don’t look back after months of a so-so article to see how it was improved. I can hardly keep up with the daily mailing as it is. I just hope that all this rush to put out papers and results “first” does not devaluate the quality of the data.

Also, I point out, I have received good comments from people when I put my papers on astro-ph after they have been submitted and they can still be revised before the final journal version.


34 Rob Seaman December 13, 2011 at 11:37 am

An interesting discussion with some legs, since this is at least the third venue it’s jumped to this week with many dozens of thoughtful posts. Some observations. Whatever the wisdom of waiting until acceptance to post a preprint, unless arXiv itself were to enforce such a policy, some will jump the gun. At that point it is pure natural selection. If an early posting strategy preferentially pays off, then more authors will do it. Second, this is clearly an effect that varies between disciplines and likely between neighboring sub-disciplines. Astronomy as a whole appears to be very much in the vanguard here, but different topical communities will either randomly or on purpose make very different choices about the wisdom of early posting of preprints. And third, preprints of the refereed literature are not the only content appropriate for arXiv. “Priority” is not the only game in town. And researchers who aspire to tenure are not the only stakeholders.


35 Phil Massey December 13, 2011 at 1:54 pm

Well, my experience has been that even the worse referee helps me improve a paper, even if it’s only forcing me to explain things in such a way that they are clearer. And, that is the version of the paper I want people to be reading. As Lee Anne notes, you want people to be using and thinking about the “final” version of the paper, not just something they found lying around on the internet. It would feel like cheating to me to post to astro-ph before having the paper formally accepted by the journal. I do pick out several people from the reference list whose work I cite, especially if I criticize it in any way, so that they can get in their two cents worth before the paper is published. That’s only fair. Should they read it on astro-ph instead? I’d feel like a jerk. I also find that many of my colleagues feel strongly enough about this subject to have adopted the same rule of thumb I have, namely that I will never referee a paper that has already been thrown up on astro-ph. If they have that little respect for the refereeing process, fine, let them “publish” on the web. But it doesn’t (and shouldn’t) count. IMNSHO.


36 Scott Ransom December 13, 2011 at 2:24 pm

Phil, the funny thing is that I like to post early on the arXiv early because I have tremendous respect for the value the refereeing process, not because I lack respect for it. And I don’t mind updating the version of my papers on the arXiv once official refereeing has been completed — I have no problem if people see the improvements or changes that were made along the way. That’s how science works.

37 August Muench December 13, 2011 at 4:19 pm

So the list of papers here: Recent Astrostatistics Papers should mostly be ignored? I see.

38 August Muench December 13, 2011 at 3:44 pm

while we are tilting at windmills, what’s up with this papers as “final” premise?


39 Peter Melchior December 13, 2011 at 7:56 pm

There is that fundamental difference: arXiv has revisions (and encourages us to make use of them), while a journal typically has not. I think that being able to correct a flaw or an error on arXiv is important and useful, since we all know accepted papers in journals that should have been revised.

40 Rob Seaman December 13, 2011 at 4:04 pm

I highly recommend Dava Sobel’s “A More Perfect Heaven”. On his deathbed, Copernicus was handed the proofs for “De Revolutionibus”, thus marking a rather final copy. Remarkably the manuscript itself still exists – basically a nearly 500 year-old preprint. As with Galileo’s work, the Church provided referees almost as challenging as Phil has had to deal with. Will astro-ph’s seven-journals-you-can’t-say-on-TV continue to fill the same role in the astronomical community of 500 years hence? Or will complete online chaos/nirvana rule? One suspects that the reality will fall somewhere between – but that Sobel’s numinous books will still be “in-print” in some format.


41 August Muench December 13, 2011 at 4:23 pm

hi rob,

every time i hear this print/pre print anecdote i have to respond with this:

The Art of Computer Programming (TAOCP) by Donald E. Knuth.

yeah, the web is a ridiculous place to publish. the rights and responsibility are overwhelming. ;-)

42 Peter Melchior December 13, 2011 at 7:32 pm

I’m strongly in favor of submitting to arXiv and to a journal at the same time. Doing it this way implies that my co-authors and I are confident that our work will survive the critical assessment of the referee AND the community. In turn, as a reader, I don’t mind reading papers, that have the “submitted to MNRAS/A&A/ApJ…” tag line; if I think their results make sense, I would also not shy away from citing such a paper.

As several people have said before, there is the chance to get feedback on your work before it gets accepted by a journal/referee. If a paper is changed from its submission to arXiv to its final publication, I don’t see this as a problem per se, on the contrary, to me it shows a heathy scientific process. That doesn’t mean, you are a free to submit any rubbish to arXiv because you’ll ruin your reputation if you do this too often.


43 Jeremy December 14, 2011 at 12:10 am

A few thoughts…

First, I think there is a very serious generational divide here. In my experience, older scientists often view arXiv as a kind of global version of the rack of re-prints in the department library, and to therefore consider that they ought to be accepted papers, while younger scientists see it kind of as a blog, which you expect to attract comments and corrections that act as part of the refereeing process. And that explains why there are such strong feelings – there are fundamentally different views of what it is.

Personally, I don’t have a hard and fast rule – it varies from paper to paper. Sometimes there are external reasons for immediately needing to have a globally-accessible document that you can refer to in another document. Sometimes you feel hesitant about a result and need to hear what an anonymous referee has to say before you’re willing to have the wider community see it. Sometimes you think the wider community will act as a much more useful source of comments than one random referee. Sometimes you are submitting to a journal that forbids early posting. The pro/con balance is not the same for every paper.


44 Ellie Newton December 14, 2011 at 10:31 am

I can’t say that I (as a younger scientist) have ever viewed the arXiv as a blog – most of what I know about astronomy originated on the arXiv. And on the contrary, in my experience graduate students are more hesitent to post to astro-ph pre-acceptance.

On a similar note, I do wish there was a certain amount of vetting on astro-ph. As a relatively new graduate student, I often find myself exploring very unfamiliar subjects on the arxiv and would appreciate outside verification (i.e. journal acceptance) that the paper I randomly picked up isn’t complete garbage. Obviously, I’m developing my own filters as I gain experience, but when I first started research as an undergrad I had very little on which to base my opinions.

45 Scott Ransom December 14, 2011 at 1:04 pm

Ellie: I’ve talked with quite a few graduate students about this and what I’ve heard agrees with what you say: grad students tend to be more hesitant to post early to the arXiv. Is it possible that that’s because advisors typically walk students through the process of producing their first couple papers, and when it comes to arXiv submission they recommend or even direct the student not to submit early? And perhaps give them the “cons” listed above? I know that I’ve seen that to some extent. I’m just wondering if it helps reconcile what you and Jeremy have suggested.

46 Peter Melchior December 14, 2011 at 1:50 pm

Jeremy: I disagree with the generational divide, and also with the notion of the younger generation using arXiv as some sort of blog. In my experience, it’s mainly the approach of your advisor that shapes your own. In the group where I did my PhD it was common practice (but not a general rule) to submit to arXiv early, so I followed this example and was never disappointed. I also know students whose advisors where reluctant to submit early (as Scott mentioned). But the age of the advisor was not the driving factor, I’ve seen examples of either approach from younger and older researchers alike.

47 Jeremy December 14, 2011 at 2:29 pm

Interesting! My perception was based on postdocs and younger faculty, rather than grad students, so it may be that there are actually 3 different generations: the two I was thinking of, followed by grad students who are more conservative in this regard because they are not yet confident in their abilities. It would be interesting to do a poll and find out people’s perceptions as a function of their career status.

48 D December 14, 2011 at 2:49 pm

I also disagree that this is a generational thing. Peter’s hypothesis seems more likely, and is certainly true for me. My advisor was very cautious and only submitted after acceptance, which is what I’ve done ever since. I’m still relatively young (first postdoc), and definitely do not view arXiv as a blog. For one, it’s acceptable practice to cite arXiv papers in ApJ, but I think they’d frown on citing random-blog-on-the-internets, or, you know, Wikipedia :)

49 Erik T December 14, 2011 at 6:30 pm

There are plenty examples of both, but I think the question is whether or not the younger folks *on the mean* submit before/after acceptance. I think Peter is probably right that graduate students almost always follow their advisors, but that doesn’t mean they continue to do so after they go on to somewhere else. I don’t know if its confidence in their ability so much as just there’s no one looking over their shoulder anymore, so they’re free to do whatever seems right to them.

These are mostly testable hypotheses, though, so data would be very interesting… the question is: will the paper on the topic be posted on the arXiv before, or after acceptance? :)

50 D December 14, 2011 at 2:43 pm

Clearly there are both benefits and drawbacks, but one drawback I haven’t seen clearly articulated is that non-experts will read (and maybe cite) your arXiv paper.

Sure, there will be a few experts who can offer you great feedback on your astro-ph version (I’ve offered such feedback myself), but the vast majority of people who read your paper won’t. A lot of them will be observationalists/theorists reading your theory/observation paper, and will just take what you say at face value if it seems reasonable. They don’t have the expertise to correct you on your code/data analysis, the way a referee certainly should.

And when they cite you later on, they probably will look up your paper to see if there’s now a journal version to cite, but will they read the whole paper again? Unlikely.


51 August Muench December 14, 2011 at 4:33 pm

A lot of them will be observationalists/theorists reading your theory/observation paper and will just take what you say at face value if it seems reasonable. They don’t have the expertise to correct you on your code/data analysis, the way a referee certainly should.

Ignoring the fact that the OP was about what you do with your paper on arXiv, I have to say that I really appreciate the problem expressed repeatedly already about having zero domain knowledge when trying to understand unfamiliar fields using arXiv. Nightly I try to understand the papers in cs.CY or cs.DL and have no idea which papers to trust and which to ignore. In astronomy we have ADS to help us figure out who to trust and to delve into people’s track records but I don’t know what tool to use in digital humanities or CS.

Nevertheless the articulations about how any document, published, arXived or blogged, are used by scientists in some of the previous posts are downright disturbing to me and I’m hoping people can clarify their approach a bit. Does the little gold star on a Nature paper really change our belief prior by so much that we quote its result at face value? Is this really how we use the literature? Are we being lazy or trapped by TMI or are we buying into a trust model for journal orchestrated peer review that maybe we shouldn’t? It gives me an earworm:

And You May Ask Yourself
How Do I Work This? …
And You May Ask Yourself
Where Does That Highway Go?
And You May Ask Yourself
Am I Right?…Am I Wrong?
And You May Tell Yourself

If I really make this about non-experts — about the general public for example — then I appreciate the point a bit more. It would be awesome to go down that rabbit hole — the public understanding of science via arXiv versus blogging versus journal published results — but maybe someone should write a cover story first so we have some better parameter space to comment in. I am not volunteering.

I also look forward to the day when refereeing includes people’s actual code.

52 Erik T December 14, 2011 at 6:21 pm

There’s a sub-point to the second pro that I think is very important: posting on astro-ph can bring to the submitter’s attention related work that they didn’t know about… While in some cases this is a sign that the submitter was sloppy about doing an in-depth lit search when writing the intro, there are also papers out there that have sort of fallen off the network, but still are of great relevance. So in this culture where “credit where credit is due” is (at least in principal) a central value, the extra time to have more people see the paper before it’s locked in stone is quite valuable.

On the other hand, sometimes this is a bad thing – e.g., the e-mail I’m sure many of us have gotten that says “My paper from 1986 that showed your main conclusion in footnote 12 of page 14 of the erratum, as long as you turn Figure 2 sidewise and squint just right.” But I think this is a worthwhile price to pay given that it might also catch some important past work you (and the rest of your collaborators/scientific clique) may have missed.


53 Marcel Haas December 15, 2011 at 11:07 am

Just my two cents: I agree with comments like ‘if you are confident enough to submit to a journal, you should be confident enough to show the paper to the rest of the world’. Also the early and free refereeing you get from readers of the arXiv can be very valuable. I do not agree with comments like ‘wrong results get posted, that will never get published and people are using wrong information in their work’. First: people can and should update their arXiv posting to the accepted version and indicate what has changed. If the paper does not get published, you should reconsider citing it when you are >yr or so into the future (as there probably is a reason why acceptance takes so long), and you should do some effort to find the paper you have found on the arXiv on ADS in order to see if the paper somehow did not get linked to the arXiv submission. In all those cases: check the paper to see if what you use is still valid. I think more care has to be taken by people who blindly cite papers, not by people who share their science in an early stage (other than that they have to make sure they update the info they put out).


54 Warrick December 16, 2011 at 6:45 am

A question and a comment.

For those who upload to arXiv when they submit papers, how many serious comments do you receive from unfamiliar authors? Following my supervisor’s example (I’m still a PhD student), I only upload after a paper is accepted, but I do circulate the submitted version to a few people outside the author list to get some extra input.

As for why grad students are timid about uploading on submission, I think that being accepted in a journal means there is one quasi-independent person who has declared that the work is of publishable quality. That makes it a bit more defensible when presented to the rest of the world.


55 Sarah December 23, 2011 at 5:30 am

Great discussion. I agree with many posters that there’s nothing wrong with posting to Arxiv on submission, and that it can carry significant advantages. But that statement comes with important sidenotes. A submission should always be labelled with its publication status so that readers know if contents might change with time (and authors should preferably upload a replacement once those have been made!). I think that peer review, while usually resulting in a better paper, does not ensure accuracy of the results or of their interpretation – that’s not saying the referee did a bad job, but (s)he is just one person with a limited eye and their own personal bias and that’s simply the reality.

Personally I’m actually more likely to contact an author about an Arxiv posting that is not yet accepted, as there’s the chance that my questions or comments might be addressed in the final paper. If it’s already labelled as “accepted”, well, what’s the point? (unless I really *need* the info, obviously, or I want to say something nice about the paper)

To those concerned about non-final plots or results propagating in the community, I think Marcel’s point is excellent that this is more a problem of blind citation. If you’re citing a result or referring to a plot without having checked on the paper’s status or made sure you understand its content, then you’ve not done a rigorous enough literature search. In any case, we cite unrefereed conference proceedings paper and as far as I know this has not resulted in a breakdown of the scientific process.

The issue of press releases and embargos is interesting, illustrated again this week with the posting to Arxiv of Murray-Clay & Loeb with an embargo label. Journals still insisting on embargos while allowing the paper to be posted to Arxiv are a little naive about the state of the media. As a blogger I’m allowed to write about it, but the press should keep shtum – why penalise the press in that way? And what if I blog on a mainstream media site, like The Guardian, is that allowed?

[NB Ivan Oransky's Embargo Watch blog is a great read on the topic of embargos in scientific publishing.]


56 KT January 16, 2012 at 2:38 pm

Making peer-reviewed papers freely available on the ArXiv is crucial as published journal costs are out of reach of many scientists. So if it was posted before peer-review, it is critical to update the paper on the ArXiv once it’s published in the journal. Hear, hear for Marcel.


57 Orsola De Marco April 24, 2012 at 7:52 am

The refereeing process is no guarantee of correctness. It just gives a false sense of security. In today’s world of huge information overload (including vast amounts of refereed papers) buyers beware. I used to advocate arXiv submission after acceptance. I completely changed my mind. I think that posting should be done at submission and clearly invite comments. I also wish that the comments to posted paper were public (threaded on the website) rather than e-mailed to the authors. The comments thread would act as a far better guarantee of quality than any refereeing. (I still would keep the refereeing process in place).


58 Pauli October 29, 2013 at 6:20 am

I submitted to arXiv after appectance to peer-reviewed journal because as an independent researcher I can’t afford journal’s publication fee.


59 Brian November 7, 2013 at 11:28 am

No one seems to have mentioned the attitudes of the journals to arXiv. Are there journals who might not want to publish a paper if it has already been ‘published’ in arXiv? My own view (from the field of Quantitive Biology) is that in many cases neither editors nor referees are doing their jobs. The editors are probably overwhelmed and the referees are not paid. I could cite recent examples from Nature and PLoS Medicine but that would probably land me in trouble. My own view is that we should consider abandoning the formal journals which are in any case proliferating at a quite alarming rate and find a way to encourage feed-back to papers published in arXiv. We all learn who is doing interesting work and who is not and with modern search engines the fact that there may be many rather weak peppers in arXIv is not an issue for me.


{ 5 trackbacks }

Leave a Comment

Previous post:

Next post: