real action : Marc : Tue, 30 Nov 1999 15:39:01 GMT

a follow up on a recent discussion in the Old Forum about PubMed Central:
Sergei got the following statements from "Grand Central"
":the peer-review system has been quite effective
: and determining an additional review system
: may be challenging."

- Along with Ron’s frustration, I add mine and say that the peer review has been quite effective indeed when one remembers that "peer" comes from the Latin and French "par-pair", meaning "equal" but also "double" or "alike". The name of the system carries its idiosyncrasy, which by definition counterselects originality.

I am now split between a total disinterest for Grand Central, oops PubMed Central, or a very aggressive involvement and attempt to lobbying. Ron switched me into the aggressive mode. There we are again then :

I was recently in a meeting where I discovered that many fairly well established and (not always) young colleagues expressed the same bitterness if not disgust towards "peer reviews". I beg then that we do something about it.

I suggest to lobby Scientific Societies such as ASM, ASH, NAS, etc. with petitions and list of signatures to adopt new review guidelines.

Once, in the old forum, I suggested an alternative procedure to the present peer review system (posted on Jan 7, 99 and called POWERS). The main goal was to avoid concentration of power to professional reviewers guarding their hunting preserve of dogmas. To that goal I suggested that:

* Reviews do not have to be a logorrheic detailed reviews, but could consist in attributing a ranking - exceptional, top, medium or mediocre- to a series of 10 criterions.

* Criterions might include : originality, scientific demonstration, ability to reproduce the experiments, didactism, writing, etc. (suggestions ?)

* Evaluation through this grid should be made (online?) by a panel of at least 15 reviewers

* Reviewers have to be picked at random among ALL peers (the 1st, 2d and last authors of articles in the field published the last five years, regardless of any other particularity of lab origin, status, etc.).

* no reviewer gets more than 5 papers a year to review

* No second hand review

Whether based on the one described above or another one, HBG should come up with alternative proposals that we should discuss and submit to petitions. I am confident that hundreds of signatures could be pooled on ANY alternative to the present scandalous system.

Then, we'll think about the next real action.

I’ll try again next year, guys :)

Marc S

RE:real action : pluie : Wed, 01 Dec 1999 14:44:52 GMT

I agree with marc

RE:real action : satamas : Wed, 01 Dec 1999 15:27:50 GMT

Yes, peer-review is supposed to be a communication of the equal peers, but, you know, sometimes some of us are more equal than others. So I guess it is up to us to develop something new instead of peer-review.

Mark already suggested a way to go in his earlier POWERS proposal ( http://www.heraclitean.com/bbs/archives/arch1/messages/94.html). Jim Till and myself commented there ( http://www.heraclitean.com/bbs/archives/arch1/messages/98.html and http://www.heraclitean.com/bbs/archives/arch1/messages/96.html). That was a good start in addressing this important issue. Now Mark goes on in an attempt to make his POWERS idea more realistic, and I like that very much. Let me tell you though, that based on my brief experience with what we have published on HBG, finding even two or three reviewers is a real challenge. No one wants to be bothered. Will we find 15 people to review an article, even on-line? That is why I believe that the idea of pre-prints is great (thanks to Jim Till). My other concern is about the suggested originality criterion. For example, Dov Henis is being very original, suggesting that nucleotides are living beings. I would not say that everyone is very enthusiastic about this case. What is original to one person, may sound crazy to others. Lastly, 15 reviewers is definitely much better than 2, but even 15 people can be wrong all together.
realistic real action : sitbon : Fri, 03 Dec 1999 18:14:07 GMT

A more detailed response to Sergei’s and Jim Till :

In my mind POWERS (Plenty Of RevieWERS) would apply more readily to the evaluation of experimental reports than to theoretical reports such as most of the HBG exchanges. With this restriction in mind, the following points might answer more precisely your and Jim Till’s objections :

Sergei, you wrote: "No one wants to be bothered. Will we find 15 people to review an article, even on-line?"

Marc: AN OVERWHELMING MAJORITY of " relevant " scientists contributing " relevantly " to their fields in " relevant " journals are NEVER OR VERY SELDOMLY solicitated for manuscript reviews. Those would be delighted to be part of the decisional process and get some of the acknowledgments. The real questions are who are they, where are they, how recruiting them, and mostly how to overcome the cliques which will argue on their "hyper"competence to maintain their scandalous and counterproductive privileges?

1-The procedure : One should not have to combine Heraclit, Shakespeare and Darwin’s skills to be part of the review process. Appropriate reviews of EXPERIMENTAL REPORTS can be achieved by TICKING BOXES. The result won’t be less relevant than the confrontation of two or three big egos...

2-The criterions : I’m presently reviewing two grant proposals which, combined, ask to rate :

Innovation
Feasability
Originality
Soundness
Methodology and
Relevance to the Organization/Journal

one could also add

Accurate Referencing
Information sufficient to reproduce the experiments
Presentation (didactism, writing, and Figs and tables)

With few other criterions (suggestions?) a thorough evaluation of any experimental report could be achieved. By doing so, you allow the recruitment of scientists all around the world and not just those able to master the use of which and that (I wish I was, though).

3-The Journal editorial policy: Each BOX WOULD HAVE A DIFFERENT PONDERATION FACTOR dictated by the Journal editorial policy. This would help maintaining editorial choices through this weighing. This system will reduce the sprinkling of the review with irrelevant manifestations of egos and etc. see my former complaints.

4-The process: Such a procedure would allow a statistical and computed process of the review. Specific changes could be suggested by reviewers but frankly, if it is not a consensus, WHO CARES if those changes are taken into account or not! ! !

5-The panel of reviewers: The broadest panel possible should be seeked. The goal is to reach a non-cliquish-biased " random selection " :) through a databank of potential reviewers. This databank of, for instance, relevant authors (1st, 2nd and last ?) of articles referenced in PubMed can be achieved by a search engine adapted for PubMed to this purpose (Sergei, THIS IS THE KEYSTONE OF THE PROPOSAL). The goal is to scatter the initial roadblock of a truly scientific evaluation : the entirely biased selection of reviewers). The absence of any other selective parameter, besides being a major author in the field in the last 5 years, multiplies the potential number of reviewers by the hundred.

Sergei: "My other concern is about the suggested originality criterion. "

Marc: see 3- above

Sergei: "Lastly, 15 reviewers is definitely much better than 2, but even 15 people can be wrong all together."

Marc: you bet ! But I do think that in contrary to very powerful and less powerful barons, who are necessarily busy defending their barony or their patrch, average scientists will be more in favor of discovery and stimulating innovations that you seem to believe.

And as a common friend of ours would say : what’s smarter than 2 or three scientists ?

15 or 20 of them! sometimes… :)

marc