http://acdlabs.typepad.com/my_weblog/2007/06/robiens_and_mod.html

There are still some remaining questions about how Modgraph came to an average deviation of 1.40 ppm.

So before we can make a final decision on performance, I think Modgraph needs to make very clear the following:

1. What is the overlap between NMRShiftDB and Modgraph’s NMR prediction databases? Further, with several different database sources how much duplication of data exists across the databases and within the entire package?

2. Once that overlap is removed from the dataset, what is the final deviation produced by NMRPredict?

I think this information needs to be made very clear from Modgraph before they can claim to be, “the most accurate carbon 13 NMR predictor in an independent evaluation?”

CSEARCH performs at 2.22ppm/2.19ppm before/after correction with a certain
network and a certain parameter setting.

ACD’s CNMR-Predictor performs at 1.59 as can be seen from the table above.

MODGRAPH’s NMRPredict performs at 1.40ppm as can be seen from their
website.

That’s the facts, which came out from the discussion now ….

I think, it’s good to know that, verified on the same dataset. I initiated
this discussion, therefore I want to recall the starting point of this
discussion:

The central question of my webpage at nmrpredict.orc.univie.ac.at was:
(Statements copied from:
nmrpredict.orc.univie.ac.at/csearchlite/Robien2Ryan_May31_2007.html
nmrpredict.orc.univie.ac.at/csearchlite/enjoy_its_free.html)

“Is there a visible improvement when you spent a few hours on
data-correction, both on a statistical basis and also with a few specific
examples?”

The answer was:

“This small improvement – which is far away from being perfect
and/or complete – without the use of one, single literature citation
improves the assignment quality by ca. 6,200 ppm !”

“My finding was, that spending less than an afternoon you can improve a
collection of some 20,000 NMR-spectra having more than 200,000 shiftvalues
by 0.03ppm or ca. 6,200ppm in total.”

My summary was:

“That’s an amazing result ! NMRShiftDB has been online for about 5 years,
the 20 most important contributors are mentioned on their homepage by name
and I think many other people have contributed into this project. All
these people together were unable to spend less than an afternoon on
data-correction within 5 years! That’s exactly the point nobody seems to
(be willing to) understand !”

Peter Murray-Rust stated on his blog with respect to NMRShiftDB-data:

“…… It contains mechanisms for assessing data quality automatically.
For example software can be run that will indicate whether values are
seriously in error………”

My answer was:

“That’s really a great idea AFTER being 5 years on the web ! ……. let
me know which data-correction protocol has been applied to the
NMRShiftDB-data leading to deviations of about 130ppm (= 2/3 of the usual
C-NMR shift range) BEFORE you put them on the web ! ……”

Final remarks:

Only 4 main-contributors are involved in this discussion:

That’s me – I have started this ‘avalanche’
Antony Williams and Ryan Sasaki from ACD
Peter Murray-Rust from Cambridge

I couldn’t find any comment by NMRShiftDB people including Christoph
Steinbeck.

In order to improve science with a project like NMRShiftDB, the first step
should be to look around what is already there (“State-of-the-art”), the
second step should be to avoid the errors other people have already paid for
(“Avoid starting from scratch”) and the third step should be avoiding
hacking the same numbers/strings into a computer (“share resources”)
The community needs chemical data, THEY SHOULD BE VALIDATED ACCORDING
TO THE STATE-OF-THE-ART (when you cite me, please use both parts of this
sentence!) – I have shown by spending a few minutes CPU-time, that the
second part of this sentence was not true for NMRShiftDB as downloaded
on March 10th, 2007.

What’s the result of this discussion:

a) We know how accurate 3 programs perform on the same dataset
b) There was severe error-correction performed on NMRShiftDB after
my analysis

Both items are valuable contributions for the scientific community.

At the end a personal remark to Christoph Steinbeck:

Copied from: sourceforge.net/forum/forum.php?forum_id=681882

Start citation —-

….. We also feel that this makes a strong case for our open access,
open source policy, which gave our reviewer the chance to access our full
material and run this test. As Eric Raymond puts it: “Given enough
eyeballs, all bugs are shallow”

End of citation —-

I clearly state here: Don’t use the “open access, open source policy”
as excuse. You simply have the responsibility as database supplier
and/or project manager to apply basic statistical tests on the data,
BEFORE you make them available to the scientific community in order
to obtain a reasonable quality of the data you provide. I am
talking only about errors, which can be found just by ‘snipping
fingers’. Obviously this point has been missed over 5 years.

Wolfgang Robien, June 6th, 2007

Peter Murray-Rust and Tony Williams have added their two cents to this debate on their respective blogs. Peter provides a great justification on providing open access to scientific information:In the case of NMRShiftDB I am firmly of the opinion that