Professor Edward J. Wegman’s independent report falsifying Michael Mann’s climate change “hockey stick,” famously concluded:
When massive amounts of public monies and human lives are at stake, academic work should have a more intense level of scrutiny and review.
In Part I of this commentary, we explained that the key to honest science is independent replication of the analysis and evaluation of important assumptions. But, we noted, that, alone, is not enough. The Wegman report also documented the need for the insulated communities of researchers to be willing to give serious attention to independent replication of their work so as to profit there from. Here’s how science can do that.
The “Team B” Idea
In the cold war days, the military used an approach called “Team B.” This team (alternatively called the Red team) developed approaches to counter the efforts of Team A (the Blue team). Still in use today, it has some strengths and some weaknesses. Without question, however, it always sharpened the quality of Team A’s work, finding errors in analysis, weaknesses of assumptions and entire areas of interest that opened up Team A’s thought processes and analytical options.
Team B operated within the professional “community,” making its contributions instantly credible and mandatory for review by Team A and the eventual policy makers.
The Marshall Institute had engendered discussion of this approach for today’s scientific policy issues, as have members of the American Physical Society. In particular, Dr. William Happer, Cyrus Fogg Brackett Professor of Physics at Princeton University, member of the National Academy of Sciences and Fellow of the American Physical Society (APS), has proposed a Team B approach within the APS in order to address climate issues. Without question, the APS has enough independent and qualified members to have been able to audit the critical work underlying the IPCC reports and inform journal publisher’s of the important analytical elements to which regular peer-reviewers should give special attention.
A Proposal
I have little confidence in Happer’s approach, however. It stands too far back from the critical scientific process and will make contributions too late in the policy process. Instead, we can take a page from the book of the existing science policy bureaucracy – environmental assessments.
Under the National Environmental Policy Act, a project using significant federal funds cannot move forward until some form of environmental assessment has been completed and approved. These assessments have, of course, been misused to delay and prevent projects. However, at their heart, they have significantly improved proposals with the potential to adversely impact the environment. Correcting to eliminate the pathologies of the process, something similar could be used in other areas.
In the context of science-based policy today, Team B would be the peer-review group required for any scientific or engineering initiative (including grants and grant programs) that Congress or the Federal bureaucracy planned to fund above a trigger level, or which would have such an impact on the private sector. The trigger might be $500 million, for example.
Before Congress authorized expenditure of such funds, the bill sponsor would need to offer a Team B report that validated the basis otherwise offered to justify the expenditure. A National Science Foundation program manager wishing to fund a subject area that expands high-cost research would need a valid Team B analysis of the science to ensure the area of investigation is justified. The Department of Energy, the Interior Department and the Environmental Protection Agency would need a Team B report on their major research initiatives and regulatory proposals. Any major studies commissioned or conducted by the National Research Council (and its subsidiaries) would require a Team B review.
In every case, a Team B report would involve an independent replication of the analysis, including assumptions, selection and screening of data, logic of computer code, statistical analysis and presentation of data. The Team B report would have to demonstrate its independence, using the tools discussed in the Wegman report.
This independence is a key aspect of a Team B approach. It would not be helpful to simply pit antagonists against each other; and, of course, one cannot permit the incestuousness observed in the Mann-Jones et al network. Rather, one wants a “one-off” approach. In nearly every case, one would want a Team B to include a statistician familiar with the statistical methods used, but not familiar with the authors and perhaps only peripherally knowledgeable about the scientific subject. They can obtain the information about the subject from the subject matter specialists on the team.
The real challenge is selecting subject matter specialists. In cases where funding opportunities are controlled by cliques, the challenge increases. Because all Team B work should be done in the light of day, perhaps all one needs is an honest Team B – (OK, call me Diogenes). An open process, one that access the entire planet’s available and experienced scientific community through the internet, may be the guarantor of a quality project.
The Upside of Reform
While Wegman did not believe peer review ought to be through blogs, without question, without McIntyre’s efforts, as recorded on his blog site, Mann’s corrupt hockey stick would never have been exposed. A competent Team B could harvest the intellectual contributions from an engaged web-community to good effect, especially with regard to highly technical issues. Further, the anonymity of the web would allow specialists to provide critical information without having to be worried about being outcast by the specialists’ cliques.
Surely a Team B approach would evolve and mature over time. What is certain, however, is that even the threat of a Team B review would occasion a return to trustworthy science and a fundamental return to the bedrock principles of scientific inquiry. As Wegman, Montford and McIntyre have made unutterably clear, if we need heightened oversight of financial institutions, we surely need such heightened oversight of the science upon which we found science-related policies having large economic consequences.
- The Most Progressive Budget in Virginia’s History - December 21, 2019
- When is a Clean Water Act Permit Needed? - December 21, 2019
- Should U.S. Consider Modern Monetary Theory to Improve Economy? - December 21, 2019