Talk:Peer review and the Wikipedia process

Ref.: Defense AT&L (November-December 2004 Vol. 33, No.6), p. 58.

The Dancer and the Piper: Resolving Problems with Government Research Contracting

In the years following World War II, there were collegial relationships between researchers in government laboratories and scientists in academe and industry. Today, however, the practice is to contract for services, and the governing public laws have become so complex that government project leaders responsible for initiating and managing contracts must have not only an advanced technical degree but also extensive training in finances, contract law, security, document control, ethics, fraud-waste-abuse, technology transfer, equal employment opportunities, small business, historically black colleges, etc. Scientists from academia and industry who com- pete for these contracts often lack similar training, and this contributes to conflict and confusion when a contract proposal is rejected. There are government management practices that also contribute to post-award disarray, and three are summarized herein together with hypotheses on root causes and suggestions for resolution. The problems discussed are not new, but they have become so pervasive over the years that the authors believe new approaches are worth serious consideration.

Proposal Evaluation

A persistent problem that faces all project leaders is how to conduct fair evaluations of contract proposals when leading technical expertise doesn’t exist within the contracting agency. The most common practice has been to solicit volunteer reviewers from “peers” in the scientific community and then hold the evaluator names anonymous to avoid undue pressure during and after the review. There are three problems with this practice: (1) the “peers” are often competitors who abuse their anonymous position to further personal research interests; (2) they are not always as qualified as needed; and (3) there is no accountability of the reviewers to assure their best performance because their reviewing effort is a “donated” service. Our suggestion is for the project leader to recruit higher levels of talent among the “peers” by offering financial payment to those who agree to perform the review and who are both free of conflict of interest and willing to publish their names and credentials.

Management Bias

Another nationally pervasive problem in competitive contracting occurs when a bidder who fails to win an award believes the competition was unfair because the project leader was biased. Reputations about bias invariably arise when one person in a competitive pool is perceived to have greater access to a project leader than others. Although project leaders are honor-bound to behave according to the agency standards of conduct, experience has shown that it is best for upper management to verify as well as to trust. Our recommendation is to have project leaders present frequent in-house reviews—and even for independent offices, such as the legal office, comptroller, contracts office, and merit pay supervisors— prior to the award of a contract.

Level of Funding

In recent years, the Department of Defense, National Aeronautics and Space Administration, National Institutes of Health, and other government agencies have been identifying gaps in the U.S. technology base that are critical to their missions. The solution in many of these agencies has been to set aside limited undesignated funds and issue generic broad agency announcements soliciting open research proposals from scientists in academe and industry. A significant problem with this practice has been a tendency to spread the funding too thinly, as a result of which, the research is incomplete, or there is no effective technology transfer, or the investment is wasted. The root cause among bidders is that the primary focus is on developing the technical content of the proposal, and cost estimation is a low priority; whereas the problem with project leaders is that they tend to spread the available funding over too many studies.

Our recommendation is for project leaders to abandon their traditional go-it-alone approach and team with other government agencies with common interests to lay out a life-cycle plan that will ensure the new technology is not only studied, but also developed and transferred into a useful government or industry application. For example, a recent U.S. Army research program (joining of metals) was forwarded to a U.S. Navy project leader with mission funding for developing process controls and then to a U.S. Air Force project leader for commercialization in a small business program. Since activity of this nature is beyond a project leader’s normal job description (and is difficult, time-consuming, and prone to failure), we recommend that upper management set up a reward system for those persons willing to look outside the envelope.

Dr. Ronald W. Armstrong, professor emeritus, University of Maryland, College Park, Md.

Dr. Roger B. Clough, (retired) National Institute of Standards and Technology, Gaithersburg, Md.

Dr. Laszlo B. Kish, associate professor, Texas A&M University, College Station, Texas.

George K. Lucey, project leader (retired) Army Research Laboratories, Adelphi, Md.

END