Science: How Do We Know What We Know?
By Guy Meilleur
Standing on its dripline with a clipboard or a tablet, we cannot understand the tree. Surfing the internet, we don’t know all there is to know about a tree care practice. Science (literally “knowing”) is the “observation, identification, description, investigation, and explanation of natural phenomena…an activity that requires study and method. Knowledge gained through experience…Scientific means having an exact, objective, factual, systematic, or methodological basis.” Scattered impressions do not form knowledge. There must be a method for us to know what we know.
What’s called “the scientific method” favors controlled experimentation to answer big questions about physical phenomena, like “Why did that apple fall on my head?” Based on Galileo’s experiments rolling different sized balls, Isaac Newton mathematically proved that the same law of gravity governs the movement of apples, the moon, the planets, and all those other lights in the sky. Little did he know that, on that very same moon, astronaut David Scott would confirm that Law by dropping a hammer and a feather. They hit the ground at the same time.
Questions about natural phenomena, like “Can we make big old trees safer and healthier by pruning?” have not been answered with controlled experiments. Too many variables, years, and dollars would be involved. But that question can be answered by systematically pruning, and then methodically observing, describing, and explaining the tree response. Once the results are in, how are they processed so a reader can consider that scientific knowledge? The work must be reviewed by other experts in the field, known as peers.
Peer review can improve the quality of submittals, add credibility, and maintain standards of quality in any profession. Understanding the various ways that peers review different subjects, the standards to which studies and articles are held, and the methods they follow, can provide confidence that what we are reading is reliable, useful, scientific knowledge. Following are some peer review processes that are recognized and utilized in a variety of industries.
Medical peer review can truly be a matter of life or death so, it is considered first. A medical peer review process was detailed in the Practical Ethics of the Physician written by Ishāq ibn 'Alī al-Ruhāwī (854–931). Al-Ruhawi synthesized historical study and practice, drawing from the Greek Hippocrates and the Roman Galen, among many others. That process started with physicians methodically documenting the patient's condition, and every treatment.
The notes were then examined by a local medical council and compared to existing standards of medical care. If the patient died, woe them who fell short of those standards! If the patient was cured, that experience could inform an improvement of those standards. Today, this clinical peer review applies to different disciplines, so there is physician peer review, nursing peer review, etc.
Educational peer review achieves learning objectives, reaching for higher order processes in three domains: knowledge, emotion, and action. Like doctors and nurses working together, these domains must interact for successful outcomes. Educational peer review can be an isolated process with an end product, as in science and medicine. It can also be a collaborative teaching tool that helps students improve through collaboration.
Science: How Do We Know
Professional peer review processes are used in many other professional fields: accounting, law, and engineering. There is technical peer review in more practice-oriented fields like aviation, forest fire management, and arboriculture. As in education, professional peer review can work peer-to-peer. Gear inspections doublecheck equipment reliability and reinforce a safety culture for arborists and their teams. Peer review of practices can improve standards, and validate professional credentials.
Peer reviewed credentialing generally involves some kind of testing. Committees of professionals collaborate on the design of the test, the subjects that it covers, and the degree of difficulty of the material. Certification testing can be done online. Qualifications like TRAQ and TPAQ involve a practical component, so those tests are in person. Tests and processes can be revised over time, after review by a community of subject matter experts (SMEs).
Peer review for publication also requires a community of SMEs with qualifying education and experience, who are able to perform a reasonably impartial review. The reviewer is typically not identified to the author, so the review is “single blind.” Less commonly, if the author is also unknown to the reviewer, the review is “double blind.” Different publications in the arboriculture industry have different peer review processes:
• Tree Care Industry magazine uses in-house SMEs, and members in their editorial committee.
• Journals like Urban Forestry and Urban Greening, Journal of Urban Ecology, Phytopathology, Trees: Structure and Function, and Journal of Environmental Horticulture have single- or double-blind reviews.
• The International Society of Arboriculture (ISA) journal is Arboriculture and Urban Forestry (AUF). It is open source (no paywall) and its searchable archives go back 50 years. If your study is just starting, AUF considers short communications on original investigations with preliminary results, and analysis of procedures, equipment, or management systems that will lead to the improvement of best management practices.
• ISA’s Arborist News used at least three reviewers. Other SMEs were recruited for depth on technical articles, like the Detective Dendro (DD) series. The DD peer review process was single-blind, highly educational for this author, and mostly enjoyable. The rigor often matched or exceeded the review process for some standards, BMPs, and conference proceedings that are called “peer-reviewed”.
• Smaller publications can have excellent editorial control. Reviewers can guide authors to make their work the best it can be. Reviewers and editors alike can also lack the ability to engage the material. There may be bias. When feedback from reviewers is confined to brief opinions without substance, the readers lose.
• The Consultant uses a team of ASCA members and outside SMEs. Content is checked for accuracy, relevance to consulting arborists, and scope. Could the author broaden the view? Or delve deeper into a topic of interest? Authors can grow in new directions and become recognized experts. If you are interested in a topic, why not submit a draft to a publication where it can be improved and approved?
Academic peer review, despite the old saying “Publish or Perish!,” is not a matter of life and death. But tenure–a lifetime appointment with more money, status, and academic freedom–is largely based on publication in academic journals. In 1940, university professors and over 250 schools wrote The Statement of Principles on Academic Freedom and Tenure: “The common good depends upon the free search for truth and its free exposition." This search requires having the author's methods and findings reviewed by academic peers.
As in other professions, editors consult these reviews and decide whether the work should be published. In the academic world, the stakes, and the pressure, are very high. Academic journals primarily serve academics. Citations are gold to authors and to journals. The more references that are cited, the higher the Impact Factor and scientific ranking for a journal. Unfortunately, academic peer review does not guarantee quality in publications:
• In 2019, Duke University agreed to pay $112,500,000.00 to resolve damages from faulty research tied to federal grants.
• The design of an experiment can bias its findings. One study on compost tea factored the $5,000 equipment expense into the cost of one batch of tea. That data did not support the conclusion that compost tea was not cost-effective.
• Narrow expertise in an editor or reviewer can severely limit outcomes. New or unfamiliar ideas may be delayed or fundamentally altered. Novel findings may be overlooked or rejected.
• Flooding the zone of attention with scattered impressions makes verification impossible. Authors might cite a lack of experimental studies, and grasp for extraneous or spurious references, sometimes under pressure from editors. A recent article on pruning, for instance, cited 32 references. Most did not support the claims within. Some contradicted them.
• No source is safe from bias. On the US Forest Service webpage, a banner popped up that said “The Radical Left Democrats shut down the government. This government website will be updated periodically during the funding lapse for mission critical functions…”
Even in .edu or .gov domains, reading only the abstract of a document does not empower us to quote the conclusions with any confidence. Only by examining the contents and the references can we trust the conclusions. Now more than ever, relying on a single source in our search for knowledge can badly mislead us. Cross-referencing multiple sources to form our opinions takes time. But to understand complex and everchanging subjects like mature trees, it’s time well spent.
Despite its potential flaws, peer review is a vital process for any profession. Combining it with other forms of scientific validation, like documenting observations noted over time and protocols derived from analysis of data and experience, can provide rigor and confidence. Wherever we look, caveat lector--let the reader beware! Look carefully, analyze rigorously, and check those references before leaping to conclusions.
