Keywords: tools of doubt, experts, peer review, amount of knowledge
Try to imagine, if you can, how much knowledge there is in the world. Here’s an image I was given to try to illustrate how awesomely large the amount of knowledge is. Imagine collecting a set of textbooks that would give a reasonable overview of all of mathematics (or, if that turns you off, pick a subject more palatable to you.) Just selecting one textbook from each reasonably separate area of math (geometry, trigonometry, topology, combinatorics, etc.) would easily fill a football field. One copy of each math textbook ever written would stuff the average town library. And this is just for one subject! Another image is to imagine the largest library you’ve ever been in or seen in a movie, and realize that that is only a tiny fraction of all the books ever written. Even the Library of Congress (shown below and which has an estimated 167 million books, maps, recordings etc.) is only a small portion of all the documents ever created. The point is, there is a lot of knowledge. And just because I’m a nerd that likes to throw big numbers around: the first article I came across regarding the total amount of data in the world gave an estimate of 18 zetabytes in 2018 (1 ZB = 10007 bytes = 1021 bytes = 1000000000000000000000 bytes = 1000 exabytes.) The article’s prediction for 2025 is 175 zettabytes which, “at the average current internet connection speed, would take 1.8 billion years to download.” Isn’t that awesome?
(In the face of all this information, how is it that some people think a single ancient text contains the wisdom of the ages?)
The point is that nobody has the time to research everything. Nobody can know everything. So, how do we know what the facts are? Where do we find trustworthy information that we can act on? The answer, of course, is to use experts. Realistically, we don’t need to depend on experts for everything. We are all experts in some things; we couldn’t survive the day without expertise about basic daily actions. But there are so many things to know that we have no choice but to accept expert opinion, even if it is simply something like trusting a mechanic to fix your car. But one of the conundrums of being a doubter is that argument from authority (argumentum ab auctoritate) is not valid. Just because someone says it is true doesn’t mean it is. So, we have to be careful in how and when we choose an expert. How do we know who to trust?
Wikipedia is a good example to look at to evaluate this question. As useful as Wikipedia is, you have to be cautious. There are risks to crowdsourcing information. If you don’t know, just about any person willing to learn the ropes of how to provide input into Wikipedia (and has internet access) can do so. So non-experts (even nefarious people) can add to Wikipedia. There are mechanisms to moderate entries including behind the scenes debates that can be reviewed. But there are some really good indications of the quality of Wikipedia entries. Two main indicators are warnings at the top of the page and the number of citations. The moderators actively put comments at the top of pages that seem a little thin or suspicious. Sometimes there are also warnings on the right hand side such as “considered pseudoscience.” And, in terms of any claim of expertise, clear citation of research or original works is a must. Interestingly, I recently came across an entry that had both warnings and a ton of references. So they aren’t mutually exclusive.
These indicators have analogs when evaluating the expertise of individual people or organizations. Are there major criticisms of them? Have they authored peer reviewed papers or are they able to cite published peer reviewed papers to support their theories and opinions? Also, do the people making criticisms, or the publishers involved, have the credentials to be taken seriously?
A person’s education can also be a good indicator. Not every highly educated person has a degree, but degrees and certificates are still an indicator of expertise – assuming the bestowing organization has its own pedigree. Someone with a PhD really is an expert – or, at least, more expert than the average person – in their subject of study. But that expertise can be very narrow. Here’s an illustration I was shown that illustrates the knowledge learned while obtaining university degrees. Draw a circle representing all the knowledge in a particular subject (like math.) Put the smallest dot you can in the center of the circle. That dot represents the amount of knowledge gained by obtaining a bachelor’s degree (traditionally four years). Now draw the smallest circle you can around that dot. That represents the amount of knowledge gained by obtaining a master’s degree (often two more years). Now draw a line from the center of the circle to the perimeter. This represents the knowledge gained by obtaining a PhD (often three to six more years.) Now draw a tiny arc over the end of the line where it intersects the perimeter. This represents what a PhD candidate contributes to the body of knowledge in order to receive the degree. PhD students are very focused.
Experience also plays a major part in developing expertise, so non-academic activities should be considered. After getting a Phd it is not unusual to go into a different area of study then the thesis. And one of the main things a PhD student learns is how to learn. So, post PhD work can expand expertise quite a bit. But, in general, experience is harder to evaluate (since certificates aren’t always given out.) But the same types of criteria can be used. Did the person work in a single area for a long time (the 10,000 hours hypothesis)? Was there career distinguished? Do other people recognize their expertise?
Unfortunately, it is not unusual for well-educated and experienced people to claim expertise beyond what they truly have. That’s why it’s important to not rely on a single source of expertise unless you have to.
A cautionary tale: I was once at a science based symposium where a presenter made what seemed to be questionable assertions. So I asked for references and I was given the name of a supposed expert. I found the “expert’s” website, and, although they were selling books they had written, there were no other cited publications at all (let alone peer reviewed papers.) Further, I was able to find evidence that they, at one time, placed “PhD” after their name but stopped doing so when it was pointed out they didn’t actually have one. It is unfortunate that an otherwise creditable symposium included a not so creditable presentation.
Another cautionary tale: One of the topics that came up during my yoga teacher training was ayurvedic “science” or “medicine.” This is an ancient practice that basis medicine on the five elements of water, fire, earth, air, and spirit (or variations thereof.) Many of the claims made seemed questionable. So, I went looking for peer reviewed papers, I found a curious situation. There are plenty of magazines dedicated to ayurveda that contain articles on medicine. What is curious is that the ones I found were magazines dedicated to ayurveda that cover no other topic. Further, I could find no articles on ayurveda in any established peer reviewed medical publication. Having worked for some large companies, I once came up with what I think of as a group version of Peter’s Principle (“People in a hierarchy rise to their level of incompetence”. The problem is that people can be really good at their job but then get promoted beyond their abilities.) My generalization is that an organization can grow to the point of having so much internal paperwork that they cease to provide value to the rest of the company (I’ve never come up with a cute name for this.) The ayurveda literature (and community) is self-contained. It feeds itself to the point of having no value outside itself. This closed self-validation (among other things) makes these articles not creditable.
A major point here is that you have to check both the credentials of an expert and the credentials of the credentials – potentially with multiple iterations. Fortunately, this chain is usually not very long since there are well established creditable publications and sources. One of the practices of a doubter is to develop a list of experts (people and organizations) that have a history of creditability.
Starting From Doubt 