Background:

If you don’t already know it, I am trained as a scientist.  I worked in research – medical, physiological, and husbandry, to be exact.  I grew up in a rigorously technical and scientific culture.  My father, and most of my male relatives, work in physics – my father was a nuclear and space physicist.  One sister is an environmental scientist.  I bring these associations up because I have come from a multi discipline culture.  I am not mired in any one world view or discipline and I try to regularly visit many disciplines to find understanding of, and solutions to, problems.

So imagine my feelings when some trainers declare that I, and my methods, are ‘unscientific’.  Science is not an attribute, like an ingredient, that you add to your product label.  Science is a process and skill set which allows us to study life and to garner useable information and, possibly,  improve how we do things.  Science is not the easiest discipline to master, and many never even give it a try.  I invite you to buck the trend and become literate in science, scientific process and scientific method.  Like me, you might really enjoy the fruits of your labor.  And, you will gain respect and authority when you are able to confidently find your way through misinformation and manipulation, and claim true understanding.

Let’s declare war on ‘sciency’ bullying and manipulation.  “How?”  you may ask?

DARE to THINK!!!  And, dare to take action to refresh, or advance, your abilities.

Embrace the journey

As when walking out on a frosty winter day, it can be hard to walk into a real study of science.  But, the process is full of special experiences and encompassing beauty.  Let’s start this journey together.

Challenge:  put thought to action!

Here is a simple challenge.  Choose a scientific paper.  Record the title and authors, using scientific citation formatting.  Simply stated, this is ‘title, authors, journal, date’ but the details can vary.  Here is a resource you might like to have:  How to cite a paper

Then watch my 14 minute presentation to help you navigate and assess science papers.  It is about 4 important things to look for whenever you read a scientific paper.  

Then report on your chosen paper.  Don’t worry about analyzing the whole paper.  Focus on sample size, sample selection, and internal and external validity.

Report back.  Use the form!

Tools for your Journey!

To support you doing this, I am providing some tools.

First is a form to use to record your paper:  Get it here

Next is the link to my video:    Tips on how to Evaluate Science Papers

Finally, the transcript of the video, for your later reference:  (see below)

“Hi, it’s Kayce, and thanks for joining me.  I wanted  to talk briefly about research and knowing how to understand the validity of what you’re reading, how to test it, how to assess it. And this is part of the path and becoming a critical and independent thinker. And my first class, and this didn’t come till I was taking classes on the doctoral level.

So, I think it should be covered before that. So I like to just share with you a few things that are easy to look for that will help you to evaluate what you read. And the first thing is many of us read reviews, right? Somebody else has done all the analysis. And so we read that and we go, Oh, goody.

Now I don’t have to do the analysis myself.  Wrong!  Because many people who write reviews wrote a review because they didn’t want to do the research themselves. And they make a critical mistake when they treat all research as being equal.  It is not.  Every situation will be different. So what are some of the things you can look at that can tell you quickly,

Well, whether or not that research might be good and valid?I’m going to talk about four things. Sample size, sample selection, and internal and external validity. Now sample size depends on various aspects of your research in general, the more the merrier, right? Not only the greater number, but the greater diversity and the selection should be random. Now that means if you’re going to study people,

you study all races, all locations, all types of work, all types of genders, right? Whatever. And the closer you get to, you know, a million people, the closer you are to being able to make conclusions about all the rest of the people in the world, But what can often happen, especially in animal studies where, you know,

the researcher doesn’t have unlimited space and so on, you might see a study and it has one animal. So Irene Pepperberg is a really rigorous researcher. She did beautiful stuff with Alex, the parrot.  He was a single subject. So what you could say about Alex, the parrot,  is what HE could do, but you couldn’t say that what Alex could do, all the other African gray parrots could do.

You just couldn’t make that conclusion. It was enough to show that Alex could do it. And it was rigorously shown. If you haven’t read the book on Irene Pepperberg’s work with Alex, do it. It’s great!  And of course, read the studies. You’ll learn an awful lot about not only the subject of the studies, but also how good research is conducted.

If you’re studying collies, for example, you can’t choose all the collies from the same breeding kennel.   They’re too closely related, or,  you can, but you can only make conclusions about that particular breeding line. You can’t extrapolate to all dogs, certainly, and not even to all collies. So the sample selection is very important. Okay? Now let’s talk about validity internal and external.

Internal is where your research design is a carefully constructed, carefully thought out so that your research is actually valid. So if you’re going to count behaviors, it’s going to be a lot better if you video the behavior and count from the video, because then you’ll be able to have lots of people count the same behavior session. And if all those people agree,

you have a good case for showing that it was a valid count. And it was an accurate count because all these people agreed. Whereas if you have some person that’s sitting at a zoo, clicking a little thing, when they see a specific behavior and the day has warm and they’re tired and they wish they had lunch. And Ooh, that’s a cute person walking – and you get the drift.

So internal validity is very important. Other problems with internal validity could be things like you’re studying bacteria and your sample got contaminated, or you’re trying to study what happens to animals in a lab, but you don’t have a lab to conduct your research in, so you’re doing your study someplace outside. One of the reasons that research is done inside of labs is because a lot of times the results are a little bit fragile and you’ll only see them if  you can keep everything constant, except the actual thing that you’re studying. Now, fortunately, with a lot of behavior studies, the results can be very robust and you can often see the, the results no matter where you do the study, right? So we have an advantage there, but you have to demonstrate that that’s the case. So you have to demonstrate that you have internal validity,

External validity gets back to do … do you, … did you demonstrate that you can justify the conclusions you made? So if you say I studied  52 dogs and a bunch were border collies and a bunch were livestock guardians, you can’t then extrapolate to say that all dog act this way. If you want to do that, you, you might be able to say all,

you know, dogs that are herders say that, but probably not. Unless you had a lot of different kinds of collies and a lot of different kinds of livestock guardians. So you have to make sure that the conclusions that are made in the paper were justified by the research. Okay? Now those were the four things I wanted to cover. Let me just also say a word about what you do with the hard work of analyzing these papers.

First of all, you really need to analyze every paper that you cite. And I know most of us don’t do that and you sure don’t do it at your lower levels because you don’t even know how to do it yet, right? We’re going to give you a spoiler alert. Now, you know. Now you can do it better, and you want to make a library.

And I use a spreadsheet, and let’s say the first column in the spreadsheet would be the title of the paper. And the second one would be the authors and maybe next I’d have the date. And then I’d have the school where it came from or the schools. Then maybe I would say the main conclusions of it. And then I would have the different factors.

Yeah. The internal validity, external validity, research design, and materials and methods, discussion, statistical analysis. That’s another very important one, but that’s too complex to go into now. So this is what I’m going to put in my spreadsheet ‘library of citations.’ There’s another thing I’m going to check. If I possibly can. When I was doing a lot of this,

there was a service called Science Citation Index, and that told us how many times a paper had been cited in publications. So if you read a paper and you thought it was really great, but it had only been cited one time, maybe you need to look further. If it was cited 2000 times, that’s probably a pretty strong paper. And as you go through the bibliographies,

you’re going to see, okay, here’s, here’s a paper that I’m interested in. And so you go pull it up and you read it and they list their bibliography and you find out, okay, what’s the earliest person. And you find the Seminole papers. What were the papers that this whole discipline is based on? You need to know those papers.

And once you go through the work of knowing them, because it’s not an easy task, you want to take a snapshot, put that in your library and give yourself a basis to easily refresh yourself on the features of it, and the validity of it. You know, I mean, it’s hard to do, right? You don’t want to have to do the whole process again.

So record your conclusions and keep them at your fingertips. It’s the biggest regret of my life, not the biggest regret of my whole life, but it’s a significant regret,  that while I was doing all this research (and that was before the internet, and don’t ask me when that was!) but I didn’t have a copy of all this. Back then, these things were done in a mechanical way,

and I couldn’t afford the kit and duplicate the records on all the papers I analyzed. It was over 350. And so those papers also weren’t on the internet because there wasn’t an internet yet. Then later on, when I wanted to talk about some of these great papers –  and did you know that a guy named M. E. Bitterman was training frogs and tuna fish and gold fish,

and you know, who knows what else? Amazing stuff, really. And this was long time ago. There was a veterinarian in South Africa named Hawthoorne with two O’s in the last name, and he showed that the safest way to catch an antelope was NOT to go in and do it really quickly and expertly because no matter how short and gentle it was, the stress that hit that antelope because of imagining what might happen next could, kill it on the spot.

And that by far, the safest way to catch an antelope was just to keep walking up on it, just putting on enough pressure that the antelope would just walk and you just kept walking until the antelope got tired and laid down, and then you could go up and catch it and move it and everything else. And it would be fine. You just had to walk a long time to do it.

But once I knew that, can you imagine how important that was working with exotic animals? Do you think I can find that paper now (and I may end up finding it because I believe I did take a copy of it and I am about to work on digitalizing my entire office. So I’ll get back to you if I find it), but I really wish I had that paper at my fingertips.

I can’t find it anywhere. All right. Just in case I didn’t make this point. Well enough, I’m going to also just end with the point on, don’t just rely on reviews. If you’re going to read a review, you need to review each paper that that review is based on because there are some lazy or insufficiently trained reviewers that will just lump all these studies together and treat them equally.

And they’re not necessarily at all equal. So to review, so to speak. When you look at a paper, you want to look at the sample size sample selection. Was it truly broad and random? Validity: Internal? In other words, was your internal design, correct, and protected from contamination and confounding of your results?   External:  does your research actually justify the conclusions that you make and the populations that you make them for.

And then,  do your own research, analyze the papers for all the reviews that you look at and analyze them independently and store the results of your analysis. It will pay big dividends later. All right. I hope this helps take care.”

Embrace the journey, and share your experiences with us.  I am eager to hear them.

Best wishes,

Kayce

copyright 2021 Kayce Cover