The Scientific Method

We’re all taught in elementary school about the scientific method:

1) Ask a question
2) Make observations and/or do some background research
3) Develop a hypothesis to explain observations
4) Test hypothesis
5) Draw conclusion
6) Report results

The Scientific Method, as shown in typical textbooks

Seems pretty cut-and-dried, doesn’t it?

As a practicing scientist, I know that science doesn’t always follow the scientific method.

It sure seems like we do, though. Every professional publication we write is done in a particular order, with an introduction (steps 1, 2, and 3), methods (step 4), results (also step 4), and discussion and conclusions section (step 5). But if you knew how things really worked…

There have been a few opportunities for non-scientists to get a glimpse into the wacky world of ‘real’ science. The hashtag #overlyhonestmethods went around Twitter a few months ago, wherein scientists around the world gave some insight into how their research really went.

But how else does practical science differ from the scientific method? For one, steps 5 and 6 above usually generate as many questions as they answer. A person could easily never get around to reporting the results, because they’re so busy trying to answer the new questions and test the new hypotheses that arose during the original analysis.

It’s a little misleading as well, to think that a scientist always has a well-formulated question before he or she starts making observations. Sometimes we make some weird observations, and spend months trying to formulate the question and hypothesis at the same time. For example, one might be handed a huge data set and be told to ‘do something with it.’

Two summers ago I was handed a mammoth second molar. “Do something with it.” So I did. I collected data, because I could, and then looked at the data and tried to formulate a hypothesis about the results. It may or may not ever get published, but it was interesting and fun.

Recently, I’ve been working on a paper about climate change at the Paleocene-Eocene Thermal Maximum (PETM). The idea is to collect a bunch of fossil mussels and look at the carbon and oxygen isotopes from the shells as a means to get at ancient climate and weather patterns. We already know that there was rapid global warming at the PETM. We want to know what that did to the weather.

Oh hey! A question! Did weather patterns change during the PETM? Did they go back to pre-PETM patterns when the PETM was over?

This is a really great question, but it arose after I made a bunch of observations: I knew the PETM was recorded in the Hanna Formation. I knew that there were some mollusks in those same rocks. And I knew that there was some way to get climate information from shells.

That’s fine. We can still investigate this question. Let’s go get some fossils! And some rocks, too, because we need carbon isotopes from the rocks to actually find the extent of the PETM. Ok, got it.

Back to the lab. Drilling, drilling, drilling.

Poof! Data.

All right. So what does it mean?

Here’s where science isn’t quite like the scientific method. I’ve now got a ton of data, thanks to the tireless work of two students (Jen and Julia). We sampled thirteen shells from all over the Hanna Formation. We have almost 200 data points. So what do you do?

The scientific method says “test hypothesis.” I’ve got a bunch of squiggly lines. Besides, what’s our hypothesis?

Hypothesis: Weather patterns during the PETM were different than those before and after the PETM.

Secondary hypothesis: Weather patterns from before and after the PETM were the same.

OK. Do all the squiggly lines look the same?

Well, no. But some of the shells were kinda… crappy.

Can I account for that?

Sure. Use statistics.

OK. I’ll calculate averages and standard deviations. And correlation coefficients, too. Why not? Excel can do that.

OK. Are they all the same?

Well, no, but the differences might have to do with rock type and environment more than climate. Some of the mussels came from lake beds and others from river beds.

And there’s the possibility that these mussels aren’t all the same species either. Would the same species be able to live in both lake and river environments? That would make the isotopic values different, I would guess.

Well crap, there’s a whole ‘nuther paper right there. A brand new question. I don’t have time for that. This needs to get published! What can I do with these data right now?

I dunno. Plot it up and see what it looks like. (Yes folks. An important step sometimes is just to plot your data and see. Most of the time, I have no idea what I’m about to see.)

OK. Here it is. Averages for each shell plotted against stratigraphic level.

Averages for the 13 mussels analyzed in this study, plotted against stratigraphy in the Hanna Formation. Grey line is carbon isotope stratigraphy used to identify where the PETM is.

Averages for the 13 mussels analyzed in this study, plotted against stratigraphy in the Hanna Formation. Grey line is carbon isotope stratigraphy used to identify where the PETM is.

Well, there’s an interesting pattern, I think. Things go more positive for both carbon and oxygen, then go back to more negative. Then they go more positive again. Is that an important pattern? How does it compare to the rock types? Or the position of the PETM?

Same figure as before, but now highlighting where the PETM is (orange box), the different depositional environments (lakes in blue and rivers in purple). The red arrows show some interesting trends in the data. I wonder what they mean?

Same figure as before, but now highlighting where the PETM is (orange box), the different depositional environments (lakes in blue and rivers in purple). The red arrows show some interesting trends in the data. I wonder what they mean?

Rats. One thing I see it means is that I need to find more fossils from between 2500 and 2600 meters (which is in the PETM and in a river environment).

But what if I don’t find more fossils? I’ve been looking for YEARS!! This is what I have. What does all this mean?

So this is where I am right now. I’ve got to do some more research on the different species of mussel that may or may not be present in the Hanna Formation. I also need to do some more research on what kind of effect the environment really has on how a mussel records weather patterns.

The point in showing you this is to illustrate part of the inner dialog that goes on with doing science. Sometimes you just get a bunch of data and start plotting it in various ways to see what patterns arise. Once you observe a pattern then you can try to understand it. And, as is almost always the case, new questions arise.

I don’t know if the mussels are different species between the lake and river environments. I don’t know what difference that would make.

I don’t know if the same mussel species when living in lakes and rivers record weather in the same way.

I don’t know how much it matters that the shells from the lower lake unit are ‘crappy’ compared to those from the upper lake unit. I have done some spectroscopy to show that there has been not a lot of alteration of the shell material, but there is a chance that weather is recorded differently in different parts of the shell as well, due to the age of the mussel.

Well, crap. I have more questions than answers. But I’ve got to write this up. This process doesn’t feel very much like the scientific method that I learned about in elementary school.

My experience isn’t unique. Keep that in mind. A lot of science is done this way. The natural world doesn’t fit conveniently into our notions of how it should work. Opportunities arise, and we jump at a chance to collect new data. Sometimes unexpected results fall from data that we hadn’t anticipated. Sometimes results fly in the face of conventional logic.

Sometimes you spend years staring at data waiting for something to make sense. Then once day while you’re taking a shower, years after you’ve given up on it,  it all makes sense!

I started this project in 2007. I’m still working on it. That’s pretty typical, too.

4 thoughts on “The Scientific Method

  1. What’s always impressed me about scientists is the questions they have to ask and answer along the way to THE answer. Many of the advances in technology we’ve seen over the course of history came from someone trying to get to an answer about something else. Take your mass spectrometer for example. Some engineer didn’t wake up one morning and say, “I’m going to build a device that measures the mass of atoms and molecules.” It was some scientist who said, “I need to measure the mass of an atom. How the heck do I do that?” And then figured it out. A long series of questions like that one led to the development of the Large Hadron Collider.

  2. Such a nice commentary on how science actually works! It’s so messy! A few things I would add from my experience: 1) I collect a lot of data that I am not sure at the time whether or not it will be useful. It may not tie directly to any hypothesis, but I can collect the data, it might be interesting, so I do (usually I regret this–does relative humidity really have a measurable effect on lizard behavior? How accurate should I be in measuring distance to nearest vegetation, and the height of that vegetation, particularly since I have no idea if I will ever analyze that data?); 2) statistical analysis can get messy fast. Whereas you searched for years for a small sample of mussel fossils, I can accumulate way too much data in a matter of months (at least on some variables). Then comes the analysis–I try one thing and realize that maybe I should approach it differently, so I try a different method. Then I think maybe I need to account for some covariates, or combine variables in a principal components analysis, or maybe I need to transform the data to normalize it, . . . in the end I have hundreds of pages of output from SAS, most of which I don’t understand! So then I start having the internal dilemma of which analysis was best/most correct/done properly, or maybe there is some other analysis I don’t even know about that I should have done instead. Even when there is a very clear pattern and the analysis looks fantastic I find myself doubting that it is real because I am not confident enough in my statistical ability; 3) Publication bias! Most of the data I have collected has never been analyzed. Of that which I have analyzed, only a fraction got written up for publication. Of the papers that an editor receives for publication, only the most interesting ones get selected. At every step there is real data that never makes it to the public eye. Ben Goldacre gives a great TED Talk about the dangers of publication bias in medicine:

    Last commentary is that asking the right question and creating a very clean, very testable hypothesis is very difficult. Most people also want to ask a big question but soon realize it is too big and too untestable–they have to be content with answering one tiny little sliver of their original question. Sometimes I think we are real gluttons for punishment!

  3. Science is such an amazing thing. I feel like if more people understood the *real* nature of it, rather than the textbook version, there would be a lot more interest. It’s exploring, just like Lewis and Clark. You don’t know what you’re going to find, or where it will be. It’s never a cut-and-dried process!

  4. Hi there, I am an Instructional Designer for CSU-Global. I am seeking copyright permission to include your Scientific Method graphic in IPS450 Interdisciplinary Portfolio. Please email if approved or denied

Leave a Reply

Your email address will not be published. Required fields are marked *