“Backed by Science” Doesn’t Mean What You Think, with Dr. Stephen Cheung 

• Listen On...
• Apple
• Soundcloud
• Spotify

Science isn’t perfect; humans can bring bias and poor design. Today, Dr. Stephen Cheung helps us understand the potential pitfalls of scientific inquiry, and the things we can trust.

Episode Transcript

[00:00:00] Trevor Connor: Hello and welcome to Fast Talk your source for the science of endurance performance. I’m your host, Trevor Connor. Here with Chris Case. The phrase backed by science has become an increasingly popular expression that brands tout go to the website of any new product, and there’s likely a page of both scientific research that proves it works.

Are these companies trying to swindle you? Well, sometimes, but most of the time these brands are referring to real studies conducted in legitimate labs. But a single study, no matter who conducted it or how well it was designed, doesn’t add up to backed by science. There are just too many factors that affect even the most well-intentioned studies, including the skill sets and style of the research team, the number of subjects and a whole host of bias that even the best researchers fall prey to.

Then there are the journals who publish the science, which can also introduce bias. Exercise science is notorious for having small numbers of subjects in their studies. This means that a high percentage of studies should find no effect on what’s being studied, but that’s not the case. Most published studies show an effect.

Too often no effect is confused with bad research. So researchers don’t even bother submitting those studies studies. Thus, the one study showing that Product X improves performance gets published, but the four studies that found that Product X didn’t help were never even submitted. That is critical to understand as new products hit the shelves and you try to decide if they will help you or not.

Here to help us understand all the biases and issues and inheritance science. As friend of the show, Dr. Steven Chung, he has led his own research lab for over two decades and recently became editor-in-chief of the respected International Journal of Sports Physiology and Performance. Every day he reviews studies for publication, so he is not here to bash on science.

He simply recognizes as well as anyone that science is not perfect and the more discerning an eye you can view it with. The more likely you are to find good answers. He’ll talk with us about the replication crisis in science. Why a result? Lacking significance doesn’t mean what people think it means, why it’s critical for multiple labs to research the same subject, how marketing can complicate things and why it’s nearly impossible to create an unbiased study.

He’ll discuss all of this using examples from trending science, like heat adaptation, core temperature devices, and the use of AI to give answers to science. Joining Dr. Chung will also hear from Pat Warner, vice President of Product at Stages Cycling, who will talk about balancing bias, marketing and experience.

Dr. Jamie Whitfield, a senior lecturer at Exercise Physiology at Australian Catholic University, will share his thoughts on heat training. Finally we’ll hear from statistician, Dr. Michael Rosenblatt, who does research at the Sylvan Adams Sports Institute. He’ll go in depth about the challenges of designing a good meta analysis, so put on your nerd tinted glasses and let’s make you fast.

[00:02:53] Chris Case: Dr. Chung, welcome back to Fast Talk. It’s been a while. Yes, very happy to be back, Chris and Trevor. Thank you. January. You took a new position as the editor in chief of a journal. Tell us about that and tell us how that pertains to this conversation we’re gonna have today.

[00:03:09] Stephen Cheung: Sure. So in January I became editor in chief for the International Journal of Sports Physiology and Performance, and I was the associate editor for it for six years before taking up this role.

And it’s, um, really not a niche journal. I think a niche journal in the best possible way for our conversation and for our focus on endurance sports because the journal is all about athletes. It’s all about sports science, and it’s all about physiology combined with performance. And so we’re really looking at all sorts of sports from endurance sports, team sports, power lifting, et cetera.

And what we’re really looking at is not necessarily biomechanics, not the motor control, but really physiology. So we want in all of our studies, some aspect of physiology and also how it relates to performance. And the other mandate for our journal is that it has to be on trained individuals. And uh, it doesn’t have to be world class, although some of our papers certainly are with the top level athletes, but they can’t be sedentary individuals, can’t be untrained individuals.

So I think all of those together make it a great journal for understanding sports science. And in terms of my background, I’ve been a researcher as an independent scientist for 28 years now. So I’ve been on the other end of writing a lot of papers, submitting them to journals. But this was also a great chance to.

In addition to just helping out the scientific community to really in a sense improve myself, because I get about two dozen papers that land in my inbox every week that I have to triage. I have to look at their research design. I have to look at kind of their fit with the scope of our journal. I also have to look at just the kind of the quality of the findings also.

So I really have to assess them and then make a decision on it to send it either to the next stage or to reject them. So it’s been a crash course over the last four and a bit months on. Really just intensive peer review, which I’ve done a lot before, but certainly not to this extent. So it’s also given me the chance to really kind of sit back and think, okay, what does make for good science?

[00:05:42] Trevor Connor: It makes you perfect for this episode. And as we were talking about before we got on mic here, this episode has evolved a little bit because the original intention behind it was to address the fact that, you know, you’re seeing this acceleration of new techniques, new devices, new science in the field, and a lot of people are trying to figure out, should I try this?

Should I use this device? Is there signs behind it? And whenever there’s a new device, you can go to the company’s website and they usually will point to, well, here’s research that backs our device. But as we dived into this and had this back and forth with you on the outline and through email, what we are really seeing is, and I’d say this is kind of probably gonna be the theme of today, that just because there is a scientific study, just because the study has been published, doesn’t necessarily mean that it’s good research.

Or that is definitive evidence that whatever it is actually works.

[00:06:46] Stephen Cheung: Yeah, absolutely. And we probably see this all the time in the news. There’s almost every week, if not every day, a news article that says, oh, coffee does this for you, and then next week you hear coffee does that for you. And they’re all from, you know, a single study.

And so you really have to take that with a grain of salt and you have to place it in a larger context of the whole literature as an entirety. And that’s certainly one of the things we’ll hopefully be talking about weighing the impact of a single paper versus mm-hmm the entire body of evidence.

[00:07:22] Trevor Connor: Yeah, so I think it’s gonna be an interesting conversation.

And like I said, you’re perfect for this ’cause this is now what you are doing. You are reading studies and assessing. Is this good research? Can they say what they’re saying? So hopefully we’re gonna dive into this a little bit and give some guidance to our listeners on, Hey, you’re interested in X, maybe it’s a, a skin temperature device, or this new technique of producing EPO benefits from heat training, you know, all these new techniques and fads.

You saw a study on it. How do you interpret that study? I still remember we had you on the show a couple years ago and Ryan was on the show with us and Ryan’s gonna kill me for sharing this story. But we were looking at this one study that Ryan had asked us to review, and as we were talking offline about this study, you started tearing the methodology apart.

I started tearing the methodology apart and I looked at Ryan and he’s just kind of slumping over, and I finally lean over to him. I go, Ryan, what’s wrong? And he just goes, I really like that study.

[00:08:31] Stephen Cheung: Yeah, that’s also a typical grad student thing too. I have students coming to me with, you know, great ideas and stuff, or what they feel is great ideas and either tell ’em no, that’s completely been done before, or else, yeah, no, the methods you’re thinking about isn’t either viable or just isn’t a good method.

So yeah, you end up as a scientist having developed pretty thick skin and ideally you go for, yeah, it’s, it sounds corny, but you aim for the truth rather than just kind of looking good, so to speak.

[00:09:06] Trevor Connor: Yeah, but maybe this is a good place to start. You raised something, you said it’s been done before, but there is something very important in science that can help you trust studies, which is replication, that it is actually important to reproduce studies to see if you can come to the same conclusions.

Correct?

[00:09:27] Stephen Cheung: Yeah. And the listeners may be familiar with this idea of the replication crisis in science. And what that means is exactly what you say, your bread and butter, you almost see every paper saying, this is the first study, this is novel because, and that’s how you get into a journal nowadays. You have to have some intriguing result.

Ideally, it’s new and never been done before. And whereas if you just write to a journal, submit a paper saying, I just copied the methods of, you know, so and so from 1986. And I did exactly what they did, and I would like to get my paper published. Most journals would say thanks, but no thanks. So that’s the replication crisis, right?

So you, you are, as a scientist, you are not. Incentivized to do a study that has been done before. So you wanna tweak it a little bit. There are ways to gain the system. You might update the methods a little bit by, you know, changing one measure. You can also say, well, there’s now new and technology to measure variable X, and that can be a problem because the whole point of doing a study is to test an idea statistically.

And most of the time we say we have an alpha of 0.05, which means that 5% of the time we’re willing to be wrong. We’re willing to do our study and accept the 5% risk that we are gonna be wrong. So that means really one out of 20 times we’re gonna make a claim that. Something works to improve your training when in fact it’s just a statistical artifact.

But again, no one is gonna go back and redo that study because they’re not gonna get it published. So you can have this accumulation just. From the probability of statistics that you are gonna end up going and thinking something actually works, when in fact it, it actually doesn’t. It was just a statistical probability that you’re gonna be wrong one in 20 times.

So that is where the problem is with this replication crisis. If we are not replicating studies, if we are not kind of going back and redoing things again, and there’s certainly been a push, I know in the field of psychology there is a major recognition of this replication crisis, and there are research groups that.

Have just said, you know, damn the torpedoes, we are going to completely replicate kind of this sampling of experiments and see what we come up with. And they’ve staked their careers really on that and you know, more power to them.

[00:12:31] Trevor Connor: Yeah. Well, I mean, I read this thing in a review about all this last night that I had never thought about it this way.

And it’s in line with what you were just saying, but really made me think, because we talk about power in studies, so a lot of ways you can achieve power, but let’s keep this simple. If you’re doing an exercise science study, the more subjects you have in the the study, the greater the power. And the idea is if there’s an actual effect, if whatever you are studying produces a result, the higher the power, the more likely you are actually going to show that effect.

I hope that made sense. Mm-hmm. ’cause this stuff gets complicated. So I’ll use a simple example. We all know that if you spent six months doing bicep curls, you’re going to see improvements in the strength in your bicep. So, ridiculous study. Really simple, but that’s an effect. Mm-hmm. So you’re studying bicep curls, you wanna see an effect.

The recommendation in research is for 80% power, which means if you repeated the study, this bicep curl study 10 times, two of the times, you would show there’s no significance that you couldn’t prove there was a benefit to doing bicep curls. Even though we know, yeah, there is an effect that of course you’re gonna see a benefit.

[00:13:50] Chris Case: Mm-hmm.

[00:13:51] Trevor Connor: So as you said, what happens if you do one of those two studies and you’re the only one to research that particular protocol?

[00:13:59] Stephen Cheung: Yeah, absolutely, and I try to make it as simple as possible. The difference between when we say an alpha of 0.05 and kind of that statistical significance versus statistical power, what you refer to is the flip side of what I just talked to you about, statistical significance.

When I say one in 20 times or 5% of the time, we’re willing to take the risk that we see in effect and we say this is a real effect, when in fact it’s not a real effect. It’s statistical artifact. Statistical power is the flip side is how often or what is your risk tolerance for saying something is the same.

When in fact it is different. So in this case, what you just said, that doing this training in this study showed that there is no effect. Well if you are only doing it on four participants as opposed to doing it on 400 participants, you can already intuitively understand that you know, with four people, there’s a strong odds that you’re gonna have a very low power that you may end up seeing in these four people that doing these bicep curls at 80%.

Didn’t have an effect on them, but that’s because they’re so underpowered, they’re four people as opposed to doing it on the 400 people. If you see the same no training effect, then you can be pretty confident that there is no training effect.

[00:15:37] Trevor Connor: And something that’s really important to understand here is, I always think of this like the criminal justice system is when somebody is found innocent of a crime, it doesn’t mean a hundred percent that they didn’t commit the crime.

It just means that we couldn’t prove definitively that they committed the crime.

[00:15:58] Stephen Cheung: Mm-hmm. Yeah.

[00:15:59] Trevor Connor: And it’s the same thing with research. Reasonable doubt. Yeah. What you want is if there, if the study demonstrates an effect, there is absolutely an effect there. There is a benefit to this, but if they find no significance, that doesn’t mean there isn’t an effect there.

It just means that they couldn’t find it in this particular study.

[00:16:18] Stephen Cheung: Yeah. And let’s look at what most scientists are dealing with. We often, in most physiology studies, a large sample size is maybe a dozen participants. We’d be absolutely thrilled if we can get 20, 25 participants to do a study. So by its very nature, many sports science studies tend to be a little bit underpowered because we’re dealing with a very small group of individuals.

And the more elite of an athlete you wanna study, the harder it is to get them in the lab. Mm-hmm. The harder it is for them to be willing to give you a muscle biopsy, to be tracked in their nutrition for 24 hours or 72 hours or whatever. So. That’s always a challenge, and that’s why oftentimes studies take relatively untrained or much lesser trained individuals because they’re much more readily available and you can get a larger sample size compared to the true kind of really elite individuals.

So it all goes back to the research question. If we’re just looking at the general effect of, let’s say, blood flow restriction training on health and outcomes, then we don’t necessarily need those really elite outdoor trail runners. We can get by with relatively sedentary or less trained individuals.

And that’s certainly one of the things that we look at when we triage journal articles for I-J-S-P-P is, you know, is the design appropriate? Is the research participants appropriate for answering this question?

[00:18:06] Chris Case: Can you, Dr. Chung, relate this to some of the new devices, the new technologies that we’re seeing, and how this impacts the trends, the uptake of these, the fact that maybe marketing is a little bit more powerful than the science supporting these devices in some cases.

Sure. And

[00:18:25] Stephen Cheung: that’s the case with a lot of wearables, right? And, uh, I believe, I was on the podcast, um, a year, a couple of years ago talking about wearables. And one of the big things is that their algorithms are based on a very broad population, and it can range from very sedentary individuals through to elite athletes.

So it’s a large population that you’re dealing with, and the algorithm is basing its recommendations on you, not just on your own data, but in reference to this large database. So it may not have. I don’t wanna say, may not have your best interest in mind, but it may not know you. Yeah. As well as you think it knows you.

It because it is saying, okay Steven, I am recommending saying your recovery score is X. Not just based on your own past history, but this all the history of other people. And again, I don’t know who those other people that you’re basing the algorithm on. So that’s something that you always have to keep in mind when you’re looking at these devices.

The power of them is that they’re easy to use and they give you a number, whether it’s recovery score, whether it’s heart rate variability, sleep quality, all of those things. But again, they are. Looking at you, but they’re also basing the recommendations not just on your patterns, but on the entire population’s pattern.

Yeah.

[00:19:58] Trevor Connor: So here’s the really big question I have for you. I did find this really interesting review. The title of it is Replication Concerns in Sports and Exercise Science, A Narrative Review of Selected Methodological Issues in the Field. The main theme of this review is we know that exercise science studies tend to be underpowered.

Far more than 20% of the studies published should find non-significant. But when they did an analysis of exercise science studies, they looked at a total of 129 studies from four different journals. 82% showed positive results, benefits of whatever they were studying. A certain percentage, very small percentage, showed negative results of whatever the thing was that they were studying.

And then it was well under 20%. Were showing nons significance and I, I can’t find the percentage, but they basically said there’s like a 1% chance. That if all the research is being conducted correctly and there’s no publication bias that that’s the distribution you would see.

[00:21:06] Chris Case: Mm-hmm.

[00:21:07] Trevor Connor: They’re basically saying there is something wrong in the publication, and we’re being biased towards getting too much in the way of positive results when studying different protocols.

So as somebody who’s now editor of a major publication, what’s your feeling about this?

[00:21:20] Stephen Cheung: Yeah. The problem here is different from the replication crisis, right? First off, no one is going to publish a paper that’s exactly the same as before. And then the other problem that you’re alluding to here is the no finding kind of bias, and that if I did a study.

And I showed no effect of whatever intervention. I, as a scientist, I’m probably gonna say, well, you know, this isn’t gonna be really interesting to a journal. So sometimes I would just say, eh, I’m just not even gonna submit it for publication. Or I might submit it to a relatively lower tier journal because I know the high end journals aren’t gonna like papers that, you know, don’t show this works or that works.

Those journals tend not to favor as much, you know, like we did our best to answer this question and we came up with the answer of no effect, right? So there’s that bias against no findings. And so some of it comes from the scientists themselves, some of it comes from the journal itself. I know certainly as editor in chief for I-J-S-P-P, I’ve really tried to triage papers by looking at.

Is the research question interesting? Is it good solid design and then not necessarily base my decision on whether, yeah, they found an effect or no, they didn’t find effect. So I think it’s a systematic problem and I think it’s both self-censoring from the scientists themselves and also some systematic bias from the journals.

And yeah, I, I guess in a sense there’s no real way around it sort of changing the system, like in different editors of journals being much more open to just good, well designed science as opposed to is this a positive finding versus a negative or non-significant finding.

[00:23:24] Trevor Connor: So not to put you on the spot, and I’m not gonna ask you if you do this, but the final recommendation of this paper is what they call registration reports, which is where if you’re conducting a study.

You come up with your study design, including how you’re gonna analyze the data, you submit it to a journal before you conduct the research and the journal, based on whether they think is a good study design or not, agrees that they’re gonna publish the study before the study has been conducted. So that way if it ends up there’s non-significant results, they’re committed to publishing it.

What’s your feeling about that sort of approach?

[00:24:04] Stephen Cheung: Yeah, I think that’s the best way forward. And those are pre-registered kind of studies where, like you say, your entire research question methodology and statistical approach is peer reviewed. Before you even do the study and the journal says yay or nay to, we will publish this study based on what you intend to do, regardless of your findings.

And this is different from the typical. How journals work where you complete the study, you do the whole thing, and then you submit it. So kind of in a sense, the cat is outta the bag already. Whereas with this pre-registered approach, which I think is great, and I would love to move our journal eventually to, and I think this is where journals should be heading, is to have these pre-approved and peer reviewed, pre accepted methods, and then the scientist goes off and does the study and then we publish the results regardless.

So yeah, absolutely. In full support of that.

[00:25:12] Chris Case: Yeah. You’re making an assessment on the merits of the science, supporting the hypothesis. You have the methodology that you propose to get to that. Answer to that question. And it has nothing to do with whether you’re going to show a result or not. I mean that in itself, that sentence, I hesitate to say it that way because it’s not, not getting a result, it’s just not getting the result that maybe you hoped to get.

Mm-hmm. Sometimes like, oh, this device should be accurate and it should be consistent, and so we’re gonna prove it through this experiment. And then when it doesn’t happen, you’re like, oh shit, we didn’t show what we’d wanted to show. Well that’s great ’cause that’s still a result and that’s still worthy data and that is worthy of publication.

And so judging an experiment on its merits before it’s conducted is what we’re talking about. And that seems very like a very sound choice to sort of eliminate some of this bias towards, ah, we’re only gonna publish things when they show kind of what we want or hope to see in the science.

[00:26:20] Stephen Cheung: Yeah, because you can also, I mean by the very nature of research design, you can really kind of put your thumb on the scales, right?

We have a hypothesis going into a study, and it’s very easy to design a study that’s almost going to guarantee that you see an effect. So, you know, Trevor talked about it before. If you did bicep curls and nothing but heavy bicep curls for six months, well guess what? You know, you are gonna come to the conclusion that doing bicep curls is an effective way to improve your strength or power, as opposed to, you know, does anyone actually do that in real life?

Right? So you’re really kind of, again, putting your thumb on the scale and biasing towards seeing an effect, as opposed to, well, you know. Doing bicep curls as part of a overall training program.

[00:27:16] Trevor Connor: Oh, and that’s a really important point. I mean, I’ll, I’ll take all this one step further that, fortunately it was less of an issue in exercise science, but a lot of the people listening to the show are also trying to look at what should I be eating?

What should I be consuming, both on the bike and off the bike? And there’s a real issue in nutritional science with the fact that most of our research is epidemiological. For anybody who doesn’t know what that means, it’s you’re looking at correlations. So you might use something like the NHANES data that just tracks everything possible about a very large cohort of people over 20 years.

And then you can grab that data, they’ll let you use it and go, okay, I wanna look at people that eat a lot of eggs compared to people that eat. Almost no eggs or no eggs and over 20 years, see if one of those groups has higher rates of heart disease. Mm-hmm. So you’re looking at those correlations.

Another really interesting study that I read, and I’m not gonna tell you what it specifically covered ’cause this can cause a bit of an emotional response, but the, the title of this, this study is grilling the data and they pointed out this issue that there are hundreds of ways of taking epidemiological data and analyzing it.

So they took data that was used in a meta-analysis that showed that this thing that they were looking at X had a small effect on all cause mortality. So seemed to contribute to all cause mortality. They then showed that there were, let me just find the exact number, ’cause it was kind of jaw dropping, 1,440 unique analytic specifications that they could use.

And they went through all of them. Mm-hmm. And of those 1,440, only 48 had significance. Of those 48, 40 actually showed that X was protective against all cause mortality. Eight showed that it was causative, and I shouldn’t use causative in epidemiological, but showed higher rates of all cause mortality.

But this meta-analysis drew the conclusion. Yeah. And so the point they are making is in nutrition with, you have so many of these epidemiological studies, you have this issue that some researchers might be taking that data and just trying analytic technique after analytic technique until they get the result that they want.

[00:29:50] Chris Case: You could call it cherry picking, I suppose, right? Yeah,

[00:29:53] Stephen Cheung: yeah. Cherry picking. We also call it kind of a, in some sense, a shotgun approach too, right? That we’re just throwing a whole bunch of kind of data or collecting a whole bunch of data and seeing if anything is significant. And then we focus on that, and that’s where the cherry picking mm-hmm.

Comes in and yeah, you can measure something. 10, 20 different ways. And again, as we talked about, chances are just from statistic probability that one of those 20 will show you a significant finding. And if that’s all you focus on, then you’re getting a really biased view. The way to look at statistics and different iCal techniques is there are different glasses that you put on, right?

Some have a red filter, some have a blue filter, some have a green filter. So how you see the data, it really depends on what filters that you are putting on when you look at the data. So, you know, how do we get around it? Well, it really comes down to the weight of evidence. It’s to not just rely on one study as the definitive be all and end all for a topic is to really.

Challenging for many athletes and many kind of people who just want to know what to do, but to be more well read and understand the broader literature rather than just focusing on this. Yeah, this one single study said this is effective, so that’s what I’m gonna do. And you may, if you just focus on that, you may miss the 15 other studies that says, well, it doesn’t really work or have a big benefit.

[00:31:37] Trevor Connor: You, I’ll admit, and I’m glad you’re bringing this up, and maybe we can talk further about this, or maybe that’s the whole answer, but I know personally, after reading all these reviews and seeing all these issues, you know, in exercise science you have this bias towards positive results, epidemiological studies, the ability to cherry pick, or it’s called p Hacking the data.

How do you know to trust what science of trust, or if you can trust the science at all at this point?

[00:32:03] Stephen Cheung: Well, I think there’s a few things. One is really. Give it time and weigh the entire literature. This is where we may get into this whole idea of heat training, right? The first studies from Ben Ronstadt’s group, and that showed that really long term heat training, and we’re not talking about the typical two week heat adaptation study, but five weeks or more where we see an effect not just on tolerance to exercise in the heat, but we’re seeing, or Ronstadt’s group is showing that there is potentially an increase in total hemoglobin mass and therefore an improvement in aerobic capacity for it.

So in some senses that’s a classic example of, okay, here is really interesting kind of intriguing data and. Those studies first started coming out in the late 2010s, I think about 20 18, 20 19 were the first studies and the challenges even to this date now, seven odd years later, those studies haven’t really been replicated by a lot of other labs, and most of the studies that show that heat training is effective have come from that same research group.

And again, I’m certainly not implying that there science is not good or incorrect or improperly done, but I would still hesitate as an athlete of kind of just accepting that, yeah, heat training is gonna increase my total hemoglobin mass. I don’t have to go at altitude. I can get all these benefits. If all of the data is just coming from one lab or one research group, I would have much more trust in the overall data and the overall efficacy of this intervention if it was being replicated by a lot of other labs that are shown the same thing.

And again, I’m absolutely, I’m a big fan of Ben’s work in multiple fields, and I’m absolutely not implying that their data is faulty or misleading. I’m just saying that this is an example of where data has come largely from one lab. It hasn’t really been truly replicated by a lot of other labs, and yet, you know, it’s taken on a life of its own, whether in social media, in training circles and in marketing that this is a great thing to do for athletes.

So I would still have some hesitation about it and I would assess it, kind of that, yeah, this is intriguing. This may work, but. Just because of one paper or one series of papers from one group, I wouldn’t place my entire kind of hopes and desires on that idea.

[00:35:02] Trevor Connor: And I get that. I mean, Dr. Ronstadt is the best of the best, but in scientific research, researchers have a style.

Each researcher has their style, they have their strengths and weaknesses. And you could have somebody who’s a fantastic researcher who is trying to put together the best study possible and their style just might not be the best style for that particular research.

[00:35:26] Stephen Cheung: Yeah, absolutely. I know I have a certain style of designing studies everybody does, and also I have tools that.

I love and I keep going back to over and over. So it goes back to, in our analogy of looking at the world through different colored glasses and you know, I may have green tinted glasses and I tend to always see things as green. I always use this green set of glasses. Another person may always use red and we’re gonna see the world slightly differently as a result.

So, yeah, absolutely. And I do think as we, you know, morph more into a talk of heat adaptation and then heat training is, this is also a key example of where we have really intriguing data.

[00:36:18] Trevor Connor: So I actually think you and Dr. Larson are a great example of where you can have research that comes outta one group or a couple groups that points in one direction and then somebody else comes in.

And just has a different take or a different approach that can actually come to different conclusions. So what I’m thinking of is hydration. There was a kind of a standard belief that about 3% body weight loss in, in water causes a decrement in your performance. And you and Dr. Larson both said, well, wait a minute.

Those studies weren’t blinded. People knew whether they were hydrating or not hydrating and came up with fairly ingenious ways to blind people so they didn’t know and ended up with very different results.

[00:37:08] Stephen Cheung: Yeah. So you’re referring to study wall etal 2015 from Paul Larson’s lab and then my Chung et all 2015 obviously from my lab.

And we kind of came to it about the SA idea about the same time, this idea that all of the previous studies in assessing the effects of hydration on exercise performance in the heat. Never blinded participants, right? They just said, Hey Trevor, today you get to drink from this water bottle. Go and ride a 20 K time trout.

And you come back in a week’s time and I say, Trevor, today, you don’t get to drink and go ride the same 20 K time trout. Good luck and. We’ve been so inundated with this idea that we have to drink that just my telling you, you get the drink today or you don’t get the drink today, can set your mental template for how you’re gonna exercise.

So we felt it was really important to blind the participants to whether they were actually being hydrated or not. And coincidentally, we both came up with the same method of blinding, which was to have an IV in your arm and have a saline drip and either have the saline drip going or not to replace the fluid that you have lost or not.

So the participants truly were blinded. They had no idea whether they were at normal hydration levels or 3% dehydrated. In my case, I had them do a 20 K time trial. I can’t quite recall what Pulses group had them do, but what we found, 20 K time trial in the heat, no difference in. 20 K time, no difference in power output or the pacing strategy.

So that came out again at the same time in 2015. That caused a lot of media frenzy. So I was actually on a lot of media interviews saying. Essentially the pitch was that you didn’t need to hydrate as much or focus on it as much as possible. I was always very careful to caveat that this is really one study and we are the first to blind participants to hydration.

I believe in the data that we got. But this is one study, so you know, take it with that heavy grain of salt. And I think one of the great things is that both of our studies encouraged or really forced other labs to come up with ways to blind participants, to hydration, and they came up with different ways, which was great.

Right. Again, we talked about. Each scientist having different styles, having different ideas, their ideas. A lot of these other studies, rather than saline drip in your arm, actually had people drink fluids in both conditions, but they also had essentially a really uncomfortable esophageal tube into their stomach, and it was either sucking out the fluid or not.

So, yeah, that, that was definitely not the way I would wanna be as a participant. Yeah. But it’s a different way of blinding participants to hydration. Mm-hmm. And several follow up studies using variations of those tools found that there was still impairment at two point a half, 3%. So, you know, how would I sum up the data?

Do I believe my data and the data from Paul Larson’s group? Yeah, absolutely. I stand by it. Do I have any reason to not believe those other studies? No. Also, like I, I’m sure they did their science at a very high quality methodologically and everything else. So it goes back to our original question, right?

There’s the weight of evidence, and I would say the weight of evidence is still probably that. Ultimately, hydration is critical. Like if you never drink over the course of an Iron Man bike leg for example, you’re gonna be suffering. But you know, is it as critical in terms of you have to drink a massive amount?

Maybe not. So again, that’s how I would ultimately assess the evidence.

[00:41:25] Trevor Connor: We could do a whole episode on being careful about what studies you choose to volunteer for. ’cause I volunteered for a lot and I can remember a few times sitting there in the lab with them telling me what they’re about to do to me, and I’m going, say what?

[00:41:39] Stephen Cheung: Yeah,

[00:41:40] Trevor Connor: seriously.

[00:41:41] Stephen Cheung: Yeah. Well, I’m on sabbatical right now and working with different colleagues who do some very invasive studies, and I love the science. I usually would be the first to volunteer for everything, but at the same time, I’m also kind of glad I aged out of both of these big studies.

[00:41:57] Pat Warner: Yep.

[00:42:02] Chris Case: If you’ve been listening to Fast Talk for a while, you’ve probably heard this said before, what gets measured gets improved, but in endurance sports, how you measure matters just as much. That’s where training with power really changes the game. Heart rate tells you what’s going on with your body, but power, power tells the truth about what’s going into the bike in every single pedal stroke and not all power meters are built the same.

One of the companies that helped define this category is stages. Cycling

[00:42:30] Trevor Connor: stages built their reputation on an accuracy you can trust in the real world, not just in the lab. Every single meter is individually calibrated before it leaves the factory. That’s not a batch test, that’s your meter calibrated and verified.

They also pioneered active temperature compensation, so whether you’re climbing in the heat or descending into cooler air, your data stays consistent. And here’s something a lot of people don’t think about. Stages locks down key calibration factors, so they can’t be user adjusted. That means no accidental tweaks, no corrupted data.

What you see is actually what you did.

[00:43:05] Chris Case: Plus you’ve got flexibility. You can go left only, right only, or dual-sided measurement depending on your goals and your budget. At the end of the day, if you care about training with intent and getting faster, accurate data isn’t optional. Check out stages, cycling@stagescycling.com and see how training with a stages power meter can change the way you ride.

[00:43:27] Trevor Connor: As Dr. Chung pointed out, the experience and style of the research team can have a big impact on the research, and that’s before marketing considerations come into play. Here’s Pat Warner talking about how the ideal study would balance all three of these factors.

[00:43:41] Pat Warner: I think a lot of the times that comes to independent testing, which is sometimes challenging because people don’t wanna spend the money or time without any bias.

That’s the hardest part about getting a new technology launched and have it a third party review without any bias. ’cause a lot of times people don’t wanna spend the money to do the research and the time without being paid to do it. And a lot of times when it’s paid to do it, it comes out being a little bit biased.

So I love when there’s scientific reports that a third party’s done because they care about the category, or it’s a research project funded by a school or an institute or something like that. It really helps understand what the category and the products are doing versus marketing.

[00:44:23] Trevor Connor: So it’s somebody who has nothing to do with the manufacturer.

They’ve done a study.

[00:44:27] Pat Warner: Yeah, and the tough part is it really helps if they understand the category. Sometimes they do things for cycling and you look at it and go, how did they get there? That is not any use case scenario that we would see or wanna be testing for.

[00:44:39] Trevor Connor: So that’s tough. ’cause you want somebody who’s gonna be unbiased, but they know the category, they’re not involved.

[00:44:44] Pat Warner: Yeah. It is tough, but as a consumer on that side of it, that’s great. As a manufacturer, we, you know, we can do it, but it’s great if that’s available. So everybody has the same set of data.

[00:44:56] Stephen Cheung: So, yeah, you also have to really understand kind of the marketing forces behind, whether it’s different wearables or different supplements or anything like that.

You know, they have a vested interest in using science to market their product and they can very much fall prey to cherry picking of, here’s one study that shows our product. And again, it can be a nutritional supplement, it can be a wearable, you know, works. And we’re gonna plug that heavily, but not necessarily talk about the 15 studies that show maybe it doesn’t work or they’re not gonna highlight that.

What we’ve just talked about, that all of the heat training studies have largely come from one lab.

[00:45:42] Trevor Connor: So I think one of the things that we’re hearing from you is if you’re interested in a particular device and you go to their website and they go, there are science behind this, and you go to their science page and there’s one study.

[00:45:52] Chris Case: Mm-hmm.

[00:45:52] Trevor Connor: You should be going beyond just looking at that.

[00:45:55] Chris Case: I think it’s wise to probably look at any commercial enterprise with skeptical lenses. Yes. To begin with. And yes, if they only have one study or if it’s a, this has been submitted for publication, but it has yet to be accepted, that’s their basis for their science.

That’s even more questionable.

[00:46:16] Stephen Cheung: Yeah. There are companies that have non-invasive or have commercialized non-invasive core temperature sensors now, so you don’t need a direct probe in your body. You don’t need to ingest a very expensive radio pill that only lasts for a day or two. You can have a single unit that can give you, or purportedly give you your core temperature.

They again. As a, the physiologist by training. I would love nothing better than than to have a noninvasive core temperature sensors that would make my life with ethical review boards a lot easier. It would make a participant kind of willingness to use these measures so much easier. But you know, you have to go with the data and as it stands right now, I would assess the weight of evidence of these non-invasive core temperature sensors.

And I’m not talking just singling out one particular device. I’m talking about the entire field as a whole. You know, the reliability may be okay, but the accuracy of these systems still has not been shown. Again, really with the consensus of. Of a large body of evidence, and that includes some of the work that I’ve published myself showing that not one, we tested two different systems.

They were on females and they were on well-trained females over heat stress tests, done over different parts of two different menstrual phases. And we showed that, again, the reliability is pretty good, but the accuracy isn’t for those systems. So I’m not the only lab that has shown that there’s been a number of labs that have shown this overall limited accuracy.

So, you know, I feel confident in saying that. Core temperature sensors, non-invasive core temperature sensors. You know, I wouldn’t jump on them as the greatest wearable as, uh, being the most accurate right now. And again, that goes back to marketing. The companies that have these systems, they obviously heavily hyped them.

They. Them to be sold. And I know in the particular case of the core system, CORE, that’s the name of the company, they started out by, you know, really getting a lot of pro teams on board to be using them. And their marketing is heavily based around all these pro teams are using them and, you know, implying that therefore you should also, and I know I have constantly gone to them and say like, show me the independent science that that shows me these work.

I would love to see it. And again, to this date they haven’t produced it. And the independent science that has been done, again, including some from my lab, have shown that their accuracy is still lacking. So. That’s a real challenge for athletes, right? You are. You see your favorite cyclists or your favorite cycling team using it.

You see it kind of highlighted in their advertising, but you know, again, teams are also sponsored and that’s how pro cycling works. They have equipment sponsors too, so they may be using it, not necessarily. Because of the science that it provides, but because they are getting a product and sponsorship for it.

So that’s a constant challenge. And how should an athlete kind of inform themselves is really through podcasts like these, through, you know, getting into the actual science itself. You know, at the very least, like reading some of the abstracts that are all publicly available and really kind of making a more informed decision than just, oh, my favorite team or athlete is using it and I’m hearing all this hype about it.

Therefore it must be good.

[00:50:27] Chris Case: What’s interesting is how many times have we said on this program. That this device might have some merit. This other methodology for training might be good, but what you really should do is focus on the fundamentals. And there’s no question about those, right? There’s literally hundreds, if not thousands of studies that say, you do this thing and it works for you.

What we’re talking about here in terms of questioning the science, and you gotta be careful of the marketing hype, are all these new things that may work. They might not, they might give you a percent here or a percent there, but they’re not the core of. Or the fundamental training methodologies, that’s gonna get you 90 to 95% of where you need to be.

[00:51:12] Stephen Cheung: Yeah, absolutely. And you also have to understand where you are. Right. Again, I’m a age group enthusiast. I wouldn’t even call myself a really highly competitive athlete anymore. Do I do heat training? No. Even though I have access to a lab, I have my own environmental chamber. You know, am I gonna put myself through heat training?

And for a few reasons, no. Again, where am I in my kind of competitive ability and level, like you say, Chris, just the basics is gonna get me 90, 95% of to my potential and it’s gonna do it in a much more sustainable manner as opposed to heat training. I can tell you, I’ve run studies in it, it’s horrible.

It’s, you are on an indoor trainer bundled up. Mm-hmm. As if you were going to do the, I did a bike and you are pedaling and just sweating buckets and with no fan nothing. It is a miserable experience. So. Yes, it may give you over five weeks of doing this, that extra one, maybe 2% of extra total hemoglobin mass increase in aerobic capacity.

But is it worth the discomfort of doing that for five weeks and then maintaining it through a competitive season? For me, no. Like again, even though I have full access to my own personal environmental chamber that I can use anytime that I feel like, so there’s what you can do sustainably versus, you know, doing an intervention, especially like heat training, which is really uncomfortable.

Even if the weight of evidence says absolutely, you are guaranteed to have. Two, 3% increase in in performance. Like again, for most of us, is it worth that? And there’s also research that shows that if you are doing heat adaptation or heat protocols, you are not getting, even if you’re doing the same wattage, you are not getting the same training stimulus from it because your body is so stressed from being hot that it is not having the same benefits on mitochondrial kind of density or improvement as opposed to, again, your question, Chris, like stick to the fundamental, stick to the key training, stick to quality nutrition, recovery, sleep, decrease in kind of external stress, and you’re gonna be 90 to, I would argue, like more than 95% of your potential and you’re gonna do it in a much more sustainable manner.

[00:53:58] Trevor Connor: You know, my one disagreement with you is I’ve been having this conversation with my fiance and showing her the evidence. If you get the two to 3% benefit, therefore I do need to go and spend five weeks on a beach in the Caribbean.

[00:54:09] Chris Case: Mm-hmm. To

[00:54:10] Trevor Connor: get those.

[00:54:11] Chris Case: Mm. Yes. I can see your rational now there. Yes. I gotta

[00:54:13] Trevor Connor: immerse myself in the heat.

Yes. It’s not just go in the basement and do it for an hour.

[00:54:17] Chris Case: Yep. Not put the garbage bag on and crank up the heat and pedal.

[00:54:22] Trevor Connor: No, I actually love that you said that because we published an episode or two ago, a potluck, where we discussed heat training and said in that episode, we’re not gonna address the science of it today because we’re talking with Dr.

Chung in an episode or two and he’ll talk about the science. But we said, let’s talk about it as a coach, and Grant was here, and he immediately brought up the, is there benefits to this? Probably. Is it worth it? Probably not. For all the things you just said. So I love that grant as a coach and you, the scientists end up in the same place on this, even if there is an effect here.

[00:54:59] Stephen Cheung: Yeah, if it’s a, you know, again, and heat training is probably an extreme example because of the time requirement, because of the discomfort, because of the potential impairment on the rest of your training and just enjoyment of training. There are certainly other interventions or tools that you know, maybe have a much lower cost.

[00:55:25] Trevor Connor: Heat training in your basement may actually be so tough that the benefits may be more mental than physiological. Here’s Dr. Jamie Whitfield explaining why.

[00:55:35] Jamie Whitfield: I definitely think training in the heat, not only just for things like plasma volume expansion and potentially hemoglobin increases, but I also think that perhaps what is underappreciated with something like that and something that people in the Northern Hemisphere will know intuitively is that training in the heat and training on the trainer is.

One of the most physiologically difficult, but also mentally difficult things that you’ll do. And so I think there’s definitely something to be said for the mental aspect of things, and that’s really hard to measure as an adaptive response as such. But I went through various lockdowns here where all I had was a trainer on my patio.

’cause I was only allowed to be outside for one hour a day. And so the alternative was you could ride on the trainer for as long as you wanted. And probably some of the best riding I ever did competitively was when we got out of lockdown. And that was just because everything felt so much easier than doing a three hour long ride sitting on my patio in in sweltering heat.

So I think it just fosters a certain level of mental resilience and toughness. And so I think that. Certainly something that that can’t be underestimated. And certainly the same is true of training in the heat that just turns the dial up for RPE all the way up. And so I think that’s one of the things that perhaps is under explored in sports science research is the mental load to training and also the responses to training and how that can impact subsequent performance.

[00:57:09] Chris Case: Well, now I’m curious, as a discerning scientist, what are the few things that you might say, ah, this is, this is worth a shot. Some of these fringe things. What have they, what? What is up to your muster in terms of science for some of these new trends?

[00:57:25] Stephen Cheung: Coincidentally, it’s another paper that I really love from Bent Ronstadt’s Group, and this came out in 2017 or so, and it’s the whole idea of periodization, of periodizing, your meso cycle, of your four weeks of training and front loading the intensity in that.

And hopefully we can put a link in the show notes about it. But I read that paper and it says, I lo, I love this idea. And the whole idea is to, rather than space out over four week period. Two hard interval sessions every week and just have that kind of even distribution. The idea is to front load the intensity and the intense workout.

So in this study by Ben Ronstadt, he had, instead of, he still had eight total hard efforts over the four weeks, but he front loaded it with five hard efforts over the first week, and then it was just easier endurance workouts and one intense effort over the remaining three weeks. So. Literally like no change to the overall volume or intensity, but just in the way they were distributed.

So I read that study and I was like, well, hey, I’m just in kind of a recreational, you know, enthusiast cyclist who loves to keep fit. Why don’t I try this? And it’s relatively low cost. I’m gonna do that training anyways. I typically do a few hard rides a week anyways. Why don’t I just try tweaking this?

And so I did try that and I wrote a blog post about the before and after, and it really worked for me. It increased my sustainable kind of eight minute power by. A really significant amount. I can’t recall the exact amount when I first did it, but it was definitely eyeopening and it was something I could actually do.

And I saw improvement in week two in my test time, week three and week four. So that’s something that I think is relatively, the opportunity cost is relatively low to try and experiment on myself. And so that’s what I did. So it’s still something that I go back to that I know from experience works for me.

So if I want to get fit really quickly and rather than again, just doing a two a week kind of intense workout schedule, I go and do a front load of workout and I know it’s gonna get me. Much fitter in four weeks. So that’s an example of where I took the science. I found it intriguing. There might not have been a huge weight of evidence behind it.

It was a one-off study from one particular research group. But I also waited of, you know, where am I in my training career. It’s relatively low cost. It’s something, this kind of workouts, these kind of workouts they were recommending. I typically do twice a week anyway, so it’s relatively low cost. So let’s see what happens.

And I found it was effective and it’s something I still kind of incorporate. Nice. And I guess my other example would be treating myself the ice cream every year. Ooh, yeah. Once a week.

[01:00:50] Chris Case: Mm-hmm.

[01:00:51] Trevor Connor: Absolutely beneficial.

[01:00:52] Stephen Cheung: Yes. I mean, twice a week. Why

[01:00:54] Trevor Connor: not? A hundred percent.

I’ve got one last question here. ’cause it is something I’m very biased about, but as somebody who knows infinitely more than me. Wanna get your take on this? Uh, in terms of knowing what science to trust, what is your feeling about researchers who say that meta-analyses are really the gold standard of scientific research?

[01:01:21] Stephen Cheung: Well, first off, what is a meta-analysis? It is a review where you take. Not just one study and tell the reader what it’s about, but you take as many studies as are relevant. And so we talked about limited sample sizes before. You have one study that did something this way with 10 participants. You have another study that did something slightly differently with 12 participants.

So you can just look at and kind of look at those two papers individually, or you can take those papers and statistically kind of analyze ’em together along with a whole bunch of other studies. So you may end up with, you know, 10 studies. I’ve done meta-analyses on heat adaptation with 130 studies, and we’re analyzing objectively, numerically, and statistically the data of those 135 papers together to try to come up with a larger sample size.

So again, in that 135. Paper meta-analysis on heat adaptation, we had about 1500 to 2000. I can’t recall the number, but we had a much a 2000 participant sample size rather than 10, right? So that’s the power of meta-analysis. The challenge with meta-analysis, or the caveat you always have to think is goes back to what we talked about before, how you search for papers or the search parameters that you use to decide which papers you include is gonna affect your end result, right?

So if I say, well, like I don’t want to look at any studies done, let’s just be completely nonsensical. Done before 2000. Right then that’s a decision I made and that’s gonna affect the, which studies I end up with and the data that I’m gonna get. If I say I’m only gonna look at completely untrained individuals, well again, that’s another decision that I’ve made and that’s gonna affect the data.

And it talks about what we discussed earlier about everybody using slightly different tools, slightly different filters. So you still, meta-analyses are great because you can have a much larger data set and you can do much bigger and stronger statistics on them. But you also have to understand that how the par, how the scientists chose those papers can be a cause of bias.

So, and also how they choose to statistically analyze the data. So. They’re not perfect, but they are. I would argue better than, you know, just a narrative review. Just if I was writing to you saying, do I think heat adaptation works? Because that can be very biased towards the papers that I know the best.

As opposed to, no, I’m not just gonna go for the papers. I know the best. I’m gonna look for every paper that I can find with this search parameter, let’s say on females. And I’m gonna come up with that rather than just again, myself, oh, I know this one paper that that talks about females and heat adaptation, and I’m gonna base my entire kind of opinion or my general bias towards it here.

I’m being much more objective, hopefully in my selection. So I think meta-analyses are. Strong. I think they are an important tool, but again, just like any scientific article, there can be unconscious bias or very conscious bias based on what papers are being selected.

[01:05:08] Trevor Connor: Meta-analysis address a lot of the issues with underpowered studies, but they pose a whole new set of challenges that get complex.

Here’s Dr. Rosenblatt, an expert on meta-analysis talking about the challenges warning nerd bomb ahead.

[01:05:23] Michael Rosenblatt: There’s several things when you’re looking at a meta-analysis that you need to consider starting with the I Square or the statistical heterogeneity. If they don’t report an I squared value or another form of statistical heterogeneity, and there are other ways to do this, the results are garbage because you don’t know if those studies should be pooled together or not, and you don’t know if there’s some contributing factor.

I’m gonna just say that off the get go. I don’t care. It’s true. The next thing is if you do a subgroup analysis or a meta regression, meaning that you’re looking at some sort of continuous variable like age or VO two max or sex or anything, and they say, well, we did these analyses but we didn’t look into if it altered the statistical heterogeneity, those results are also garbage.

And the reason why the garbage is because they’re not saying, well, did this actually influence the results? You do a subgroup analysis or a meta regression because you want to determine what causes that statistical heterogeneity, meaning are there clinical factors around the study? That maybe influences those studies.

Okay, so that’s really important. The next thing is looking at change from baseline or percentage change from baseline. Now, I’m certainly somebody who’s done this too in previous studies, but we should only be looking at, are there differences between groups at follow up? We shouldn’t be looking at within group changes from baseline to follow up unless you’re doing it specifically for looking at relationships.

Maybe you can use those, but even then there’s some limitations to that. The magnitude of the effect is garbage if you’re just looking at change from baseline, and there’s several reasons for that. Most importantly is that if you’re looking at a change from baseline. The follow up result that you’re looking at, that change score is correlated with the baseline value, meaning that maybe some people respond differently based on their baseline values, and if there’s some degree of correlation, and there is in most cases, the result is meaningless.

There was a study that came out and I spoke with, uh, the author about why there’s problems with meta-analysis and, and sports science. And I said to him, I’m like, well, I hate the title of what you wrote for this because there’s nothing wrong with meta-analysis. There’s problems with how you do them. And so that’s what he ended up getting to.

And I spoke with him about it and it’s like, well, there’s nothing wrong with the methodology. It’s are you doing the methods correctly? And then there’s this thing about p hacking and p hacking is if you, you know, you’re looking for a result after you do your analysis because there’s no statistically significant difference.

Well, the thing is, is, and so you’re doing some sort of subgroup analysis, but if you ran, if you do a randomized trial and you randomize participants into different groups, there should be no difference between those groups other than the intervention. However, when you’re doing a meta-analysis, you are combining studies that are all different, and you’re not randomizing how you combine those studies.

So it’s called the Oxford Approach. First, you combine all studies that meet your eligibility criteria no matter what. Then what you do is you look at the degree of statistical heterogeneity and you look for what we call clinical heterogeneity. What are the possible reasons that can explain this variation?

And so you’d look at study design, participant characteristics, intervention characteristics, and outcome characteristics. And you start investigating to see, well, where might these differences be? And you wanna get as much as you possibly can, and you can’t just have a statistician do this. You need to have somebody who’s a statistician within the field that you’re looking at so that you can specifically say, what about periodization?

What about the wash period? Well, if we’re looking at periodization, was it a two to one or three to one? Okay. How many days after they did their intervention period did they measure VO two max? Because that could influence the results because maybe they haven’t recovered. So when you’re actually doing these ways to assess for clinical heterogeneity, you wanna look at every possible variable that you can, which is what makes this type of a project ridiculously complex to do, which is why, again, I say if you’re going to look at a meta-analysis and they don’t talk about heterogeneity, and they just show a result, you can’t trust that result.

If they do a subgroup analysis, but they don’t say, Hey, it decreased the statistical heterogeneity, maybe that subgroup analysis is irrelevant.

[01:09:26] Trevor Connor: So there is then the really important question, is it possible to do research intentionally or unintentionally without some bias?

[01:09:35] Stephen Cheung: I guess it would depend on how you slice or dice or define that.

I would ultimately, if you want to be super technical, like I can say no ’cause, because every choice that you make as a scientist is a bias, right? So by definition you can’t. I know personally what I teach my students and what I try best to do is come up with as neutral a design as possible to answer a question, and where if I see a positive effect, it’s an interesting story if I see a knot non-positive or non-significant, or even a negative effect.

It’s an interesting story. Ultimately, as an editor in chief, when I triage and review papers, what I look for is, okay, what’s the research question? Is it an interesting question for our journal’s audience and. Is the design in general a good design to answer that specific question. If I was doing the same study, is this a generally an appropriate design that I would use?

I may not make a hundred percent of the same choices, but in general, do I agree that this is an overall good approach? So again, the challenge is unless you have a lot of experience in reading scientific papers, you know, and irrespective of what degree you have, you may not have that ability to really discern that.

So what would I do as an athlete? I would. Talk to people who you really trust in terms of their knowledge of the science that have also your best interest at heart, whether it’s a coach, whether it’s a mentor of some kind, or a sports scientist or a nutritionist that you know isn’t just trying to sell you something really does have your best interest in mind.

And I would also say, don’t be afraid to. Read the science. You don’t have to read the entire paper, but certainly again, understand the abstract. The abstract is typically a 250 word summary of the rationale for the study, the main methods, the main results, and the main conclusion. And just be willing to read those.

[01:11:59] Trevor Connor: And I think something that’s important here, you know, as you said, read the abstract and all of us read a lot of studies or I’m trying to think of the right wording here. ’cause we don’t read the study, we just read the abstract and never get to the actual study. And that’s inevitable because there’s so much new research coming out.

There’s no way you can read it all. But I can’t tell you how often I’ve read the abstract of a study, then read the study and go. Wow. That’s not what I got from the abstract. Hmm. When you dive into it, you just get a different story. Mm-hmm. Because it is really hard to summarize it in 250 words.

[01:12:31] Stephen Cheung: Yeah. And I wouldn’t rely on just one paper.

Right. Just get used to reading abstracts in an area and you’ll very quickly get a sense of. Even if all you do is read the abstracts of what’s the general idea that seems to be out there about, again, let’s say heat training or the validity of tool X or Y, and you’ll be able to at least get the broad view.

And then again, if you can talk to someone who may be more expert in that, who can guide you into understanding it further. Then even better, again, whether it’s a coach, whether it’s a nutritionist, whether it’s a healthcare professional. Yep.

[01:13:18] Trevor Connor: And then maybe as we’re wrapping things up, I have to ask the question, because this is the big topic nowadays.

What is your feeling about asking AI to summarize the research for you? And I will. Start with my story. The first time I tried that just to see I was interested, I went to chat. GPT asked it to summarize the research on a particular subject. Read. It went, that’s very interesting. Please provide me with the studies that you got all this from.

And it gave me a list of 10 studies. And then I went to look at those studies and every single one of them was made up.

[01:13:52] Stephen Cheung: Yeah, that’s a big issue. And I know that’s the same issue in healthcare. Should you use chat GPT to, you know, instead of your doctor. And you’ve highlighted it perfectly, Trevor. It’s a, it’s garbage in, garbage out.

So one is what papers, what data is informing the algorithm for the AI to give you results. And also it can also be an echo chamber. So I’m starting kind of to do a lot more work on female physiology and sports science, but if you talk to chat GPT or talk to AI and ask for the general knowledge about concept X, while most of the data is gonna be coming from male data, it’s probably gonna be very limited coming from female data.

And that doesn’t even get into what you just highlighted, that a lot of these studies can be made up, or it can be in the gray literature, it may not be in kind of actually peer reviewed journals, and it can be someone’s random blog post, for example. Right. So I personally, certainly as a scientist, I wouldn’t ask AI to summarize a field for me, but even as an athlete, I would very much advise against it.

I think you’d need to go and. Do some of your own research and again, read some of the abstracts. At least get a sense of the field before you maybe ask AI to refine your question further.

[01:15:31] Chris Case: All right, Dr. Chung, you’ve done this before. It’s been a while, but you’ve done it. We wanna get to our take homes.

Would you like to go first or would you like to go last today?

[01:15:41] Stephen Cheung: I’m happy with whichever. I’d love to hear your, uh, opinions first and take homes. So why don’t I go last? Sounds good. Sounds good. I’m gonna put the spot on Trevor, then go first, Trevor.

[01:15:51] Trevor Connor: Alright. My take home is something that took me a long time to realize about the science, but I think is really important and anybody who listens to our show knows I’m a science geek.

I love the science. But I had that honeymoon phase with the science where I’m like, Ooh, if it’s a, there’s a study on it. It has to be true. It has to be real. It has to be great. And you hear people talk all the time out in social media about, oh, this is true because there’s science behind it. As soon as you cite a study, it has to be true.

And that’s actually why I loved last night going into these studies and reviews that we’re talking about the nature of exercise science, and it was a little depressing just seeing all these issues, but just that recognition that science isn’t perfect. You can’t read one study and go. That’s definitive.

Now we know this helps you or this doesn’t help you. There needs to be a body of science and you do need to learn how to read it. Or if you can’t read it yourself, find those people who, like Dr. Chung, who are really good at reading the science and can say, yeah, this is a good study. That’s a bad study.

This is a good body of evidence that you can trust. We need that with the science. You can’t just immediately say, oh, this study drew that conclusion. Therefore, and certainly, you know, we’ve had that on the show where we’ve had people with products, new products come to us, say we want to advertise, and they show us some scientific studies and our response is, well that’s great.

That’s interesting science. And they go, so can we advertise? And we go, no, it’s interesting. It’s pointing in the right direction, but the science isn’t there yet. And you have to be skeptical like that, saying this looks promising. There’s a couple studies. It’s not there yet.

[01:17:42] Chris Case: Yeah. For me, the skepticism comes naturally, I suppose.

And so without discounting this whole discussion, my personal approach here is maybe it’s just boring. Like the new stuff doesn’t interest me. It’s the science behind it. Maybe the study of it is interesting, but I don’t latch on right away. I’m not a new adopter, so to speak, so call it boring, call it just traditional or something like that.

But I always look at things skeptically weight. Quite a while until that body of evidence grows and it’s confirmed and replicated and repeated before I would ever latch onto any of these things because I just find that I get everything I need from just the basics. Dr. Chung,

[01:18:32] Stephen Cheung: I love what both of you have said and so mine’s gonna be really to tag on and amplify.

That is Science isn’t perfect and I know we spent a lot of time this episode really apparently or seemingly to trash science and trash what we as scientists do. That is not the intent. It is to educate the listeners on the limits of what science can and can’t do. But I really love what you just said, Chris, that it’s the weight of evidence.

It’s don’t go for the one-off really kind of viral study that takes off whether in social media or amongst your friends really. Don’t be afraid to get into the literature. Again, the abstracts generally are fairly readable and if you read enough of them, you will get a general idea of the overall concept.

And then don’t be afraid to self experiment to really weigh the cost and benefits of whether you want to try something, whether it’s gonna work for you in the right situation. And again, I’m a heat the physiologist and I don’t do heat training, and I’ve outlined the reasons why, so. So really assess whether it’s the right thing for you.

And then don’t be afraid to ask questions, whether it’s to the scientists themselves or to people who have maybe more direct knowledge in a particular field.

[01:20:04] Trevor Connor: Fantastic. Dr. Chung. Always a pleasure.

[01:20:07] Stephen Cheung: It’s always great to be back and love what you guys do. Thanks so much.

[01:20:13] Trevor Connor: That was another episode of Fast Talk.

Subscribe to Fast Talk wherever you prefer to find your favorite podcast. Be sure to leave us a radiant or review. Don’t forget, we’re now on YouTube. Give us a like and subscribe there too and help us reach new audiences. As always, remember that the thoughts and opinions express in Fast Talk are those are the individual.

Join us on social media at at Fast Talk Labs for access to our endurance sports knowledge base. Continuing education for coaches as well as our in-person remote athlete services. Head to Fast Talk labs.com for Dr. Steven Chung, pat Warner, Dr. Jamie Whitfield, and Dr. Michael Rosenblatt. I’m Trevor Connor.

Thanks for listening.