Garmin Fénix 7^® Underestimates Performance at the Lactate Threshold in Comparison to Standardized Blood Lactate Field Test

Introduction

Performance diagnostics are a central element for many professional and recreational athletes throughout the season. These diagnostics usually apply advanced technologies such as calorimetry, photoplethysmography, near-infrared-spectroscopy (NIRS), blood lactate analysis, and ergospirometry. Among the numerous measures, cardiorespiratory fitness (VO_2max) and lactate threshold (LT) are crucial performance indicators, particularly in endurance disciplines.^1,2 Previously, athletes depended on time-consuming and costly laboratory-based tests to obtain these metrics. The gold standard procedures for ergospirometry for VO_2max and lactate testing for LT³ involve wearing uncomfortable masks and invasive blood sample acquisition. After testing, the analysis and implementation of the acquired data for training and racing were reserved for sports scientists and coaches. In recent years, different non-invasive methods for LT determination that use surrogate markers and algorithms to predict lactate levels have been introduced.⁴ One example is the use of wearable NIRS devices attached to prominent locomotor muscles during a given exercise. During running, NIRS can be used to track changes in muscle oxygenation in the calf or quadriceps muscles in response to running speed.⁵ Despite the non-invasiveness of these methods and evidence of their validity and reliability in detecting LT,⁵ their application is still costly and rather laboratory-based. It requires trained personnel for the measurement and interpretation. With the advent of wearables such as global positioning system (GPS)-coupled fitness watches, performance testing should become accessible to a broad range of customers. These smartwatches aim to master the advanced features of performance testing, such as VO_2max and LT determination, and provide training and recovery recommendations, as well as race result predictions.

Among the general population, smartwatches are popular for their daily activity tracking with measures like step count, climbed floors, heart rate (HR), and, based on that, an estimate of daily energy expenditure. However, smartwatches play a more profound role for recreational and professional athletes. The features of advanced models have evolved to include complex physiological measures such as heart rate variability, VO_2max, and LT determination. Based on frequent monitoring of VO_2max and LT, the effectiveness of the training strategies can be analyzed. Furthermore, these measures allow for the setting of individual intensity zones, representing the basis of endurance training.² Smartwatches therefore constitute a powerful tool for monitoring, tracking, planning training, races, and recovery. Especially in endurance disciplines like running, cycling, swimming, or triathlon, wearables are used daily. In 2022, Helsen et al reported that in a sample of 2146 active runners, as much as 84.8% used a smartwatch during weekly exercise.⁶

The VO_2max estimate of smartwatches from the Garmin^® Fenix family (Garmin Deutschland GmbH, Garching) uses extrapolation of running speed and individual HR based on an algorithm developed by Firstbeat Technologies, Finland.⁷ To determine the LT, the previously calculated VO_2max estimate, running speed acquired by GPS, and HR measured with a chest strap were combined.

If accurate, LT identification using smartwatches can alter the performance testing field. However, the question remains whether these wearables can detect individual thresholds that are valid and reliable enough to replace repeated laboratory-based performance tests, which in Germany, on average, cost around 190–395 € (own market research). Before relying on sports success based on the metrics estimated by a smartwatch, independent research should ensure their accuracy and uncover their potential weaknesses. As the underlying algorithms for wearables are constantly evolving, the need for further evaluation in field tests in step with actual practice is evident.⁸ This study aimed to compare the estimates of individual running pace and HR of the Garmin Fenix 7^® to individual running pace and HR obtained during a standardized capillary blood lactate field test using the modified Dmax method for analyzation.⁹

In the corresponding literature, previous Garmin Fenix^® models have been tested for accuracy in complex physiological measures (Table 1). Wagner et al investigated the reliability of the Garmin Fenix 5^® in detecting energy expenditure and VO_2max in a sample of 34 male runners.¹⁰ They concluded that the measures were unreliable compared to the corresponding reference methods, calorimetry, and ergospirometry. Similar results were reported by Anderson et al, who compared the Garmin Fenix 5x^® to the ParvoMedics TrueOne 2400^® (PMT) in 17 male and 8 female recreational runners. Carrier et al were concerned with the validation of the LT estimate of the Garmin Fenix 6^® in comparison to a treadmill-graded exercise test with blood lactate monitoring. Compared with the lab-based test, Fenix 6^® underestimated the LT speed and HR at LT; however, it did not differ significantly.¹¹ Indeed, this study was based on measures of six subjects; therefore, more robust findings are desirable. Another study by Carrier et al reported a mean VO_2max overestimation of 8.05% in the Garmin Fenix 3 HR^® model compared to measurements obtained from a laboratory-based graded exercise test.¹² A recent systematic review and meta-analysis by the INTERLIVE Consortium concluded that wearables could generally be applied at the population level. However, improvements must be made before their use in sports and clinical purposes is possible.¹³

Table 1 Corresponding Literature on the Accuracy of Physiological and Biomechanical Estimates of Different Garmin® Smartwatch Models

Thus, the present study aimed to tie in with the corresponding literature and investigate the LT speed and HR estimate of the latest Garmin^® Fenix model, the Garmin Fenix 7^®. Therefore, we compared the estimated LT speed and HR with values obtained during a field-based graded exercise test. This should provide insights into potential improvements concerning the accuracy of estimated physiological parameters calculated by wearables, thus enabling athletes, coaches, and researchers to make well-informed decisions on how to implement these wearables in practice, racing, and research.

Methods

Transparency and Openness

The study was conducted following the guidelines of the Declaration of Helsinki and was approved by the Ethics Committee of the University of Bundeswehr Munich (M 23–28). Consent was obtained from all the participants before participation in the study. Anonymized data can be obtained upon request.

Participants

We performed an a priori power analysis using G*Power¹⁵ to estimate the required sample size. To compare the smartwatch’s estimated lactate threshold and the threshold according to the regular field test, we used a dependent means t-test. We adopted an effect size of 0.5, an alpha error probability of 0.05, and a power of 0.8. These input parameters resulted in a priori recommendations for the 34 participants in this study. The inclusion criteria were: (a) all participants were adults and (b) were healthy. Due to incomplete data (48 %), that is, no lactate threshold was ascertained, a resting lactate level ≥2 mmol/l, or dropout, 26 datasets were completed and incorporated into the statistical analysis. The calculated post-hoc power was sufficient (β= 0.99). Only the differences in HR values between the watch and Dmax methods were low (β = 0.42).

Materials

Participants received a Garmin Fénix 7^® smartwatch and a HR chest strap (Garmin HRM-Pro) during the study period. Garmin Fénix 7^® is a high-end smartwatch product¹⁶ with lactate threshold pace and HR estimation capability. Like many wearables, Garmin uses a light-based photoplethysmography (PPG) sensor to estimate the HR on the wrist. However, the LT measurement only works in combination with the chest strap HRM-Pro.¹⁷ The HRM-Pro sensor measures with a frequency of 2.4 GHz at 8 dBm nominal. It is licensed under us8386009. The instruction was to use the smartwatch and the chest strap in each training session (see 2.4 Study Design). The smartwatch must be connected to an application. As the use of the corresponding Garmin application Garmin Connect is prohibited by the institution owing to data security concerns, we used the provider Fitrockr.¹⁸ Theoretically, all aggregated data from the watch were stored on German-based servers and exportable However, we did not analyze the synced data because the watch displayed all the required data. For the subsequent field test, the Lactate Scout+ Solo or the Lactate Scout 4 (EKF Diagnostics, Cardiff, UK) were used to measure blood lactate samples from the earlobe. These portable devices need to absorb 0.2 microliter of capillary blood onto a pre-calibrated enzymatic-amperometric sensor. For each measurement, a fresh sensor is connected to the device. They applied a measurement range of 0.5 to 25 mmol/L. The Lactate Scout+ was tested in a study by Bonaventura et al (2015) and four other portable lactate analyzers. Overall, the results showed that all portable analyzers have a negative bias underestimating the values either in low blood lactate concentrations or in high concentrations, depending on the device. For the Lactate Scout+ the measurement error was 3.5 %. However, the authors suggested that portable analyzers are still reliable for determining the LT and added that devices from one manufacturer can be used interchangeably.

Study Design

The study was divided into four parts: (a) introduction to the smartwatch and design, (b) calibration phase, (c) LT test with the watch, and (d) LT field test with blood lactate. First, all participants were introduced to how and when to use the smartwatch. Here, we helped them synchronize their smartwatch with the Fitrockr hub and explained the procedure according to the Fenix 7^® manual.¹⁶ The watch uses the profile information entered in the initial setting and VO_2max calculation to calculate the lactate threshold. Nevertheless, the participants had to run some sessions with the HR chest strap to determine accurate maximum HR and VO_2max calculations. The Garmin^® manual does not specify the number of training sessions that must be completed to begin the lactate threshold test. Second, to standardize the validation, we instructed participants to run twice per week for 5 km. 1.) At 70% of the individual’s maximum HR (70% HR_max), and 2.) at 85% HR_max. The use of smartwatches for other exercise activities was prohibited. We calculated the maximum heart rate (HR_max) using the standard sports science formula (220 minus age). Third, after five weeks, they should execute the lactate threshold test with the watch. Because this test required a maximum load run, one of the authors was presented to provide first aid if needed. After activating the GPS, the participants started the lactate threshold test on the Garmin smartwatch as follows (Figure 1):

Figure 1 Procedure on Garmin watch face: getting to the guided lactate threshold test.

1. From the watchface, press START
2. Navigate to the activity outdoor running,
3. Hold MENU
4. Select training
5. Select lactate threshold test > start test following the instructions on the screen.

During the test, the watch displayed both current and intended target HR. The task was to run the given intervals at each displayed target HR for a specific time. After completing the test intervals, the participants were notified by the vibration signals from the watch. They had to stop the timer and save the activity. Subsequently, the watch allows users to update their HR zones for future training based on the ascertained lactate threshold HR. However, users can always decide whether to accept or reject the latest lactate threshold calculation. In this study, all the participants were instructed to accept the calculated values. Fourth, the lactate threshold values, lactate threshold pace, and lactate threshold HR determined by the Fenix 7^® were compared with a graded test in the field using blood lactate taken from the earlobe ≥ 48 hours later.

Participants were instructed not to fast before the testing (no food <1 hour before testing). A maximum of one cup of coffee was allowed in the morning before testing. We did not control for the female participants’ menstrual cycle phase (follicular or luteal). The participants arrived at the test location in groups of two. Resting lactate was taken from the earlobe and analyzed using Lactate Scout+ Solo or Lactate Scout 4 before the first stage started. We chose a starting velocity of 8 km/h for the male participants and 6 km/h for the female participants. To control velocity, one investigator functioned as a pacemaker using an accelerometer on a bike. Each stage was run 2 x 400 m. After completing the stage, the participants stopped at the test location. Subjective exhaustion was measured using the Borg CR10 scale to ensure maximum load.^19,20 HR was read from the watch display, and blood lactate was measured. A standardized break of 30s was used for this purpose. The Borg CR10 scale indicates the rate of perceived exhaustion (RPE) from zero (no exhaustion) to ten (maximum exhaustion).^19,20 At the end of the break, the next stage, which was one km/h faster, commenced. This procedure was repeated until the individual performance limit was reached for each participant. Immediately after completion, the last lactate probe and heart rate (HR) were measured. The modified Dmax method^9,21 was used to analyze the lactate threshold. The method contains:

1. The first increase in lactate (> 0.4 mmol/l compared to the previous stage) served as a rule for determining the aerobic threshold
2. The point on the lactate curve at which the aerobic threshold is determined is related to the lactate termination value.
3. Next, we determined the point on the lactate curve with the maximum perpendicular distance to the connection between the two points.
4. Consequently, the anaerobic threshold corresponds to the load at the point determined in step 3.

For a visual workflow, see Figure 2. If the resting lactate value of one participant was ≥ 2 mmol/l, they were excluded from the study, as it could be problematic when calculating the anaerobic threshold using this method. After conducting the lactate threshold test once on the watch, the values were updated for each run. Therefore, the field test was tracked as training so that the updated value from the day of the field test, and not from the lactate threshold test of the watch, could be compared with the field test data. Hence, we compared the data obtained from the same day using the watch and the regular used field test with Dmax method.

Figure 2 Study Design.

Statistical Analysis

We used Excel (Microsoft, 2019), SPSS version 29 (IBM Statistics, 2023), and JASP Version 0.16.4 (JASP Team, 2022) for statistical analysis. We compared the ascertained lactate threshold pace of the watch, respectively the km/h, with the pace of the field test (Dmax) as we used this method to calculate the values. In addition, we compared the estimated lactate threshold HR of the watch with the HRs of the field test. All dependent variables, lactate_PaceWatch, lactate_HRWatch, lactate_PaceDmax, and lactate_HRDmax were normally distributed, and we used dependent t-tests to determine if the values differed. Additionally, we calculated the mean differences between the two measurements ((Watch-Dmax)/Watch*100) and plotted the mean differences and levels of agreement in a Bland-Altman Plot. All data can be obtained upon request.

Results

Descriptive Statistics

Descriptive data of the 26 participants are listed in Table 2.

Table 2 Descriptive of Study Sample

Subjective exhaustion was measured using the Borg CR10 scale to ensure maximum load.^19,20 All participants, except for two, reached a maximum exhaustion score of 10. Therefore, we assume all participants reached their individual capacity. We found that the pace at the LT calculated by the Garmin Fenix 7^® (M 11.87 km/h ± 1.26 km/h) is 11.96 % lower than the pace at the LT obtained during the field test (M 13.28 km/h ± 1.72 km/h). Similarly, the HR acquired by the Garmin Fenix 7^® at LT (174.42 bpm ± 4.62 bpm) is 1.71 % lower than the HR at LT calculated with the Dmax Method (177.42 bpm ± 9.99 bpm). We observed a higher variance in pace and HR at LT for the data gathered during the lactate field test. For an overview, see Tables 3 and 4.

Table 3 Descriptive Values of the Dependent Variables as Mean (Std)

Table 4 Differences Between the Measurements for All Participants

Pace and HR Differences

In general, we observed a significant difference between male and female participants in their maximum km/h at LT as measured by the watch (t(24)= 4.303, p <0.001, d = 2.0) and the Dmax method (t(24)= 2.802, p = 0.01, d = 1.3) (see Figure 3). Garmin underestimated the values for male and female users by 11.48% (male) and 13.58% (female) for the pace at LT, and 1.47% (male) and 2.5% (female) for HR. By comparing the calculated pace at LT of the watch with the value gained during the lactate field test, we observed a significant difference (t(25)= −6.089, p < 0.001, d = −1.19). This equals an average increase of 11.96% in the measured km/h at LT during the lactate field test. Although we also detected a 1.71% difference in the HR at LT, the effect was not statistically significant (t(25)= −1.837, p = 0.078, d = −0.36) (see Figure 4). Interestingly, the visualization of all data points and their distribution shows that the differences may be because the smartwatch does not seem to differentiate as accurately in the marginal areas as the Dmax method does for pace and HR. However, smartwatch values accumulated in the middle range (see Figures 5–7).

Figure 3 Differences between the LT pace output from the smartwatch and the modified Dmax method. Notes: Differences between the LT pace output from the smartwatch and the modified Dmax method. Error bars show the SE. * = p < 0.001. Graphs were created using Microsoft Excel (Microsoft, 2019).

Figure 4 Differences between smartwatch LT HR output (bpm) and the modified Dmax method. Notes: Differences between smartwatch LT HR output (bpm) and the modified Dmax method. Error bars show the SE. Graphs were created using Microsoft Excel (Microsoft, 2019).

Figure 5 The distribution of all data points (pace). Notes: The distribution of all data points (26 per comparison group and graph) was illustrated using rain-cloud plots.22 Here, we compared individual km/h measured with the Garmin Fénix 7® (green) and the Dmax Method (orange), not stratified by sex. The graph was created using JASP (JASP Team, 2022).

Figure 6 The distribution of all data points (HR). Notes: All the 26 data points were included in the analysis. Illustrated by rain cloud plots.22 Here, the comparison of individual HR measured with the Garmin Fénix 7® (green) and the Dmax Method (orange), not stratified by sex. The graph was created using JASP (JASP Team, 2022).

Figure 7 Bland-Altman Plot for differences in pace (A) and differences in HR (B) plotted with mean values. Notes: Bland-Altman Plot for differences in pace (A) and differences in HR (B) plotted with mean values. It shows that the deviations between the measurement methods (here: smartwatch and field test with Dmax method) for pace and HR at the lactate threshold are within the level of agreement.

Discussion

In this study, we aimed to compare the running pace and HR at LT estimated by Garmin Fenix 7^® with a standardized graded field test using capillary blood lactate. We found that Fenix 7^® underestimated both pace and HR at LT compared to the field test. The mean difference for pace at LT was 11.96%, and 1.71% for the HR. Differences in pace and HR during LT were evident in both genders.

In recent years, wearables have gained popularity because they provide data that can help improve performance or augment healthy habits.⁶ Therefore, several studies have investigated the validity and reliability of different variables obtained from wearables, such as VO_2max or LT, and compared them to laboratory standards.¹³ Similar to our findings, they reported deviant values. However, only a few variables can estimate the pace or HR at LT. Hence, existing data are limited. A study by Carrier et al compared Garmin Fenix 6^® to a treadmill-based graded exercise test.¹¹ In line with our results, they found that Garmin Fenix 7^® underestimated the pace and HR at LT. However, as the calculated difference was only 8.38% for pace at LT and 6.20% for HR at LT, they concluded that the wearable device was valid. In contrast, we found a greater difference (11.96%) when comparing the estimated pace at LT with that at LT quantified during the field test.

Interestingly, while the calculated LT values seemed to underestimate the participants’ potential, studies on Fenix 5^® and 5x^® showed the opposite for VO_2max estimates. Additionally, Anderson et al (2019) and Wagner et al (2020) reported mean differences of 2.16 mL/kg/min and 4.3 mL/min/kg for the Garmin Fenix 5x^® and 5^® in recreational runners, respectively.^10,14 These findings are supported by a recent systematic review and meta-analysis that compared the literature on VO_2max estimates for 14 different wearables.¹³ As there are only some smartwatches that can provide LT estimates, such collective comparisons for this parameter are still missing. However, because the LT estimate is based on a combination of VO_2max, pace, and individual HR, it can be expected that the error observed in VO_2max estimates will likely be resolved.

For recreational runners, these differences may not cause serious problems. The degree of agreement between LT pace and HR determined with the field test vs estimated by the Garmin Fenix 7^® was visualized using a Bland-Altman plot (see Figure 7). For LT pace, most data points are close to the mean without a specific trend. For the HR at LT, however, the two methods seem to deviate with a noticeable trend. At lower HRs, the difference between the means tends to be positive, while at higher HR of >180 bpm, the difference between the means appears to be negative. This underlines the difference in HR variance between the two conditions (see also Figure 6). It is likely that the estimated LT pace and HR still positively influence training, racing, and recovery compared to having no information on these performance metrics. The gathered estimates provide a basis for classifying individual training intensity zones and designing training cycles. If the deviations in LT pace and HR are constant, potential training adaptions, performance gains, or losses can still be detected over time. However, considering a competitive or professional athlete, the same estimation error may have more extensive consequences. The pace at LT was, on average, underestimated by 11.96% by the smartwatch compared to the field test. For example, at an actual LT pace of 4:00 min/km, this deviation can be as high as 28s/km. This distinct underestimation might lead to the permanent application of insufficient or inadequate training stimuli, which would prevent athletes from achieving their best possible performance enhancement within a given training period. In contrast, a potential overestimation of LT pace or HR could lead to unplanned fatigue or over-training symptoms. These assumptions are also supported by the higher variation in the data obtained by the standardized field test. The variance (SD) was greater at both pace and HR at LT when quantified using the field test and modified Dmax method. In this case, a higher variance may indicate a better representation of the inter-individual performance spectra using the modified Dmax method. The lactate field test and subsequent calculation of pace and HR may thus enable scientists and performance coaches to detect finer nuances in performance within and between individuals. This can be used to improve athletic training and consequential performance. Improving performance by paying attention to details and improving minor factors has already been applied by many large teams in competitive sports, such as Team Sky.²³

Although there is a growing interest in noninvasive measurements to determine LT,⁴ data comparing the indirect LT estimates of smartwatches with laboratory tests are scarce. Other indirect concepts use wearable devices with NIRS. In contrast to smartwatches such as Garmin Fenix 6^® or 7^®, which base the estimate of HR, VO_2max, pace, and an algorithm, these devices use optical sensors to quantify the local muscle oxygenation status to obtain the LT. Several studies using NIRS-based principles concluded it to be a low-cost and reliable tool for estimating the lactate threshold, with high correlations for HR and power compared to a capillary blood lactate test.^24–27 Therefore, the assembly of NIRS devices and wearables may represent a promising principle for future developments.²⁸ As the Garmin^® Fenix series can quantify blood oxygenation through optical sensors,²⁹ a potential way to improve LT estimation could be incorporating blood oxygenation in the calculation.

However, there are some general limitations that impede data interpretation. As previously mentioned, there is a lack of studies testing smartwatch LT estimates in validated laboratory settings or field tests. This is directly linked to a second limitation concerning the study population, which is caused by the lack of available studies and is very homogenous.^24–27 Future studies are needed in the field of competitive sports, recreational sports, and clinical settings across the board. Another concern is the variety of methods applied in studies dealing with wearable devices compared with gold standards. This included both data acquisition and analysis. This problem was addressed by Molina-Garcia et al who presented guidelines for the validation and testing of wearables including reference standards and statistical testing.¹³

Additionally, we want to mention that Garmin^® handles their used algorithms completely. This means they can change their calculation bases at any time without providing reasons. This may limit comparability between older and future studies. Another important limitation of the methods applied in this study is the use of the Lactate Scout+ and the Lactate Scout 4 devices for assessing blood lactate during the field test. A study by Bonaventura et al (2015) showed that portable blood lactate analyzers underestimate either low or high values, depending on the device. To the best of our knowledge, today, there is no study on the reliability and accuracy of the Lactate Scout 4 compared to a gold-standard stationary lactate device. Furthermore, this study primarily focused on comparing pace and HR at LT between the two conditions, not the blood lactate values from the field test. Additionally, we did not ask for the female participants’ menstrual cycle phase (follicular or luteal).

There is a growing interest in non-invasive methods for determining physiological parameters, such as lactate and the associated thresholds. Various direct and indirect advances have made these variables easily accessible to athletes and coaches. To conclude, supported by our findings and the literature, it seems that the LT estimates calculated by smartwatches, such as Garmin Fenix 7^® are not yet comparable to laboratory and standardized field tests. However, in recreational settings, using smartwatches can still serve as a basis for the monitoring and planning of training, racing, and recovery. As the number of available studies in this context is still small, future work is needed to address the current accuracy of LT estimates using smartwatches and ways to improve their underlying principles.