On Tuesday at The Athletic, Corey Pronman released a ranking of NHL teams by their under-23 players. The Lightning landed at number 20 on his list. Pronman is one of the most well-known prospect writers and he does an admirable job of covering the prospect pipelines for every NHL team.
As a Lightning fan, my eyebrows rose when I saw the Lightning in the 20th spot. The Bolts have both Brayden Point and Mikhail Sergachev, two high-end under-23 NHL players. They also have a deep pipeline with Cal Foote, Boris Katchouk, Taylor Raddysh, Alex Volkov, and Mathieu Joseph leading the way.
Mustering up all the self-awareness I could, I resisted the urge to yell at Pronman on Twitter and instead tried to think of how I would go about creating my own list. I don’t have the depth of knowledge of these players or scouting in general that Pronman does so sitting down and creating a ranking using only what I know about teams’ prospects was not an option.
Instead, I sought to come up with a structured way to assess the values of the players in each team’s pipeline and then use that value to rank them from top to bottom. Fortunately, we as hockey fans have some tools available to us to do just this.
WAR as a Common Currency
Corsica launched a prospect and draft tool earlier this summer that includes, among lots of other useful things, a projected wins above replacement (WAR) measure for almost every skater playing junior hockey in a known league. That’s handy for our purposes here because we can use those projections along with NHL WAR to create a new under-23 ranking.
The goal here isn’t to make a better list than Pronmans but rather to make a different list using a different approach and see how it compares to his. As a scout, he relies on his own observations and the stats he feels are important to create a mental model. He then used that model to rank the teams.
In this exercise, we’re going to be explicit in our methodology and use a single measure of value (WAR) as a common currency that we can use for the ranking. WAR is ideal for this purpose because it takes lots of information and weighs it quantitatively to arrive at one number. In Pronman’s approach, he takes lots of information and weighs it qualitatively to arrive at one number.
The WAR calculations for NHL players and prospects are different. For NHL players, Corsica’s WAR is focused on measuring a player’s impact on both shot quantity and quality. It has offensive and defensive components. It also accounts for contextual factors like quality of teammates, quality of competition, zone starts, etc.
For prospects, the WAR calculation works differently. It uses production stats like scoring combined with other information such as the player’s league, their physical traits, their age, and other inputs to project how much WAR that player will generate in 82 NHL games. This is a challenging thing to model as the data outside the NHL is limited and lots of things can impact a prospect’s development. Because of this, the model’s estimates are conservative.
Analysis
To conduct our analysis, we’re going to rely on some of the information that Pronman gave us in his article. He included ten players for each team to create his ranking. We’re going to trust his eye to say that those are the ten best prospects in each team’s pipeline and use those same ten players to create our rankings.
This presents a challenge in that Corsica does not provide projected WAR for goalie prospects. In Pronman’s rankings, eight teams have a goalie included in their ten prospects. To account for this, I’ve given each goalie an estimated projection based on where they were ordered in the the ten prospects. For example, if the goalie for a given team was their fourth best prospect, I assigned them the average projected WAR among all of the fourth-ranked prospects in Pronman’s list.
That approach probably undersells the goalies a bit. They are the most valuable position. But we also know that projecting goalies is notoriously difficult so perhaps a conservative approach is justified. In any case, this is the approach used here and we can now move on to seeing some results.
To start, we can pull the total NHL WAR for each of the under-23 players. We can then pull the projected WAR for the prospects. Because both of those numbers are in the same currency, we can simply add them together and come up with a ranking. And when we do, it looks like this.
This is interesting and we already see some notable divergences. The Lightning have moved from 20th to 9th. Buffalo has gone from 2nd to 23rd. Nashville has gone from 26th to 6th. But looking a little closer, this chart isn’t really getting at what we were trying to measure.
The way we’ve approached the question heavily weighs NHL players over prospects. This is for two reasons. First, the WAR estimates for prospects are conservative so even the best prospects aren’t being given anywhere near the same weight as the best NHL players. And second, while both of our measures are WAR calculations, adding them together probably isn’t a good way to do this.
For NHL players, we have all of their career WAR. That could include multiple seasons. For prospects, the best they could have included here is an 82-game projection that is already conservative. So to get closer to what we’re trying to measure, we need to account for these issues.
One way to do this would be to use WAR per 82 games for NHLers the same as we do for the prospects. But that creates its own problems. Players who have played well in a small sample will look similar to players who played well in a large sample. By moving purely to rate stats, we lose some information that is valuable in this type of analysis.
Let’s go back to Pronman’s article to see if we can find any guidance on how he designed his mental model.
My same preferences and ranking philosophies exist from previous pieces but, with the addition of proven NHL players, extra value is given to players who have established themselves in the league and especially if they have proven themselves to be good players.
This bit is helpful. Pronman says that he gives extra value to NHL players who have established themselves in the league. That supports the idea that we don’t want to move purely to rate stats. We want players who have been in the NHL for multiple seasons to keep that value. It also signals how we might take the next step.
Instead of working with the raw WAR calculations, we can scale them according to Pronman’s description. While he doesn’t give us numeric values, a fair reading of that paragraph suggests that giving NHL players 60% of the weight in this analysis and prospects 40% of the weight would be appropriate.
Doing this will decrease the importance of the NHL players and increase the importance of the prospects relative to the first chart. But we still maintain NHL performance as being more important and align with Pronman’s description of his approach.
Below is the chart scaled to the 60/40 weighting. Note that because of the scaling, we are no longer looking at wins above replacement. We’re looking at a new scaled stat based on wins above replacement. The numbers on the y-axis serve only to provide a value that we can use to rank the teams and have no meaning outside of this specific exercise.
Using this new scaled stat, we have similar differences in ranking to the first version but this one makes more sense in the breakdown of values and is true to the approach in the original article.
With these ranks, we can look specifically at how the resulting rankings from our approach and Pronman’s approach differed.
Some teams like Edmonton, Toronto, and Winnipeg stayed largely the same while Minnesota, Nashville, Arizona, and Buffalo had big changes.
The gaps here are interesting and we can address some of them specifically. Nashville is almost entirely due to Juuse Saros. The young goaltender has been excellent in Nashville and goalies are the most important players on the ice. A WAR model is always going to assign lots of value to that position and that creates the gap here.
Buffalo is probably the most interesting case. This is where a specific scout’s preferences become important. Pronman is high on Jack Eichel and Casey Mittelstadt. And he has good reason to be. They both have exceptional talent. But thus far, the production hasn’t matched that talent. Eichel hasn’t reached his potential in Buffalo yet and Mittelstadt’s numbers in the NCAA last year were good but not great. Pronman knows that and is betting that production to match the talent will come. The WAR models are less confident.
Any situation where we have an expert’s opinion and a purely quantitative analysis that differ significantly begs for combining those two approaches to arrive at a final output. Ideally, we would have a structured way to weigh them based on the effectiveness of each’s predictions historically. But given that we don’t have that information, we can use a brute force method and take an average of the rankings.
Below is a heatmap where the combined ranking is an average of Pronman’s rankings and the scaled WAR rankings. The teams are sorted from first place to last place by the combined rank and the colors help show the difference according to each method. Blue is a high ranking and orange a low one. Look at Buffalo and Nashville as examples.
In this final version, we now have a ranking that blends information from an informed scout (Pronman) and statistical models (WAR). Tampa was 20th in Pronman’s rankings and 9th in the WAR rankings. They now sit 13th. Buffalo lands in 12th after being at the top of Pronman’s list and in the bottom third in WAR.
Conclusion
Taking information from multiple sources both qualitative and quantitative can lead to better outcomes than either individually. Too often in sports analysis, we see scouting pitted against statistical modeling as if the two are somehow incompatible. Smart organizations recognize that the two are complimentary and use both when evaluating players.
Scouts have experience and can identify things that models miss. Especially in situations with limited data availability. Models can process far more information and make better decisions about how to weight that information than scouts. Figuring out a method that utilizes the strengths and mitigates the weaknesses of each approach is the best path to arriving at quality outcomes.
This article is a simplistic version of how that can be accomplished. A far more rigorous approach would be needed to understand the appropriate way to blend the information from the scouts with the outputs of the model. Getting better at melding quantitative and qualitative assessments into holistic player evaluations in a structured way is one area where hockey analytics can continue to grow.
Extra Stuff:
This is the chart for U23 NHL players alone according to WAR.
This is the chart for U23 prospects alone according to WAR.
