2025 ARTICLE 6: THE KP INDEX (KPI): THE COMMITTEE'S MISTAKE AND THE BIG 10'S OWN GOAL

 From Report of the Women's Soccer Committee December 9, 2024 and January 29, 2025 Meetings:

"KPI, and other ranking systems.  The Committee felt the KPI should be used more in selections as a valued tool in the process.  The Committee reviewed the Massey Ratings and decided not to request it as an additional criterion at this point."

Thus, as it has done for the past two years, the Committee intends to supplement the NCAA RPI with the KPI as a system for ranking teams.  It intends to use the KPI more, however, than it has in the past.  This is a change from a year earlier, when the Committee reported that it had found the KPI not useful and proposed use of Massey beginning in 2026.

The Committee for a while has pushed for use of more than just the NCAA RPI for team ratings and rankings.  The Committee two years ago finally received approval to use the KPI.  After it received that approval, I asked a Committee member where the decision to use the KPI, as the particular system approved for use, came from.  The member advised me it did not come from the Committee, so far as he recalled.  My assumption, therefore, is it came from the NCAA staff.

What Is the KPI? 

The KPI is a product of Kevin Pauga, the Associate Athletic Director for Strategic Initiatives and Conference Planning at Michigan State.  It appears the KPI is something he produces outside his formal work for Michigan State.  Pauga also is regarded as a scheduling expert, employed by the Big 10 and other conferences to produce conference schedules for their teams.  His scheduling system considers many parameters.  I do not know if it considers the relationship between teams' schedules and their NCAA RPI rankings.

According to a New York Times article dated March 26, 2015, Kevin Pauga started using the KPI for college basketball ratings and rankings in 2013.  According to the article, a KPI rating is a

"number that, essentially, correlates to how valuable a team’s wins are versus how damaging its losses are. The ratings run from negative-one (bad) to plus-one (good), and combine variables that might change over the course of a season. His goal is to quantify so-called good wins and bad losses, and to assess how teams compare with one another."

And, according to an NCAA release dated March 5, 2025, for NCAA men's basketball:

"The Kevin Pauga Index metric ranks team resumes by assigning a value to each game played. The best win possible is worth about +1.0, the worst loss about -1.0, and a virtual tie at 0.0.  Adjustments are made to each game's value based on location of the game, opponent quality and percentage of total points scored. Game values are added together and divided by games played to determine a team's KPI ranking."

Beyond these descriptions, it appears the KPI is proprietary, so I do not know exactly what its formulas are.

Is the KPI a Good Rating System for Division I Women's Soccer?

I don't know how well the KPI functions as a rating system for other NCAA sports.  For Division I women's soccer, however, it is a poor rating system.  It has the same defects as the NCAA RPI.

In 2025 Article 2, I explained my system for grading Division I women's soccer rating systems and did a detailed review of how the NCAA RPI performs as a rating system.  As I showed:

"Based on the ability of schedulers to "trick" the NCAA RPI and on its conference- and region-based discrimination as compared to what the Balanced RPI shows is achievable, the NCAA RPI continues to get a failing grade as a rating system for Division I women's soccer."

The following review of the KPI is similar to the review of the NCAA RPI in 2025 Article 2.  I will not go through the same detailed explanation of the grading system here as I gave in Article 2, so you might want to review Article 2 before proceeding further.

Ability of the System to Rate Teams from a Conference Fairly in Relation to Teams from Other Conferences

This table has the conferences arranged in order from those with the best KPI average rating from 2017 through 2024 at the top and those with the poorest at the bottom.  (KPI ratings are available only for years since 2017.)

In the table:

The Conference NonConference Actual Winning Percentage column shows the conference's actual winning percentage against non-conference opponents.  In calculating Winning Percentage, I use the NCAA RPI Winning Percentage formula in effect as of 2024.

The Conference NonConference Likelihood Winning Percentage column shows the conference's expected winning percentage against non-conference opponents, based on the differences between opponents' KPI ratings as adjusted for home field advantage.  The expected winning percentage for each game is determined using a Result Probability Table for the KPI.  The table comes from an analysis of the location-adjusted rating differences and the results of all games played since 2017.  This method for determining expected winning percentages is highly precise when applied to large numbers of games as shown by the following table for the KPI:

 

In this table, the Total Win, Tie, and Loss Likelihood columns show the expected wins, losses, and ties by the higher rated team, after adjustment for home field advantage, for all games since 2017.  The columns show these in absolute numbers and as a percentage of all games played.  The Total Actual Wins, Ties, and Losses columns show similar numbers, but based on the actual results.  As you can see, out of the more than 22,000 games played, the difference between the expected wins and actual wins is 11 games, between the expected ties and actual ties is  6 games, and between the expected losses and actual losses is 4 games.  In other words, as I stated, the Result Probability Table method is a highly precise way of determining expected winning percentages for the KPI.

Returning to the larger table above, the Conference NonConference Actual Less Likely Winning Percentage column shows the difference between the conference teams' actual winning percentages in non-conference games and their expected winning percentages based on their games' location-adjusted KPI rating differences.  A positive difference means the teams' actual winning percentages are better than expected based on the KPI and a negative difference means the teams' actual winning percentages are poorer. 

This chart is based on the conferences table.  In the chart, conferences are arranged in order of those with the highest average KPI ratings on the left to those with the poorest ratings on the right.  The vertical axis is for the difference between a conference's actual and its KPI-expected winning percentage.  As you can see from the chart, stronger conferences (on the left) tend to perform better than their KPI ratings say they should and weaker conferences (on the right) tend to perform more poorly.  In other words, the KPI underrates teams from stronger conferences and overrates teams from weaker conferences.  This is the same pattern the NCAA RPI has.  The downward sloping straight line is a trend line showing the pattern of the data; and the formula on the chart is a formula whose use can tell what the data indicate the actual v expected difference should be for any particular conference average KPI rating.

This table draws from the data underlying the chart and from the chart itself, as well as from similar data and charts for the NCAA RPI and the Balanced RPI.  It thus compares the KPI to the NCAA RPI and also includes the Balanced RPI (which is similar to what the Massey rating system would show), to show what a good rating system can do.

In the first two color coded columns, the "Spread" column shows the performance percentage difference between the conference that most outperforms what the KPI says its performance should be and the conference that most underperforms.  The "Under and Over" column shows the total amounts by which all conferences either outperform or underperform.  Both of these columns are measures of rating system fairness in relation to teams from the different conferences.  As you can see, the KPI and the NCAA RPI both do a poor job, as compared tto what a good rating system can do.

The color coded column on the right comes from the chart and shows the rating system pattern in relation to conference strength, using the chart's trend line formula.  It shows the extent of discrimination in relation to conference strength.  As the column shows, the KPI and NCAA RPI have similar discrimination based on conference strength, whereas a good system as exemplified by the Balanced RPI has virtually no discrimination.

Ability of the System to Rate Teams from a Geographic Region Fairly in Relation to Teams from Other Geographic Regions

The following table and chart are similar to those above for conferences, but are for geographic regions:

The table and chart for geographic regions show an overall pattern similar to that for conferences.  The data point pattern, however, is up and down enough that I do not have a high level of confidence in the relationship the chart shows between regions' average ratings and their actual performance as compared to their KPI-expected performance.  This is indicated by the R squared value I have included in the chart below the trend line formula.  The R squared value is a measure of how well the data fit the trend line, with an R squared value of 1 being a perfect fit and of 0 being no fit.  The R squared value on the chart indicates a mediocre fit.

Here is another table-chart set.  They is based on the relationship between the proportion of games in each region that are ties, as a measure of the amount of parity within the region.

This chart shows a similar patter in relation to region average KPI rating, but with a significantly better R squared value.

Altogether, the tables and charts for regions result in the following comparison table:

As the first two highlighted columns show, the KPI and the NCAA RPI have relatively similar levels of discrimination among conferences, in terms of general fairness.  And as the Balanced RPI row shows, this could avoided by using a better rating system.  And, as the two color coded columns to the right show, to the extent that the KPI discriminates in relation to region parity or region strength, its discimination is similar to that of the NCAA RPI.  Again, the Balanced RPI row shows this could be avoided by using a better rating system.

Ability of the System to Produce Ratings That Will Match Overall Game Results

This table simply shows the extent to which the higher rated team, after adjustment for home field advantage, wins, ties, and loses for the particular rating systems.  Thus it is a gross measure of how well game results correlate with the systems' ratings.  Since the NCAA RPI and Balanced RPI numbers are based on the No Overtime rule having been in effect for all seasons, whereas the KPI numbers for 2017 through 2021 use overtime games' results, the best comparison is in the highlighted column on the right, which disregards tie games.  As a point of reference, a difference of 0.1% represents a difference of roughly 3 games per year.  As you can see, the KPI gets the correct result about 15 times fewer per year (0.5%) than the NCAA RPI and about 39 times fewer (1.3%) than the Balanced RPI.

This is similar to the previous table but limited to games involving at least one team in the system's Top 60.  Again, the best comparison is from the color coded column on the right.  Here, a difference of 0.1% represents one game per year.  Again, the KPI performs more poorly than the NCAA RPI and than the Balanced RPI.

Conclusion About the KPI as a Rating System

As the above information shows, the KPI has the same problems as the NCAA RPI, with higher levels of discrimination among conferences and geographic regions than can be achieved by a good rating system.

So why would the NCAA staff agree to use of the KPI, if it is the NCAA staff that selected the KPI as an additional system as I suspect?  And why would the Women's Soccer Committee double down on the KPI as it intends to do this coming year?  The answer is simple: Because the KPI has the same problems as the NCAA RPI.  It will not rock the NCAA RPI boat, but rather will make the NCAA RPI look legitimate.  Whereas if the Committee were to use Massey, it would have significant differences from the NCAA RPI and thus would expose the NCAA RPI as a poor rating system.  Indeed, I suspect that the staff and Committee looked at Massey ratings as compared to the NCAA RPI, saw that there were significant differences, and did not want to have to deal with the fallout those differences exposed.  I do not know this, I could be wrong, but it is the best explanation I can come up with for the Committee's reversal of its position on the KPI and Massey over the last year.

The Big 10's Own Goal

Finally, why do I say the Big 10, in all of this, has committed an "own goal" blunder?  Remember, Ken Pauga is an Associate Athletic Director at the Big 10's Michigan State.  If you look at the first table above, you will see that the Big 10 is the second most discriminated against conference by the KPI, actually winning 4.1% more games per year against non-conference opponents than the KPI ratings say they should win.

The following table breaks the numbers down by Big 10 team, based on the teams in the conference before absorption of the Pac 12 teams in 2024:

As you can see from the column on the right, the KPI discriminates against 11 teams (including Pauga's Michigan State) and in favor of 3.  If I were to add the four teams absorbed from the Pac 12 in 2024, there would be 3 more discriminated against and 1 more in favor of.

Thus so far as Division I women's soccer is concerned, the Women's Soccer Committee now will be using, apparently with added emphasis, a rating system that comes from a Big 10 school administrator yet discrimates against the Big 10 as a whole and even against his own school.  When it comes to the "last in" and "first out" for NCAA Tournament at large selections, that looks like an "own goal" to me.