REPORT 16: BIG NEWS ABOUT THE NCAA TOURNAMENT AT LARGE SELECTION CRITERIA

This past week, the NCAA finally published its Report of the NCAA Division I Women's Soccer Committee January 30-31, 2023, Meeting.  (It still has not published the report on its August 2023 meeting.)  The Report has two big news items:

1.  The Committee voted to recommend using the KPI (a rating/ranking system) as another consideration tool for ranking and selection for the NCAA Tournament.

2.  The Committee recommended expanding the NCAA Tournament seeding from 32 teams to 48 teams.

This past week, the NCAA also published its NCAA Tournament Pre-Championship Manual.  The Manual states that the Committee will seed 32 teams (as it first did in 2022), so the recommendation to expand the seeding to 48 teams was not approved.  The Manual, however, in its description of the criteria the Committee will use for at large selections, includes a new criterion: "Results of the Kevin Pauga Index (KPI)."  The Manual identifies this as an additional "Secondary" criterion, but my statistical analyses of past Committee decisions indicate that the Committee considers all of the criteria and I have seen no indication that it considers the Secondary criteria any less important than the Primary criteria (except for the Secondary criterion of results over a team's last eight games, which appears unimportant to the Committee).

Thus the Committee did get approval to use the KPI rating and ranking system in addition to its use of the NCAA's RPI.  On paper, this is a big change.  In real life, it remains to be seen how much difference it will make, if any, in the Committee's decision making.

You can find the current KPI Rankings here.  At the top of the page, you may need to fill in the query boxes with KPI Rankings, NCAA D-I, Women's Soccer, 2023.

I have not yet had time to do a full analysis of how well the KPI system does at rating DI women's soccer teams.  As I have written about extensively, the current NCAA RPI does not properly rate teams from conferences in relation to teams from other conferences and likewise does not properly rate teams from geographic regions in relation to teams from other geographic regions.  Rather, the current NCAA RPI, on average, discriminates against stronger conferences and regions and in favor of weaker conferences and regions.  On the other hand, both my Balanced RPI and the Kenneth Massey ratings do not have these problems: On average, teams grouped by conference all perform pretty much as their ratings under those systems say they should and the same is true of teams grouped by geographic region.

Knowing that the current NCAA RPI has the fairness problem I just described and my Balanced RPI and the Massey ratings do not, I have done a quick test of the KPI system in relation to the current NCAA RPI, my Balanced RPI, and the Massey ratings, to see where the KPI system is likely to fall in relation to the fairness problems.  The test is based on the systems' ranks of teams for games played through October 10.

First, as a piece of general information, the average difference between the current NCAA RPI rankings and the KPI rankings is 11.3 positions.  This compares to a difference between the current NCAA RPI and the Balanced RPI rankings of 17.4 positions and of 26.2 positions for Massey.

Second, below I am providing three tables.

1.  The first table, for individual teams, shows how the four systems rank each team.

2.  The second table looks at the conferences, showing the four systems' average ranks for the conferences.  In this table, in the Conferences column, I have red highlighted conferences where it appears to me that the current NCAA RPI and the KPI overrate the conferences and yellow highlighted conferences where it appears they underrate the conferences.  I have assigned the highlighting based on a comparison to how my Balanced RPI and Massey rate the conferences, knowing that on average my Balanced RPI and Massey rate conferences properly in relation to each other.  As you will see from a review of this table, it appears that the KPI has a problem similar to that of the current NCAA RPI (or possibly an even bigger problem), in terms of properly rating teams from conferences in relation to teams from other conferences.

3.  The third table looks at geographic regions, with the same format as the table for conferences. Here too, it appears that the KPI has a problem similar to that of the current NCAA RPI, in terms of properly rating teams from geographic regions in relation to teams from other geographic regions.

My above observations about how the KPI appears to perform in relation to conferences and regions are based only on compared rankings for the current season based on games played through October 10.  The KPI has ratings and ranks available for seasons since 2017, so it is possible that after the 2023 season is over, I will be able to do a detailed analysis of how the KPI performs as compared to the current NCAA RPI, my Balanced RPI, and Massey.  In the meantime, I consider my above observations as only preliminary suspicions about the KPI.