Comparing the NCAA's 2015 ARPI and the 5 Iteration ARPI, 2009 BPs to the Committee's Actual Decisions

In the preceding post, I provided information about changes to the RPI that I've recommended to the Women's Soccer Committee.  The change is from the NCAA's current 2015 ARPI to what I call the 5 Iteration ARPI with the 2009 Bonus and Penalty regime.  As I discussed, the 5 Iteration ARPI, 2009 BPs system performs far better than the 2015 ARPI.

To add more information, I decided to compare the Committee's decisions on at large selections and seeds over the last 10 years to what the rankings would have been over those 10 years for the 2015 ARPI and for the 5 Iteration ARPI, 2009 BPs systems.  The purpose of this is to see which system's ratings match better with the Committee's actual decisions.

To do the comparison, I determined, for each rating system, the average rank of the teams the Committee gave at large selections, the average rank of the teams in the Top 60 (using the 2015 ARPI's ratings to determine the Top 60) to which the Committee denied at large selections, the average ranks of the teams to which the Committee gave #1, #2, #3, and #4 seeds respectively, and the average rank of the 16 teams to which the Committee seeds as a group.  The following table shows the results of the comparison:

Starting with the at large selections, the 5 Iteration ARPI, 2009 BPs ranked the teams to which the Committee gave selections 1.46 positions better than the 2015 ARPI.  What this means is that the Committee's at large selections matched better with the 5 Iteration version than with the NCAA's current version.

Moving on to the teams to which the Committee denied at large selections, the 5 Iteration version ranked those teams 3.73 positions more poorly than the 2015 ARPI.  Again, this means the Committee's decisions -- this time at large rejections -- matched better with the 5 Iteration version than with the NCAA's current version.

For seeds, for the #1 and #4 seeds, the Committee's decisions matched slightly better with the 5 Iteration version than with the NCAA's current version.  For the #2 and #3 seeds, on the other hand, the Committee's decisions matched slightly  better with the NCAA's current version.  When looking at seeds as a whole group, the Committee's decisions match very slightly better with the 5 Iteration version.

Looking particularly at the Committee's at large decisions, and assuming that those decisions in most cases were the right ones, the above numbers show that the 5 Iteration version's rankings come closer to being the right rankings than the current NCAA's version's rankings.  They certainly come closer to the decisions the Committee believes are the right decisions.