How accurate is the Elo system and its predictions

Many sites offer predictions for football matches, very few of them actually review and publish analyses of their predictions. Having over 260000 football matches in the database, I wanted to see how my system stands the reality check. Does a team really win in 70% of the cases (draw counting as half-win half-loss) if my system estimates the probability at 70%? Let's find out.
I went through all the matches in my database with the exception of 2nd leg matches. The reason for this is that teams do play for an overall victory depending on the first leg and might afford to lose the second leg. Then I grouped the matches that were left by percentile (probability for the home team to win the game). Then I calculated the average outcome of the matches in each percentile. The result is shown in the following graph:

100%80%60%40%20%0%20%40%60%80%100%Calculated probabilityAverage result

The ideal curve is shown in grey, in that case all the probabilities would be realistic and we would have a perfect prediction model. The curve for the Elo system is deviating from the ideal, even beyond statistical effects. It appears that especially in games with an 80-20 favourite the odds are a bit too one-sided, the actual results from these games are closer to 75-25. Games between teams of similar strength where the home team is the slight favourite seem to be well estimated though.

Created: 23 Mar 2013 - Modified: 14 Jul 2012