PRO Referee assessment system
PRO Referee Manager Sandra Serafini – Part 2
To be an effective soccer player, especially at the professional level, requires a thorough understanding of the sport: reading the flow of the game, knowing what to look for, knowing where to position yourself, and having practiced enough repetitions to be able to execute properly in any situation.
All these things are also true for the people on the third team on the pitch – the officials.
Recognizing this, the Professional Referee Organization (PRO) has developed a computer-based assessment system with the intent of developing officials’ skills to the same high degree as the players. This state-of-the-art system combines a cloud-based video database from TeamXStream with PRO’s assignment and evaluation platform developed by Edge10.
PRO’s Sandra Serafini has talked with 10Soccer previously about her role at PRO, and the growing opportunities for women as professional soccer referees. This time, Sandra was kind enough to give me a tour of PRO’s very impressive referee assessment system.
The combined system incorporates contractual data, sports science, and video-based performance evaluation to raise the level of professional soccer officiating in the U.S.
PRO’s foray into computer-based referee management started with the not-so-simple task of assigning referees to NWLS, USL, NASL, and MLS games. Just the number of games and referees alone make that a daunting task. And it’s a lot more complex than that.
“We’ve got the collective bargaining agreement [CBA] which has a lot of rules in it,” Sandra explained to me, “and we also wanted to coordinate our assignments with our sport science data to make sure we weren’t giving people too much – taking their travel, game load, recovery, and training into account.”
The CBA rules (for example, a referee cannot officiate the same team within nine days) and sports science data were programmed into the Edge10 system, allowing PRO to take all the contractual and fitness parameters into account when making referee assignments. The Edge10 system provides a visual dashboard showing which officials meet the required criteria and can be assigned to a game.
But PRO took it a step further. “We felt we should probably also do something with the evaluation system, “ Serafini told me. “We essentially needed a way to be able to share video with the officials and have that help drive our education.”
PRO worked with TeamXStream, who has an established video database platform that has been used by teams, coaches, and players in many sports.
A soccer team might use the TeamXStream system to track things like passes in the attacking third or defensive shape or connected passes. PRO took the same system and configured it for evaluating fouls, misconducts, offside, and penalty kick calls made by officials.
The results have been fantastic. PRO used the system for NWSL games for about a year. It was working so well they pulled in some NASL and USL games last summer. This season they added MLS games.
The details of the system are amazing. Video from each game is reviewed by the PRO assessor. Individual video clips are made of any key match incidents. Key match incidents include things like penalty kick decisions, decisions to send players off, and offside decisions that lead to obvious goal scoring opportunities or goals.
The assessor then uses the Edge10 system to evaluate each official. There are specific criteria for the referee, assistant referee, and fourth official that are evaluated on match difficulty and the official’s performance. The assessors rate each of those criteria along with providing an explanation of the rating.
Whenever a rating is impacted by a key match incident, the assessor puts the URL of the incident’s video clip into the Edge 10 assessment form. When the official gets their assessment back they can read the ratings, and they have immediate access to the pertinent video clips.
The system has been received very positively by PRO staff, including Peter Walton (PRO General Manager) and Greg Barkey (Manager of Assistant Referees), as well as the officials.
“What the officials like about it is the evaluations now are much more analytical,” Sandra told me. “The criteria are all laid out and customized for each officiating position. It’s the evaluator’s job to go in an analyze the game and address each of those criteria, taking into account match difficulty and performance.”
The assessment criteria combined with the video allows for very specific performance reviews. “We’re trying to make it more objective,” Serafini said. “Here are the facts. Here’s the timestamp. Here’s what occurred.”
Sandra gave an example of how this specificity helps. “I did a performance analysis of one official looking at misconduct; looking at yellow cards and red cards and accuracy. When we broke it down we found the errors were happening with contact above the shoulders. It turned out he was stopping short by about four or five yards and narrowing his angle of view when the contact occurred. So what seemed to be a misconduct recognition issue turned out to be a positioning issue. He just needed to make an adjustment in where he chose his position in a given moment of time.”
And Sandra pointed out, “We also let them know what they’re doing right. We say ‘Here’s what happened that was good. You made these considerations and came to the correct conclusion. That’s great. Keep doing that.’ We want the officials to be conscious of the things they’re doing right, and why they’re doing them right, so they can keep doing them and we turn those into habits.”
Towards that end, the assessment and video systems are being incorporated into PRO training for officials. PRO is tailoring their education, field drills, and exercises based on what they’re learning from the system.
As American soccer has grown, the stakes for the players, coaches, and clubs have risen dramatically. This puts even more emphasis on having the game officiated correctly. PRO is using state-of-the-art technology, plus lots of old fashioned hard work, to train, assign, and evaluate their officials in order to maximize their success.
Our thanks again to Sandra Serafini for taking time out of her busy schedule to talk with us.