Tully and Fabius-Pompey played a back-and-forth boys soccer game Oct. 30. It was for the Section III Class C title. The teams were closely matched, the score tight, the energy high. Crowds were loud and trying to one up each other.
It was hard to tell the two teams were ranked in the middle of the pack — a No. 7 and No. 8 seed — not at the top.
Tully and Fabius-Pompey took unlikely routes to the finals. No. 7 Tully topped No. 10 Syracuse Academy of Science for the third time this year in the first round, then went on the road to beat No. 2 Sauquoit Valley. Tully’s Black Knights eliminated No. 3 Westmoreland/Oriskany in the semifinals.
Tully was also a top 10 team in the state’s Class C poll at the time of the seeding meeting, putting the Black Knights as the No. 7 seed in sectionals with a 13-2-1 record and the No. 8 spot in the state.
That a No. 7 seed and a No. 8 seed met in a sectional final, rather than higher seeds, is thanks to Section III’s newest seeding model –– the Bradley-Terry Model. But it was a statistical outlier.
The Bradley-Terry Model, known to some as the BTM, was first studied by German mathematician Ernst Zermelo in the 1920s. It was named for American mathematicians Ralph A. Bradley and Milton E. Terry, who presented the model in 1952.
According to a lecture at Stanford University by Professor Zhou Fan, “the Bradley-Terry model assigns scores to a fixed set of items based on pairwise comparisons of these items.” The model is typically run by computer programs such as R and C#.
HOW IT STARTED
Connor Meissner, then a master’s student at Syracuse University, was conducting research last summer for a paper about the Bradley-Terry Model. Section III was looking for a new way to seed teams and happened to ask Syracuse representatives if they had any ideas. Meissner tagged in and had all the knowledge needed to sell them on the model..
“I had happened to be working on a paper that will be published soon involving this ranking system and empirically showing how it was more accurate that an alternative in a New England league,” Meissner said. “It was a perfect coincidence that I was working on that paper, as they were looking for an objective way to rank teams that was a little more advanced than wins and losses.”
Todd Mulvaney took over as Section III’s executive director in August and heard the section had been looking for a new, consistent and objective seeding model. When he heard about Meissner’s paper, Mulvaney knew he needed to hear him out.
“The first thing was the section has been looking at coming up with a consistent seeding system for quite some time,” Mulvaney said. “Last year, when the state rolled out the six classifications, it seemed like the timing was perfect. Not only was the timing perfect, but there was a gentleman who presented to our athletic council, which is our governing body, last spring the BTM model.”
Section leaders agreed to a one-year pilot.
HOW IT WAS APPROVED
Bill Donald, Tully’s boys soccer coach and the Section III Boys Soccer chair, didn’t know how the new system would work or what it would look for, but got to vote for it as the Section III Boys Soccer Committee chair.
“They gave all the committees opportunities to vote on it,” Donald said. “It’s a trial to see what’s different. We all had a vote and I voted for it.”
Mulvaney, Associate Executive Director Jason Czarny and the rest of the sport committees approved the Bradley-Terry Model, so it was rolled out for the 2023-24 school year. Czarny committed himself to understanding the model so he could explain it. It takes the human element almost entirely out of seeding.
“We were really looking for something that would be more objective than subjective,” Czarny said. “What we had was, after seasons would play out, people would apply their own versions of their own eye tests to some of the teams as you look at records. You could say these are the strongest teams based off the schedules they played.”
“One of the best models we had was the boys lacrosse model,” Czarny added. “It started to take out some of the subjectivity and be more objective. The BTM came to us as people started to discuss it and find what options were out there. It was one of the first ones that came across meeting what we were looking for.”
Mulvaney and Czarny said Section III is monitoring the model and making sense of it using an ad hoc committee and the Athletic Council.
HOW IT’S GOING
The Tully/Fabius-Pompey game was the only one in Section III that featured two teams seeded fifth or lower. Beaver River’s girls soccer team reached the Class C final as the No. 6 seed.
Across Section III’s soccer and field hockey sectionals, the number of upsets –– lower seeds beating higher seeds –– dropped from a three-season average of 26.9% to 19.6% in 2023, Meissner’s data shows. The average seed of the semifinalists was 2.88 this season as opposed to 3.5 in the previous three years, the average seed of finalists dropped from 3.18 to 2.42 and the average seed of champions fell from 2.52 to 1.92.
“I think we’re extremely pleased with it and I think a lot of athletic directors are hearing the same things from their coaches,” Czarny said. “There was a little hesitancy before we rolled out into sectional play. As things have gone along and brackets have played out, it’s really put a lot of ones and twos in the finals and threes and fours in the semis.”
Jamesville-Dewitt’s boys soccer team shows the model’s effectiveness. The Red Rams went 5-10-1 during the regular season, but ended up with the No. 4 seed in the Class A tournament. They rewarded the computers by beating No. 5 Carthage 3-0 before losing to No. 1 Westhill, the eventual champions, in penalty kicks.
But how?
According to an article on collegehockeynews.com by developer John Whelan, the Bradley-Terry Model is simply broken down to winning ratio times strength of schedule. A winning ratio is wins divided by losses, rather than the winning percentage, wins divided by games played. The strength of schedule, the weighted average of the opponents’ ratings, is what requires the computer.
“That initial value is essentially a placeholder,” said Meissner, who works with High School Sports Stats on the coding of the model in its system. “Within the R code itself they have their own kind of mechanics for that initial value. At the end of the day, no matter what initial values you put, the recursive method will always eventually land on the same ratings as the final output.”
Section III is only seeding based on in-section games because the data of every team in the state would be needed to use all games on the schedule. The model can’t account for partial data; it’s every team in the state or every team in the section.
THE J-D 'OUTLIER'
J-D found the one loophole the system. The team had a lot of “quality losses,” beat a couple good teams and did what it needed to against weaker competition. That kept the Red Rams’ winning ratio high enough to let the strength of schedule work its magic.
J-D’s and Tully’s seeds seemed to come out of nowhere. Coaches, players and spectators usually know where the team stands at all times. In the past, High School Sports Stats kept up-to-the-minute rankings in each class. But the new model not completely and accurately implemented for most of the fall, so coaches and players didn’t have that information, aside from one release Oct. 1, two weeks before seeding day.
“Our Athletic Council, with a recommendation from our ad hoc committee, decided to post that just because people were still wrapping their heads around what the BTM was,” Mulvaney said. “They wanted to have an idea of where they stood heading into the seeding meeting. We thought to help ease their anxieties with it that we’d post it Oct. 1 before the seeding meeting.”
Meissner said all the bugs in High School Sports Stats’ system have been resolved and the model can now be run instantaneously. It can update the website rankings within 3 minutes of entering the final score.
KEEPING UP WITH THE MODEL
Despite that, Section III is still unsure how often it will release the updated rankings. The model needs a large sample size, so much data before the midpoint of the season would be misleading. Mulvaney and Czarny are talking about a more frequent release.
“We’re looking at all those things,” Czarny said. “How often do we want to post? Do we want to increase the amount of times we post throughout the regular season? It’s our pilot year, so we’re taking it all in stride and having these thoughtful conversations.”
Cortland baseball coach Ben Albright initially had concerns about the Bradley-Terry Model, but was soothed by seeing how things worked in the fall. One of his primary concerns as a coach remains the lack of awareness of a team’s current standing.
“Baseball is a little bit of a unique sport where whoever is on the mound pitching for you can be a difference maker,” Albright said. “If you can line your pitching up going into sectionals, that bodes well for how you attack each game that you may have. I know a lot of coaches will look ahead and say, ‘We can play these three teams based on where we’re at right now.’ Then you start to create your plan looking forward.”
“Now, you can have a great record but because you’ve maybe played weaker opponents according to the Bradley-Terry Model, then you have a lower seed and you won’t play those teams you anticipate,” Albright added. “It kind of takes that out, where before you could figure out the points.”
THE SCHEDULING DILEMMA
The Bradley-Terry Model has shown the value of playing quality, in-section non-league opponents. Czarny said that will be one of the things stressed to athletic directors when they put together next year’s schedules.
“What it taught us going forward is that if there are gaps in scheduling across Section III, it can expose a lack of games for comparison points,” Czarny said. “We’re learning from that and we’re going to be advising all of our leagues as they go to schedule later in the year to consider that fact –– getting more games across the wider scope of leagues.”
That would create a dilemma for Donald and his Tully team, as they got to play only three non-league games this season. Donald elected to play Westhill, the Class A champion, Cazenovia, a traditionally strong Class B team, and Fabius-Pompey, Tully’s biggest rival, a second time. All three non-league games were against fellow OHSL members.
Now that he knows what the model accounts for, Donald has a choice –– take a risk and play a team from a different league that could have a down year or continue playing strong, known commodities.
“Now the ADs know, they can’t schedule all against themselves,” Donald said. “You have to go out and play other leagues and you have to play comparable teams. Instead of me going and playing Westhill, we should’ve played a team from out east. That would’ve made more sense.”
“But me playing Westhill made us a lot better, so you have to weigh that,” he added. “What’s better for you as a team? I say playing a competitive schedule is better, so I guess the system is the system and I don’t worry about it.”
A team still needs to win two to four postseason games to win a sectional title, regardless of its seed.
“I think we just have to get used to it,” Donald said. “People fear the unknown and I think that’s the biggest thing.”
