Five Million Games Later

The question I seem to get asked most often, in one form or another, is whether the FOH sim itself is any good. It’s a fair question, and “yeah, it’s solid” is a wack answer. So, I’d like to show you some actual data instead. This is probably the least glamorous part of building a sim, but it’s just as important as getting the UI and features right and it’s definitely the part I’ve spent the most time on…by a wide margin.

Here’s the overall workflow. There’s a second, headless version of the game that runs seasons without the UI, just the engine itself. Every new feature release that touches the engine goes through a series of sims: a quick single-season shape check, then multiple leagues running 8-10 seasons each for widespread confirmation. On top of that, I periodically run 25-to-50 season marathon sims to prove the changes hold long-term. These sims ultimately come back with a report card comparing about 40 different measurements against bands I’ve calibrated to real-NBA shape. Champions, playoff victories, scoring and player stat distributions, payroll usage, injury rates, trade volume, coach turnover, and more. If something drifts outside of its band, it gets flagged and I dig in to try to figure out why.

Not that anyone is counting, but if you add up every version of that harness plus the various overnight marathon leagues and calibration sweeps I’ve been running, the running tally is just north of 3,700 full test seasons and roughly 5.4 million games. Both numbers continue to climb, mostly because I keep on breaking shit. Just for the time between beta launch and v1.0 launch these calibration sims accounted for roughly 1,300 of those seasons and about 2.4 million games.

What the report card says

From a recent validation run of 50 seasons across 5 separate leagues, a few of the 40 numbers I care about the most (in no particular order):

• 20 different franchises won a title in 50 seasons. This metric was important to make sure a couple of superteams didn’t exist across decades.
  • I also checked the min, max, and average number of wins for these championship teams because one of the earlier iterations of the sim had teams with 70+ wins regularly, and roughly 19% of the championships went to teams with less than 45 wins. So…uhhh…I fixed that.
• Finals series averaged 5.7 games, and 14 of the 50 went the full 7 games.
• Better seeds won: in the 2 vs 7 series, the 2 seed won 73.4% of the time, a bit more upset-prone than the NBA’s long-run rate, which fits the modern-parity era the sim is tuned toward.
  • This one is oddly important because I had a weird bug where the 7 seed was winning a ridiculously high number of these series. So it took some creativity in the playoff sim to fix this one.
• Rotation players were shooting around 45.2% from the field, and 79.4% from the FT line.
• Player minute rotations are roughly in line with the NBA, with 78 players getting 30+ minutes per game vs the NBA’s 71. This number used to be way out of sync and I actually just got it fixed recently (I speak more about that below).
• Teams made about 16.4 trades per season, and about half of them landed within 2 weeks before the deadline.

The bug that took 4 rounds of fixes

This one haunted me for weeks. League payroll was hovering at roughly 70% of the cap no matter what I did. In real life, the NBA has a salary floor at 90% of the cap. Teams are literally required to spend that much, and in practice the league sits well over the cap through Bird rights and exceptions. My teams were hovering below the mandatory minimum. Across 4 straight rounds of fixes, I threw everything I could at it: I rewrote how CPU teams priced free agents, I added bidding premiums so contenders would overpay the way contenders do, I built a market-heat system that inflates prices when there’s money chasing a thin pool of talent, I even tried rebuilding the re-sign phase so teams would fight harder to keep their own guys instead of letting them walk.

Every one of those fixes “worked”, in the sense that the thing it targeted specifically got better. Vet-minimum contracts went from 80% of all signings down to nearly zero, mid-tier salaries more than doubled, and Bird retention recovered.

But after all that, overall payroll did not move. Seven marathon fleets in a row, all between 69% and 71%.

In the end, it all came down to a bug in cap holds. In the NBA, when a free agent walks his old team can renounce the hold on their books. In FOH, there was a bug where the renounce didn’t stick: a team would renounce the player, and the next season the player’s cap hold would quietly reappear on the team’s books. When I finally went and looked, 196 of 289 hold candidates in one league were stale, and some of them belonged to contracts that expired 12 seasons before. The average team carried $127 million in ghost contracts. Oops.

And that’s why nothing worked the way I had intended. The market-heat system I had built to inflate prices took league-wide cap room as its input, but cap room was negative everywhere, so the system decided that the market was frozen and kept pinning itself to the minimum multiplier value. I built an auction that was running upside down, and then spent 4 rounds of feature fixes and marathon sims to tune other systems against that broken auction.

After all that work, the fix was one small rule: a player only generates a cap hold if his contract expired entering this season. Holds tanked from $127 million per team to roughly $1.3 million. Payroll immediately jumped to 89.5-94.1% of the cap on the next fleet. Keeper re-signings went from around 300 per league to over 1,000. Teams under the salary floor dropped from 25-29 down to 6-13. Later retention work carried it the rest of the way; the current fleet sits around 117%, in the real league’s over-the-cap range.

4 rounds of fucking around with different systems trying to outsmart the game, and the answer was a stale row in a database. Granted, the additions ended up being helpful towards making things more realistic and enriched the salary cap moves by the AI throughout the league, but a nice lesson for next time (a lesson I will definitely not learn).

The realization tax

For the generated player leagues, I wanted more stars in the league. So, naturally, I raised the potential ratings on incoming draft classes and ran a few fleets expecting more stars. But the number of stars in the league per season barely moved.

What I found is that players systematically peaked below their potential, and the gap was worse the higher the player’s ceiling. A prospect with an 81-83 potential was peaking about 5-6 points below that, and only a quarter of them ever reached even an 80 overall. I kept playing around and found that to reliably make a player who peaks at around 85, you had to give them a potential of about 88+.

I’ve come to call it the realization tax (realized potential) and it’s one of the first items in the list that I check whenever rolling new features or changes to potential, player growth, drafts, etc.

It’s in my list of things to figure out. Luckily, there’s a bit of a hacky workaround (bumping the potential higher than the expected peak of the players), but it’s in the FOH backlog for further review once I get the time.

What’s still wrong

Back to the original question: is the FOH sim itself any good? Yes, I think it’s great. But things aren’t perfect, and there are a few things that I’m still iterating on to get the sim there while also not breaking anything else in the process.

The biggest one is around scoring breadth. The top-line scorers, overall team ppg, and player stats in general are mostly in line with NBA bands. But the one item that I keep cracking away at is the number of players scoring above 20ppg in the league. In the NBA, under the qualification rules my testing fleet uses, there are 40 players scoring 20+ppg; in FOH’s fully generated leagues we’re hovering closer to about 24. Worth noting this is a generated-league problem: leagues started from today’s NBA rosters or an imported file come in much closer to the real number, since they arrive with real scoring hierarchies already in place.

Just this week I threw a large calibration sweep at it, 240 test seasons alone across nearly every knob the engine exposes. The short answer? The knobs I have implemented currently don’t fix it. The players who should be doing that scoring exist: trace the guys with 20-point talent on 20-point minutes and they’re coming in at 15-19 a game. The top of the scoring curve is squashed while the bands below it hold close to real-NBA counts. Every lever that I try that does push them back over 20ppg ends up breaking something else. The real problem is that the second option on the team doesn’t convert their possessions into points the way a real second option in the NBA does. This involves changing FOH’s scoring math a bit, and it’s the top priority on the sim side right now. The only reason it didn’t hold up v1.0 is that I want to be very careful with it, a change there touches everything downstream, and rushing it breaks more than it fixes.

The efforts with those 240 test seasons weren’t for nothing though, I did end up fixing a bug around player playing time where star players were getting fewer minutes than expected. That’s the tradeoff I took into v1.0 as I still think the overall distribution is close, just not quite perfect.

Two smaller ones while I’m at it: three-point percentage runs about two points cold (32.3% against a 34-38% band), and the aging curve peaks a shade late, with the 27-29 and 30-32 age bands in a dead heat when the real league clearly peaks at 27-29. Both are filed.

So, yes, there are still some edges that I’m working through here. And if you notice anything strange, point it out, I’d love to make things as realistic as possible.

Why I’m posting the numbers

There are a lot of new sports sims popping up right now and most of them arrive with either some big round number or no data at all around how much testing they’ve had. I’m well aware that a headline like “5 million games” is exactly that and shouldn’t convince you of anything on its own.

What I hope does convince you that I’m taking this seriously is the second half of this post. Anybody can claim a big test count, or “the world’s best sim”. Fewer people will tell you that some of their shit is still broken, or admit that they spent weeks building features to fix a bug in a stale database row. The fleet is only worth something because I publish what it catches, including the stuff that makes me look like a dumb dumb.

So, that’s the deal. The game is free, it runs in your browser, there’s no signup, and the numbers above are the ones I’m holding myself to. If you play a few seasons and something feels wrong, tell me. Half the report card exists because somebody said “this seems off” and they were right.

play.frontofficehoops.com to play, r/FrontOfficeHoops to yell at me. I read everything.