Our Job Match Score Changed When You Reordered Your Skills: A Postmortem

Written by Best JobFinder’s founder.

Same job. Same skills. One run scored 211, the next scored 478. The only thing that had changed between them was the order of the user's skills array.

That was the bug report, more or less, behind "sometimes the score is different". This post is what caused it, how it hid inside a passing test suite, and what the score looks like now. I build Best JobFinder, and this is the number the whole product hangs on: it is the gate for "auto-submit above 85, review 50 to 84, skip below 50". A gate that moves is worse than no gate.

The formula

Every job gets a score from the skills extracted from its description, against the user's own skill list, where each skill carries a competency (0 to 100) and a weight (up to 10, negative allowed):

score = [ Σ(competency% × weight)  −  30 × missingCount ] ÷ 10

• A matched skill contributes competency × weight. Unrated competency (0) counts as 100: if you listed it, you have it.
• A missing skill, one the job wants and you did not list, contributes −30, so −3.0 to the final score.
• The denominator is always 10, the maximum weight, never the sum of actual weights.
• A negative weight is used as-is. PHP at weight −5 and 100% competency is −50 in the numerator. That is how you say "please do not show me PHP jobs".

So the score can exceed 100 and can go negative. A separate value, the ratio, clamps it to 0 to 100 for progress bars and colours. Worked example: .NET 100% × 5, SQL 100% × 4, Java 50% × 1, Kubernetes missing. Numerator 500 + 400 + 50 − 30 = 920. Score 92.0.

The real implementation, trimmed:

static compute(jobSkills: string[], userSkills: UserSkill[]) {
  const uniqueLower = [...new Set(jobSkills.map((s) => s.toLowerCase()))]
  const skillMap = new Map(userSkills.map((us) => [us.name.toLowerCase(), us]))

  let numerator = 0
  const matched: string[] = [], missing: string[] = []
  for (const lower of uniqueLower) {
    const us = skillMap.get(lower)          // exact, case-insensitive. Nothing else.
    if (us) {
      matched.push(lower)
      const comp = us.competency > 0 ? us.competency : 100
      numerator += comp * (us.weight ?? 1)
    } else {
      missing.push(lower)
      numerator -= MAX_WEIGHT * 3            // MISSING_PENALTY × MAX_WEIGHT
    }
  }
  const score = +(numerator / MAX_WEIGHT).toFixed(2)
  return { matched, missing, score, ratio: Math.max(0, Math.min(100, Math.round(score))) }
}

The line that matters is skillMap.get(lower). It was not always that line.

What the mirror did instead

The score is computed in two places: the shared TypeScript module the app uses, and a mirror inside the Chrome extension's background script, because the extension scores jobs on the page in front of you before anything is saved. The mirror had been "improved":

userSkills.find((u) => u.norm === n || u.norm.startsWith(n) || n.startsWith(u.norm))

Plus some .js stripping, so react.js would match react. It read as helpful. It produced two bugs, one of which is in the headline.

Non-determinism. find() is first-hit-wins over an array in whatever order the API returned it. If the job wants C and the user has C++, C# and Cypress, the match is whichever of those happens to come first. Reorder the list, or edit an unrelated skill so the API returns a different order, and the job scores against a different weight. Verified: 211 versus 478 on identical input with the array reversed.

Borrowed weights. REST matched RESTful. AI matched Airflow. React matched React Native. C matched anything that starts with a C. Each of those scored the job with another skill's competency and weight. Then the side panel's own breakdown table, which looks rows up by exact name, could not find the row, so the footer that reads "numerator ÷ 10 = score" could not reproduce the headline number above it.

Three screens, three numbers

That was the extension's own inconsistency. There was a second class of bug that spanned surfaces. The injected panel on the job page, the side panel's Skills box and the job card chip each had their own idea of where the score comes from:

• The page panel preferred the stored score from the database when one existed.
• The side panel always rendered the live score from the background.
• The job card read job.skillMatchScore raw, which the server computes from the saved row's skills alone.
• And when the background result was absent, the side panel fell back to scoring the description by itself, over a strictly different skill set (description only, no page-DOM skills, no stored skills).

One Indeed job rendered 286.0 in the page panel and on the card, and 569.0 in the side panel, at the same time. Equal formulas are not sufficient. The two sides also have to score the same set of skills, and only one of them is allowed to produce the number.

Why the tests did not catch it

There were parity tests. The extension's test directory was not in the Vitest include glob at the repo root. npm test reported green while never running a single extension test. The mirror had drifted for weeks under a passing suite.

This is the lesson I would keep if I had to drop all the others: before you cite a gate as evidence, confirm the gate actually inspects the files you changed. We found our typecheck script in the same state, compiling zero files and exiting 0.

The fix

1. Exact, case-insensitive matching only. A job skill matches a user skill when jobSkill.toLowerCase() === userSkill.name.toLowerCase(). No prefix, no suffix, no .js stripping, no token overlap. Aliases are a data problem (the admin skills catalog has canonical names and aliases) and are resolved at extraction time, not by fuzzing the scorer.
2. A parity test with a trap corpus. The extension's computeSkillMatch and the canonical SkillScorer.compute must return identical score, ratio, matched and missing across cases built from the bugs above (REST/RESTful, AI/Airflow, C), and the same result with the array reversed. The two extractSkillsFromText mirrors are asserted against each other too.
3. One producer per tab. The background script resolves the score through a single function: live score first, stored score as the fallback, and if the live score is missing it rebuilds it lazily from the same union the lookup uses, memoises it and broadcasts it. No surface computes its own. No surface renders the stored score when a live one exists. And a score broadcast is never dropped for arriving before the "current job" message it belongs to; it is tagged with the job id and staleness is decided at render time.
4. The server scores the same set. When the extension's merged skill set is a strict superset of what the database holds, it pushes the merge back, the server re-scores and returns the stored result, and the page is re-injected immediately after an auto-save instead of showing the pre-save number until some later lookup silently swapped it.
5. The extension tests run. include covers chrome-extension/**/*.test.ts. There is a comment in the config saying not to narrow it again.

What a deterministic score buys you

Thresholds. The whole supervised auto-apply model is "submit above this line, review between these, skip below that", set per run. That only means something if the number is a function of the inputs and nothing else. A score that depends on array order, on which surface you are looking at, or on whether a background message arrived first is not a number you can put a rule on.

It also buys you explainability. Every surface that shows the score can show the same breakdown table underneath it, per skill: competency, weight, contribution, and the missing penalties, summing to the number above. When a user asks "why 78?", the answer is a table, not "the model felt that way".

The score is at bestjobfinder.net, on every job, with that table under it. Seven-day trial. If you find a job where the table does not add up to the headline, that is a bug and I want the job id.