Jev Job Screening Evaluation / Field notes
I built the test before the automation
I got early access to Jev, TypeSafe's System One decision model. I tested it as a job-posting screener: a blind evaluation against my career copilot across 65 postings, sealed verdicts, and an evidence review of every disagreement. It was safe, but not ready.
- Role
- Job seeker and evaluator
- Stack
- TypeSafe Jev (early access), Google Sheets, Python
- Timeline
- September 2026
- Status
- Evaluation complete; not deployed
- Scope
- Blind shadow evaluation of 65 job postings across two batches.
Where I started
I got early access to Jev, TypeSafe's System One decision model. It doesn't generate text: you give it a question with fixed answers, and it returns a typed verdict with a calibrated confidence score. I pointed it at my job search as a screening test, sorting postings into strong fit, weak fit, or reject.
I was tempted to plug it into my job search right away. I am looking for my next role and anything that cuts screening workload helps. But a screener that quietly discards the right opportunity is worse than no screener at all. So I decided to test it first.
My Hermes career copilot already evaluates postings for my tracker. That gave me a reference evaluator and a way to run the whole test blind.
How the test was designed
The question was not whether Jev agrees with my copilot. Agreement is not correctness. The question was whether Jev adds safe, net value beyond plain eligibility rules: can it save evaluation effort without screening out anything worth my attention?
Collect fixed batches
Postings came from LinkedIn Jobs. No seniority, pay, or remote filters, so the sample included roles with restrictions and unclear eligibility.Evaluate blind
The career copilot assessed every posting first, with Jev's verdicts hidden. Jev ran only after the copilot's judgments were locked, and its verdicts were written to a sealed tab.Adjudicate disagreements
Every disagreement was decided against the posting evidence and my corrected profile, not against the copilot. The copilot is a reference, not ground truth.Apply the promotion bar
The protocol required at least 20 natural Jev rejects to judge the reject lane. Fewer than that meant insufficient evidence, not a pass.
Two batches ran under this protocol: 50 collected postings (39 unique after dedupe) and an extension batch of 30 (26 unique), authorized when the first batch did not produce enough natural rejects.
What I found
Across 65 unique postings, Jev rejected nothing worthwhile. All 13 of its unique rejects were adjudicated non-worthwhile, and it produced zero false negatives. That is the safety result, and it held across both batches.
The value result is thinner. Of 65 postings, the two systems agreed outright on 43: 35 passes and 8 rejects. The copilot was right on 16 disagreements, Jev on 6. Five of Jev's six were cases where it caught hard blockers the copilot missed: in-person requirements, security clearances, an on-site mandate. The sixth was the most interesting single result of the whole test. The copilot rejected a Developer Community Manager role at Cognition on a title heuristic, and Jev kept it in weak fit. The posting asked for 2+ years of DevRel or community experience on an AI agents team, and it was relevant to my profile. That was the copilot's only false negative, and Jev preserved it.
Then the weaknesses. Both systems underweighted geographic eligibility. Six roles with strict non-US physical requirements were passed by both evaluators and only caught at adjudication. The earlier pilot had already flagged this pattern once, with a London-based role Jev rated a strong fit. It reproduced.
Jev also issued only two strong fit calls across Phase A, both at low confidence (0.53 and 0.29). There is no basis here for letting Jev promote anything on its own.
Decisions and tradeoffs
Blind before anything
- Why
- If the copilot had seen Jev's verdicts, its judgments would have anchored on them and the comparison would be worthless. Locking the copilot's assessments first kept the two evaluations independent.
- The tradeoff
- Blind evaluation doubles the work. Every posting gets two full assessments plus adjudication of the disagreements.
Notes & evidence
The blind held for both batches. The copilot's columns were locked before Jev ran, and Jev's verdicts stayed sealed until the copilot's work was done.
A promotion bar with a real number
- Why
- A vague sense of safety is how tools get deployed on vibes. Requiring 20 natural rejects before judging the reject lane forced a concrete evidence threshold, agreed in advance.
- The tradeoff
- The extension batch still fell short at 13. Running more batches to chase the number would have cost more adjudication effort for a tool that was already showing thin value.
Notes & evidence
Phase A closed with insufficient evidence for promotion. No live routing, quarantine, holds, or automated triage were approved.
Disagreements decided by evidence, not by the copilot
- Why
- The copilot misread my profile in places, including a rationale claiming I was explicitly avoiding senior roles. That is not in my profile. Adjudicating against posting evidence and the corrected profile kept one system's mistakes from becoming the verdict.
- The tradeoff
- Adjudication is manual and judgment-heavy. It does not scale the way an automated metric would.
Notes & evidence
The misstatement was recorded in the adjudication notes, and outcomes were decided on actual posting requirements.
No live routing during the test
- Why
- A shadow test that quietly affects the real tracker is not a shadow test. Nothing Jev produced entered my real job tracker or changed any queue order during Phase A.
- The tradeoff
- The test measured evaluation behavior, not live workflow savings. The effort-saved question is still estimated, not observed.
Notes & evidence
This was a condition of the experiment from the start, and it held.
What the evidence says
- Phase A adjudication recordverified
65 unique postings adjudicated: 35 agree-pass, 8 agree-reject, 16 copilot-correct, 6 Jev-correct; 30 worthwhile, 35 not.
The sheet holds the collected postings, the sealed Jev verdicts, the locked copilot assessments, the adjudicated outcomes, and a version log of the protocol decisions. The 20-reject bar and the insufficient-evidence verdict are recorded there.View evidence ↗- Phase A final reportverified
Numbers checked against the experiment sheet 2026-09-18: 80 rows, 65 adjudicated (35 agree-pass, 8 agree-reject, 16 copilot-correct, 6 Jev-correct), worthwhile 30 yes / 35 no, 13 Jev rejects all non-worthwhile with 0 false negatives, the Cognition save preserved, gate closed with insufficient evidence for promotion.
The full 8-page report with the combined metrics and the gate decision. Its numbers were checked against the experiment sheet.View evidence ↗- Live workflow savingsnot run
- Phase A was blind shadow only. No live routing, quarantine, or automated triage was tested, so effort saved in a real workflow remains estimated.
What I learned
The test did its job. It told me not to ship. Jev proved safe on the reject lane across 65 postings, which is genuinely useful information, but safety is not the same as value. Six correct disagreements in 65 postings, one of them a real save, does not clear the bar for automation.
The most durable result is the method. A blind protocol with a pre-registered evidence bar caught weaknesses in both systems: the copilot's title heuristic, the profile misstatement in its rationales, and the geographic eligibility gap they shared. If I test another tool, this is the rig I will reuse, with geography extraction fixed first.
For now, the career copilot keeps doing the evaluations and Jev stays a manual aid I consult, not a filter I trust.
Better next steps.