Case Study
Forge Intel
A job intelligence pipeline that scrapes, scores, and surfaces high-fit roles at 5 target companies — bypassing ATS auth walls with headless rendering and AI scoring, then delivering instant Telegram alerts when a match exceeds threshold.
The Problem
Staff+ job searches at Anthropic, Netflix, and Nvidia require monitoring dozens of postings across Workday, Greenhouse, and Ashby — all platforms designed to wall off machine-readable job data. Checking manually wastes hours. Missing a posting means applying late or not at all.
The specific constraint: ATS platforms like Workday serve HTML challenge pages to automated requests, return empty-body SSR shells, or block headless browsers entirely. A scraper that only works on static HTML is dead on arrival against 80% of the target job boards.
Solution Architecture
Forge Intel runs as a weekly cron job with a multi-strategy fetching layer, SQLite persistence, and an AI scoring pipeline backed by the damilola.tech /api/v1/score-job endpoint.
- ›Multi-strategy fetch: plain HTTP first, then mixed-SSR heuristic detection (Ashby/Workday empty-shell fingerprinting), then Playwright headless fallback for SPA/ATS URLs that require full JavaScript execution.
- ›Job content bypass: when URL-based fetching hits an auth wall, the pipeline falls back to
job_contentmode — passing raw JD text directly to the scorer, decoupling scraping from scoring. - ›SQLite persistence: every scraped job gets a hash-keyed row. Re-runs are idempotent — already-scored roles are skipped, new postings trigger scoring automatically.
- ›AI scoring: damilola.tech
score_jobMCP tool evaluates fit against resume, returns 0–100 score with gap analysis and a failure_mode taxonomy for diagnosing extraction failures. - ›Telegram alerts: any posting scoring above the threshold triggers an immediate DM to D via the Telegram Bot API — job title, company, score, and top 3 fit bullets.
Quantified Impact
Anthropic, Netflix, Nvidia, Airbnb, and Vercel — all monitored from a single weekly cron run without manual checking.
3 bypass strategies: plain HTTP, mixed-SSR heuristic, and Playwright headless — covering Workday, Greenhouse, Ashby, and Lever.
From cron fire to Telegram DM for any posting that scores above threshold. No polling delay — immediate on each scrape run.
Only high-score matches surface. Low-fit roles are silently stored in SQLite for audit but never interrupt D's focus time.
Tech Stack
Lessons
- ›Scraping strategy is a decision tree, not a single tool. The first scraper worked on 2 of 5 targets. Only by layering HTML → SSR heuristic → Playwright did coverage reach the full target set. Build the fallback chain before claiming coverage.
- ›Decouple scraping from scoring. The content bypass mode (
job_content) unlocked the full scoring pipeline for job descriptions that cannot be machine-fetched at all. Treating “I have the text” and “I fetched the URL” as two separate inputs made the system more resilient. - ›Idempotency compounds value over time. The SQLite hash-key design means each weekly run builds on the previous one. Score history, new-posting detection, and trend analysis are all free consequences of getting idempotency right on day one.
- ›Failure modes need taxonomy, not just handling. Adding
failure_modeto error responses (auth_wall, empty_shell, timeout, parse_error) made debugging silent failures 10x faster and drove the decision to build each bypass strategy.