Signal collects job postings daily, models them in a warehouse, and publishes what they say about the market. This page covers how each number is produced, what the data cannot support, and the mistakes caught before they were published.
| Stage | What happens |
|---|---|
| Collect | Job-board API plus company career boards (Greenhouse, Lever, Ashby). Raw responses land in object storage unmodified, partitioned by date, so any downstream decision can be reprocessed. |
| Snapshot | Every morning, how many US openings mention each of 51 technologies. No public source publishes this over time, so it only exists because it is recorded daily. |
| Load | Flattened into a warehouse with idempotent upserts — re-running a day changes nothing. |
| Extract | Technologies matched against a fixed vocabulary of 120 tools with alias mapping. |
| Model | 100 tested dbt models: staging, a star schema, and the marts behind every page here. |
| Sponsorship | 240,097 US Department of Labor visa filings aggregated with Spark, matched to employers. |
| Publish | This site is generated from the warehouse and served as static files. Search runs in your browser. |
Technology extraction originally used a language model. Its output was unusable: "Analytics" and "analytics" came back as different things, and "AI", "AI/ML" and "AI/LLM" as three. A fixed dictionary replaced it — cheaper, deterministic, and reproducible, which published numbers require. Anyone re-running it gets the same answer.
A model still does the work it is genuinely better at — reading a posting to judge whether a candidate could hold the role — but nothing you see here was generated by one. Every figure is a query.
An early version of the trend model reported that Python demand had fallen 70%. It had not. The months behind that comparison held between 45 and 241 postings, where a single posting moves a share by a full percentage point — and older months are biased anyway, because a posting only appears in them if it is still listed, so roles that filled quickly have vanished.
The sample threshold was raised and the claim withdrawn. Two demand series now exist and are never mixed: market-wide counts, which are trustworthy, and a collected sample, which is labelled as one.
Trend claims on this site are gated on how many days the collection has completely covered, not on how many dates happen to appear in the data. Right now that is 2 days of 28 required, so every rate and trend figure is suppressed — including on pages where there is clearly enough data to draw something.
The distinction matters because the warehouse keeps one "last seen" timestamp per posting and overwrites it daily, so history has to be accumulated deliberately rather than queried. Where it was reconstructed after the fact, only a posting's first and last sighting could be recovered — which means a role open for three weeks contributes two days and nothing in between. Counting open roles across those days would undercount by an unknown amount, and an undercount in a time series reads as decline rather than as missing data.
So those days are excluded from every rate, rather than drawn as a dip. The count above rises by one for each day the collection completes a full pass, and nothing that depends on it appears until the threshold is met.
Under 4% of job postings mention visa sponsorship at all, even at full length — so any tool inferring it from the job text is mostly guessing. Signal uses 240,097 filings employers actually made to the Department of Labor, including the wage they legally attested to.
Matching those to job postings is the hard part: boards say "Capital One", the government says "CAPITAL ONE NATIONAL ASSOCIATION". Names are normalised and matched exactly, with a guarded fallback for legal-entity prefixes, and a hand-written list for the cases no rule can reach — Esri files as "ENVIRONMENTAL SYSTEMS RESEARCH INSTITUTE ESRI", and short brands like Oracle and TD Bank sit below the length floor that stops "Apple" matching "Apple Movers".
One employer often files under several legal entities, so the filings are summed across all of them rather than read from one. Capital One files under two, PwC under five, Cognizant under four — reading a single entity understated the largest employers by two to six times.
Confidence is graded rather than assumed, and ambiguity is left undecided rather than guessed. "Cognizant" prefixes twelve different legal entities, and the alphabetically first has 13 filings against the right one's 15,274 — so where a brand matches several entities, nothing is claimed until a person has looked. Sampling also found "Lighthouse" resolving to "LIGHTHOUSE BEHAVIORAL SOLUTIONS", which are different organisations. 1,473 employers qualify to be stated as fact.
A company with no match is reported as having none, not as not sponsoring. Employers file under legal names that do not always resolve to the name on a job posting. But some absences are real and worth trusting: the two largest unmatched employers here appear nowhere in the filings under any name, because export-control rules require them to hire US persons only.
A filing record proves an employer has sponsored. It does not mean they will for a given role, and citizenship or clearance requirements override it.
No aggregator sees every job. Startups in particular post to their own boards, which is why company career boards are collected directly.
Absence is weak evidenceA technology missing from a posting does not mean the employer lacks it, so comparisons are only made in the positive direction.
The models run on two different warehouse engines and are verified to produce identical results. That check earned its place: the same SQL once classified 1,590 postings as internships on one engine and zero on the other — compiling cleanly on both and disagreeing silently, because their regular-expression functions anchor differently.
Every model has tests covering grain, ranges and referential integrity, and they run on every change. Pipeline-level checks catch what per-build tests cannot: whether the daily capture has stopped, whether volume has drifted, whether scores have collapsed onto a single value.
Source and full commit history: https://github.com/ZohaibA365/Signal