Methodology

WASDE, 2021 onwards: USDA's official machine-readable CSV, one file per report. July 2010 – 2020 (and any month without a CSV): the XML edition archived by ESMIS (Cornell / USDA National Agricultural Library), discovered through its public API. Earlier reports exist only as PDF/text and are out of scope.

The WASDE dataset currently holds 194 reports (July 2010 – November 2026) and 857,965 values. Months without a WASDE (the October 2013 and October 2025 government shutdowns) are simply absent. The other four reports (Export Sales, NASS crush, Crop Progress, Grain Stocks) are described below; every source is US Government work in the public domain.

The XML edition does not label commodities or units for most tables. Its layout was learned from the 53 months (2021–2025) published in both formats: 97% of CSV values are reproduced exactly from the XML, and 100% of the grain and oilseed balance sheets. Label variants (e.g. "Hard Winter" vs "Hard Red Winter") are aliased in versioned configuration files.

Every table, commodity, line and unit gets a canonical id (e.g. "Ending Stocks, Total" and "Ending stocks" → ending_stocks). A label never seen before stops the run until it is mapped, so nothing is silently dropped.

Continuity choices: "EU-27" (2010–2013), "European Union" and "EU-27+UK" (Jan–Apr 2021) form one European Union series. Older reports publish projected farm prices as ranges (e.g. 7.50–8.90); these are stored as the midpoint. Units follow the official CSV, and labels that are wrong at the source (cotton farm price tagged as bales) are corrected.

Known gaps in the pre-2021 history: WASDE's projection-reliability tables, some quarterly livestock lines, and the U.S. milk table, whose fat- and skim-solids blocks share labels in the XML. Footnoted "negligible" cells in old cotton tables are stored as missing, not zero.

• Stocks-to-use: ending stocks ÷ use. For World and World ex-China, use = domestic use. For the U.S. and individual countries, use = domestic use + exports. Where a table publishes no domestic total (U.S. soybeans), it is total use − exports.
• Revision path: the value USDA published for one marketing year in each successive report. Month 0 is the first projection in May; month 12 is when the projection becomes an estimate.
• Drivers: Δ ending stocks = Δ beginning stocks + Δ production + Δ imports − Δ domestic use − Δ exports + other. For U.S. tables production splits into an area effect (Δ area × previous yield) and a yield effect (the rest).
• Percentile: shown only once at least 10 comparable values exist; otherwise ranks read "k of n".

1. Candidates: ending stocks, production, yield, area, exports, crush and farm price of the focus commodities, for the new- and old-crop years.
2. Each change must clear an absolute materiality floor (e.g. 50 mil. bu for U.S. corn ending stocks, 2 MMT for world stocks).
3. Score = half "size" (percent change) and half "surprise" (change ÷ the median change of the same line in the same month of past years).
4. One headline per commodity, region and marketing year, at most two per commodity, six in total.

Weekly Export Sales Reporting (ESR) data from the FAS API, published Thursdays at 8:30 ET for the week ending the previous Thursday. Metric tonnes for the current marketing year; next-marketing-year sales are kept separate. Commitments = accumulated exports + outstanding sales. ESR commitments are not Census exports.

Pace vs USDA (E1): commitments ÷ the U.S. export projection of the latest WASDE released before that ESR (an as-of join on release timestamps, so no look-ahead), converted to metric tonnes. Each of the 5 previous marketing years is measured the same way, at the same week, against the WASDE public at the time. Five years is a small sample: shown as a mean and a min–max range, never as a percentile. Required weekly sales (E2): (as-of target − commitments) ÷ weeks left, vs the 4-week average of net sales; not shown with fewer than 6 weeks left. China share (E3): always next to the share of unknown destinations, which can hide China.

Vintages: the API serves only the latest revision of past weeks, so history before the first capture is latest-only; every pull is stored as an immutable snapshot from then on. Shutdown catch-up weeks (2013, 2018–19, 2025) are pinned to their actual FAS release dates.

The monthly NASS report Fats and Oils: Oilseed Crushings, Production, Consumption and Stocks, released about the first business day of each month at 15:00 ET for the month two months earlier. Each release zip is found through the ESMIS API and stored byte-for-byte; every printed value is kept with the release that printed it. Tons are short tons (2,000 lb), oil is in 1,000 lb, bushels = short tons × 33.3333. History window: 2015-10+ (ESMIS holds NASS files from the October 2015 release only).

Crush pace (F1): marketing-year-to-date crush ÷ the WASDE crush projection public when NASS first printed that month; the 5 previous years measured the same way. Still needed (F2): (projection − crush to date) ÷ months left, vs the same months last year; not shown with fewer than 5 months left. Oil yield (F3): crude oil × 1,000 ÷ bushels crushed, vs the same month of 5 years and the WASDE-implied yield.

NOPA members represent roughly 95% of US soybean crush and publish about two weeks earlier; NOPA data is proprietary and not redistributed here. See nopa.org.

The weekly NASS Crop Progress report, released Mondays at 16:00 ET from April to November (Tuesday after a federal holiday), covering the week ending the previous Sunday. Release zips are found through the ESMIS API and stored as published; the national figure is NASS's own "selected states" aggregate (e.g. 18 states for corn). Kept: corn, soybeans, winter wheat and spring wheat; the stage tables (cumulative % of acreage planted, emerged, … harvested) with NASS's previous week, previous year and 5-year average, and the condition table (very poor … excellent). History starts with the first file the pipeline reads in full (June 2010).

• Good/excellent (C1): GE = good% + excellent%, against NASS's published previous-year value and against the 5 previous seasons. NASS publishes no 5-year average for condition, so that one is computed here: for each prior season, the week whose day of the year is nearest (within ±3 days). n is the number of seasons found; fewer than 5 suppresses the finding.
• Stage pace (C2): % of acreage at the stage in progress minus NASS's own published 5-year average (used exactly as published; no recomputation).
• Condition index (C3): CI = (1·very poor + 2·poor + 3·fair + 4·good + 5·excellent) ÷ 100, from 1 to 5; vs the 5 prior seasons at the same week (same ±3-day rule).
• Season closed: after the last scheduled release of the season, the card reads "Season closed: final GE X% vs 5-yr Y%" (the final rating of each crop against the final ratings of the 5 prior seasons); C2 and C3 are not shown, the report is marked off-season rather than stale, and the card leaves the home page in months 12, 1, 2, 3 (December to March).

The quarterly NASS Grain Stocks report (12:00 ET): stocks of corn, soybeans and wheat on December 1, March 1, June 1 and September 1, on and off farms. History from the June 2010 release; each release is stored as published, and a quarter revised by the next release keeps its first print.

• Stocks vs WASDE (G1): only for the stock date that matches a WASDE ending-stocks estimate (corn and soybeans September 1, wheat June 1). Gap = NASS total (first print) − the old-crop ending stocks of the WASDE public at the Grain Stocks release (as-of join, no look-ahead), in million bushels and %, ranked by the size of the gap among every year since 2010 (n ≈ 16).
• Implied use (G2): stocks at the start of the quarter + production − stocks at the end. Production is the WASDE figure public at the release, in the harvest quarter only (corn and soybeans September → December 1, wheat June → September 1), otherwise 0. Compared with the same quarter of the 5 previous years. Caveats: imports omitted; small for corn/soy. NASS may revise production on the same day (Small Grains Summary / annual Crop Production); this uses the WASDE figure public at release.
• On-farm share (G3): on-farm stocks ÷ total, vs the same date of the 5 previous years, and total stocks vs a year earlier.

/commodities/soybeans and /commodities/corn put the latest number of every report that covers the commodity on one page (WASDE balance sheet, Export Sales commitments, crush, Crop Progress, Grain Stocks), each with its release date, a link to the report chapter it comes from and the files in the catalogue. Nothing is recomputed there: the numbers are the report pages' own.

• Insights are computed in Python as typed parameters (ranks, shares, n); the site only turns them into words in English or Portuguese, and a test checks that the page matches the JSON.
• Ranks read "k of n". A percentile appears only when n ≥ 10. n is always shown, with a "small sample" chip below 10.
• An insight is not shown when n < 5, when an input report is older than its cadence plus grace (checked against the release that uses it), or when no USDA projection was public at the time.
• The home row holds at most 5 cards: one per report plus one cross-report card; a report left out by that cap appears under "Also released" with the same rules.
• Multiple comparisons: this build screened W1 66, W2 3, W3 3, W4 1, E1 5, E2 5, E3 1, F1 1, F2 1, F3 1, C1 4, C2 4, C3 4, G1 3, G2 3, G3 3 series or candidates.

A scheduled GitHub Actions workflow runs every 30 minutes on weekdays. A light gate checks the release calendars first and stops in seconds when nothing is due. When a new report appears it rebuilds bronze → silver → gold (dbt + DuckDB, with data tests such as the balance-sheet identity, monotonic crop progress, condition shares summing to 100 and a no-look-ahead check), redeploys this site, verifies it live and only then publishes the versioned GitHub release. Crop Progress is caught on Monday evening runs; a late NASS posting is picked up by Tuesday's first run.

A new WASDE becomes the data-YYYY-MM release, always marked Latest. Any other build with new data publishes an immutable weekly-YYYY-MM-DD prerelease, never marked Latest. Each report is isolated: a failed pull keeps the last good data, is flagged on the Reports page and never blocks the others. Superseded weekly prereleases are pruned after a documented window; data-* releases are kept forever.