Grassroot Institute article dataset

A separate, local dataset of individual posts identified in the preserved GRI website. It includes articles, blog posts, columns, news releases, testimony and multimedia posts, with formats recorded separately. The capture files and existing databases were read without modification. No network requests were used to build this dataset.

Open the searchable article index or Topics over time. This share copy includes all 3,949 index records and their excerpts. Index CSV downloads the fields shown by the browser. Full-text JSONL, SQLite and the original research CSV remain in the local research dataset.

Scope and provenance

The extraction checks all 6,859 saved HTML responses in capture_2026-09-20/coverage_review/coverage.sqlite3. Individual posts are identified by WordPress's single-post marker and Article/NewsArticle/BlogPosting structured data. An Open Graph article label alone is insufficient: GRI also uses that label on static pages.

The result contains 3,949 distinct posts represented by 3,972 saved responses. The 23 duplicate URL responses are grouped by WordPress post ID. All variants remain in source_responses.csv. Static pages and archive/category/date listings are excluded; all 2,887 exclusions are accounted for in page_classification.csv. One distinct article was outside the seed-only text index previously used for podcast research.

Displayed publication dates range from 2008-12-28 to 2026-09-21. These are the dates in the captured site, not a claim that the organization began publishing in 2008. Deleted pages, uncaptured historical posts and unsaved external media are outside this dataset.

Each article retains its primary coverage-database source ID, original URL, capture timestamp, response reference and extracted-text hash. source_responses.csv retains the archive response hash and all duplicate response references. The extraction verifies archived response integrity using the existing archive reader.

Main fields

CSV list fields use | delimiters. JSONL and SQLite retain those fields as JSON arrays. Blank metadata is not filled with guessed values.

Attribution

The body byline takes precedence over a site upload credit. For example, the Jamie Story interview is attributed to Mark Coleman, while GRI's institutional site credit remains recorded. Explicit republication introductions recover additional writers. A small number of articles identify their contributor through a closing biography; those cases remain flagged and their attribution method is explicit.

article_authors.csv keeps each observed credit, its role, the exact supporting excerpt and whether it was selected for the main record. Named testimony senders remain distinguishable from bylined authors. Speakers, interview subjects, quoted experts and authors of reports merely discussed in a post are not automatically made authors of that post.

authors.csv groups spelling variants through conservative punctuation/case/credential normalization. It is an attribution index, not a fully resolved person registry: it does not automatically equate different first names or merge uncertain identities.

Dates

The date visible to readers is the primary publication date. The structured publication timestamp is retained alongside it. Some timestamps fall on a different day when converted to Honolulu time; those discrepancies are flagged, not silently resolved. Original publication notices and release datelines can also differ from the website publication date.

Topics

All article bodies are processed locally. The baseline classification maps GRI's substantive categories to topics and applies a documented vocabulary to the title and full text. Title hits and site categories receive greater weight than passing body mentions. Format labels such as “Multimedia” and “President's Corner” are not themselves substantive topics.

topics.json contains the baseline vocabulary. article_topics.csv retains the selected topic, score, site category, matched terms and a body excerpt. Scores rank lexical evidence; they are not calibrated probabilities. A short description that supplies only weak topical signals receives a low_signal_topic flag. review_overrides.json records targeted corrections after reading problematic source text, including author attributions and topics. This is automated corpus classification with targeted review, not a claim that every topic label has been individually reviewed.

The separate key_terms field uses local TF-IDF weighting over literal words and adjacent word pairs. It provides a more specific search aid alongside the broader topic labels; it is not a generated summary.

Embedded audio, video and image contents have not been transcribed or interpreted. An article with no substantive saved prose can still have a title, date and site author. Where neither text nor metadata establishes a topic, it remains unassigned and flagged.

Files

Rebuilding

Requires Python 3, lxml, the preserved capture folder and its existing coverage_review/audit.py. No model service, API key or network access is required.

python3 article_dataset/extract_pages.py
python3 article_dataset/build_dataset.py
python3 article_dataset/validate_dataset.py
python3 article_dataset/build_browser.py

The first command can be skipped when the extraction cache is unchanged. Rebuilding replaces only generated files in article_dataset/.

Example SQLite queries:

SELECT title, publication_date, authors, key_topics
FROM articles
WHERE publication_date BETWEEN '2009-01-01' AND '2009-12-31';

SELECT a.title, a.publication_date, t.topic, t.method
FROM articles a JOIN article_topics t USING(article_id)
WHERE t.topic = 'Think-tank and advocacy networks';

SELECT a.title, a.url
FROM article_search s JOIN articles a USING(article_id)
WHERE article_search MATCH '"Atlas Network"';