Decoding the Feed: An Empirical Investigation into RSS and Atom Syndication Reliability

Executive Overview

For decades, Really Simple Syndication (RSS), Atom, and JSON Feed have served as the silent plumbing of the web, routing millions of blog posts, news updates, and essays to RSS readers, automated scrapers, newsletter generators, and Retrieval-Augmented Generation (RAG) pipelines. Yet, despite its ubiquity, syndication is governed by fragile conventions rather than rigorous enforcement. Developers building automated ingestion systems have long operated under a loose set of heuristics—chief among them the traditional advice: "Check for content:encoded to extract full articles."

A comprehensive empirical study conducted by Vellum Labs across 32 major public web feeds challenges these foundational assumptions. Operated by an autonomous LLM agent (Claude Code) that systematically polled 32 high-profile public feeds two seconds apart on October 12, 2026, and again a month later on November 11, 2026, the investigation reveals a fragmented landscape where standard rules fail in both directions.

The findings expose critical vulnerabilities in automated data pipelines: misleading field allocations, inconsistent paywalls disguised as feed items, and a widespread disregard for bandwidth-saving HTTP conditional requests. This report breaks down the anatomy of modern web syndication, offering an investigative look at how technical drift affects data extraction, large language model ingestion, and the future of web monitoring.

Detailed Chronology of the Investigation

The investigation was designed with methodological rigor to capture temporal consistency. Vellum Labs deployed a standard-library Python script to poll 32 distinct public feeds twice: once on October 12, 2026 (analyzing 1,991 total items), and precisely one month later on November 11, 2026 (analyzing 2,095 total items).

Phase 1: The Initial Baseline (October 12, 2026)

On the initial run, the script systematically interrogated 32 feeds representing a cross-section of tech blogs, independent publications, corporate engineering logs, and news aggregators. The formats broke down into 22 RSS 2.0 feeds, 8 Atom feeds, and 2 JSON Feeds, with zero legacy RSS 1.0 feeds encountered. Remarkably, one feed was served incorrectly with a text/html MIME type yet parsed as valid RSS.

The primary objective was to determine whether a feed delivered the complete text of an article or merely a teaser, necessitating a secondary scraper to hit the web page. The baseline data revealed that 19 of the 32 feeds consistently provided full posts. However, the mechanism varied wildly: 15 relied on standard content fields, while 4 put the entire post inside the <description> or <summary> tags with no designated content field whatsoever.

Phase 2: The Re-Run and Stability Check (November 11, 2026)

Thirty days later, the exact same script was executed against the same 32 endpoints. Across 2,095 items, the structural breakdown remained remarkably stable. The same 19 feeds delivered full posts, the same 3 mixed-content feeds maintained their split, and the same 7 summary-only feeds remained restricted.

However, subtle shifts occurred in payload sizes and server-side HTTP header handling. For instance, Cloudflare’s blog feed dropped support for caching headers on the second run, joining the ranks of feeds that require full downloads on every poll. The stability of the data confirmed that syndication formats, while messy, operate on rigid, long-term publishing templates unique to each content management system.

Supporting Context & Metrics: Deconstructing Feed Architectures

To understand why traditional heuristics fail, we must look closer at how content creators package data. The 32 feeds evaluated by Vellum Labs fall neatly into five distinct operational categories.

The Five Kinds of Feeds

The Myth of the Universal Content Field

The common programming wisdom—if content:encoded exists, you have the whole article—proved structurally flawed in both directions:

1. False Negatives: Four major feeds (Dan Luu, Will Larson, dev.to, and Simon Willison) completely omit content fields, opting instead to place thousands of words of raw HTML inside <description>. Automated ingestors looking strictly for content:encoded would incorrectly classify these as summaries and attempt redundant web scrapes.
2. False Positives: Three feeds (The Verge, Ars Technica, and Stratechery) include valid content fields for nearly every item, but populate them exclusively with promotional teasers.

Identifying Teasers Programmatically

For systems attempting to process articles locally without hitting external web servers, distinguishing between a full post and a teaser requires algorithmic parsing. Teasers frequently announce themselves via trailing metadata or specific linguistic markers.

An analysis of the last 80 characters of teaser items revealed recurring patterns:

• Substack feeds (The Pragmatic Engineer): … Read more
• The Verge: … Read the full story at The Verge.
• Ars Technica: Read full article Comments
• Martin Fowler: … more…
• Stratechery: Relies entirely on character length rather than explicit footer strings.

Vellum Labs devised a robust heuristic rule to flag teasers: strip WordPress-style footers (e.g., "The post X appeared first on Y") and trailing link signatures, then evaluate whether the remaining text falls below a threshold (e.g., 400 characters) or terminates in a known teaser marker. Items meeting these criteria are automatically routed to a web-scraping queue to fetch the canonical URL.

Bandwidth Optimization: Caching and Conditional Polling

Polling web feeds at scale introduces a massive bandwidth overhead if managed naively. Across the tested corpus, payload sizes varied dramatically. For example, Smashing Magazine transmitted 1.4 MB for just 40 posts, while Dan Luu’s comprehensive HTML-laden feed clocked in at 6.7 MB for 128 posts. Polling Dan Luu hourly without caching validators would consume roughly 160 megabytes of bandwidth daily—almost entirely for un-updated content.

Fortunately, HTTP protocol features offer a solution. Of the 32 feeds monitored:

• 26 feeds initially transmitted caching headers (ETag or Last-Modified).
• When these tokens were bounced back to the servers in subsequent requests via If-None-Match or If-Modified-Since headers, 24 of the 26 responded with a 304 Not Modified status code accompanied by an empty body.
• By November, this efficiency improved slightly, with 25 out of 25 responding with 304s when queried with valid cached headers.

However, exceptions persist. Cloudflare’s blog feed dropped its caching headers between October and November, joining a stubborn cohort of 6 feeds (including the Netflix Tech Blog, Go blog, NYT, BBC, Google blog, and OpenAI news) that consistently omit ETag and Last-Modified headers. For these endpoints, automated scrapers have no choice but to download and hash the full payload to detect changes.

Data Integrity: The guid versus link Dilemma

Deduplication is a core challenge when ingesting multiple syndication streams. Systems typically rely on a unique identifier to prevent processing the same article twice. The study found that 15 out of 32 feeds use a guid or id that is not the item’s public URL.

• WordPress Feeds (e.g., GitHub Blog, The Verge): Use internal database identifiers disguised as query parameters (e.g., https://github.blog/?p=99381) while assigning a clean canonical URL to the <link> tag.
• Medium Feeds (e.g., Netflix Tech Blog): Use internal hash IDs for the GUID (https://medium.com/p/516d5a29b252) while appending tracking parameters to the link.
• Atom Feeds (e.g., Go blog, DHH): Employ Tag URIs (tag:world.hey.com,2005:World::Post/49362) as permanent identifiers.

To build a robust aggregation engine across mixed sources, developers cannot rely solely on the native guid. The optimal strategy requires normalizing identifiers: fallback logic must prioritize the canonical item URL, falling back to the guid only when the URL is absent or malformed.

Conversely, date parsing presented zero friction across the dataset. All 1,991 timestamps in October and 2,095 timestamps in November parsed cleanly using standard libraries (email.utils.parsedate_to_datetime for RSS and datetime.fromisoformat for Atom/JSON Feeds).

Future Outlook: The Evolution of Web Syndication

The findings from Vellum Labs’ empirical study carry profound implications for the future of automated information extraction, AI training data pipelines, and RSS reader architecture.

As Large Language Models increasingly rely on real-time web feeds to ingest current events, documentation updates, and technical essays, relying on naive parsing heuristics will lead to incomplete data ingestion, hallucinations driven by truncated text, and wasted compute resources. Syndication standards—while flexible by design—suffer from systemic divergence driven by content management system defaults (predominantly WordPress and custom publishing engines).

Recommendations for Feed Publishers

1. Embrace Full-Text Syndication: Publishers seeking maximum reach through automated AI agents and modern readers should include the full article body within standardized content fields (content:encoded or Atom <content>) rather than forcing readers to scrape web pages.
2. Standardize Identifiers: Feed generators should ensure that the guid or id maps directly to the canonical public URL whenever possible to streamline automated deduplication.
3. Honor Caching Headers: Content hosts must implement robust ETag and Last-Modified header support to prevent unnecessary bandwidth consumption by automated monitoring scripts.

Recommendations for Consumer Developers

Developers building RAG indices, newsletters, or RSS clients must abandon monolithic parsing assumptions. Ingestion engines must implement multi-tiered validation layers: checking multiple content nodes, inspecting trailing character strings to detect truncated teasers, and maintaining intelligent fallback queues to scrape full web pages when feeds fall short.

As the web continues to transition from human-centric browsing to agentic, machine-to-machine data exchange, the humble RSS feed remains a vital infrastructure layer. Upgrading its reliability is no longer just a matter of developer convenience—it is a structural necessity for the intelligent web.