yudopr.dev
Back to all posts

Bypassing the Walled Gardens: The Real Trade-Offs of Turning Social Media into RSS

2026-08-14•Updated 2026-08-31•14 min read
RSSOpen-SourceWeb ScrapingNext.jsSelf-Hosted

When I first set out to aggregate my short-form content and follow key creators without falling into algorithmic rabbit holes, I ran straight into the modern web's most annoying reality: social platforms have turned into fiercely guarded walled gardens.

First, platforms killed off native RSS feeds to push users back into app-centric, algorithmic timelines. Then, many official APIs became increasingly gated behind developer approvals, authentication flows, quotas, and paid access.

I didn't want to maintain heavy SDKs, register OAuth apps, or pay enterprise rates just to display my own latest videos on my website.

I wanted a simple, resilient, and open protocol: RSS (Really Simple Syndication).

Social Media to RSS Architecture

For ordinary websites, RSS is still surprisingly easy to recover with tools like RSS-Bridge or OpenRSS. Social media is a different problem. Platforms like TikTok, Instagram, and X aggressively restrict automated access, so a feed that works today can easily break tomorrow.

For my own use case, that changes the decision completely: if I need a reliable feed from a social platform, I either pay for a managed service such as RSS.app or build a small scraper specifically for my own public profile.

Here is how I think about those trade-offs, and how I use RSS to power the video shelf right here on this website.

1. Why I Prefer RSS Over Official APIs

Dealing with official platform APIs for simple read-only feeds is almost always overkill. Here is how I weigh the trade-offs:

Aspect Official Platform APIs RSS & Bridge Feeds
Authentication Developer account, OAuth2, refresh tokens Zero auth required
Cost Free tiers disappearing / exorbitant subscriptions Free for many normal websites; reliable social feeds may require a paid service or custom scraper
Timeline Order Algorithmic, engagement-weighted, ads 100% Reverse-chronological
Privacy & Tracking Tracked per app client ID and user session Decoupled (my server/reader queries the bridge)
Interoperability Siloed SDKs and custom JSON schemas Universal standard (XML/Atom/JSONFeed)

By converting everything into RSS, I can pipe all my updates into a single RSS reader (I use Miniflux and NetNewsWire) alongside tech blogs, newsletters, and release notes, without ever opening an algorithmic feed.

graph LR
    subgraph Sources
        Web[Normal Websites]
        TT[TikTok Profiles]
        IG[Instagram Profiles]
        X[X / Twitter]
        YT[YouTube Channels]
    end

    subgraph Feed Options
        OR[OpenRSS]
        RB[RSS-Bridge]
        RA[RSS.app Paid Social Feed]
        CS[Purpose-Built Personal Scraper]
        Native[Native XML / Atom]
    end

    subgraph My Ecosystem
        NextJS[Next.js Homepage Shelf]
        Reader[Miniflux / NetNewsWire]
        Bot[n8n / Webhook Automations]
        FB[GitHub feeds Branch]
    end

    Web --> OR
    Web --> RB
    YT --> Native
    Native --> FB
    CS --> FB
    FB --> NextJS

    TT --> RA
    IG --> RA
    X --> RA

    TT -. personal profile .-> CS
    IG -. personal profile .-> CS
    X -. personal profile .-> CS

    OR --> Reader
    RB --> Reader
    RA --> NextJS
    RA --> Bot

2. Method 1: RSS-Bridge for Sites That Are Still Scrape-Friendly

When I want full control and zero recurring software cost, RSS-Bridge is still one of the first tools I try. It is an open-source PHP project that turns supported web pages and endpoints into RSS 2.0, Atom, or JSON feeds.

The important caveat is that having a bridge implementation does not guarantee that the upstream platform will continue allowing automated access. This matters especially for large social networks, where login walls, anti-bot systems, rate limits, and frontend changes can make a bridge unreliable even when the bridge itself is correctly implemented.

For normal websites and less restrictive sources, RSS-Bridge is still extremely useful. For TikTok, Instagram, or X, I now treat it as an experiment rather than something I want to depend on for a production integration.

Quick Testing with a Public Instance

If I need to test whether a bridge currently works, I can use a public instance such as rss-bridge.org before deciding whether self-hosting is worthwhile.

If the target site is already blocking automated requests, however, moving the same bridge to my own server does not magically make the upstream restriction disappear.

Self-Hosting RSS-Bridge with Docker

For sources that do work reliably, self-hosting is simple:

docker run -d \
  --name rss-bridge \
  -p 3000:80 \
  rssbridge/rss-bridge:latest

The benefit is control: I own the cache, deployment, update schedule, and feed endpoint. The downside is that I also own every breakage caused by upstream HTML or anti-bot changes.

3. Method 2: OpenRSS for Websites Without Native Feeds

Another option worth trying is OpenRSS.

The workflow is intentionally simple: for a supported public website, prepend openrss.org/ to the page URL and OpenRSS attempts to expose that page as a feed. For blogs, news sites, documentation pages, and other public web content, this is a great low-friction alternative to maintaining my own parser.

For example, conceptually:

https://example.com/news

becomes:

https://openrss.org/example.com/news

OpenRSS is especially attractive because it is focused on open feed access rather than forcing every integration through a proprietary API.

Why I Don't Rely on OpenRSS for Major Social Platforms

The limitation is the same one that affects most community RSS bridges: the social platform still controls whether automated requests can reach its content.

That makes OpenRSS useful for the wider web, but not a dependable replacement for RSS.app for the social networks I care about:

  • Instagram has repeatedly blocked feed generation. OpenRSS documents the issue and, as of 2026, offers Instagram feeds to monthly donors while warning that they may still be unreliable.
  • X / Twitter has blocked OpenRSS from obtaining the content needed to generate feeds.
  • TikTok support is listed by OpenRSS in its feed directory, but OpenRSS itself describes the integration as unstable.

So I keep OpenRSS in my toolbox, but I don't design a critical TikTok, Instagram, or X integration around it.

4. Method 3: RSS.app for Reliable Social-Media Feeds

For social networks that rely heavily on client-side rendering, login walls, and aggressive bot protection, a managed service can be cheaper than continuously fixing my own general-purpose scraper. In those situations, RSS.app is the practical option.

There is an important pricing detail, though: RSS.app's free plan does not include social-media feed generation. The free tier is limited to native RSS feeds. If I want feeds from services such as TikTok, Instagram, or X, I need a paid plan after the trial. RSS.app confirms this in its account FAQ.

At the time of writing, the Basic plan supports up to 15 feeds. Its pricing page lists the annual-billing equivalent at $8.32/month, so I think of it as roughly a $10/month small-project expense depending on billing cadence.

For my use case, that is still relatively inexpensive. I am not paying hundreds of dollars for an official social API or maintaining a generic scraping platform. I am paying a small subscription for a handful of feeds that are maintained for me.

When I use it: When the social feed needs to keep working and the value of my time is higher than the subscription cost.

5. Method 4: Build a Small Scraper for My Own Public Social Profile

The other realistic option is to stop trying to build a universal social-media-to-RSS service.

If I only need updates from my own public TikTok, Instagram, or X profile, I can build a purpose-specific scraper for exactly that page and expose the result as RSS myself.

That architecture is much smaller:

graph LR
    Profile[My Public Social Profile]
    Scraper[Small Purpose-Built Scraper]
    Cache[Cached Post Metadata]
    RSS[RSS / Atom Endpoint]
    Site[My Next.js Website]

    Profile --> Scraper
    Scraper --> Cache
    Cache --> RSS
    RSS --> Site

The scraper only needs to collect the fields my site actually uses:

post id
title or caption
post URL
published timestamp
thumbnail URL
platform

Then I can transform those records into RSS or Atom and cache the result so I am not requesting the social platform on every page load.

This approach gives me more control and removes a recurring SaaS fee, but the trade-off is obvious: I become responsible for maintaining the scraper whenever the platform changes its HTML, endpoint behavior, or bot protection.

I would only choose this route for a narrow personal integration. Building a public, general-purpose scraper for arbitrary users brings back the exact maintenance burden I was trying to avoid.

My Decision Rule

For the web in general:

Native RSS available?
    -> Use it.

No native RSS?
    -> Try OpenRSS or RSS-Bridge.

For a major social platform:

Need a reliable social feed?
    -> Pay for RSS.app.

Only need my own public profile and don't mind maintaining code?
    -> Build a small purpose-specific scraper.

That is the trade-off I find much more realistic than assuming every social network can still be bridged for free.

6. Real-World Case Study: How I Built the Video Shelf on My Homepage

Rather than just consuming these feeds in a desktop RSS reader, I use this architecture directly in the codebase of this website.

If you check the Video Chronicles shelf on my homepage, those cards are generated through RSS feeds with zero official social-platform API tokens. YouTube comes from its native Atom feed, while TikTok comes from a small purpose-built Playwright scraper for my own profile.

Neither platform is touched while your browser loads the page. A daily GitHub Actions workflow fetches and validates both feeds, publishes them as two XML files on a dedicated feeds branch, and my Next.js server reads those cached files through the GitHub Contents API.

sequenceDiagram
    participant GA as GitHub Actions Job (daily cron)
    participant YT as YouTube Atom Feed
    participant TT as TikTok Profile (@yudopr)
    participant FB as GitHub feeds Branch
    participant Next as Next.js Server

    GA->>YT: GET /feeds/videos.xml?channel_id=UCofUYbRmLxAqny842JE6xsA
    YT-->>GA: Atom XML (new videos)
    GA->>GA: Validate every entry
    GA->>TT: Scrape profile (headless Chromium)
    TT-->>GA: Latest public videos
    GA->>FB: Publish youtube-rss.xml + tiktok-rss.xml
    Next->>FB: GET Contents API (5-min cache)
    FB-->>Next: Cached XML feeds
    Next-->>Browser: Render Video Cards

Here is how the pieces fit together in my codebase.

1. The Publisher: a Daily GitHub Actions Workflow

The workflow that used to scrape only TikTok is now the single publisher for both platforms. Every day at 21:30 UTC it runs both fetch scripts, and only if both succeed does it commit the two XML files to the feeds branch:

# .github/workflows/tiktok-rss.yml
- name: Scrape TikTok profile and build RSS
  run: pnpm exec tsx scripts/scrape-tiktok.ts dist/tiktok-rss.xml

- name: Fetch and validate YouTube Atom feed
  run: pnpm exec tsx scripts/fetch-youtube.ts dist/youtube-rss.xml

- name: Publish feed to feeds branch
  run: |
    mkdir -p /tmp/feed-publish
    cp dist/youtube-rss.xml /tmp/feed-publish/youtube-rss.xml
    cp dist/tiktok-rss.xml /tmp/feed-publish/tiktok-rss.xml
    # ...one commit containing both files, pushed to origin/feeds

The important property is that this is fail-closed: if the YouTube source is unreachable, parsing fails, or no video passes validation, the script exits non-zero and the workflow stops before the publish step. A stale feed is better than a broken one, so the feeds branch simply stays as it was.

2. Validating YouTube Without an API

YouTube maintains a public Atom feed for every channel. The fetch script in scripts/fetch-youtube.ts reads it, then validates every entry before it is allowed into the cached feed:

// scripts/fetch-youtube.ts — the validation core
const YOUTUBE_FEED_URL =
  'https://www.youtube.com/feeds/videos.xml?channel_id=UCofUYbRmLxAqny842JE6xsA';
// Statuses that are definitive unavailability, not anti-bot walls:
const UNAVAILABLE_STATUSES = new Set(['LIVE_STREAM_OFFLINE', 'UNPLAYABLE', 'ERROR']);

for (const entry of entries) {
  const watchUrl = `https://www.youtube.com/watch?v=${entry.id}`;
  const html = await fetchText(fetchImpl, watchUrl); // up to 3 attempts, 10s timeout
  const status = html === null ? null : getYouTubePlayabilityStatus(html);

  if (status !== null && UNAVAILABLE_STATUSES.has(status)) continue;

  if (status === 'OK') {
    validVideoIds.add(entry.id); // explicitly playable
    continue;
  }

  // Login/consent walls and non-2xx responses say nothing about the video
  // itself — YouTube serves them to datacenter IPs (CI runners). Confirm
  // via oEmbed, the same endpoint CMSes use for link previews.
  const oembed = await fetchText(fetchImpl, oEmbedUrl(entry.id));
  if (oembed !== null) validVideoIds.add(entry.id);
}

That last branch is the part that bit me in production. On a GitHub runner, every watch page came back as LOGIN_REQUIRED — Google walls datacenter IPs with a login/consent page. I proved it was the IP, not the videos: the same script from my home connection reported status: OK for the same videos on the same day, and the oEmbed endpoint returned HTTP 200 for all 15 of them. The lesson I encoded: treat a login wall as a question, not an answer. Definitive statuses (LIVE_STREAM_OFFLINE, UNPLAYABLE, ERROR) are trusted; walls get double-checked through a channel that does not wall servers.

3. The App Reads Only Cached Feeds

Server rendering never requests YouTube or TikTok. src/lib/videos.ts loads both XML files from the Contents API of the feeds branch and caches them in-process for five minutes:

// src/lib/videos.ts
const YOUTUBE_FEED_API_URL =
  'https://api.github.com/repos/yudopr11/yulog-v2/contents/youtube-rss.xml?ref=feeds';
const TIKTOK_FEED_API_URL =
  'https://api.github.com/repos/yudopr11/yulog-v2/contents/tiktok-rss.xml?ref=feeds';
const FEED_CACHE_TTL_MS = 5 * 60 * 1000;

async function loadFeedXml(feedUrl: string): Promise<string | null> {
  const cached = feedXmlCache.get(feedUrl);
  if (cached && Date.now() - cached.fetchedAt < FEED_CACHE_TTL_MS) return cached.xml;

  const headers: Record<string, string> = {
    'User-Agent': 'yulog',
    Accept: 'application/vnd.github.raw',
  };
  if (process.env.GITHUB_FEED_TOKEN) {
    headers.Authorization = `Bearer ${process.env.GITHUB_FEED_TOKEN}`;
  }

  const res = await fetch(feedUrl, { headers, cache: 'no-store' });
  if (!res.ok) return null;
  const xml = await res.text();
  feedXmlCache.set(feedUrl, { xml, fetchedAt: Date.now() });
  return xml;
}

The YouTube mapper feeds this XML through the shared pure parser (parseYouTubeFeed, the same module the workflow script uses, so parsing behavior is identical in CI and in production). The TikTok mapper is private to videos.ts and handles RSS-item quirks: strip hashtags and mentions from captions, truncate long ones, and pull the thumbnail from media:content. Then both streams are merged chronologically with a maximum of three cards and two per platform:

export async function fetchAllVideosFromCache(): Promise<VideoCardItem[]> {
  const [ytVideos, ttVideos] = await Promise.all([
    fetchYouTubeRSSFromCache(3),
    fetchTikTokRSSFromCache(3),
  ]);

  // Sort newest-first, then select at most 3 cards with at most 2 per
  // platform, filling from the other platform when one has fewer videos.
  // ...

  return selected.length > 0 ? selected : FALLBACK_VIDEOS;
}

If the GitHub read fails for either feed, the shelf falls back to static cards — readers never see a broken shelf, just a slightly stale one. The token in .env.example (GITHUB_FEED_TOKEN, a fine-grained PAT with Contents read-only) is optional when the feeds branch is publicly readable and required when it is not.

This used to be a paid RSS.app feed on the TikTok side. When I moved both feeds into one publisher workflow, I applied the decision rule from the previous section and replaced it with the purpose-built scraper: it only ever needs my own public profile, and I am the one who maintains it. The subscription fee disappeared, and the maintenance surface became a few hundred lines of scraper plus the validation logic above. That validation is the part I would warn anyone about — "reliable" feeds fail in unglamorous ways, and the fix was treating a login wall as a question rather than an answer.

Final Thoughts

RSS is still one of the simplest ways to pull content back out of algorithmic interfaces, but I no longer assume that every social platform can be bridged reliably for free.

For ordinary websites, native RSS, OpenRSS, and RSS-Bridge are still excellent options. For heavily protected social platforms such as TikTok, Instagram, and X, I think there are two practical choices: pay for a managed service like RSS.app, or build and maintain a small scraper specifically for the public profile I actually need.

For a personal project, paying roughly the cost of a small SaaS subscription for up to 15 feeds can be a very reasonable trade. If I only need one or two of my own profiles and want full control, a purpose-built scraper can make more sense.

The important part is not pretending the walled gardens are open. RSS gives me a clean interface on my side; I still have to decide who is going to maintain the bridge to the platform on the other side. ::: ​​