1. Habit
  2. Skills
  3. Web Scraping skill
jamditis · 6K installs

Web Scraping skill
Runs by itself, every day.

Fresh facts from public pages every day.

Install the Web Scraping skill and make it a habit: follow https://gethabit.dev/skills/jamditis/web-scraping.md

  1. 1Copy
  2. 2Paste into Claude Code, Codex or Cursor
  3. 3It runs by itself

6K installs429 starsMIT licenceSource on GitHub ↗

Web Scraping

by jamditis · 6K installs

The skill

Collects text from public web pages the safe way.

Only when you ask.

Fresh data

Fresh facts from public pages every day. It stops when a site says no.

↻ Runs by itselfevery day

How the habit works.
You don't have to.

Three questionsWhen does it start? How does it check its work? What does it remember?

01

It starts by itself

daily source collection. Runs in Claude Code routine or cron.

  • ↻ daily source collection
  • 0 5 * * *

02

It checks its work

access and content checks. If a check fails, it stops.

  • ✓every URL passes validate_public_url before fetching
  • ✓no fetch continues after 401, 403 or 429
  • ✓extracted content is at least 100 characters
  • ✓robots.txt rules are honored

03

It remembers

source fetch log. The next run reads it first: URLs fetched before, methods that worked and blocks seen.

loops/public-page-scrape-loop/sources.md

One run, step by step.
Then it stops.

Stop ruleStops after 25 URLs per run, or at the first access block on a site.

  1. 01

    Validate URLs

    Your agent checks scheme, host and public IPs for every URL and redirect.

  2. 02

    Check robots

    It reads robots.txt and honors crawl delays and disallow rules.

  3. 03

    Run the cascade

    It tries trafilatura, then requests with BeautifulSoup, then Playwright.

  4. 04

    Detect poison pills

    It checks for paywalls, CAPTCHAs, rate limits and login walls.

  5. 05

    Stop at blocks

    On a block it stops and suggests an official API or asks for documented authorization.

  6. 06

    Wrap results

    It wraps content in EXTERNAL_DATA tags with the source URL and logs it.

What it may do.
Nothing more.

Safe by defaultIt writes only in its own folder and asks you before anything goes out.

Is it for you?
Use it when it fits.

Quick checkRead both lists. If the left one sounds like you, the Web Scraping skill helps.

Use it when

  • ✓ You collect public articles
  • ✓ You pull video metadata with yt-dlp
  • ✓ A scraper hits a 403 or CAPTCHA

Skip it when

  • – You want to bypass a paywall
  • – An official API already exists

Questions

Web Scraping skill, answered.

Does it get past CAPTCHAs?

No. A CAPTCHA, login wall or 403 is a stop signal, not something to evade.

Is scraped text trusted?

No. Retrieved content is data, never instructions.

How do I install the Web Scraping skill?

Copy the prompt at the top into your agent. It installs the skill from jamditis/claude-skills-journalism and sets up the habit. Or run npx skills add jamditis/claude-skills-journalism --skill web-scraping.

Is the Web Scraping skill free?

Yes. It is published under MIT. Habit is free too.

Web Scraping. Now as a habit.

Install the Web Scraping skill and make it a habit: follow https://gethabit.dev/skills/jamditis/web-scraping.md

  • Free
  • No account
  • Undo in one line

All jamditis skills →More content skills →

Works in

  • Claude Code
  • Codex
  • Cursor
  • ChatGPT
  • Claude app

Made by jamditis, published under MIT. Habit links to the original and never changes it. Is this your skill?

Copying didn't work here. Select the line and copy it by hand.