scrapling-scraper

Standalone Python scraper feeding deal_offer from Cloudflare- and JS-gated travel sites the TS pipeline can't reach.

Overview

scrapling-scraper is a standalone Python scraper built on Scrapling for the two travel sources the TS Flue/n8n pipeline cannot reach: AirGuru (Cloudflare-challenge gated, via StealthyFetcher) and Novaturas (JS-rendered Magento PWA, via DynamicFetcher). It parses listing pages into the canonical Offer shape and POSTs them to the existing apps/web endpoint /api/internal/deals/upsert (header x-internal-rpc-token), feeding the same deal_offer table without touching any existing TS code.

This app is deliberately outside the pnpm/turbo workspace — it has no package.json and never runs through the TS lint/typecheck/test gates. Its own checks are ruff + pytest, and its commands are plain Python, not pnpm -F.

Stack

Python 3 + Scrapling (StealthyFetcher for Cloudflare, DynamicFetcher for JS rendering)
Pydantic for the Offer model (mirrors deals-schema.ts)
httpx to POST parsed offers to the upsert endpoint
pytest for offline parser/validation tests, ruff for linting
Docker for containerized runs

Develop

cd apps/scrapling-scraper
python3 -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"
scrapling install            # one-time: download the stealth browser
ruff check .                 # lint

Run it locally against /api/internal/deals/upsert:

cp .env.example .env         # point UPSERT_URL at local pnpm dev:web or prod
set -a && . ./.env && set +a
python -m scrapling_scraper novaturas    # or: airguru | all

Expect output like novaturas: scraped 12 offers, upserted 12. Verify rows with pnpm db:studio (deal_offer, source_id 1=airguru, 3=novaturas).

Test

pytest                        # offline parser + validation tests, against
                               # committed fixtures in tests/

Build & deploy

docker build -t scrapling-scraper apps/scrapling-scraper
docker run --rm --env-file apps/scrapling-scraper/.env scrapling-scraper all

Not deployed yet — prod hosting (Fly/Railway, or triggering it from the n8n box via cron) is deferred. Today it runs locally or as the Docker image above, invoked ad hoc or from a scheduler.

Structure

src/scrapling_scraper/offer.py - Pydantic Offer model + parse_money
src/scrapling_scraper/config.py - env (UPSERT_URL, INTERNAL_RPC_TOKEN) + source ids/urls
src/scrapling_scraper/upsert.py - httpx POST to /api/internal/deals/upsert
src/scrapling_scraper/main.py - CLI entry: python -m scrapling_scraper [airguru|novaturas|all]
src/scrapling_scraper/sources/airguru.py - StealthyFetcher fetch + parse a.single-offer
src/scrapling_scraper/sources/novaturas.py - DynamicFetcher fetch + parse a[href^="/search/"]
tests/ - offline parser tests against committed fixtures

See the apps overview and the monorepo overview.