travel-scraper

Cloudflare Worker that extracts and normalizes travel-operator offers into the shared Offer schema for the web app's Travels feature.

Overview

travel-scraper is a Flue-based (Cloudflare-targeted) extractor that turns raw Lithuanian tour-operator content into a normalized Offer[]. Its single extract-offers workflow receives raw operator content (or a URL to self-fetch), parses each source's feed/JSON/HTML into the shared Offer schema from @repo/api, and returns a decoded ExtractResult. It feeds apps/web's Travels feature (deal watches, offer matching, Telegram alerts), typically invoked from an n8n + Supabase pipeline.

The novaturas source requires the listing URL with date params (?hcf=…&hct=…) — the parameterized URL must be sent, not the bare catalog path, or the rendered page has no offers.

Stack

Flue workflow runtime + CLI (--target cloudflare)
Cloudflare Workers + Durable Objects (SQLite-backed), Wrangler
@cloudflare/puppeteer Browser Rendering for JS-heavy operators
Effect (Schema) and valibot for boundary decoding
@repo/api for the shared Offer schema; @repo/db (dev) for ids
Vitest for tests

Develop

pnpm -F travel-scraper dev              # flue dev --target cloudflare
pnpm -F travel-scraper pipeline:dry-run # dry-run the parse pipeline against fixtures, no worker

Test

pnpm -F travel-scraper test           # vitest (watch)
pnpm -F travel-scraper test:ci        # run once, junit output
pnpm -F travel-scraper test:coverage  # coverage
pnpm -F travel-scraper test:ui        # vitest UI

Build & deploy

pnpm -F travel-scraper build   # flue build --target cloudflare

Deploys as a Cloudflare Worker. Flue auto-generates the Worker config and merges wrangler.jsonc (Browser binding + the v1 DO SQLite migration) on top; deploy with Wrangler.

Structure

flue.config.ts — Flue build config (Cloudflare target)
src/workflows/extract-offers.ts — the discovered workflow + HTTP route
src/parse/ — per-operator parsers (airguru, novaturas, teztour, westexpress) and shared offers normalization
src/sources.ts — operator slug -> deal_source.id map and schema
scripts/pipeline-dry-run.ts — local fixture-driven pipeline run
wrangler.jsonc — Worker overrides merged into Flue's generated config

See the apps overview and the monorepo overview.