Web Scraping Engineer (Scrapy + Zyte API) – Long-Term Project Overview We are building a large-scale data aggregation system across 200+ listing / marketplace websites and are looking for an experienced web scraping engineer to own and maintain the scraping layer. This is a serious, long-term project focused on reliability, cost control, and clean data. Responsibilities Build and maintain Scrapy spiders (one per site or site family) Integrate Zyte API / Zyte Smart Proxy Manager Extract structured data: Address / location Price (hourly / daily / monthly where applicable) Listing URL Handle: Pagination / infinite scroll JS-heavy sites (React / Vue / Angular) XHR / JSON endpoint discovery Export normalized JSON Monitor and fix spiders when websites change Optimize Zyte usage to minimize costs Technical Requirements (Must-Have) Strong experience with Scrapy Experience using Zyte API or similar proxy/rendering services Proven ability to scrape JavaScript-rendered websites Clean, maintainable Python code Understanding of robots.txt and ethical scraping Git-based workflow No Selenium-only approaches No “browser rendering for everything” Nice to Have Marketplace / booking / real-estate scraping experience XHR / API reverse-engineering skills Cost-optimization mindset Playwright knowledge (fallback only) Deliverables Scrapy spiders (clean, modular) JSON exports per website Configuration for: Entry URLs Pagination logic Zyte usage Basic documentation for maintenance Project Scale ~200 websites total Some sites: hundreds of pages Others: tens of thousands of listings Ongoing maintenance and expansion expected Scrapy framework Zyte integration Standard JSON schema Monthly retainer available for maintenance We are cost-conscious and expect efficient Zyte usage. Required Screening Questions (Do Not Skip) In one sentence, explain the difference between Scrapy HTTP requests and browser rendering. Name one JS-heavy website you have scraped and explain briefly how you extracted the data. List two techniques to reduce Zyte API costs. (Optional but recommended) Paste a short Scrapy spider skeleton (10–20 lines). Generic answers will be rejected. How to Apply Please include: Years of Scrapy experience Example projects or GitHub (if available) Your preferred pricing model (per-site / hourly / retainer) Your estimated time per website Why Work With Us? Long-term collaboration Clear technical direction No shady scraping Real production-scale system Room to grow into a lead role