When the data you need has no API
A lot of valuable information lives on web pages and portals that were built for humans: supplier catalogues, public tenders, job boards, regulatory registers, partner portals that only offer a “download CSV” button after six clicks. Teams end up copying it by hand, or buying brittle scripts that break every time the site is redesigned.
We build collection pipelines that are reliable, polite and maintainable. In our own product work — the job-lead collection behind OptyCV — we have learned the unglamorous parts: rate limits, deduplication, change detection and keeping collectors alive as sources evolve.
AI makes extraction resilient
The hard part of scraping is not fetching pages; it is turning inconsistent HTML into consistent records. Instead of hard-coding where each field sits on the page, we pass cleaned page content to a language model with a strict output schema: title, price, currency, location, date posted. The model finds those fields even when the layout shifts. Every record is validated — types, ranges, required fields — and low-confidence results are flagged rather than silently stored.
For high-volume sources we use the AI layer to generate and repair selectors, then run cheap deterministic extraction most of the time, falling back to the model only when something changes. That keeps per-record cost low.
Compliance and good manners
We start every project by asking whether we should collect the data at all. That means reading terms of use, honouring robots rules, avoiding personal data unless you have a lawful basis, and choosing official APIs or data feeds whenever they exist. Collectors identify themselves where appropriate, throttle requests and run off-peak. We do not bypass CAPTCHAs, paywalls or access controls.
Browser automation beyond scraping
The same tooling drives web workflows: submitting forms on a supplier portal, downloading statements, checking order status on a logistics site. Combined with an agent, a browser becomes another tool — used sparingly, with screenshots and logs of every action for audit.