Scale & operate

Trim Network Traffic

View as Markdown

Residential proxy bytes are usually the most expensive line on a scraping bill, and most of those bytes are ads, trackers, and images your automation never looks at. Turn them off at the runtime and measure the difference on the run itself.

1import { Bctrl } from "@bctrl/sdk";
2import { chromium } from "playwright";
3
4const bctrl = new Bctrl({ apiKey: process.env.BCTRL_API_KEY! });
5
6const runtime = await bctrl.runtimes.create({
7  type: "browser",
8  name: "lean-scraper",
9  config: {
10    proxy: { type: "managed-rotating", country: "US" },
11    networkTraffic: {
12      blockAds: true,
13      blockTrackers: true,
14      blockResourceTypes: ["image", "media", "font"], // pages you read, not look at
15      urlBlocklist: ["https://metrics.acme-cdn.com"],
16    },
17  },
18});
19const { connectUrl, runId } = await bctrl.runtimes.start(runtime.id);
20
21const browser = await chromium.connectOverCDP(connectUrl);
22const context = browser.contexts()[0] ?? (await browser.newContext());
23const page = context.pages()[0] ?? (await context.newPage());
24
25for (const url of urlsToScrape) {
26  await page.goto(url);
27  // ...extract what you need...
28}
29
30await browser.close();
31await bctrl.runtimes.stop(runtime.id);
32
33// The receipt: what the run actually consumed.
34const usage = await bctrl.runs.usage(runId);
35console.log(usage.proxyBytes, usage.runtimeSeconds, usage.creditsUsed);

Run it once with networkTraffic and once without, on the same URL list, and compare proxyBytes - on media-heavy sites the cut is usually dramatic.

📸 Content TODO: do that comparison on a real site list and put the two proxyBytes numbers here as a small before/after table - a concrete “X% fewer proxy bytes” beats the adjective.

The knobs

  • blockAds / blockTrackers - filter-list blocking, the safe default for almost everything.
  • blockResourceTypes - drop whole resource classes (image, media, font, ping, beacon, prefetch, texttrack). Don’t block image if a flow needs to see the page - vision agents and screenshot steps included.
  • urlAllowlist / urlBlocklist - surgical control when one host is the problem (or the only thing allowed).
  • saver - preset data-saver levels (light / medium / high, or none to disable) when you’d rather not hand-pick.

Blocking also speeds pages up - fewer requests through the proxy means faster loads, which compounds across a long crawl.

Next