macOS Containers / Notes

Headless browsers and crawlers in containers

After build toolchains, a headless browser is the most common thing anybody containerises on a laptop. It is also the workload that punishes a sloppy container configuration fastest.

Why containerise a browser at all

The honest reason is version pinning. A browser engine is a moving target, and headless mode itself has been rewritten in living memory, so rendering behaviour changes between releases in ways that quietly invalidate whatever you concluded last month. Baking a specific engine build into an image is the only practical way to make a rendering result reproducible six months later. Everything else follows from that: a crawl or a test suite that produced a result you cannot reproduce did not really produce a result.

The secondary reasons are parallelism and hygiene. Twenty browser instances with independent profiles, caches and cookie jars are easier to reason about as twenty containers than as twenty processes fighting over one home directory, and a crawler that runs in a container is not accidentally authenticated as you because it inherited your browser profile.

The parts people get wrong

The gap on macOS

Chromium and Firefox are the easy cases, because Linux builds exist and a Linux container is the natural home for them. WebKit is where it breaks down. Safari's rendering is a first-class target for anything user-facing, and the version of it you care about only exists on macOS. A Linux container cannot give you it, and the Linux WebKit builds that automation frameworks ship are close relatives rather than the same engine.

So the practical setup for a lot of teams is split: Linux containers for the Chromium and Firefox matrix, and a separate long-lived Mac doing the Safari runs by hand, with all the configuration drift that implies. A macOS guest cheap enough to create per run is the thing that would close that gap, which is the case for this project beyond development builds. The notes on isolation boundaries cover why a guest and a namespace are not interchangeable here, and the requirements page covers what the current release costs you.

Where the output goes

A rendering pipeline is only useful if somebody acts on the difference between the raw response and the rendered page. The people who do this most rigorously are technical search teams, because for them the gap between the two is the gap between what exists and what gets indexed. It is a well-covered problem outside the testing world: 420 SEO's write-up on what a crawler sees when content arrives after the initial response works through the same diff from the indexing side, on catalogues large enough that the answer matters commercially.

The complement to crawling your own site is reading what actually hit it. Your server logs record every request a real crawler made, in order, with response codes, which is ground truth that no synthetic crawl can give you. The same team's notes on using log files to find where crawl budget is being spent cover the parsing and the aggregation, and the technique is identical whether the pages are products or documentation.

Both of those are containerisable jobs in their own right, and both are better as scheduled runs against a pinned image than as something anybody does by hand.