review · segments
Build browser pool daemon with stealth hardening
pi 290 events 2 segments
segment 1 of 2
Build the persistent browser pool daemon with PHP integration
Created a Node HTTP daemon (scripts/browser-pool-daemon.js) that manages M persistent browser contexts each with proxy, stable UA, and userDataDir profile. Added BrowserPoolClient.php, config in auctionbot.php, and bot:browser-pool artisan command for start/stop/status/benchmark. Minimal feature-flagged integration in Scraper.php. Documented in BROWSER_POOL.md. Committed in 5 logical commits. Benchmarks showed pooled scrapes through proxies slower (~17s) than cold unproxied scrape (~3.5s) because the cold path previously did not pass proxies to Puppeteer.
outcome
Browser pool daemon, PHP client, artisan command, and scraper integration are built and committed on branch browser-pool.
next steps
—
key decisions
- Use environment variables passed from PHP artisan command to configure the Node daemon, not a separate config file.
- Use Puppeteer's userDataDir for persistent cookie/session profiles.
- Feature-flag the pool with fallback to cold launch in Scraper.php.
- Keep proxies config-driven so residential proxies can be swapped later without code changes.
open questions
—
1 month ago → 1 month ago
segment 2 of 2
Add stealth hardening with puppeteer-extra and test CloudFront block
Installed puppeteer-extra and puppeteer-extra-plugin-stealth via yarn. Created scripts/browser-pool-fingerprints.js with Chrome 131 desktop fingerprints including matching Sec-CH-UA client hints. Updated daemon to use puppeteer-extra, apply stealth plugin, set Accept-Language, Referer, and timezone per page. Updated config user-agents and removed BROWSER_POOL_USER_AGENTS env var from artisan command. Empirical test through DigitalOcean proxies showed persistent CloudFront 403 block on both homepage and lot page. Updated BROWSER_POOL.md with measures and test results.
outcome
Stealth hardening implemented and tested; CloudFront block not bypassed; documentation updated with honest results.
next steps
- Acquire residential proxy sources and swap into config('auctionbot.proxies') to bypass CloudFront IP-reputation block.
key decisions
- Use built-in fingerprint module (scripts/browser-pool-fingerprints.js) instead of config user-agents for the daemon, keeping fingerprints self-contained in JS.
- Remove BROWSER_POOL_USER_AGENTS env var from artisan command since daemon no longer reads it.
- Keep proxies config-driven so residential proxies can be swapped later without code changes.
open questions
- Whether a residential proxy provider will be acquired and integrated.
1 month ago → 1 month ago