nandi/glean · 4e13e05

⑂ 4e13e05

Register the scraper probe in the test suite

The previous commit landed the scraper but not these: the shell line that
made the edits had a bad cd and never ran, while the commit that followed it
did.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

nandithebull committed 2026-09-21T18:36:03-07:00 Browse files

4e13e05 parent: e933815

modified spike/nim/glean_spike.nimble +1 -0 Viewed

@@ -30,6 +30,7 @@ const networkProbes = @[
30	30	"jetstream_probe",
31	31	"feedparser_probe", # fixtures are offline, but it also fetches real feeds
32	32	"feedfetcher_probe", # retry policy is scripted; one check hits the network
	33	+ "scraper_probe", # fixtures are offline; one check fetches a real article
33	34	]
34	35
35	36	proc runProbe(name: string) =

    "jetstream_probe",
    "feedparser_probe",   # fixtures are offline, but it also fetches real feeds
    "feedfetcher_probe",  # retry policy is scripted; one check hits the network
++  "scraper_probe",      # fixtures are offline; one check fetches a real article
  ]
  proc runProbe(name: string) =

modified spike/nim/readme.md +39 -0 Viewed

@@ -446,6 +446,45 @@ is why it is written down both here and at the top of the module.
446	446
447	447	Worth fixing on the Go side too; it is a handful of lines.
448	448
	449	+## 13. The scraper
	450	+
	451	+`atproto/scraper.nim` ports `internal/scraper/scraper.go`. Feeds routinely
	452	+ship a first paragraph and a "read more" link, so the reader view needs the
	453	+page itself: find the node most likely to be the article, strip the furniture
	454	+around it, and re-render a whitelisted subset of HTML. Output stays HTML
	455	+rather than text, because paragraphs, headings, lists and code blocks are
	456	+most of what makes a long article readable.
	457	+
	458	+### Sanitisation is the real test
	459	+
	460	+This renders HTML fetched from an arbitrary site into a reader's client, so
	461	+the probe weights toward what must not survive: `script` and `style`
	462	+content, `iframe`, event handlers, unknown data attributes, `javascript:`
	463	+URLs. Attributes are whitelisted rather than blacklisted, so an attribute
	464	+nobody anticipated is dropped by default instead of needing to be added to a
	465	+list. A page that renders correctly proves nothing about what was stripped,
	466	+so each of those is asserted separately.
	467	+
	468	+Links carry `rel="noopener noreferrer"`, and a link whose target is not
	469	+http(s) is flattened to its text -- the words survive without the anchor.
	470	+
	471	+Two smaller details worth keeping:
	472	+
	473	+- A lazy-loaded image resolves through `src` → `data-src` → `data-lazy-src`
	474	+ to the first reachable URL. Rendering the placeholder gives a broken
	475	+ image where the picture should be.
	476	+- Removal walks children back to front. Deleting by index shifts every
	477	+ sibling after it, so a forward walk skips one after each removal -- the
	478	+ probe has four consecutive `<nav>` elements specifically to catch that.
	479	+
	480	+The extractor refuses rather than returning something thin. A caller handed a
	481	+nav menu cannot tell it from a short article; an error lets it fall back to
	482	+the feed's own summary.
	483	+
	484	+The live check takes its URL from the Rust blog's feed rather than
	485	+hard-coding one, after a hard-coded guess 404'd -- a probe that rots when
	486	+someone reorganises their permalinks is worse than no probe.
	487	+
449	488	## What this does and does not prove
450	489
451	490	Proven: the cryptography, the storage layer, and the OAuth flow up to user

  Worth fixing on the Go side too; it is a handful of lines.
++## 13. The scraper
++
++`atproto/scraper.nim` ports `internal/scraper/scraper.go`. Feeds routinely
++ship a first paragraph and a "read more" link, so the reader view needs the
++page itself: find the node most likely to be the article, strip the furniture
++around it, and re-render a whitelisted subset of HTML. Output stays HTML
++rather than text, because paragraphs, headings, lists and code blocks are
++most of what makes a long article readable.
++
++### Sanitisation is the real test
++
++This renders HTML fetched from an arbitrary site into a reader's client, so
++the probe weights toward what must *not* survive: `script` and `style`
++content, `iframe`, event handlers, unknown data attributes, `javascript:`
++URLs. Attributes are whitelisted rather than blacklisted, so an attribute
++nobody anticipated is dropped by default instead of needing to be added to a
++list. A page that renders correctly proves nothing about what was stripped,
++so each of those is asserted separately.
++
++Links carry `rel="noopener noreferrer"`, and a link whose target is not
++http(s) is flattened to its text -- the words survive without the anchor.
++
++Two smaller details worth keeping:
++
++- A lazy-loaded image resolves through `src` → `data-src` → `data-lazy-src`
++  to the first *reachable* URL. Rendering the placeholder gives a broken
++  image where the picture should be.
++- Removal walks children back to front. Deleting by index shifts every
++  sibling after it, so a forward walk skips one after each removal -- the
++  probe has four consecutive `<nav>` elements specifically to catch that.
++
++The extractor refuses rather than returning something thin. A caller handed a
++nav menu cannot tell it from a short article; an error lets it fall back to
++the feed's own summary.
++
++The live check takes its URL from the Rust blog's feed rather than
++hard-coding one, after a hard-coded guess 404'd -- a probe that rots when
++someone reorganises their permalinks is worse than no probe.
++
  ## What this does and does not prove
  Proven: the cryptography, the storage layer, and the OAuth flow up to user