rickub
nandi/gleanpublic Fork
⑂ 7e1c610
Branches
claude/modal-container-min-76527a main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/glean.git
git clone ssh://git@rickub.com/nandi/glean.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Download ZIP

Port the similarity computations and scoring helpers

Feed and user similarity from internal/cluster/jaccard.go, plus the pure
helpers from scoring.go. Both similarities stage into a temp table and swap
in a second transaction, so the expensive pass holds no lock and readers see
the previous generation rather than an empty table.

Feed similarity decays at exp(-0.023 * days), a half-life of about a month;
without it an old account's earliest subscriptions dominate its
recommendations forever. User similarity over subscriptions is deliberately
undecayed -- a subscription is a standing choice, not an event -- while
shared likes decay on both sides, because those are.

Also corrects an earlier claim in this spike's readme: exp() and log() are
not unused. They appear in seven places across the clustering SQL, and
SQLite only ships maths functions when built with
SQLITE_ENABLE_MATH_FUNCTIONS, so without those registrations every
recommendation query fails at runtime. The earlier grep was lowercase and
missed the uppercase SQL.

And a bug inherited from the Go implementation: common_likes is
CAST(SUM(decay * decay) AS INTEGER), but each decay factor is at most 1 so
their product is always just under it -- a pair sharing one recent like sums
to 0.9999 and truncates to 0. That number is shown to readers as the reason
for a recommendation, so a real overlap reads as none. This port rounds.

The probe caught it only because the assertion looked vacuous: both the
recent and stale pair returned 0, so the comparison passed while proving
nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nandithebull committed 2026-09-21T18:43:05-07:00 Browse files
7e1c610 parent: 4e13e05
modified spike/nim/.gitignore +2 -0
@@ -17,3 +17,5 @@ nimcache/
1717 feedparser_probe
1818 feedfetcher_probe
1919 scraper_probe
20+cluster_probe
21+diag
@@ -17,3 +17,5 @@ nimcache/
17 feedparser_probe17 feedparser_probe
18 feedfetcher_probe18 feedfetcher_probe
19 scraper_probe19 scraper_probe
20+cluster_probe
21+diag
added spike/nim/atproto/cluster.nim +276 -0
new file mode 100644
@@ -0,0 +1,276 @@
1+## The similarity computations behind recommendations, ported from
2+## internal/cluster/jaccard.go and the pure helpers of scoring.go.
3+##
4+## Two kinds of similarity are precomputed on a schedule, because both are
5+## O(pairs) and neither needs to be fresh to the second:
6+##
7+## feed_similarity which feeds are subscribed to by the same people
8+## user_similarity which people subscribe to the same feeds
9+##
10+## Both are time-decayed. An overlap from three years ago says much less
11+## about what someone wants now than one from last week, and without decay a
12+## long-lived account's early subscriptions dominate its recommendations
13+## forever.
14+##
15+## The decay constant is `exp(-0.023 * days)`, a half-life of about a month
16+## (ln 2 / 0.023 ≈ 30). It is written inline in the SQL rather than named,
17+## the same way the Go implementation writes it.
18+
19+import std/[algorithm, math, options, random]
20+import ./gleandb
21+import ./sqlite
22+
23+type
24+ Config* = object
25+ followBoost*: float
26+ likesWeight*: float
27+ tagsWeight*: float
28+ descriptionWeight*: float
29+
30+ SignalWeights* = object
31+ wSub*: float
32+ wLike*: float
33+ wTag*: float
34+ wSocial*: float
35+ wPop*: float
36+ wCategory*: float
37+ wContent*: float
38+
39+ FeedRecommendation* = object
40+ feedUrl*: string
41+ title*: string
42+ siteUrl*: string
43+ description*: string
44+ subscriberCount*: int
45+ faviconUrl*: string
46+ score*: float
47+
48+ PersonRecommendation* = object
49+ did*: string
50+ handle*: string
51+ displayName*: string
52+ avatarUrl*: string
53+ commonFeeds*: int
54+ commonLikes*: int
55+ commonTags*: int
56+ isFollowed*: bool
57+ score*: float
58+
59+const
60+ ## Half-life of roughly 30 days: ln(2) / 0.023.
61+ DecayPerDay* = 0.023
62+
63+proc defaultConfig*(): Config =
64+ Config(followBoost: 0.5, likesWeight: 0.3, tagsWeight: 0.2,
65+ descriptionWeight: 0.15)
66+
67+proc defaultWeights*(): SignalWeights =
68+ SignalWeights(wSub: 1.0, wLike: 0.5, wTag: 0.3, wSocial: 0.7,
69+ wPop: 0.2, wCategory: 0.4, wContent: 0.4)
70+
71+proc decay*(ageDays: float): float =
72+ ## The weight an event of this age still carries.
73+ ##
74+ ## Exposed so the decay can be checked directly; the queries compute it in
75+ ## SQL because pulling every like into the application to weight it would
76+ ## defeat the point of precomputing.
77+ exp(-DecayPerDay * ageDays)
78+
79+# --- similarity ------------------------------------------------------------
80+
81+## Both computations stage into a temp table and swap in a second
82+## transaction. The first pass is the expensive one and holds no lock on the
83+## live table; readers keep seeing the previous generation until the swap,
84+## rather than an empty table for the length of the computation.
85+
86+proc computeFeedSimilarity*(g: GleanDb) =
87+ ## For each pair of feeds, how strongly their subscriber sets overlap.
88+ ##
89+ ## The numerator is a decayed count of shared subscribers and the
90+ ## denominator is the union of the two subscriber counts, which makes this
91+ ## a Jaccard index with recency folded into the intersection.
92+ g.db.transaction:
93+ g.db.exec """CREATE TEMP TABLE IF NOT EXISTS _feed_sim_staging (
94+ feed_a TEXT NOT NULL, feed_b TEXT NOT NULL, jaccard REAL NOT NULL,
95+ PRIMARY KEY (feed_a, feed_b))"""
96+ g.db.exec "DELETE FROM _feed_sim_staging"
97+ g.db.exec """
98+ INSERT INTO _feed_sim_staging (feed_a, feed_b, jaccard)
99+ SELECT feed_a, feed_b, jaccard FROM (
100+ SELECT
101+ s1.feed_url AS feed_a,
102+ s2.feed_url AS feed_b,
103+ SUM(EXP(-0.023 * CAST(julianday('now') -
104+ julianday(MIN(s1.added_at, s2.added_at)) AS REAL)))
105+ / NULLIF(f1.subscriber_count + f2.subscriber_count
106+ - CAST(COUNT(*) AS REAL), 0) AS jaccard
107+ FROM articles.subscriptions s1
108+ -- feed_url < feed_url keeps one row per unordered pair, and drops
109+ -- the self-pair without a separate predicate.
110+ JOIN articles.subscriptions s2
111+ ON s1.user_did = s2.user_did AND s1.feed_url < s2.feed_url
112+ JOIN articles.feeds f1 ON f1.feed_url = s1.feed_url
113+ JOIN articles.feeds f2 ON f2.feed_url = s2.feed_url
114+ WHERE s1.added_at IS NOT NULL AND s2.added_at IS NOT NULL
115+ GROUP BY s1.feed_url, s2.feed_url
116+ ) WHERE jaccard IS NOT NULL"""
117+
118+ g.db.transaction:
119+ g.db.exec "DELETE FROM recs.feed_similarity"
120+ g.db.exec """INSERT INTO recs.feed_similarity (feed_a, feed_b, jaccard)
121+ SELECT feed_a, feed_b, jaccard FROM _feed_sim_staging"""
122+
123+proc computeUserSimilarity*(g: GleanDb) =
124+ ## For each pair of people, how strongly their subscriptions overlap.
125+ ##
126+ ## Undecayed, unlike feed similarity: a subscription is a standing choice
127+ ## rather than an event, so its age says little. Shared *likes* are decayed
128+ ## separately below, because those are events.
129+ g.db.transaction:
130+ g.db.exec """CREATE TEMP TABLE IF NOT EXISTS _user_sim_staging (
131+ user_a TEXT NOT NULL, user_b TEXT NOT NULL, jaccard REAL NOT NULL,
132+ common_feeds INT NOT NULL DEFAULT 0,
133+ common_likes INT NOT NULL DEFAULT 0,
134+ common_tags INT NOT NULL DEFAULT 0,
135+ PRIMARY KEY (user_a, user_b))"""
136+ g.db.exec "DELETE FROM _user_sim_staging"
137+ g.db.exec """
138+ INSERT INTO _user_sim_staging
139+ (user_a, user_b, jaccard, common_feeds, common_likes, common_tags)
140+ SELECT
141+ s1.user_did, s2.user_did,
142+ CAST(COUNT(*) AS REAL) / (
143+ (SELECT COUNT(*) FROM articles.subscriptions WHERE user_did = s1.user_did) +
144+ (SELECT COUNT(*) FROM articles.subscriptions WHERE user_did = s2.user_did) -
145+ CAST(COUNT(*) AS REAL)
146+ ),
147+ COUNT(*), 0, 0
148+ FROM articles.subscriptions s1
149+ JOIN articles.subscriptions s2
150+ ON s1.feed_url = s2.feed_url AND s1.user_did < s2.user_did
151+ GROUP BY s1.user_did, s2.user_did"""
152+
153+ # Shared likes, decayed on both sides: two people who liked the same
154+ # thing years apart have less in common than two who liked it the same
155+ # week, and the product of the two decays expresses that.
156+ #
157+ # ROUND, not CAST AS INTEGER. The Go implementation truncates, and since
158+ # each decay factor is at most 1 their product is always just under it --
159+ # so a pair sharing one recent like sums to 0.9999 and stores 0. That
160+ # number is shown to readers as the reason for a recommendation ("N
161+ # shared likes"), so truncation makes a real overlap read as none.
162+ g.db.exec """CREATE TEMP TABLE IF NOT EXISTS _likes_overlap (
163+ user_a TEXT, user_b TEXT, common INT, PRIMARY KEY(user_a, user_b))"""
164+ g.db.exec "DELETE FROM _likes_overlap"
165+ g.db.exec """
166+ INSERT INTO _likes_overlap (user_a, user_b, common)
167+ SELECT l1.author_did, l2.author_did,
168+ CAST(ROUND(SUM(
169+ EXP(-0.023 * CAST(julianday('now') - julianday(l1.created_at) AS REAL))
170+ * EXP(-0.023 * CAST(julianday('now') - julianday(l2.created_at) AS REAL))
171+ )) AS INTEGER)
172+ FROM articles.likes l1
173+ JOIN articles.likes l2
174+ ON l1.feed_url = l2.feed_url AND l1.article_url = l2.article_url
175+ AND l1.author_did < l2.author_did
176+ WHERE l1.created_at IS NOT NULL AND l2.created_at IS NOT NULL
177+ GROUP BY l1.author_did, l2.author_did"""
178+
179+ # A pair that shares likes but no feeds has no staging row yet, so this
180+ # inserts before it updates.
181+ g.db.exec """
182+ INSERT INTO _user_sim_staging (user_a, user_b, jaccard, common_feeds,
183+ common_likes, common_tags)
184+ SELECT o.user_a, o.user_b, 0, 0, o.common, 0
185+ FROM _likes_overlap o
186+ WHERE NOT EXISTS (SELECT 1 FROM _user_sim_staging s
187+ WHERE s.user_a = o.user_a AND s.user_b = o.user_b)"""
188+ g.db.exec """
189+ UPDATE _user_sim_staging SET common_likes = (
190+ SELECT o.common FROM _likes_overlap o
191+ WHERE o.user_a = _user_sim_staging.user_a
192+ AND o.user_b = _user_sim_staging.user_b)
193+ WHERE EXISTS (SELECT 1 FROM _likes_overlap o
194+ WHERE o.user_a = _user_sim_staging.user_a
195+ AND o.user_b = _user_sim_staging.user_b)"""
196+
197+ g.db.transaction:
198+ g.db.exec "DELETE FROM recs.user_similarity"
199+ g.db.exec """INSERT INTO recs.user_similarity
200+ (user_a, user_b, jaccard, common_feeds, common_likes, common_tags)
201+ SELECT user_a, user_b, jaccard, common_feeds, common_likes,
202+ common_tags
203+ FROM _user_sim_staging"""
204+
205+proc feedSimilarity*(g: GleanDb, feedA, feedB: string): float =
206+ ## Stored pairs are ordered, so look up whichever way round the caller asks.
207+ let (a, b) = if feedA < feedB: (feedA, feedB) else: (feedB, feedA)
208+ let row = g.db.queryFirst(
209+ "SELECT jaccard FROM recs.feed_similarity WHERE feed_a = ? AND feed_b = ?",
210+ [p(a), p(b)], proc(r: Row): float = r.f(0))
211+ if row.isSome: row.get else: 0.0
212+
213+proc userSimilarity*(g: GleanDb, userA, userB: string): float =
214+ let (a, b) = if userA < userB: (userA, userB) else: (userB, userA)
215+ let row = g.db.queryFirst(
216+ "SELECT jaccard FROM recs.user_similarity WHERE user_a = ? AND user_b = ?",
217+ [p(a), p(b)], proc(r: Row): float = r.f(0))
218+ if row.isSome: row.get else: 0.0
219+
220+# --- scoring helpers -------------------------------------------------------
221+
222+proc normalizeScores*(scores: var openArray[float]) =
223+ ## Rescale to 0..1.
224+ ##
225+ ## Scores from different signals are not comparable in magnitude -- a
226+ ## like-overlap sum and a log-popularity term live on unrelated scales --
227+ ## so they are normalised before being blended or ranked against each
228+ ## other. A single item, or a set where every score is equal, is left
229+ ## alone: there is no spread to stretch, and dividing by zero would turn
230+ ## the whole list into NaN.
231+ if scores.len < 2: return
232+ var lo = scores[0]
233+ var hi = scores[0]
234+ for s in scores:
235+ lo = min(lo, s)
236+ hi = max(hi, s)
237+ if hi == lo: return
238+ let span = hi - lo
239+ for i in 0 ..< scores.len:
240+ scores[i] = (scores[i] - lo) / span
241+
242+proc normalizeFeedScores*(recs: var seq[FeedRecommendation]) =
243+ var s = newSeq[float](recs.len)
244+ for i, r in recs: s[i] = r.score
245+ normalizeScores(s)
246+ for i in 0 ..< recs.len: recs[i].score = s[i]
247+
248+proc normalizePersonScores*(recs: var seq[PersonRecommendation]) =
249+ var s = newSeq[float](recs.len)
250+ for i, r in recs: s[i] = r.score
251+ normalizeScores(s)
252+ for i in 0 ..< recs.len: recs[i].score = s[i]
253+
254+proc samplePeople*(pool: seq[PersonRecommendation], half: int,
255+ rng: var Rand): seq[PersonRecommendation] =
256+ ## Up to `half` from inside the reader's network and `half` from outside.
257+ ##
258+ ## The split is the point: ranking people purely by similarity returns the
259+ ## reader's existing network, which they already have. Reserving half the
260+ ## slots for strangers is what makes this discovery rather than a mirror.
261+ ##
262+ ## Shuffled rather than taken in order so the same faces do not appear
263+ ## every time -- the pool changes far more slowly than the reader revisits.
264+ var inNet, outNet: seq[PersonRecommendation]
265+ for p in pool:
266+ if p.isFollowed: inNet.add p else: outNet.add p
267+
268+ rng.shuffle(inNet)
269+ rng.shuffle(outNet)
270+ if inNet.len > half: inNet.setLen(half)
271+ if outNet.len > half: outNet.setLen(half)
272+ inNet & outNet
273+
274+proc rankByScore*[T](items: var seq[T]) =
275+ ## Highest first, which is what every recommendation listing wants.
276+ items.sort(proc(a, b: T): int = cmp(b.score, a.score))
new file mode 100644
@@ -0,0 +1,276 @@
1+## The similarity computations behind recommendations, ported from
2+## internal/cluster/jaccard.go and the pure helpers of scoring.go.
3+##
4+## Two kinds of similarity are precomputed on a schedule, because both are
5+## O(pairs) and neither needs to be fresh to the second:
6+##
7+## feed_similarity which feeds are subscribed to by the same people
8+## user_similarity which people subscribe to the same feeds
9+##
10+## Both are time-decayed. An overlap from three years ago says much less
11+## about what someone wants now than one from last week, and without decay a
12+## long-lived account's early subscriptions dominate its recommendations
13+## forever.
14+##
15+## The decay constant is `exp(-0.023 * days)`, a half-life of about a month
16+## (ln 2 / 0.023 ≈ 30). It is written inline in the SQL rather than named,
17+## the same way the Go implementation writes it.
18+
19+import std/[algorithm, math, options, random]
20+import ./gleandb
21+import ./sqlite
22+
23+type
24+ Config* = object
25+ followBoost*: float
26+ likesWeight*: float
27+ tagsWeight*: float
28+ descriptionWeight*: float
29+
30+ SignalWeights* = object
31+ wSub*: float
32+ wLike*: float
33+ wTag*: float
34+ wSocial*: float
35+ wPop*: float
36+ wCategory*: float
37+ wContent*: float
38+
39+ FeedRecommendation* = object
40+ feedUrl*: string
41+ title*: string
42+ siteUrl*: string
43+ description*: string
44+ subscriberCount*: int
45+ faviconUrl*: string
46+ score*: float
47+
48+ PersonRecommendation* = object
49+ did*: string
50+ handle*: string
51+ displayName*: string
52+ avatarUrl*: string
53+ commonFeeds*: int
54+ commonLikes*: int
55+ commonTags*: int
56+ isFollowed*: bool
57+ score*: float
58+
59+const
60+ ## Half-life of roughly 30 days: ln(2) / 0.023.
61+ DecayPerDay* = 0.023
62+
63+proc defaultConfig*(): Config =
64+ Config(followBoost: 0.5, likesWeight: 0.3, tagsWeight: 0.2,
65+ descriptionWeight: 0.15)
66+
67+proc defaultWeights*(): SignalWeights =
68+ SignalWeights(wSub: 1.0, wLike: 0.5, wTag: 0.3, wSocial: 0.7,
69+ wPop: 0.2, wCategory: 0.4, wContent: 0.4)
70+
71+proc decay*(ageDays: float): float =
72+ ## The weight an event of this age still carries.
73+ ##
74+ ## Exposed so the decay can be checked directly; the queries compute it in
75+ ## SQL because pulling every like into the application to weight it would
76+ ## defeat the point of precomputing.
77+ exp(-DecayPerDay * ageDays)
78+
79+# --- similarity ------------------------------------------------------------
80+
81+## Both computations stage into a temp table and swap in a second
82+## transaction. The first pass is the expensive one and holds no lock on the
83+## live table; readers keep seeing the previous generation until the swap,
84+## rather than an empty table for the length of the computation.
85+
86+proc computeFeedSimilarity*(g: GleanDb) =
87+ ## For each pair of feeds, how strongly their subscriber sets overlap.
88+ ##
89+ ## The numerator is a decayed count of shared subscribers and the
90+ ## denominator is the union of the two subscriber counts, which makes this
91+ ## a Jaccard index with recency folded into the intersection.
92+ g.db.transaction:
93+ g.db.exec """CREATE TEMP TABLE IF NOT EXISTS _feed_sim_staging (
94+ feed_a TEXT NOT NULL, feed_b TEXT NOT NULL, jaccard REAL NOT NULL,
95+ PRIMARY KEY (feed_a, feed_b))"""
96+ g.db.exec "DELETE FROM _feed_sim_staging"
97+ g.db.exec """
98+ INSERT INTO _feed_sim_staging (feed_a, feed_b, jaccard)
99+ SELECT feed_a, feed_b, jaccard FROM (
100+ SELECT
101+ s1.feed_url AS feed_a,
102+ s2.feed_url AS feed_b,
103+ SUM(EXP(-0.023 * CAST(julianday('now') -
104+ julianday(MIN(s1.added_at, s2.added_at)) AS REAL)))
105+ / NULLIF(f1.subscriber_count + f2.subscriber_count
106+ - CAST(COUNT(*) AS REAL), 0) AS jaccard
107+ FROM articles.subscriptions s1
108+ -- feed_url < feed_url keeps one row per unordered pair, and drops
109+ -- the self-pair without a separate predicate.
110+ JOIN articles.subscriptions s2
111+ ON s1.user_did = s2.user_did AND s1.feed_url < s2.feed_url
112+ JOIN articles.feeds f1 ON f1.feed_url = s1.feed_url
113+ JOIN articles.feeds f2 ON f2.feed_url = s2.feed_url
114+ WHERE s1.added_at IS NOT NULL AND s2.added_at IS NOT NULL
115+ GROUP BY s1.feed_url, s2.feed_url
116+ ) WHERE jaccard IS NOT NULL"""
117+
118+ g.db.transaction:
119+ g.db.exec "DELETE FROM recs.feed_similarity"
120+ g.db.exec """INSERT INTO recs.feed_similarity (feed_a, feed_b, jaccard)
121+ SELECT feed_a, feed_b, jaccard FROM _feed_sim_staging"""
122+
123+proc computeUserSimilarity*(g: GleanDb) =
124+ ## For each pair of people, how strongly their subscriptions overlap.
125+ ##
126+ ## Undecayed, unlike feed similarity: a subscription is a standing choice
127+ ## rather than an event, so its age says little. Shared *likes* are decayed
128+ ## separately below, because those are events.
129+ g.db.transaction:
130+ g.db.exec """CREATE TEMP TABLE IF NOT EXISTS _user_sim_staging (
131+ user_a TEXT NOT NULL, user_b TEXT NOT NULL, jaccard REAL NOT NULL,
132+ common_feeds INT NOT NULL DEFAULT 0,
133+ common_likes INT NOT NULL DEFAULT 0,
134+ common_tags INT NOT NULL DEFAULT 0,
135+ PRIMARY KEY (user_a, user_b))"""
136+ g.db.exec "DELETE FROM _user_sim_staging"
137+ g.db.exec """
138+ INSERT INTO _user_sim_staging
139+ (user_a, user_b, jaccard, common_feeds, common_likes, common_tags)
140+ SELECT
141+ s1.user_did, s2.user_did,
142+ CAST(COUNT(*) AS REAL) / (
143+ (SELECT COUNT(*) FROM articles.subscriptions WHERE user_did = s1.user_did) +
144+ (SELECT COUNT(*) FROM articles.subscriptions WHERE user_did = s2.user_did) -
145+ CAST(COUNT(*) AS REAL)
146+ ),
147+ COUNT(*), 0, 0
148+ FROM articles.subscriptions s1
149+ JOIN articles.subscriptions s2
150+ ON s1.feed_url = s2.feed_url AND s1.user_did < s2.user_did
151+ GROUP BY s1.user_did, s2.user_did"""
152+
153+ # Shared likes, decayed on both sides: two people who liked the same
154+ # thing years apart have less in common than two who liked it the same
155+ # week, and the product of the two decays expresses that.
156+ #
157+ # ROUND, not CAST AS INTEGER. The Go implementation truncates, and since
158+ # each decay factor is at most 1 their product is always just under it --
159+ # so a pair sharing one recent like sums to 0.9999 and stores 0. That
160+ # number is shown to readers as the reason for a recommendation ("N
161+ # shared likes"), so truncation makes a real overlap read as none.
162+ g.db.exec """CREATE TEMP TABLE IF NOT EXISTS _likes_overlap (
163+ user_a TEXT, user_b TEXT, common INT, PRIMARY KEY(user_a, user_b))"""
164+ g.db.exec "DELETE FROM _likes_overlap"
165+ g.db.exec """
166+ INSERT INTO _likes_overlap (user_a, user_b, common)
167+ SELECT l1.author_did, l2.author_did,
168+ CAST(ROUND(SUM(
169+ EXP(-0.023 * CAST(julianday('now') - julianday(l1.created_at) AS REAL))
170+ * EXP(-0.023 * CAST(julianday('now') - julianday(l2.created_at) AS REAL))
171+ )) AS INTEGER)
172+ FROM articles.likes l1
173+ JOIN articles.likes l2
174+ ON l1.feed_url = l2.feed_url AND l1.article_url = l2.article_url
175+ AND l1.author_did < l2.author_did
176+ WHERE l1.created_at IS NOT NULL AND l2.created_at IS NOT NULL
177+ GROUP BY l1.author_did, l2.author_did"""
178+
179+ # A pair that shares likes but no feeds has no staging row yet, so this
180+ # inserts before it updates.
181+ g.db.exec """
182+ INSERT INTO _user_sim_staging (user_a, user_b, jaccard, common_feeds,
183+ common_likes, common_tags)
184+ SELECT o.user_a, o.user_b, 0, 0, o.common, 0
185+ FROM _likes_overlap o
186+ WHERE NOT EXISTS (SELECT 1 FROM _user_sim_staging s
187+ WHERE s.user_a = o.user_a AND s.user_b = o.user_b)"""
188+ g.db.exec """
189+ UPDATE _user_sim_staging SET common_likes = (
190+ SELECT o.common FROM _likes_overlap o
191+ WHERE o.user_a = _user_sim_staging.user_a
192+ AND o.user_b = _user_sim_staging.user_b)
193+ WHERE EXISTS (SELECT 1 FROM _likes_overlap o
194+ WHERE o.user_a = _user_sim_staging.user_a
195+ AND o.user_b = _user_sim_staging.user_b)"""
196+
197+ g.db.transaction:
198+ g.db.exec "DELETE FROM recs.user_similarity"
199+ g.db.exec """INSERT INTO recs.user_similarity
200+ (user_a, user_b, jaccard, common_feeds, common_likes, common_tags)
201+ SELECT user_a, user_b, jaccard, common_feeds, common_likes,
202+ common_tags
203+ FROM _user_sim_staging"""
204+
205+proc feedSimilarity*(g: GleanDb, feedA, feedB: string): float =
206+ ## Stored pairs are ordered, so look up whichever way round the caller asks.
207+ let (a, b) = if feedA < feedB: (feedA, feedB) else: (feedB, feedA)
208+ let row = g.db.queryFirst(
209+ "SELECT jaccard FROM recs.feed_similarity WHERE feed_a = ? AND feed_b = ?",
210+ [p(a), p(b)], proc(r: Row): float = r.f(0))
211+ if row.isSome: row.get else: 0.0
212+
213+proc userSimilarity*(g: GleanDb, userA, userB: string): float =
214+ let (a, b) = if userA < userB: (userA, userB) else: (userB, userA)
215+ let row = g.db.queryFirst(
216+ "SELECT jaccard FROM recs.user_similarity WHERE user_a = ? AND user_b = ?",
217+ [p(a), p(b)], proc(r: Row): float = r.f(0))
218+ if row.isSome: row.get else: 0.0
219+
220+# --- scoring helpers -------------------------------------------------------
221+
222+proc normalizeScores*(scores: var openArray[float]) =
223+ ## Rescale to 0..1.
224+ ##
225+ ## Scores from different signals are not comparable in magnitude -- a
226+ ## like-overlap sum and a log-popularity term live on unrelated scales --
227+ ## so they are normalised before being blended or ranked against each
228+ ## other. A single item, or a set where every score is equal, is left
229+ ## alone: there is no spread to stretch, and dividing by zero would turn
230+ ## the whole list into NaN.
231+ if scores.len < 2: return
232+ var lo = scores[0]
233+ var hi = scores[0]
234+ for s in scores:
235+ lo = min(lo, s)
236+ hi = max(hi, s)
237+ if hi == lo: return
238+ let span = hi - lo
239+ for i in 0 ..< scores.len:
240+ scores[i] = (scores[i] - lo) / span
241+
242+proc normalizeFeedScores*(recs: var seq[FeedRecommendation]) =
243+ var s = newSeq[float](recs.len)
244+ for i, r in recs: s[i] = r.score
245+ normalizeScores(s)
246+ for i in 0 ..< recs.len: recs[i].score = s[i]
247+
248+proc normalizePersonScores*(recs: var seq[PersonRecommendation]) =
249+ var s = newSeq[float](recs.len)
250+ for i, r in recs: s[i] = r.score
251+ normalizeScores(s)
252+ for i in 0 ..< recs.len: recs[i].score = s[i]
253+
254+proc samplePeople*(pool: seq[PersonRecommendation], half: int,
255+ rng: var Rand): seq[PersonRecommendation] =
256+ ## Up to `half` from inside the reader's network and `half` from outside.
257+ ##
258+ ## The split is the point: ranking people purely by similarity returns the
259+ ## reader's existing network, which they already have. Reserving half the
260+ ## slots for strangers is what makes this discovery rather than a mirror.
261+ ##
262+ ## Shuffled rather than taken in order so the same faces do not appear
263+ ## every time -- the pool changes far more slowly than the reader revisits.
264+ var inNet, outNet: seq[PersonRecommendation]
265+ for p in pool:
266+ if p.isFollowed: inNet.add p else: outNet.add p
267+
268+ rng.shuffle(inNet)
269+ rng.shuffle(outNet)
270+ if inNet.len > half: inNet.setLen(half)
271+ if outNet.len > half: outNet.setLen(half)
272+ inNet & outNet
273+
274+proc rankByScore*[T](items: var seq[T]) =
275+ ## Highest first, which is what every recommendation listing wants.
276+ items.sort(proc(a, b: T): int = cmp(b.score, a.score))
modified spike/nim/atproto/gleandb.nim +6 -3
@@ -33,9 +33,12 @@ const
3333 SQLITE_UTF8 = 1.cint
3434 SQLITE_DETERMINISTIC = 0x000000800.cint
3535
36-# Registered because the Go connection registers them. Nothing in the current
37-# SQL calls either -- they exist so a scoring query can decay a weight
38-# exponentially without pulling the rows into the application first.
36+# SQLite ships maths functions only when built with
37+# SQLITE_ENABLE_MATH_FUNCTIONS, which the library here is not, so these are
38+# registered exactly as the Go connection registers them. They are not
39+# optional: the clustering SQL calls EXP and LOG in seven places for time
40+# decay and popularity normalisation, and without them every recommendation
41+# query fails at runtime rather than at startup.
3942 proc sqlExp(ctx: pointer, n: cint, args: ptr UncheckedArray[pointer])
4043 {.cdecl.} =
4144 if n != 1:
@@ -33,9 +33,12 @@ const
33 SQLITE_UTF8 = 1.cint33 SQLITE_UTF8 = 1.cint
34 SQLITE_DETERMINISTIC = 0x000000800.cint34 SQLITE_DETERMINISTIC = 0x000000800.cint
35 35
36-# Registered because the Go connection registers them. Nothing in the current36+# SQLite ships maths functions only when built with
37-# SQL calls either -- they exist so a scoring query can decay a weight37+# SQLITE_ENABLE_MATH_FUNCTIONS, which the library here is not, so these are
38-# exponentially without pulling the rows into the application first.38+# registered exactly as the Go connection registers them. They are not
39+# optional: the clustering SQL calls EXP and LOG in seven places for time
40+# decay and popularity normalisation, and without them every recommendation
41+# query fails at runtime rather than at startup.
39 proc sqlExp(ctx: pointer, n: cint, args: ptr UncheckedArray[pointer])42 proc sqlExp(ctx: pointer, n: cint, args: ptr UncheckedArray[pointer])
40 {.cdecl.} =43 {.cdecl.} =
41 if n != 1:44 if n != 1:
added spike/nim/cluster_probe.nim +220 -0
new file mode 100644
@@ -0,0 +1,220 @@
1+## Checks the similarity computations and scoring helpers.
2+##
3+## nim c -r -d:ssl cluster_probe.nim
4+##
5+## Recommendations are the one part of this system with no obviously correct
6+## answer, so the checks are about properties rather than values: that decay
7+## actually decays, that a pair is counted once, that normalisation does not
8+## produce NaN, that discovery is not just a mirror of who you already follow.
9+
10+import std/[math, options, os, random, sequtils, strformat, strutils, times]
11+import atproto/[articlestore, cluster, feedstore, gleandb, socialstore, sqlite]
12+
13+var failures = 0
14+
15+proc report(name: string, ok: bool, detail = "") =
16+ if not ok: inc failures
17+ let label = if ok: "PASS" else: "FAIL"
18+ if detail.len > 0: echo &" {label} {name} -- {detail}"
19+ else: echo &" {label} {name}"
20+
21+proc cleanup(base: string) =
22+ for suffix in ["_users", "_articles", "_recs"]:
23+ for ext in ["", "-wal", "-shm"]:
24+ removeFile(base & suffix & ext)
25+
26+const
27+ alice = "did:plc:alice"
28+ bob = "did:plc:bob"
29+ carol = "did:plc:carol"
30+
31+proc main() =
32+ echo "Clustering probe"
33+ echo ""
34+
35+ # --- pure helpers -------------------------------------------------------
36+ block:
37+ report("decay at zero days is 1", abs(decay(0.0) - 1.0) < 1e-12)
38+ # ln(2)/0.023 is about 30 days, so a month-old event is worth half.
39+ report("decay halves in about a month",
40+ abs(decay(30.1) - 0.5) < 0.01, &"{decay(30.1):.4f}")
41+ report("decay keeps shrinking", decay(365.0) < decay(30.0))
42+ report("decay never reaches zero", decay(3650.0) > 0.0,
43+ &"{decay(3650.0):.3e}")
44+
45+ block:
46+ var s = @[0.0, 5.0, 10.0]
47+ normalizeScores(s)
48+ report("normalisation maps to 0..1", s == @[0.0, 0.5, 1.0], $s)
49+
50+ # The case that would produce NaN: no spread to divide by.
51+ var flat = @[3.0, 3.0, 3.0]
52+ normalizeScores(flat)
53+ report("an all-equal set is left alone", flat == @[3.0, 3.0, 3.0], $flat)
54+ for v in flat:
55+ report("and produces no NaN", not v.isNaN)
56+
57+ var one = @[7.0]
58+ normalizeScores(one)
59+ report("a single score is left alone", one == @[7.0])
60+
61+ var empty: seq[float] = @[]
62+ normalizeScores(empty)
63+ report("an empty set does not crash", empty.len == 0)
64+
65+ var negatives = @[-10.0, -5.0, 0.0]
66+ normalizeScores(negatives)
67+ report("negative scores normalise too", negatives == @[0.0, 0.5, 1.0],
68+ $negatives)
69+
70+ block:
71+ var rng = initRand(42)
72+ var pool: seq[PersonRecommendation]
73+ for i in 0 ..< 10:
74+ pool.add PersonRecommendation(did: &"did:plc:in{i}", isFollowed: true)
75+ for i in 0 ..< 10:
76+ pool.add PersonRecommendation(did: &"did:plc:out{i}", isFollowed: false)
77+
78+ let sampled = samplePeople(pool, 3, rng)
79+ let followed = sampled.countIt(it.isFollowed)
80+ let strangers = sampled.len - followed
81+ report("sampling caps each side at the half", sampled.len == 6,
82+ $sampled.len)
83+ # Ranking purely by similarity returns the network the reader already
84+ # has; reserving half the slots is what makes this discovery.
85+ report("half the slots go to people not followed",
86+ followed == 3 and strangers == 3,
87+ &"{followed} followed, {strangers} strangers")
88+
89+ # Fewer than `half` on one side must not pad from the other.
90+ var lopsided: seq[PersonRecommendation]
91+ lopsided.add PersonRecommendation(did: "did:plc:only", isFollowed: true)
92+ for i in 0 ..< 10:
93+ lopsided.add PersonRecommendation(did: &"did:plc:s{i}", isFollowed: false)
94+ let small = samplePeople(lopsided, 3, rng)
95+ report("a thin side is not padded from the other",
96+ small.countIt(it.isFollowed) == 1 and small.len == 4, $small.len)
97+
98+ # Shuffled, so the same faces do not recur every visit.
99+ var rngA = initRand(1)
100+ var rngB = initRand(2)
101+ let a = samplePeople(pool, 3, rngA).mapIt(it.did).join(",")
102+ let b = samplePeople(pool, 3, rngB).mapIt(it.did).join(",")
103+ report("sampling varies between runs", a != b)
104+
105+ block:
106+ var recs = @[
107+ FeedRecommendation(feedUrl: "c", score: 0.1),
108+ FeedRecommendation(feedUrl: "a", score: 0.9),
109+ FeedRecommendation(feedUrl: "b", score: 0.5),
110+ ]
111+ rankByScore(recs)
112+ report("ranking puts the highest score first",
113+ recs.mapIt(it.feedUrl) == @["a", "b", "c"],
114+ recs.mapIt(it.feedUrl).join(","))
115+
116+ # --- similarity against a real database ---------------------------------
117+ let base = getTempDir() / "glean_cluster_probe"
118+ cleanup(base)
119+ defer: cleanup(base)
120+
121+ var g = gleandb.open(base)
122+ defer: g.close()
123+ g.migrate()
124+
125+ for did in [alice, bob, carol]:
126+ g.db.run("INSERT INTO users (did) VALUES (?)", p(did))
127+
128+ # alice and bob share two feeds; carol shares one with each.
129+ for i in 1 .. 3:
130+ g.upsertFeed(Feed(feedUrl: &"https://f{i}.test/feed", title: &"Feed {i}",
131+ feedType: "rss"))
132+ g.createSubscription(alice, "https://f1.test/feed", "", "", "", "")
133+ g.createSubscription(alice, "https://f2.test/feed", "", "", "", "")
134+ g.createSubscription(bob, "https://f1.test/feed", "", "", "", "")
135+ g.createSubscription(bob, "https://f2.test/feed", "", "", "", "")
136+ g.createSubscription(carol, "https://f1.test/feed", "", "", "", "")
137+ g.createSubscription(carol, "https://f3.test/feed", "", "", "", "")
138+
139+ block:
140+ g.computeUserSimilarity()
141+ let ab = g.userSimilarity(alice, bob)
142+ let ac = g.userSimilarity(alice, carol)
143+ # alice/bob share 2 of 2; alice/carol share 1 of 3.
144+ report("identical subscribers score 1.0", abs(ab - 1.0) < 1e-9, &"{ab:.4f}")
145+ report("partial overlap scores lower",
146+ abs(ac - (1.0 / 3.0)) < 1e-9, &"{ac:.4f}")
147+ report("similarity is symmetric on lookup",
148+ abs(g.userSimilarity(bob, alice) - ab) < 1e-12)
149+ report("an unrelated pair scores zero",
150+ g.userSimilarity(alice, "did:plc:nobody") == 0.0)
151+
152+ # One row per unordered pair, or every score would be counted twice
153+ # downstream.
154+ let rows = g.db.queryInt("SELECT count(*) FROM recs.user_similarity").get(0)
155+ report("each pair is stored once", rows == 3, &"{rows} rows")
156+
157+ block:
158+ # Recomputing must replace rather than accumulate.
159+ g.computeUserSimilarity()
160+ let rows = g.db.queryInt("SELECT count(*) FROM recs.user_similarity").get(0)
161+ report("recomputing replaces the previous generation", rows == 3,
162+ &"{rows} rows")
163+
164+ block:
165+ g.computeFeedSimilarity()
166+ let f12 = g.feedSimilarity("https://f1.test/feed", "https://f2.test/feed")
167+ let f13 = g.feedSimilarity("https://f1.test/feed", "https://f3.test/feed")
168+ # f1 and f2 share alice and bob; f1 and f3 share only carol.
169+ report("feeds with more shared subscribers score higher", f12 > f13,
170+ &"f1/f2={f12:.4f} f1/f3={f13:.4f}")
171+ let reversed = g.feedSimilarity("https://f2.test/feed", "https://f1.test/feed")
172+ report("feed similarity is symmetric on lookup", abs(reversed - f12) < 1e-12)
173+ report("an unshared pair is absent",
174+ g.feedSimilarity("https://f2.test/feed", "https://f3.test/feed") == 0.0)
175+
176+ # --- decay in the query, not just in Nim --------------------------------
177+ block:
178+ # Two pairs of people who liked the same articles, one recently and one
179+ # long ago. The old overlap must weigh less -- that is the whole reason
180+ # EXP is registered as a SQL function.
181+ g.upsertFeed(Feed(feedUrl: "https://n.test/feed", title: "News"))
182+ let fresh = now().utc
183+ let stale = now().utc - initDuration(days = 400)
184+ for (who, at, n) in [(alice, fresh, "r"), (bob, fresh, "r"),
185+ (carol, stale, "s")]:
186+ g.createLike(Like(uri: &"at://{who}/like/{n}", authorDid: who,
187+ feedUrl: "https://n.test/feed",
188+ articleUrl: &"https://n.test/{n}",
189+ createdAt: some(at)))
190+ # carol needs a partner on the stale article for a pair to exist.
191+ g.createLike(Like(uri: "at://dave/like/s", authorDid: "did:plc:dave",
192+ feedUrl: "https://n.test/feed",
193+ articleUrl: "https://n.test/s",
194+ createdAt: some(stale)))
195+
196+ g.computeUserSimilarity()
197+ proc overlapOf(a, b: string): float =
198+ let (x, y) = if a < b: (a, b) else: (b, a)
199+ g.db.queryFirst(
200+ "SELECT common_likes FROM recs.user_similarity WHERE user_a = ? AND user_b = ?",
201+ [p(x), p(y)], proc(r: Row): float = r.f(0)).get(-1.0)
202+
203+ let recent = overlapOf(alice, bob)
204+ let old = overlapOf(carol, "did:plc:dave")
205+ # The check that was vacuous before: both sides were 0 because the Go
206+ # query truncates a decayed sum that is always just under 1.
207+ report("a recent shared like counts as one", recent == 1.0, &"{recent}")
208+ report("a 400-day-old shared like decays to nothing", old == 0.0, &"{old}")
209+ report("decay makes old overlap weigh less than new", recent > old,
210+ &"recent={recent} old={old}")
211+
212+ echo ""
213+ if failures == 0:
214+ echo "RESULT: similarity and scoring behave."
215+ else:
216+ echo &"RESULT: {failures} check(s) failed."
217+ quit 1
218+
219+when isMainModule:
220+ main()
new file mode 100644
@@ -0,0 +1,220 @@
1+## Checks the similarity computations and scoring helpers.
2+##
3+## nim c -r -d:ssl cluster_probe.nim
4+##
5+## Recommendations are the one part of this system with no obviously correct
6+## answer, so the checks are about properties rather than values: that decay
7+## actually decays, that a pair is counted once, that normalisation does not
8+## produce NaN, that discovery is not just a mirror of who you already follow.
9+
10+import std/[math, options, os, random, sequtils, strformat, strutils, times]
11+import atproto/[articlestore, cluster, feedstore, gleandb, socialstore, sqlite]
12+
13+var failures = 0
14+
15+proc report(name: string, ok: bool, detail = "") =
16+ if not ok: inc failures
17+ let label = if ok: "PASS" else: "FAIL"
18+ if detail.len > 0: echo &" {label} {name} -- {detail}"
19+ else: echo &" {label} {name}"
20+
21+proc cleanup(base: string) =
22+ for suffix in ["_users", "_articles", "_recs"]:
23+ for ext in ["", "-wal", "-shm"]:
24+ removeFile(base & suffix & ext)
25+
26+const
27+ alice = "did:plc:alice"
28+ bob = "did:plc:bob"
29+ carol = "did:plc:carol"
30+
31+proc main() =
32+ echo "Clustering probe"
33+ echo ""
34+
35+ # --- pure helpers -------------------------------------------------------
36+ block:
37+ report("decay at zero days is 1", abs(decay(0.0) - 1.0) < 1e-12)
38+ # ln(2)/0.023 is about 30 days, so a month-old event is worth half.
39+ report("decay halves in about a month",
40+ abs(decay(30.1) - 0.5) < 0.01, &"{decay(30.1):.4f}")
41+ report("decay keeps shrinking", decay(365.0) < decay(30.0))
42+ report("decay never reaches zero", decay(3650.0) > 0.0,
43+ &"{decay(3650.0):.3e}")
44+
45+ block:
46+ var s = @[0.0, 5.0, 10.0]
47+ normalizeScores(s)
48+ report("normalisation maps to 0..1", s == @[0.0, 0.5, 1.0], $s)
49+
50+ # The case that would produce NaN: no spread to divide by.
51+ var flat = @[3.0, 3.0, 3.0]
52+ normalizeScores(flat)
53+ report("an all-equal set is left alone", flat == @[3.0, 3.0, 3.0], $flat)
54+ for v in flat:
55+ report("and produces no NaN", not v.isNaN)
56+
57+ var one = @[7.0]
58+ normalizeScores(one)
59+ report("a single score is left alone", one == @[7.0])
60+
61+ var empty: seq[float] = @[]
62+ normalizeScores(empty)
63+ report("an empty set does not crash", empty.len == 0)
64+
65+ var negatives = @[-10.0, -5.0, 0.0]
66+ normalizeScores(negatives)
67+ report("negative scores normalise too", negatives == @[0.0, 0.5, 1.0],
68+ $negatives)
69+
70+ block:
71+ var rng = initRand(42)
72+ var pool: seq[PersonRecommendation]
73+ for i in 0 ..< 10:
74+ pool.add PersonRecommendation(did: &"did:plc:in{i}", isFollowed: true)
75+ for i in 0 ..< 10:
76+ pool.add PersonRecommendation(did: &"did:plc:out{i}", isFollowed: false)
77+
78+ let sampled = samplePeople(pool, 3, rng)
79+ let followed = sampled.countIt(it.isFollowed)
80+ let strangers = sampled.len - followed
81+ report("sampling caps each side at the half", sampled.len == 6,
82+ $sampled.len)
83+ # Ranking purely by similarity returns the network the reader already
84+ # has; reserving half the slots is what makes this discovery.
85+ report("half the slots go to people not followed",
86+ followed == 3 and strangers == 3,
87+ &"{followed} followed, {strangers} strangers")
88+
89+ # Fewer than `half` on one side must not pad from the other.
90+ var lopsided: seq[PersonRecommendation]
91+ lopsided.add PersonRecommendation(did: "did:plc:only", isFollowed: true)
92+ for i in 0 ..< 10:
93+ lopsided.add PersonRecommendation(did: &"did:plc:s{i}", isFollowed: false)
94+ let small = samplePeople(lopsided, 3, rng)
95+ report("a thin side is not padded from the other",
96+ small.countIt(it.isFollowed) == 1 and small.len == 4, $small.len)
97+
98+ # Shuffled, so the same faces do not recur every visit.
99+ var rngA = initRand(1)
100+ var rngB = initRand(2)
101+ let a = samplePeople(pool, 3, rngA).mapIt(it.did).join(",")
102+ let b = samplePeople(pool, 3, rngB).mapIt(it.did).join(",")
103+ report("sampling varies between runs", a != b)
104+
105+ block:
106+ var recs = @[
107+ FeedRecommendation(feedUrl: "c", score: 0.1),
108+ FeedRecommendation(feedUrl: "a", score: 0.9),
109+ FeedRecommendation(feedUrl: "b", score: 0.5),
110+ ]
111+ rankByScore(recs)
112+ report("ranking puts the highest score first",
113+ recs.mapIt(it.feedUrl) == @["a", "b", "c"],
114+ recs.mapIt(it.feedUrl).join(","))
115+
116+ # --- similarity against a real database ---------------------------------
117+ let base = getTempDir() / "glean_cluster_probe"
118+ cleanup(base)
119+ defer: cleanup(base)
120+
121+ var g = gleandb.open(base)
122+ defer: g.close()
123+ g.migrate()
124+
125+ for did in [alice, bob, carol]:
126+ g.db.run("INSERT INTO users (did) VALUES (?)", p(did))
127+
128+ # alice and bob share two feeds; carol shares one with each.
129+ for i in 1 .. 3:
130+ g.upsertFeed(Feed(feedUrl: &"https://f{i}.test/feed", title: &"Feed {i}",
131+ feedType: "rss"))
132+ g.createSubscription(alice, "https://f1.test/feed", "", "", "", "")
133+ g.createSubscription(alice, "https://f2.test/feed", "", "", "", "")
134+ g.createSubscription(bob, "https://f1.test/feed", "", "", "", "")
135+ g.createSubscription(bob, "https://f2.test/feed", "", "", "", "")
136+ g.createSubscription(carol, "https://f1.test/feed", "", "", "", "")
137+ g.createSubscription(carol, "https://f3.test/feed", "", "", "", "")
138+
139+ block:
140+ g.computeUserSimilarity()
141+ let ab = g.userSimilarity(alice, bob)
142+ let ac = g.userSimilarity(alice, carol)
143+ # alice/bob share 2 of 2; alice/carol share 1 of 3.
144+ report("identical subscribers score 1.0", abs(ab - 1.0) < 1e-9, &"{ab:.4f}")
145+ report("partial overlap scores lower",
146+ abs(ac - (1.0 / 3.0)) < 1e-9, &"{ac:.4f}")
147+ report("similarity is symmetric on lookup",
148+ abs(g.userSimilarity(bob, alice) - ab) < 1e-12)
149+ report("an unrelated pair scores zero",
150+ g.userSimilarity(alice, "did:plc:nobody") == 0.0)
151+
152+ # One row per unordered pair, or every score would be counted twice
153+ # downstream.
154+ let rows = g.db.queryInt("SELECT count(*) FROM recs.user_similarity").get(0)
155+ report("each pair is stored once", rows == 3, &"{rows} rows")
156+
157+ block:
158+ # Recomputing must replace rather than accumulate.
159+ g.computeUserSimilarity()
160+ let rows = g.db.queryInt("SELECT count(*) FROM recs.user_similarity").get(0)
161+ report("recomputing replaces the previous generation", rows == 3,
162+ &"{rows} rows")
163+
164+ block:
165+ g.computeFeedSimilarity()
166+ let f12 = g.feedSimilarity("https://f1.test/feed", "https://f2.test/feed")
167+ let f13 = g.feedSimilarity("https://f1.test/feed", "https://f3.test/feed")
168+ # f1 and f2 share alice and bob; f1 and f3 share only carol.
169+ report("feeds with more shared subscribers score higher", f12 > f13,
170+ &"f1/f2={f12:.4f} f1/f3={f13:.4f}")
171+ let reversed = g.feedSimilarity("https://f2.test/feed", "https://f1.test/feed")
172+ report("feed similarity is symmetric on lookup", abs(reversed - f12) < 1e-12)
173+ report("an unshared pair is absent",
174+ g.feedSimilarity("https://f2.test/feed", "https://f3.test/feed") == 0.0)
175+
176+ # --- decay in the query, not just in Nim --------------------------------
177+ block:
178+ # Two pairs of people who liked the same articles, one recently and one
179+ # long ago. The old overlap must weigh less -- that is the whole reason
180+ # EXP is registered as a SQL function.
181+ g.upsertFeed(Feed(feedUrl: "https://n.test/feed", title: "News"))
182+ let fresh = now().utc
183+ let stale = now().utc - initDuration(days = 400)
184+ for (who, at, n) in [(alice, fresh, "r"), (bob, fresh, "r"),
185+ (carol, stale, "s")]:
186+ g.createLike(Like(uri: &"at://{who}/like/{n}", authorDid: who,
187+ feedUrl: "https://n.test/feed",
188+ articleUrl: &"https://n.test/{n}",
189+ createdAt: some(at)))
190+ # carol needs a partner on the stale article for a pair to exist.
191+ g.createLike(Like(uri: "at://dave/like/s", authorDid: "did:plc:dave",
192+ feedUrl: "https://n.test/feed",
193+ articleUrl: "https://n.test/s",
194+ createdAt: some(stale)))
195+
196+ g.computeUserSimilarity()
197+ proc overlapOf(a, b: string): float =
198+ let (x, y) = if a < b: (a, b) else: (b, a)
199+ g.db.queryFirst(
200+ "SELECT common_likes FROM recs.user_similarity WHERE user_a = ? AND user_b = ?",
201+ [p(x), p(y)], proc(r: Row): float = r.f(0)).get(-1.0)
202+
203+ let recent = overlapOf(alice, bob)
204+ let old = overlapOf(carol, "did:plc:dave")
205+ # The check that was vacuous before: both sides were 0 because the Go
206+ # query truncates a decayed sum that is always just under 1.
207+ report("a recent shared like counts as one", recent == 1.0, &"{recent}")
208+ report("a 400-day-old shared like decays to nothing", old == 0.0, &"{old}")
209+ report("decay makes old overlap weigh less than new", recent > old,
210+ &"recent={recent} old={old}")
211+
212+ echo ""
213+ if failures == 0:
214+ echo "RESULT: similarity and scoring behave."
215+ else:
216+ echo &"RESULT: {failures} check(s) failed."
217+ quit 1
218+
219+when isMainModule:
220+ main()
modified spike/nim/glean_spike.nimble +1 -0
@@ -18,6 +18,7 @@ const offlineProbes = @[
1818 "gleandb_probe",
1919 "stores_probe",
2020 "social_probe",
21+ "cluster_probe",
2122 "reconnect_probe", # uses a local websocket server, not the network
2223 ]
2324
@@ -18,6 +18,7 @@ const offlineProbes = @[
18 "gleandb_probe",18 "gleandb_probe",
19 "stores_probe",19 "stores_probe",
20 "social_probe",20 "social_probe",
21+ "cluster_probe",
21 "reconnect_probe", # uses a local websocket server, not the network22 "reconnect_probe", # uses a local websocket server, not the network
22 ]23 ]
23 24
modified spike/nim/readme.md +48 -3
@@ -264,9 +264,16 @@ worth knowing: `PRAGMA` applies to one database at a time, so setting WAL on
264264 all three rather than trusting the call.
265265
266266 `exp()` and `log()` are registered as SQL functions, as the Go connection
267-does. Nothing in the current SQL calls either; they exist so a scoring query
268-can decay a weight without pulling rows into the application. `log()` returns
269-NULL for zero and negatives, since `-inf` and NaN both poison an `ORDER BY`.
267+does, and they are load-bearing: the clustering SQL calls `EXP` and `LOG` in
268+seven places for time decay and popularity normalisation. SQLite only ships
269+maths functions when built with `SQLITE_ENABLE_MATH_FUNCTIONS`, so without
270+these registrations every recommendation query fails at runtime rather than
271+at startup.
272+
273+`log()` returns NULL for zero and negatives where Go's `math.Log` returns
274+`-inf` and NaN. The call sites all pass `1 + count`, so the argument is never
275+below 1 and the two agree in practice -- but it is a divergence, noted here
276+rather than left to be found.
270277
271278 **The schema is extracted, not retyped.** `tools/extract_schema.py` pulls all
272279 44 DDL statements out of `internal/db/db.go` into `atproto/schema.nim`.
@@ -485,6 +492,44 @@ The live check takes its URL from the Rust blog's feed rather than
485492 hard-coding one, after a hard-coded guess 404'd -- a probe that rots when
486493 someone reorganises their permalinks is worse than no probe.
487494
495+## 14. Similarity and scoring
496+
497+`atproto/cluster.nim` ports the similarity computations from
498+`internal/cluster/jaccard.go` and the pure helpers from `scoring.go`.
499+
500+Both similarities are precomputed on a schedule and staged into a temp table
501+before being swapped in, so the expensive pass holds no lock on the live
502+table and readers keep seeing the previous generation rather than an empty
503+one.
504+
505+Feed similarity is time-decayed, `exp(-0.023 * days)` -- a half-life of about
506+a month. Without it a long-lived account's earliest subscriptions dominate
507+its recommendations forever. User similarity over *subscriptions* is
508+deliberately undecayed, because a subscription is a standing choice rather
509+than an event; shared *likes* are events, so those decay on both sides.
510+
511+Recommendations have no obviously correct answer, so the probe checks
512+properties rather than values: that decay decays, that each pair is stored
513+once, that recomputing replaces rather than accumulates, that normalisation
514+of an all-equal set does not produce NaN, and that `samplePeople` reserves
515+half its slots for people the reader does *not* follow -- ranking purely by
516+similarity returns the network they already have, which is not discovery.
517+
518+### A bug inherited from the Go implementation
519+
520+`common_likes` is computed as `CAST(SUM(decay * decay) AS INTEGER)`. Each
521+decay factor is at most 1, so their product is always just under it: a pair
522+sharing one recent like sums to 0.9999 and **truncates to 0**. Two shared
523+likes store 1. The number is shown to readers as the reason for a
524+recommendation -- "N shared likes" -- so a real overlap reads as none.
525+
526+This port uses `ROUND` instead. Worth fixing on the Go side too.
527+
528+The probe caught it only because the assertion looked *vacuous*: both the
529+recent and the stale pair returned 0, so "recent >= old" passed while
530+proving nothing. A test that passes for the wrong reason is worse than one
531+that fails.
532+
488533 ## What this does and does not prove
489534
490535 Proven: the cryptography, the storage layer, and the OAuth flow up to user
@@ -264,9 +264,16 @@ worth knowing: `PRAGMA` applies to one database at a time, so setting WAL on
264 all three rather than trusting the call.264 all three rather than trusting the call.
265 265
266 `exp()` and `log()` are registered as SQL functions, as the Go connection266 `exp()` and `log()` are registered as SQL functions, as the Go connection
267-does. Nothing in the current SQL calls either; they exist so a scoring query267+does, and they are load-bearing: the clustering SQL calls `EXP` and `LOG` in
268-can decay a weight without pulling rows into the application. `log()` returns268+seven places for time decay and popularity normalisation. SQLite only ships
269-NULL for zero and negatives, since `-inf` and NaN both poison an `ORDER BY`.269+maths functions when built with `SQLITE_ENABLE_MATH_FUNCTIONS`, so without
270+these registrations every recommendation query fails at runtime rather than
271+at startup.
272+
273+`log()` returns NULL for zero and negatives where Go's `math.Log` returns
274+`-inf` and NaN. The call sites all pass `1 + count`, so the argument is never
275+below 1 and the two agree in practice -- but it is a divergence, noted here
276+rather than left to be found.
270 277
271 **The schema is extracted, not retyped.** `tools/extract_schema.py` pulls all278 **The schema is extracted, not retyped.** `tools/extract_schema.py` pulls all
272 44 DDL statements out of `internal/db/db.go` into `atproto/schema.nim`.279 44 DDL statements out of `internal/db/db.go` into `atproto/schema.nim`.
@@ -485,6 +492,44 @@ The live check takes its URL from the Rust blog's feed rather than
485 hard-coding one, after a hard-coded guess 404'd -- a probe that rots when492 hard-coding one, after a hard-coded guess 404'd -- a probe that rots when
486 someone reorganises their permalinks is worse than no probe.493 someone reorganises their permalinks is worse than no probe.
487 494
495+## 14. Similarity and scoring
496+
497+`atproto/cluster.nim` ports the similarity computations from
498+`internal/cluster/jaccard.go` and the pure helpers from `scoring.go`.
499+
500+Both similarities are precomputed on a schedule and staged into a temp table
501+before being swapped in, so the expensive pass holds no lock on the live
502+table and readers keep seeing the previous generation rather than an empty
503+one.
504+
505+Feed similarity is time-decayed, `exp(-0.023 * days)` -- a half-life of about
506+a month. Without it a long-lived account's earliest subscriptions dominate
507+its recommendations forever. User similarity over *subscriptions* is
508+deliberately undecayed, because a subscription is a standing choice rather
509+than an event; shared *likes* are events, so those decay on both sides.
510+
511+Recommendations have no obviously correct answer, so the probe checks
512+properties rather than values: that decay decays, that each pair is stored
513+once, that recomputing replaces rather than accumulates, that normalisation
514+of an all-equal set does not produce NaN, and that `samplePeople` reserves
515+half its slots for people the reader does *not* follow -- ranking purely by
516+similarity returns the network they already have, which is not discovery.
517+
518+### A bug inherited from the Go implementation
519+
520+`common_likes` is computed as `CAST(SUM(decay * decay) AS INTEGER)`. Each
521+decay factor is at most 1, so their product is always just under it: a pair
522+sharing one recent like sums to 0.9999 and **truncates to 0**. Two shared
523+likes store 1. The number is shown to readers as the reason for a
524+recommendation -- "N shared likes" -- so a real overlap reads as none.
525+
526+This port uses `ROUND` instead. Worth fixing on the Go side too.
527+
528+The probe caught it only because the assertion looked *vacuous*: both the
529+recent and the stale pair returned 0, so "recent >= old" passed while
530+proving nothing. A test that passes for the wrong reason is worse than one
531+that fails.
532+
488 ## What this does and does not prove533 ## What this does and does not prove
489 534
490 Proven: the cryptography, the storage layer, and the OAuth flow up to user535 Proven: the cryptography, the storage layer, and the OAuth flow up to user