rickub
nandi/gleanpublic⑂ Fork 0
⑂ 97314cb
Branches
claude/modal-container-min-76527a main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/glean.git
git clone ssh://git@rickub.com/nandi/glean.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Remove full content for article embedding and truncateUnverified

Julien Robert committed 2026-04-26T02:24:11+02:00 Browse files
97314cb parent: 588cf0a
modified docs/specs.md +20 -20
@@ -10,14 +10,14 @@ The core idea: your RSS subscriptions are a strong signal about your interests.
1010
1111 ## 2. Stack
1212
13-| Layer | Technology |
14-| ---------------- | --------------------------------------------------------------- |
15-| Backend | Go |
16-| Database | SQLite (3 files: users, articles, recs via `mattn/go-sqlite3` + `sqlite-vec` for vector search) |
17-| Frontend | htmx + TailwindCSS |
18-| Auth | AT Protocol OAuth / DID resolution (configurable PLC directory) |
19-| AT Protocol role | AppView for `at.glean.*` lexicons |
20-| Data source | AT Protocol Jetstream → SQLite index |
13+| Layer | Technology |
14+| ---------------- | ----------------------------------------------------------------------------------------------- |
15+| Backend | Go |
16+| Database | SQLite (3 files: users, articles, recs via `mattn/go-sqlite3` + `sqlite-vec` for vector search) |
17+| Frontend | htmx + TailwindCSS |
18+| Auth | AT Protocol OAuth / DID resolution (configurable PLC directory) |
19+| AT Protocol role | AppView for `at.glean.*` lexicons |
20+| Data source | AT Protocol Jetstream → SQLite index |
2121
2222 ## 3. AT Protocol Lexicons
2323
@@ -645,14 +645,14 @@ Glean uses a multi-signal recommendation system that combines subscription overl
645645
646646 ### 7.1 Signals
647647
648-| Signal | Source | Weight (default) | Description |
649-| ------------ | ------------------------ | ---------------- | -------------------------------------------------- |
650-| Subscription | `subscriptions` | 1.0 | Jaccard over subscriber sets between similar users |
651-| Like | `likes` | 0.5 | Time-decayed like co-occurrence (30-day half-life) |
652-| Tag | `annotations.tags` | 0.3 | Jaccard over annotation tag sets |
653-| Social | `follow_distances` | 0.7 | Follow distance: 1-hop=1.0, 2-hop=0.3, 3-hop=0.1 |
654-| Popularity | `feeds.subscriber_count` | 0.2 | `log(1 + subscribers) / log(1 + max)` |
655-| Category | `subscriptions.category` | 0.4 | Boost feeds matching user's existing categories |
648+| Signal | Source | Weight (default) | Description |
649+| ------------ | ------------------------ | ---------------- | ------------------------------------------------------- |
650+| Subscription | `subscriptions` | 1.0 | Jaccard over subscriber sets between similar users |
651+| Like | `likes` | 0.5 | Time-decayed like co-occurrence (30-day half-life) |
652+| Tag | `annotations.tags` | 0.3 | Jaccard over annotation tag sets |
653+| Social | `follow_distances` | 0.7 | Follow distance: 1-hop=1.0, 2-hop=0.3, 3-hop=0.1 |
654+| Popularity | `feeds.subscriber_count` | 0.2 | `log(1 + subscribers) / log(1 + max)` |
655+| Category | `subscriptions.category` | 0.4 | Boost feeds matching user's existing categories |
656656 | Content | `article_embeddings` | 0.4 | Cosine similarity via embedding KNN (requires embedder) |
657657
658658 ### 7.2 Feed Co-occurrence (Jaccard Similarity)
@@ -759,10 +759,10 @@ New users with <5 subscriptions get a fallback strategy:
759759
760760 A background goroutine runs on a configurable schedule (`GLEAN_CLUSTER_INTERVAL`, default 10m):
761761
762-1. **Compute feed embeddings**: Embed new feed descriptions via OpenAI-compatible API into `feed_embeddings` table (skipped if no embedder configured)
762+1. **Compute feed embeddings**: Embed new feed descriptions via embedding API into `feed_embeddings` table (skipped if no embedder configured)
763763 2. **Compute feed similarity**: Batch-update `feed_similarity` table (Jaccard over subscriber sets + embedding cosine similarity)
764764 3. **Compute user similarity**: Batch-update `user_similarity` table (subscription Jaccard + time-decayed likes + tags + follow boost)
765-4. **Compute article embeddings**: Embed new articles' full content (`title + summary + full_content + content`) via OpenAI-compatible API into `article_embeddings` vec0 table (skipped if no embedder configured)
765+4. **Compute article embeddings**: Embed new articles (`title + summary + content`, excluding `full_content` to stay within embedding model token limits) via embedding API into `article_embeddings` vec0 table (skipped if no embedder configured)
766766 5. **Compute follow distances**: Incremental BFS for dirty users (1-hop through 3-hop from `follows` table)
767767 6. **Compute signal profiles**: Per-user category/tag/like summaries
768768 7. **Auto-dismiss stale**: Dismiss items shown >=5 times over >5 days without action
@@ -854,7 +854,7 @@ CREATE TABLE user_signal_profiles (
854854
855855 When `GLEAN_EMBED_BASE_URL` is configured, article text and feed descriptions are embedded into vectors stored in `sqlite-vec` virtual tables (`recs.feed_embeddings`, `recs.article_embeddings`). The vec0 extension provides native KNN vector search via `WHERE embedding MATCH ? AND k = ?`, replacing Go-side cosine similarity for large-scale lookups. Without embeddings, recommendations rely only on subscription overlap, like patterns, and social graph — no content-based signals.
856856
857-The embedder uses the official `github.com/openai/openai-go` SDK with `option.WithBaseURL()`, so any OpenAI-compatible `/v1/embeddings` endpoint works (OpenAI, Ollama, local inference servers).
857+The embedder uses the official `github.com/openai/openai-go` SDK with `option.WithBaseURL()`, so any OpenAI-compatible `/v1/embeddings` endpoint works (OpenAI, Gemini, Ollama, local inference servers).
858858
859859 vec0 tables are created dynamically at startup with the configured dimension (`GLEAN_EMBED_DIMENSION`, default 1536):
860860
@@ -879,7 +879,7 @@ CREATE TABLE recs.feed_embedding_meta (
879879 );
880880 ```
881881
882-During cron, `ComputeArticleEmbeddings` embeds new articles in batches (using `title + summary + full_content + content` for maximum semantic coverage) and inserts them into the vec0 table. `ComputeFeedEmbeddings` embeds feed descriptions (`title || description`) and re-embeds when the source text changes (detected via `feed_embedding_meta`). During on-demand article recommendations, the user's liked article embeddings are averaged into an interest vector, then a vec0 KNN query finds the top-200 most semantically similar articles. For cold-start users (<5 subscriptions), their subscribed feed embeddings are averaged and a KNN query finds similar feeds.
882+During cron, `ComputeArticleEmbeddings` embeds new articles in batches using `title + summary + content` (the scraped `full_content` is excluded to stay within model token limits — most embedding models cap at ~8k tokens). Text is truncated to 8000 characters as a safety net. Batches are capped at 100 inputs per API call. `ComputeFeedEmbeddings` embeds feed descriptions (`title || description`) and re-embeds when the source text changes (detected via `feed_embedding_meta`). During on-demand article recommendations, the user's liked article embeddings are averaged into an interest vector, then a vec0 KNN query finds the top-200 most semantically similar articles. For cold-start users (<5 subscriptions), their subscribed feed embeddings are averaged and a KNN query finds similar feeds.
883883
884884 ## 8. HTTP API / htmx Endpoints
885885
@@ -10,14 +10,14 @@ The core idea: your RSS subscriptions are a strong signal about your interests.
10 10
11 ## 2. Stack11 ## 2. Stack
12 12
13-| Layer | Technology |13+| Layer | Technology |
14-| ---------------- | --------------------------------------------------------------- |14+| ---------------- | ----------------------------------------------------------------------------------------------- |
15-| Backend | Go |15+| Backend | Go |
16-| Database | SQLite (3 files: users, articles, recs via `mattn/go-sqlite3` + `sqlite-vec` for vector search) |16+| Database | SQLite (3 files: users, articles, recs via `mattn/go-sqlite3` + `sqlite-vec` for vector search) |
17-| Frontend | htmx + TailwindCSS |17+| Frontend | htmx + TailwindCSS |
18-| Auth | AT Protocol OAuth / DID resolution (configurable PLC directory) |18+| Auth | AT Protocol OAuth / DID resolution (configurable PLC directory) |
19-| AT Protocol role | AppView for `at.glean.*` lexicons |19+| AT Protocol role | AppView for `at.glean.*` lexicons |
20-| Data source | AT Protocol Jetstream → SQLite index |20+| Data source | AT Protocol Jetstream → SQLite index |
21 21
22 ## 3. AT Protocol Lexicons22 ## 3. AT Protocol Lexicons
23 23
@@ -645,14 +645,14 @@ Glean uses a multi-signal recommendation system that combines subscription overl
645 645
646 ### 7.1 Signals646 ### 7.1 Signals
647 647
648-| Signal | Source | Weight (default) | Description |648+| Signal | Source | Weight (default) | Description |
649-| ------------ | ------------------------ | ---------------- | -------------------------------------------------- |649+| ------------ | ------------------------ | ---------------- | ------------------------------------------------------- |
650-| Subscription | `subscriptions` | 1.0 | Jaccard over subscriber sets between similar users |650+| Subscription | `subscriptions` | 1.0 | Jaccard over subscriber sets between similar users |
651-| Like | `likes` | 0.5 | Time-decayed like co-occurrence (30-day half-life) |651+| Like | `likes` | 0.5 | Time-decayed like co-occurrence (30-day half-life) |
652-| Tag | `annotations.tags` | 0.3 | Jaccard over annotation tag sets |652+| Tag | `annotations.tags` | 0.3 | Jaccard over annotation tag sets |
653-| Social | `follow_distances` | 0.7 | Follow distance: 1-hop=1.0, 2-hop=0.3, 3-hop=0.1 |653+| Social | `follow_distances` | 0.7 | Follow distance: 1-hop=1.0, 2-hop=0.3, 3-hop=0.1 |
654-| Popularity | `feeds.subscriber_count` | 0.2 | `log(1 + subscribers) / log(1 + max)` |654+| Popularity | `feeds.subscriber_count` | 0.2 | `log(1 + subscribers) / log(1 + max)` |
655-| Category | `subscriptions.category` | 0.4 | Boost feeds matching user's existing categories |655+| Category | `subscriptions.category` | 0.4 | Boost feeds matching user's existing categories |
656 | Content | `article_embeddings` | 0.4 | Cosine similarity via embedding KNN (requires embedder) |656 | Content | `article_embeddings` | 0.4 | Cosine similarity via embedding KNN (requires embedder) |
657 657
658 ### 7.2 Feed Co-occurrence (Jaccard Similarity)658 ### 7.2 Feed Co-occurrence (Jaccard Similarity)
@@ -759,10 +759,10 @@ New users with <5 subscriptions get a fallback strategy:
759 759
760 A background goroutine runs on a configurable schedule (`GLEAN_CLUSTER_INTERVAL`, default 10m):760 A background goroutine runs on a configurable schedule (`GLEAN_CLUSTER_INTERVAL`, default 10m):
761 761
762-1. **Compute feed embeddings**: Embed new feed descriptions via OpenAI-compatible API into `feed_embeddings` table (skipped if no embedder configured)762+1. **Compute feed embeddings**: Embed new feed descriptions via embedding API into `feed_embeddings` table (skipped if no embedder configured)
763 2. **Compute feed similarity**: Batch-update `feed_similarity` table (Jaccard over subscriber sets + embedding cosine similarity)763 2. **Compute feed similarity**: Batch-update `feed_similarity` table (Jaccard over subscriber sets + embedding cosine similarity)
764 3. **Compute user similarity**: Batch-update `user_similarity` table (subscription Jaccard + time-decayed likes + tags + follow boost)764 3. **Compute user similarity**: Batch-update `user_similarity` table (subscription Jaccard + time-decayed likes + tags + follow boost)
765-4. **Compute article embeddings**: Embed new articles' full content (`title + summary + full_content + content`) via OpenAI-compatible API into `article_embeddings` vec0 table (skipped if no embedder configured)765+4. **Compute article embeddings**: Embed new articles (`title + summary + content`, excluding `full_content` to stay within embedding model token limits) via embedding API into `article_embeddings` vec0 table (skipped if no embedder configured)
766 5. **Compute follow distances**: Incremental BFS for dirty users (1-hop through 3-hop from `follows` table)766 5. **Compute follow distances**: Incremental BFS for dirty users (1-hop through 3-hop from `follows` table)
767 6. **Compute signal profiles**: Per-user category/tag/like summaries767 6. **Compute signal profiles**: Per-user category/tag/like summaries
768 7. **Auto-dismiss stale**: Dismiss items shown >=5 times over >5 days without action768 7. **Auto-dismiss stale**: Dismiss items shown >=5 times over >5 days without action
@@ -854,7 +854,7 @@ CREATE TABLE user_signal_profiles (
854 854
855 When `GLEAN_EMBED_BASE_URL` is configured, article text and feed descriptions are embedded into vectors stored in `sqlite-vec` virtual tables (`recs.feed_embeddings`, `recs.article_embeddings`). The vec0 extension provides native KNN vector search via `WHERE embedding MATCH ? AND k = ?`, replacing Go-side cosine similarity for large-scale lookups. Without embeddings, recommendations rely only on subscription overlap, like patterns, and social graph — no content-based signals.855 When `GLEAN_EMBED_BASE_URL` is configured, article text and feed descriptions are embedded into vectors stored in `sqlite-vec` virtual tables (`recs.feed_embeddings`, `recs.article_embeddings`). The vec0 extension provides native KNN vector search via `WHERE embedding MATCH ? AND k = ?`, replacing Go-side cosine similarity for large-scale lookups. Without embeddings, recommendations rely only on subscription overlap, like patterns, and social graph — no content-based signals.
856 856
857-The embedder uses the official `github.com/openai/openai-go` SDK with `option.WithBaseURL()`, so any OpenAI-compatible `/v1/embeddings` endpoint works (OpenAI, Ollama, local inference servers).857+The embedder uses the official `github.com/openai/openai-go` SDK with `option.WithBaseURL()`, so any OpenAI-compatible `/v1/embeddings` endpoint works (OpenAI, Gemini, Ollama, local inference servers).
858 858
859 vec0 tables are created dynamically at startup with the configured dimension (`GLEAN_EMBED_DIMENSION`, default 1536):859 vec0 tables are created dynamically at startup with the configured dimension (`GLEAN_EMBED_DIMENSION`, default 1536):
860 860
@@ -879,7 +879,7 @@ CREATE TABLE recs.feed_embedding_meta (
879 );879 );
880 ```880 ```
881 881
882-During cron, `ComputeArticleEmbeddings` embeds new articles in batches (using `title + summary + full_content + content` for maximum semantic coverage) and inserts them into the vec0 table. `ComputeFeedEmbeddings` embeds feed descriptions (`title || description`) and re-embeds when the source text changes (detected via `feed_embedding_meta`). During on-demand article recommendations, the user's liked article embeddings are averaged into an interest vector, then a vec0 KNN query finds the top-200 most semantically similar articles. For cold-start users (<5 subscriptions), their subscribed feed embeddings are averaged and a KNN query finds similar feeds.882+During cron, `ComputeArticleEmbeddings` embeds new articles in batches using `title + summary + content` (the scraped `full_content` is excluded to stay within model token limits — most embedding models cap at ~8k tokens). Text is truncated to 8000 characters as a safety net. Batches are capped at 100 inputs per API call. `ComputeFeedEmbeddings` embeds feed descriptions (`title || description`) and re-embeds when the source text changes (detected via `feed_embedding_meta`). During on-demand article recommendations, the user's liked article embeddings are averaged into an interest vector, then a vec0 KNN query finds the top-200 most semantically similar articles. For cold-start users (<5 subscriptions), their subscribed feed embeddings are averaged and a KNN query finds similar feeds.
883 883
884 ## 8. HTTP API / htmx Endpoints884 ## 8. HTTP API / htmx Endpoints
885 885
modified internal/cluster/article.go +23 -6
@@ -10,12 +10,28 @@ import (
1010 vec "github.com/asg017/sqlite-vec-go-bindings/cgo"
1111 )
1212
13+// embedBatchSize caps how many texts are sent in a single embedding API call.
14+// Most /v1/embeddings endpoints accept up to ~2048 inputs per request; lower
15+// values reduce payload size and memory pressure.
1316 const embedBatchSize = 100
1417
15-// ComputeArticleEmbeddings embeds new articles (title + summary) into the
16-// article_embeddings vec0 table. Skipped when no embedder is configured.
17-// Existing embeddings for deleted articles are cleaned up. Articles already
18-// embedded are not re-embedded.
18+// maxEmbedChars truncates text sent to the embedding model. Most models have
19+// a token context window (~4 chars/token); title+summary+content stays well
20+// under this, but the cap is kept as a safety net.
21+const maxEmbedChars = 8000
22+
23+func truncateForEmbed(s string) string {
24+ if len(s) <= maxEmbedChars {
25+ return s
26+ }
27+ return s[:maxEmbedChars]
28+}
29+
30+// ComputeArticleEmbeddings embeds new articles (title + summary + content) into
31+// the article_embeddings vec0 table. full_content (scraped body) is excluded
32+// because embedding models have token context limits and the feed-provided
33+// content already captures the topical signal needed for recommendation KNN.
34+
1935 func (e *Engine) ComputeArticleEmbeddings(ctx context.Context) error {
2036 if e.embedder == nil {
2137 e.logger.Debug("article embeddings skipped, no embedder")
@@ -37,9 +53,9 @@ func (e *Engine) ComputeArticleEmbeddings(ctx context.Context) error {
3753 }
3854
3955 rows, err := conn.QueryContext(ctx, `
40- SELECT a.id, COALESCE(a.title, '') || ' ' || COALESCE(a.summary, '') || ' ' || COALESCE(a.full_content, '') || ' ' || COALESCE(a.content, '')
56+ SELECT a.id, COALESCE(a.title, '') || ' ' || COALESCE(a.summary, '') || ' ' || COALESCE(a.content, '')
4157 FROM articles.articles a
42- WHERE (COALESCE(a.title, '') != '' OR COALESCE(a.summary, '') != '' OR COALESCE(a.full_content, '') != '' OR COALESCE(a.content, '') != '')
58+ WHERE (COALESCE(a.title, '') != '' OR COALESCE(a.summary, '') != '' OR COALESCE(a.content, '') != '')
4359 AND a.id NOT IN (SELECT article_id FROM recs.article_embeddings)
4460 ORDER BY a.id
4561 `)
@@ -58,6 +74,7 @@ func (e *Engine) ComputeArticleEmbeddings(ctx context.Context) error {
5874 rows.Close()
5975 return err
6076 }
77+ a.text = truncateForEmbed(a.text)
6178 batch = append(batch, a)
6279 }
6380 rows.Close()
@@ -10,12 +10,28 @@ import (
10 vec "github.com/asg017/sqlite-vec-go-bindings/cgo"10 vec "github.com/asg017/sqlite-vec-go-bindings/cgo"
11 )11 )
12 12
13+// embedBatchSize caps how many texts are sent in a single embedding API call.
14+// Most /v1/embeddings endpoints accept up to ~2048 inputs per request; lower
15+// values reduce payload size and memory pressure.
13 const embedBatchSize = 10016 const embedBatchSize = 100
14 17
15-// ComputeArticleEmbeddings embeds new articles (title + summary) into the18+// maxEmbedChars truncates text sent to the embedding model. Most models have
16-// article_embeddings vec0 table. Skipped when no embedder is configured.19+// a token context window (~4 chars/token); title+summary+content stays well
17-// Existing embeddings for deleted articles are cleaned up. Articles already20+// under this, but the cap is kept as a safety net.
18-// embedded are not re-embedded.21+const maxEmbedChars = 8000
22+
23+func truncateForEmbed(s string) string {
24+ if len(s) <= maxEmbedChars {
25+ return s
26+ }
27+ return s[:maxEmbedChars]
28+}
29+
30+// ComputeArticleEmbeddings embeds new articles (title + summary + content) into
31+// the article_embeddings vec0 table. full_content (scraped body) is excluded
32+// because embedding models have token context limits and the feed-provided
33+// content already captures the topical signal needed for recommendation KNN.
34+
19 func (e *Engine) ComputeArticleEmbeddings(ctx context.Context) error {35 func (e *Engine) ComputeArticleEmbeddings(ctx context.Context) error {
20 if e.embedder == nil {36 if e.embedder == nil {
21 e.logger.Debug("article embeddings skipped, no embedder")37 e.logger.Debug("article embeddings skipped, no embedder")
@@ -37,9 +53,9 @@ func (e *Engine) ComputeArticleEmbeddings(ctx context.Context) error {
37 }53 }
38 54
39 rows, err := conn.QueryContext(ctx, `55 rows, err := conn.QueryContext(ctx, `
40- SELECT a.id, COALESCE(a.title, '') || ' ' || COALESCE(a.summary, '') || ' ' || COALESCE(a.full_content, '') || ' ' || COALESCE(a.content, '')56+ SELECT a.id, COALESCE(a.title, '') || ' ' || COALESCE(a.summary, '') || ' ' || COALESCE(a.content, '')
41 FROM articles.articles a57 FROM articles.articles a
42- WHERE (COALESCE(a.title, '') != '' OR COALESCE(a.summary, '') != '' OR COALESCE(a.full_content, '') != '' OR COALESCE(a.content, '') != '')58+ WHERE (COALESCE(a.title, '') != '' OR COALESCE(a.summary, '') != '' OR COALESCE(a.content, '') != '')
43 AND a.id NOT IN (SELECT article_id FROM recs.article_embeddings)59 AND a.id NOT IN (SELECT article_id FROM recs.article_embeddings)
44 ORDER BY a.id60 ORDER BY a.id
45 `)61 `)
@@ -58,6 +74,7 @@ func (e *Engine) ComputeArticleEmbeddings(ctx context.Context) error {
58 rows.Close()74 rows.Close()
59 return err75 return err
60 }76 }
77+ a.text = truncateForEmbed(a.text)
61 batch = append(batch, a)78 batch = append(batch, a)
62 }79 }
63 rows.Close()80 rows.Close()