nandi/glean · 71c0a01

modified .env.example +6 -0 Viewed

@@ -18,3 +18,9 @@ GLEAN_EMBED_BASE_URL=https://llms.example.com/v1
18	18	GLEAN_EMBED_API_KEY=API-KEY-001
19	19	GLEAN_EMBED_MODEL=qwen-embedding-4b
20	20	GLEAN_EMBED_DIMENSION=2560
	21	+# LLM for article language detection and other text tasks
	22	+# Any OpenAI-compatible /v1/chat/completions endpoint works.
	23	+# Without an LLM, all articles default to English and language filtering is disabled.
	24	+GLEAN_LLM_BASE_URL=https://llms.example.com/v1
	25	+GLEAN_LLM_API_KEY=API-KEY-001
	26	+GLEAN_LLM_MODEL=qwen3.5-35b-a3b

  GLEAN_EMBED_API_KEY=API-KEY-001
  GLEAN_EMBED_MODEL=qwen-embedding-4b
  GLEAN_EMBED_DIMENSION=2560
++# LLM for article language detection and other text tasks
++# Any OpenAI-compatible /v1/chat/completions endpoint works.
++# Without an LLM, all articles default to English and language filtering is disabled.
++GLEAN_LLM_BASE_URL=https://llms.example.com/v1
++GLEAN_LLM_API_KEY=API-KEY-001
++GLEAN_LLM_MODEL=qwen3.5-35b-a3b

modified docs/specs.md +27 -6 Viewed

@@ -352,6 +352,7 @@ CREATE TABLE articles (
352	352	published DATETIME,
353	353	updated DATETIME,
354	354	fetched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
	355	+ language TEXT NOT NULL DEFAULT '',
355	356	UNIQUE(feed_url, guid)
356	357	);
357	358
@@ -361,6 +362,8 @@ CREATE INDEX idx_articles_published ON articles(published DESC);
361	362
362	363	Content is stored as raw HTML from the feed's `<content:encoded>`, `<summary>`, or JSON Feed `content_html`. `full_content` stores scraped article content fetched from the original URL. The server renders it in a sanitized view (strip `<script>`, `<iframe>`, etc.).
363	364
	365	+The `language` column stores the ISO 639-1 code detected by the LLM (e.g. `en`, `fr`, `ja`). It defaults to empty (`''`) and is populated by the cron job when `GLEAN_LLM_BASE_URL` is configured.
	366	+
364	367	### 4.5 Read State
365	368
366	369	Read/unread state is tracked per user per article:
@@ -463,7 +466,8 @@ CREATE TABLE users (
463	466	did TEXT PRIMARY KEY,
464	467	indexed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
465	468	updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
466		- follows_dirty BOOLEAN NOT NULL DEFAULT 1
	469	+ follows_dirty BOOLEAN NOT NULL DEFAULT 1,
	470	+ languages TEXT
467	471	);
468	472	```
469	473
@@ -526,11 +530,13 @@ CREATE TABLE articles (
526	530	published DATETIME,
527	531	updated DATETIME,
528	532	fetched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
	533	+ language TEXT NOT NULL DEFAULT '',
529	534	UNIQUE(feed_url, guid)
530	535	);
531	536
532	537	CREATE INDEX idx_articles_feed ON articles(feed_url);
533	538	CREATE INDEX idx_articles_published ON articles(published DESC);
	539	+CREATE INDEX idx_articles_language ON articles(language);
534	540	```
535	541
536	542	### 6.5 Read State (`<base>_articles`)
@@ -708,6 +714,8 @@ score = like_signal * w_like
708	714
709	715	Content signal uses embedding vectors: the user's liked article embeddings are averaged into a single interest vector, then a KNN query against the `article_embeddings` vec0 table finds semantically similar articles. This requires an embedder to be configured; without it, the content signal is 0.
710	716
	717	+Language filtering: Users can set preferred languages on their profile (stored in the `user_settings` table as a JSON array of ISO 639-1 codes). When set, article recommendations are filtered to only include articles whose `language` column matches one of the selected codes or whose `language` is empty (not yet classified). This ensures users with language preferences still see all recommendations when the LLM hasn't run or hasn't classified certain articles yet. If no languages are set (empty/nil), all articles are shown regardless of language.
	718	+
711	719	### 7.5 User Feedback (Dismiss)
712	720
713	721	Users can dismiss recommendations they don't want to see again:
@@ -763,9 +771,10 @@ A background goroutine runs on a configurable schedule (`GLEAN_CLUSTER_INTERVAL`
763	771	2. Compute feed similarity: Batch-update `feed_similarity` table (Jaccard over subscriber sets + embedding cosine similarity)
764	772	3. Compute user similarity: Batch-update `user_similarity` table (subscription Jaccard + time-decayed likes + tags + follow boost)
765	773	4. Compute article embeddings: Embed new articles (`title + summary + content`, excluding `full_content` to stay within embedding model token limits) via embedding API into `article_embeddings` vec0 table (skipped if no embedder configured)
766		-5. Compute follow distances: Incremental BFS for dirty users (1-hop through 3-hop from `follows` table)
767		-6. Compute signal profiles: Per-user category/tag/like summaries
768		-7. Auto-dismiss stale: Dismiss items shown >=5 times over >5 days without action
	774	+5. Detect article languages: Batch-classify article languages via LLM, updating the `language` column (skipped if no LLM configured)
	775	+6. Compute follow distances: Incremental BFS for dirty users (1-hop through 3-hop from `follows` table)
	776	+7. Compute signal profiles: Per-user category/tag/like summaries
	777	+8. Auto-dismiss stale: Dismiss items shown >=5 times over >5 days without action
769	778
770	779	Jetstream ingestion and record indexing happen in a separate persistent goroutine (the Jetstream consumer), not in the cron.
771	780
@@ -856,6 +865,14 @@ When `GLEAN_EMBED_BASE_URL` is configured, article text and feed descriptions ar
856	865
857	866	The embedder uses the official `github.com/openai/openai-go` SDK with `option.WithBaseURL()`, so any OpenAI-compatible `/v1/embeddings` endpoint works (OpenAI, Gemini, Ollama, local inference servers).
858	867
	868	+### 7.14 LLM Client (optional)
	869	+
	870	+When `GLEAN_LLM_BASE_URL` is configured, an LLM client is available for text classification tasks. It uses the same `github.com/openai/openai-go` SDK pointed at any OpenAI-compatible `/v1/chat/completions` endpoint.
	871	+
	872	+Language detection: The cron job calls `DetectLanguages` in batches of up to 100 articles per request. For each article, the title and summary (truncated to 500 characters) are sent with a prompt asking for ISO 639-1 codes. Results are written to `articles.language`. Articles that are already classified (non-empty `language`) are skipped. An empty response from the LLM defaults to English (`en`).
	873	+
	874	+Without an LLM, all articles remain at the default empty language value and language-based filtering is unavailable.
	875	+
859	876	vec0 tables are created dynamically at startup with the configured dimension (`GLEAN_EMBED_DIMENSION`, default 1536):
860	877
861	878	```sql
@@ -916,6 +933,7 @@ The server renders HTML fragments that htmx swaps into the page. No JSON API nee
916	933	\| `/library/{id}/delete` \| POST \| Delete an annotation \|
917	934	\| `/stats` \| GET \| Application metrics and performance data (Prometheus, public) \|
918	935	\| `/profile/{did}` \| GET \| Public profile: their feeds, likes, annotations \|
	936	+\| `/settings/languages` \| POST \| Save preferred recommendation languages (htmx, requires auth) \|
919	937	\| `/auth/login` \| GET \| Login page \|
920	938	\| `/auth/register` \| GET \| Register with Eurosky (OAuth flow with hardcoded PDS) \|
921	939	\| `/auth/resolve` \| GET \| Resolve handle to DID \|
@@ -979,7 +997,8 @@ glean/
979	997	│ ├── cluster/
980	998	│ │ ├── jaccard.go # Jaccard similarity computation
981	999	│ │ ├── embed.go # Embedder interface + OpenAI-compatible implementation
982		-│ │ ├── article.go # Article + feed embedding computation, vec0 KNN content boost
	1000	+│ │ ├── llm.go # LLM client for language detection and text tasks
	1001	+│ │ ├── article.go # Article + feed embedding computation, vec0 KNN content boost, language detection
983	1002	│ │ ├── scoring.go # Feed + people + article recommendation queries (on-demand)
984	1003	│ │ ├── social.go # Incremental follow-distance computation (1-3 hop, dirty-flag)
985	1004	│ │ ├── dismiss.go # Dismiss + impression tracking
@@ -997,6 +1016,7 @@ glean/
997	1016	│ │ ├── stats_handler.go # Stats handler (Prometheus metrics display)
998	1017	│ │ ├── index_handler.go # Landing page handler
999	1018	│ │ ├── profile_handler.go # Public profile handler
	1019	+│ │ ├── settings_handler.go # User settings (language preferences)
1000	1020	│ │ ├── terms_handler.go # Terms of service handler
1001	1021	│ │ ├── pagination.go # Pagination helpers
1002	1022	│ │ ├── middleware.go # Auth, logging, CSRF middleware
@@ -1077,13 +1097,14 @@ Cron (every 10m) ──► Cluster Engine
1077	1097	├─► Compute feed similarity
1078	1098	├─► Compute user similarity
1079	1099	├─► Compute article embeddings (if embedder configured)
	1100	+ ├─► Detect article languages (if LLM configured)
1080	1101	├─► Compute follow distances
1081	1102	├─► Compute signal profiles
1082	1103	└─► Auto-dismiss stale recommendations
1083	1104
1084	1105	Browser ──GET /dashboard──► Server
1085	1106	│
1086		- ├─► Compute recommendations on-demand
	1107	+ ├─► Compute recommendations on-demand (filtered by user's language preferences)
1087	1108	├─► Fetch feed metadata
1088	1109	└─◄ Render recommendation cards (htmx)
1089	1110	```

      published   DATETIME,
      updated     DATETIME,
      fetched_at  DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
++    language    TEXT NOT NULL DEFAULT '',
      UNIQUE(feed_url, guid)
  );
  Content is stored as raw HTML from the feed's `<content:encoded>`, `<summary>`, or JSON Feed `content_html`. `full_content` stores scraped article content fetched from the original URL. The server renders it in a sanitized view (strip `<script>`, `<iframe>`, etc.).
++The `language` column stores the ISO 639-1 code detected by the LLM (e.g. `en`, `fr`, `ja`). It defaults to empty (`''`) and is populated by the cron job when `GLEAN_LLM_BASE_URL` is configured.
++
  ### 4.5 Read State
  Read/unread state is tracked per user per article:
      did            TEXT PRIMARY KEY,
      indexed_at     DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
      updated_at     DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
--    follows_dirty  BOOLEAN NOT NULL DEFAULT 1
++    follows_dirty  BOOLEAN NOT NULL DEFAULT 1,
++    languages      TEXT
  );
  ```
      published   DATETIME,
      updated     DATETIME,
      fetched_at  DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
++    language    TEXT NOT NULL DEFAULT '',
      UNIQUE(feed_url, guid)
  );
  CREATE INDEX idx_articles_feed ON articles(feed_url);
  CREATE INDEX idx_articles_published ON articles(published DESC);
++CREATE INDEX idx_articles_language ON articles(language);
  ```
  ### 6.5 Read State (`<base>_articles`)
  Content signal uses embedding vectors: the user's liked article embeddings are averaged into a single interest vector, then a KNN query against the `article_embeddings` vec0 table finds semantically similar articles. This requires an embedder to be configured; without it, the content signal is 0.
++**Language filtering**: Users can set preferred languages on their profile (stored in the `user_settings` table as a JSON array of ISO 639-1 codes). When set, article recommendations are filtered to only include articles whose `language` column matches one of the selected codes **or** whose `language` is empty (not yet classified). This ensures users with language preferences still see all recommendations when the LLM hasn't run or hasn't classified certain articles yet. If no languages are set (empty/nil), all articles are shown regardless of language.
++
  ### 7.5 User Feedback (Dismiss)
  Users can dismiss recommendations they don't want to see again:
 . **Compute feed similarity**: Batch-update `feed_similarity` table (Jaccard over subscriber sets + embedding cosine similarity)
 . **Compute user similarity**: Batch-update `user_similarity` table (subscription Jaccard + time-decayed likes + tags + follow boost)
 . **Compute article embeddings**: Embed new articles (`title + summary + content`, excluding `full_content` to stay within embedding model token limits) via embedding API into `article_embeddings` vec0 table (skipped if no embedder configured)
--5. **Compute follow distances**: Incremental BFS for dirty users (1-hop through 3-hop from `follows` table)
++5. **Detect article languages**: Batch-classify article languages via LLM, updating the `language` column (skipped if no LLM configured)
--6. **Compute signal profiles**: Per-user category/tag/like summaries
++6. **Compute follow distances**: Incremental BFS for dirty users (1-hop through 3-hop from `follows` table)
--7. **Auto-dismiss stale**: Dismiss items shown >=5 times over >5 days without action
++7. **Compute signal profiles**: Per-user category/tag/like summaries
++8. **Auto-dismiss stale**: Dismiss items shown >=5 times over >5 days without action
  Jetstream ingestion and record indexing happen in a separate persistent goroutine (the Jetstream consumer), not in the cron.
  The embedder uses the official `github.com/openai/openai-go` SDK with `option.WithBaseURL()`, so any OpenAI-compatible `/v1/embeddings` endpoint works (OpenAI, Gemini, Ollama, local inference servers).
++### 7.14 LLM Client (optional)
++
++When `GLEAN_LLM_BASE_URL` is configured, an LLM client is available for text classification tasks. It uses the same `github.com/openai/openai-go` SDK pointed at any OpenAI-compatible `/v1/chat/completions` endpoint.
++
++**Language detection**: The cron job calls `DetectLanguages` in batches of up to 100 articles per request. For each article, the title and summary (truncated to 500 characters) are sent with a prompt asking for ISO 639-1 codes. Results are written to `articles.language`. Articles that are already classified (non-empty `language`) are skipped. An empty response from the LLM defaults to English (`en`).
++
++Without an LLM, all articles remain at the default empty language value and language-based filtering is unavailable.
++
  vec0 tables are created dynamically at startup with the configured dimension (`GLEAN_EMBED_DIMENSION`, default 1536):
  ```sql
  | `/library/{id}/delete`         | POST   | Delete an annotation                                                |
  | `/stats`                       | GET    | Application metrics and performance data (Prometheus, public)       |
  | `/profile/{did}`               | GET    | Public profile: their feeds, likes, annotations                     |
++| `/settings/languages`          | POST   | Save preferred recommendation languages (htmx, requires auth)       |
  | `/auth/login`                  | GET    | Login page                                                          |
  | `/auth/register`               | GET    | Register with Eurosky (OAuth flow with hardcoded PDS)               |
  | `/auth/resolve`                | GET    | Resolve handle to DID                                               |
  │   ├── cluster/
  │   │   ├── jaccard.go             # Jaccard similarity computation
  │   │   ├── embed.go               # Embedder interface + OpenAI-compatible implementation
--│   │   ├── article.go             # Article + feed embedding computation, vec0 KNN content boost
++│   │   ├── llm.go                 # LLM client for language detection and text tasks
++│   │   ├── article.go             # Article + feed embedding computation, vec0 KNN content boost, language detection
  │   │   ├── scoring.go             # Feed + people + article recommendation queries (on-demand)
  │   │   ├── social.go              # Incremental follow-distance computation (1-3 hop, dirty-flag)
  │   │   ├── dismiss.go             # Dismiss + impression tracking
  │   │   ├── stats_handler.go       # Stats handler (Prometheus metrics display)
  │   │   ├── index_handler.go       # Landing page handler
  │   │   ├── profile_handler.go     # Public profile handler
++│   │   ├── settings_handler.go    # User settings (language preferences)
  │   │   ├── terms_handler.go       # Terms of service handler
  │   │   ├── pagination.go          # Pagination helpers
  │   │   ├── middleware.go          # Auth, logging, CSRF middleware
                             ├─► Compute feed similarity
                             ├─► Compute user similarity
                             ├─► Compute article embeddings (if embedder configured)
++                           ├─► Detect article languages (if LLM configured)
                             ├─► Compute follow distances
                             ├─► Compute signal profiles
                             └─► Auto-dismiss stale recommendations
  Browser ──GET /dashboard──► Server
                                  │
--                                ├─► Compute recommendations on-demand
++                                ├─► Compute recommendations on-demand (filtered by user's language preferences)
                                  ├─► Fetch feed metadata
                                  └─◄ Render recommendation cards (htmx)
  ```

modified internal/cluster/article.go +106 -0 Viewed

@@ -364,6 +364,112 @@ func (e *Engine) ComputeFeedEmbeddings(ctx context.Context) error {
364	364	return nil
365	365	}
366	366
	367	+// DetectArticleLanguages detects the language of articles that still have the
	368	+// default language ('en') using the embedding model. It processes articles in
	369	+// batches alongside the embedding computation to reuse the same API client.
	370	+func (e *Engine) DetectArticleLanguages(ctx context.Context) error {
	371	+ if e.llm == nil {
	372	+ e.logger.Debug("language detection skipped, no LLM client")
	373	+ return nil
	374	+ }
	375	+
	376	+ conn, err := e.db.Conn(ctx)
	377	+ if err != nil {
	378	+ return err
	379	+ }
	380	+ defer conn.Close()
	381	+
	382	+ rows, err := conn.QueryContext(ctx, `
	383	+ SELECT id, COALESCE(title, '') \|\| ' ' \|\| COALESCE(summary, '')
	384	+ FROM articles.articles
	385	+ WHERE language = ''
	386	+ ORDER BY id DESC
	387	+ LIMIT 5000
	388	+ `)
	389	+ if err != nil {
	390	+ return err
	391	+ }
	392	+
	393	+ type article struct {
	394	+ id int64
	395	+ text string
	396	+ }
	397	+ var batch []article
	398	+ for rows.Next() {
	399	+ var a article
	400	+ if err := rows.Scan(&a.id, &a.text); err != nil {
	401	+ rows.Close()
	402	+ return err
	403	+ }
	404	+ if strings.TrimSpace(a.text) == "" {
	405	+ continue
	406	+ }
	407	+ if len(a.text) > 500 {
	408	+ a.text = a.text[:500]
	409	+ }
	410	+ batch = append(batch, a)
	411	+ }
	412	+ rows.Close()
	413	+
	414	+ if len(batch) == 0 {
	415	+ e.logger.Info("article languages up to date")
	416	+ return nil
	417	+ }
	418	+
	419	+ for i := 0; i < len(batch); i += embedBatchSize {
	420	+ end := min(i+embedBatchSize, len(batch))
	421	+ sub := batch[i:end]
	422	+
	423	+ texts := make([]string, len(sub))
	424	+ for j, a := range sub {
	425	+ texts[j] = a.text
	426	+ }
	427	+
	428	+ langs, err := e.llm.DetectLanguages(ctx, texts)
	429	+ if err != nil {
	430	+ return fmt.Errorf("detect languages batch %d: %w", i/embedBatchSize, err)
	431	+ }
	432	+
	433	+ tx, err := conn.BeginTx(ctx, nil)
	434	+ if err != nil {
	435	+ return err
	436	+ }
	437	+ defer func() { _ = tx.Rollback() }()
	438	+
	439	+ stmt, err := tx.PrepareContext(ctx, `UPDATE articles.articles SET language = ? WHERE id = ? AND language = ''`)
	440	+ if err != nil {
	441	+ return err
	442	+ }
	443	+ defer stmt.Close()
	444	+
	445	+ updated := 0
	446	+ for j, lang := range langs {
	447	+ if lang == "" {
	448	+ lang = "en"
	449	+ }
	450	+ res, err := stmt.ExecContext(ctx, lang, sub[j].id)
	451	+ if err != nil {
	452	+ return fmt.Errorf("update language: %w", err)
	453	+ }
	454	+ n, _ := res.RowsAffected()
	455	+ updated += int(n)
	456	+ }
	457	+
	458	+ if err := tx.Commit(); err != nil {
	459	+ return err
	460	+ }
	461	+
	462	+ e.logger.Info("article languages detected",
	463	+ slog.Int("batch", i/embedBatchSize),
	464	+ slog.Int("count", len(sub)),
	465	+ slog.Int("updated", updated),
	466	+ )
	467	+ }
	468	+
	469	+ e.logger.Info("article languages computed", slog.Int("total", len(batch)))
	470	+ return nil
	471	+}
	472	+
367	473	func (e Engine) ensureContentBoostTable(ctx context.Context, conn sql.Conn) error {
368	474	_, err := conn.ExecContext(ctx, `CREATE TEMP TABLE IF NOT EXISTS _content_boost (article_id INT PRIMARY KEY, score REAL)`)
369	475	if err != nil {

  	return nil
  }
++// DetectArticleLanguages detects the language of articles that still have the
++// default language ('en') using the embedding model. It processes articles in
++// batches alongside the embedding computation to reuse the same API client.
++func (e *Engine) DetectArticleLanguages(ctx context.Context) error {
++	if e.llm == nil {
++		e.logger.Debug("language detection skipped, no LLM client")
++		return nil
++	}
++
++	conn, err := e.db.Conn(ctx)
++	if err != nil {
++		return err
++	}
++	defer conn.Close()
++
++	rows, err := conn.QueryContext(ctx, `
++		SELECT id, COALESCE(title, '') || ' ' || COALESCE(summary, '')
++		FROM articles.articles
++		WHERE language = ''
++		ORDER BY id DESC
++		LIMIT 5000
++	`)
++	if err != nil {
++		return err
++	}
++
++	type article struct {
++		id   int64
++		text string
++	}
++	var batch []article
++	for rows.Next() {
++		var a article
++		if err := rows.Scan(&a.id, &a.text); err != nil {
++			rows.Close()
++			return err
++		}
++		if strings.TrimSpace(a.text) == "" {
++			continue
++		}
++		if len(a.text) > 500 {
++			a.text = a.text[:500]
++		}
++		batch = append(batch, a)
++	}
++	rows.Close()
++
++	if len(batch) == 0 {
++		e.logger.Info("article languages up to date")
++		return nil
++	}
++
++	for i := 0; i < len(batch); i += embedBatchSize {
++		end := min(i+embedBatchSize, len(batch))
++		sub := batch[i:end]
++
++		texts := make([]string, len(sub))
++		for j, a := range sub {
++			texts[j] = a.text
++		}
++
++		langs, err := e.llm.DetectLanguages(ctx, texts)
++		if err != nil {
++			return fmt.Errorf("detect languages batch %d: %w", i/embedBatchSize, err)
++		}
++
++		tx, err := conn.BeginTx(ctx, nil)
++		if err != nil {
++			return err
++		}
++		defer func() { _ = tx.Rollback() }()
++
++		stmt, err := tx.PrepareContext(ctx, `UPDATE articles.articles SET language = ? WHERE id = ? AND language = ''`)
++		if err != nil {
++			return err
++		}
++		defer stmt.Close()
++
++		updated := 0
++		for j, lang := range langs {
++			if lang == "" {
++				lang = "en"
++			}
++			res, err := stmt.ExecContext(ctx, lang, sub[j].id)
++			if err != nil {
++				return fmt.Errorf("update language: %w", err)
++			}
++			n, _ := res.RowsAffected()
++			updated += int(n)
++		}
++
++		if err := tx.Commit(); err != nil {
++			return err
++		}
++
++		e.logger.Info("article languages detected",
++			slog.Int("batch", i/embedBatchSize),
++			slog.Int("count", len(sub)),
++			slog.Int("updated", updated),
++		)
++	}
++
++	e.logger.Info("article languages computed", slog.Int("total", len(batch)))
++	return nil
++}
++
  func (e *Engine) ensureContentBoostTable(ctx context.Context, conn *sql.Conn) error {
  	_, err := conn.ExecContext(ctx, `CREATE TEMP TABLE IF NOT EXISTS _content_boost (article_id INT PRIMARY KEY, score REAL)`)
  	if err != nil {

modified internal/cluster/cron.go +3 -0 Viewed

@@ -44,6 +44,9 @@ func (c *Cron) Run(ctx context.Context) error {
44	44	if err := c.engine.ComputeArticleEmbeddings(ctx); err != nil {
45	45	c.engine.logger.Error("article embeddings failed", "error", err)
46	46	}
	47	+ if err := c.engine.DetectArticleLanguages(ctx); err != nil {
	48	+ c.engine.logger.Error("language detection failed", "error", err)
	49	+ }
47	50	if err := c.engine.ComputeSignalProfiles(ctx); err != nil {
48	51	c.engine.logger.Error("signal profiles failed", "error", err)
49	52	}

  			if err := c.engine.ComputeArticleEmbeddings(ctx); err != nil {
  				c.engine.logger.Error("article embeddings failed", "error", err)
  			}
++			if err := c.engine.DetectArticleLanguages(ctx); err != nil {
++				c.engine.logger.Error("language detection failed", "error", err)
++			}
  			if err := c.engine.ComputeSignalProfiles(ctx); err != nil {
  				c.engine.logger.Error("signal profiles failed", "error", err)
  			}

modified internal/cluster/jaccard.go +9 -4 Viewed

@@ -37,12 +37,17 @@ type Engine struct {
37	37	mu sync.Mutex
38	38	config Config
39	39	embedder Embedder
	40	+ llm *LLMClient
40	41	}
41	42
42		-// NewEngine creates a new recommendation engine. Pass nil for embedder to
43		-// disable content-based signals (no embedding computation, no KNN queries).
44		-func NewEngine(db sql.DB, embedder Embedder, logger slog.Logger) *Engine {
45		- return &Engine{db: db, logger: logger, config: DefaultConfig(), embedder: embedder}
	43	+func NewEngine(db sql.DB, embedder Embedder, llm LLMClient, logger slog.Logger) Engine {
	44	+ return &Engine{
	45	+ db: db,
	46	+ logger: logger,
	47	+ config: DefaultConfig(),
	48	+ embedder: embedder,
	49	+ llm: llm,
	50	+ }
46	51	}
47	52
48	53	// ComputeFeedSimilarity recomputes the feed_similarity table: time-decayed

  	mu       sync.Mutex
  	config   Config
  	embedder Embedder
++	llm      *LLMClient
  }
--// NewEngine creates a new recommendation engine. Pass nil for embedder to
++func NewEngine(db *sql.DB, embedder Embedder, llm *LLMClient, logger *slog.Logger) *Engine {
--// disable content-based signals (no embedding computation, no KNN queries).
++	return &Engine{
--func NewEngine(db *sql.DB, embedder Embedder, logger *slog.Logger) *Engine {
++		db:       db,
--	return &Engine{db: db, logger: logger, config: DefaultConfig(), embedder: embedder}
++		logger:   logger,
++		config:   DefaultConfig(),
++		embedder: embedder,
++		llm:      llm,
++	}
  }
  // ComputeFeedSimilarity recomputes the feed_similarity table: time-decayed

modified internal/cluster/jaccard_test.go +131 -38 Viewed

@@ -94,7 +94,7 @@ func seedFollowData(t testing.T, ctx context.Context, dbs db.Store) {
94	94	}
95	95
96	96	func newTestEngine(dbs db.Store) Engine {
97		- return NewEngine(dbs.SQLDB(), NewMockEmbedder(8), slog.Default())
	97	+ return NewEngine(dbs.SQLDB(), NewMockEmbedder(8), nil, slog.Default())
98	98	}
99	99
100	100	func TestComputeFeedSimilarity(t *testing.T) {
@@ -638,60 +638,153 @@ func TestComputeArticleEmbeddings(t *testing.T) {
638	638	assert.Equal(t, count, 3, "expected 3 article embeddings")
639	639	}
640	640
641		-func TestArticleRecommendationsWithContentBoost(t *testing.T) {
	641	+func seedArticleRecData(t testing.T, ctx context.Context, dbs db.Store) {
	642	+ t.Helper()
	643	+
	644	+ for _, did := range []string{"did:test:alice", "did:test:bob"} {
	645	+ _, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO users (did) VALUES (?)`, did)
	646	+ assert.NilError(t, err)
	647	+ }
	648	+
	649	+ for _, f := range []struct{ url, title string }{
	650	+ {"https://tech.com/feed", "Tech Feed"},
	651	+ {"https://dev.com/feed", "Dev Feed"},
	652	+ {"https://shared.com/feed", "Shared Feed"},
	653	+ } {
	654	+ _, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.feeds (feed_url, title, site_url, feed_type, subscriber_count) VALUES (?, ?, ?, 'rss', 2)`, f.url, f.title, f.url)
	655	+ assert.NilError(t, err)
	656	+ }
	657	+
	658	+ subs := []struct{ user, feed string }{
	659	+ {"did:test:alice", "https://tech.com/feed"},
	660	+ {"did:test:alice", "https://shared.com/feed"},
	661	+ {"did:test:bob", "https://dev.com/feed"},
	662	+ {"did:test:bob", "https://shared.com/feed"},
	663	+ }
	664	+ for _, s := range subs {
	665	+ _, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.subscriptions (user_did, feed_url) VALUES (?, ?)`, s.user, s.feed)
	666	+ assert.NilError(t, err)
	667	+ }
	668	+
	669	+ articles := []struct{ feed, guid, title, summary, url, lang string }{
	670	+ {"https://tech.com/feed", "1", "golang programming tutorial", "learn go programming", "https://tech.com/go", "en"},
	671	+ {"https://dev.com/feed", "2", "rust programming tutorial", "learn rust programming", "https://dev.com/rust", "fr"},
	672	+ {"https://dev.com/feed", "3", "cooking dinner recipes", "easy dinner recipes", "https://dev.com/cook", ""},
	673	+ {"https://dev.com/feed", "4", "python data science", "python for data analysis", "https://dev.com/python", "de"},
	674	+ }
	675	+ for _, a := range articles {
	676	+ _, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.articles (feed_url, guid, title, summary, url, published, language) VALUES (?, ?, ?, ?, ?, datetime('now'), ?)`,
	677	+ a.feed, a.guid, a.title, a.summary, a.url, a.lang)
	678	+ assert.NilError(t, err)
	679	+ }
	680	+
	681	+ likes := []struct{ uri, author, feed, article string }{
	682	+ {"at://alice/like/1", "did:test:alice", "https://tech.com/feed", "https://tech.com/go"},
	683	+ {"at://bob/like/2", "did:test:bob", "https://dev.com/feed", "https://dev.com/rust"},
	684	+ {"at://bob/like/3", "did:test:bob", "https://dev.com/feed", "https://dev.com/cook"},
	685	+ {"at://bob/like/4", "did:test:bob", "https://dev.com/feed", "https://dev.com/python"},
	686	+ }
	687	+ for _, l := range likes {
	688	+ _, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.likes (uri, author_did, feed_url, article_url, created_at) VALUES (?, ?, ?, ?, datetime('now'))`,
	689	+ l.uri, l.author, l.feed, l.article)
	690	+ assert.NilError(t, err)
	691	+ }
	692	+}
	693	+
	694	+func TestArticleRecommendations_LanguageFilter_ShowsUnclassified(t *testing.T) {
642	695	ctx := context.Background()
643	696	dbs := setupClusterTestDB(t)
	697	+ seedArticleRecData(t, ctx, dbs)
644	698
645		- _, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO users (did) VALUES (?)`, "did:test:alice")
646		- assert.NilError(t, err)
647		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO users (did) VALUES (?)`, "did:test:bob")
648		- assert.NilError(t, err)
	699	+ engine := newTestEngine(dbs)
	700	+ assert.NilError(t, engine.ComputeFeedSimilarity(ctx))
	701	+ assert.NilError(t, engine.ComputeUserSimilarity(ctx))
	702	+ assert.NilError(t, engine.ComputeArticleEmbeddings(ctx))
649	703
650		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.feeds (feed_url, title, site_url, feed_type, subscriber_count) VALUES (?, ?, ?, 'rss', 2)`,
651		- "https://tech.com/feed", "Tech Feed", "https://tech.com")
652		- assert.NilError(t, err)
653		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.feeds (feed_url, title, site_url, feed_type, subscriber_count) VALUES (?, ?, ?, 'rss', 2)`,
654		- "https://dev.com/feed", "Dev Feed", "https://dev.com")
655		- assert.NilError(t, err)
656		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.feeds (feed_url, title, site_url, feed_type, subscriber_count) VALUES (?, ?, ?, 'rss', 2)`,
657		- "https://shared.com/feed", "Shared Feed", "https://shared.com")
	704	+ recs, err := engine.GetArticleRecommendations(ctx, "did:test:alice", []string{"en"}, 10)
658	705	assert.NilError(t, err)
	706	+ assert.Assert(t, len(recs) > 0, "alice should get recommendations with language filter")
659	707
660		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.subscriptions (user_did, feed_url) VALUES (?, ?)`, "did:test:alice", "https://tech.com/feed")
661		- assert.NilError(t, err)
662		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.subscriptions (user_did, feed_url) VALUES (?, ?)`, "did:test:alice", "https://shared.com/feed")
663		- assert.NilError(t, err)
664		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.subscriptions (user_did, feed_url) VALUES (?, ?)`, "did:test:bob", "https://dev.com/feed")
665		- assert.NilError(t, err)
666		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.subscriptions (user_did, feed_url) VALUES (?, ?)`, "did:test:bob", "https://shared.com/feed")
667		- assert.NilError(t, err)
	708	+ urls := make(map[string]bool)
	709	+ for _, r := range recs {
	710	+ urls[r.URL] = true
	711	+ }
	712	+ assert.Assert(t, urls["https://tech.com/go"], "should include article with language 'en'")
	713	+ assert.Assert(t, urls["https://dev.com/cook"], "should include article with empty language (unclassified)")
	714	+ assert.Assert(t, !urls["https://dev.com/rust"], "should exclude article with language 'fr'")
	715	+ assert.Assert(t, !urls["https://dev.com/python"], "should exclude article with language 'de'")
	716	+}
668	717
669		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.articles (feed_url, guid, title, summary, url, published) VALUES (?, ?, ?, ?, ?, datetime('now'))`,
670		- "https://tech.com/feed", "1", "golang programming tutorial", "learn go programming", "https://tech.com/go")
671		- assert.NilError(t, err)
672		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.articles (feed_url, guid, title, summary, url, published) VALUES (?, ?, ?, ?, ?, datetime('now'))`,
673		- "https://dev.com/feed", "2", "rust programming tutorial", "learn rust programming", "https://dev.com/rust")
674		- assert.NilError(t, err)
675		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.articles (feed_url, guid, title, summary, url, published) VALUES (?, ?, ?, ?, ?, datetime('now'))`,
676		- "https://dev.com/feed", "3", "cooking dinner recipes", "easy dinner recipes", "https://dev.com/cook")
677		- assert.NilError(t, err)
	718	+func TestArticleRecommendations_LanguageFilter_MultipleLanguages(t *testing.T) {
	719	+ ctx := context.Background()
	720	+ dbs := setupClusterTestDB(t)
	721	+ seedArticleRecData(t, ctx, dbs)
678	722
679		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.likes (uri, author_did, feed_url, article_url, created_at) VALUES (?, ?, ?, ?, datetime('now'))`,
680		- "at://alice/like/1", "did:test:alice", "https://tech.com/feed", "https://tech.com/go")
681		- assert.NilError(t, err)
682		- _, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.likes (uri, author_did, feed_url, article_url, created_at) VALUES (?, ?, ?, ?, datetime('now'))`,
683		- "at://bob/like/1", "did:test:bob", "https://dev.com/feed", "https://dev.com/rust")
	723	+ engine := newTestEngine(dbs)
	724	+ assert.NilError(t, engine.ComputeFeedSimilarity(ctx))
	725	+ assert.NilError(t, engine.ComputeUserSimilarity(ctx))
	726	+ assert.NilError(t, engine.ComputeArticleEmbeddings(ctx))
	727	+
	728	+ recs, err := engine.GetArticleRecommendations(ctx, "did:test:alice", []string{"en", "fr"}, 10)
684	729	assert.NilError(t, err)
	730	+ assert.Assert(t, len(recs) > 0, "alice should get recommendations with multi-language filter")
	731	+
	732	+ urls := make(map[string]bool)
	733	+ for _, r := range recs {
	734	+ urls[r.URL] = true
	735	+ }
	736	+ assert.Assert(t, urls["https://tech.com/go"], "should include article with language 'en'")
	737	+ assert.Assert(t, urls["https://dev.com/rust"], "should include article with language 'fr'")
	738	+ assert.Assert(t, urls["https://dev.com/cook"], "should include article with empty language (unclassified)")
	739	+ assert.Assert(t, !urls["https://dev.com/python"], "should exclude article with language 'de'")
	740	+}
	741	+
	742	+func TestArticleRecommendations_LanguageFilter_NoPreferences(t *testing.T) {
	743	+ ctx := context.Background()
	744	+ dbs := setupClusterTestDB(t)
	745	+ seedArticleRecData(t, ctx, dbs)
685	746
686	747	engine := newTestEngine(dbs)
	748	+ assert.NilError(t, engine.ComputeFeedSimilarity(ctx))
	749	+ assert.NilError(t, engine.ComputeUserSimilarity(ctx))
	750	+ assert.NilError(t, engine.ComputeArticleEmbeddings(ctx))
	751	+
	752	+ recs, err := engine.GetArticleRecommendations(ctx, "did:test:alice", nil, 10)
	753	+ assert.NilError(t, err)
	754	+ assert.Assert(t, len(recs) > 0, "alice should get all recommendations without language filter")
	755	+
	756	+ urls := make(map[string]bool)
	757	+ for _, r := range recs {
	758	+ urls[r.URL] = true
	759	+ }
	760	+ assert.Assert(t, urls["https://tech.com/go"], "should include 'en' article")
	761	+ assert.Assert(t, urls["https://dev.com/rust"], "should include 'fr' article")
	762	+ assert.Assert(t, urls["https://dev.com/cook"], "should include unclassified article")
	763	+ assert.Assert(t, urls["https://dev.com/python"], "should include 'de' article")
	764	+}
	765	+
	766	+func TestArticleRecommendations_LanguageFilter_EmptyPreferences(t *testing.T) {
	767	+ ctx := context.Background()
	768	+ dbs := setupClusterTestDB(t)
	769	+ seedArticleRecData(t, ctx, dbs)
687	770
	771	+ engine := newTestEngine(dbs)
688	772	assert.NilError(t, engine.ComputeFeedSimilarity(ctx))
689	773	assert.NilError(t, engine.ComputeUserSimilarity(ctx))
690	774	assert.NilError(t, engine.ComputeArticleEmbeddings(ctx))
691	775
692		- recs, err := engine.GetArticleRecommendations(ctx, "did:test:alice", 10)
	776	+ recs, err := engine.GetArticleRecommendations(ctx, "did:test:alice", []string{}, 10)
693	777	assert.NilError(t, err)
694		- assert.Assert(t, len(recs) > 0, "alice should get article recommendations")
	778	+ assert.Assert(t, len(recs) > 0, "empty language list should show all articles")
	779	+
	780	+ urls := make(map[string]bool)
	781	+ for _, r := range recs {
	782	+ urls[r.URL] = true
	783	+ }
	784	+ assert.Assert(t, urls["https://tech.com/go"], "should include 'en' article")
	785	+ assert.Assert(t, urls["https://dev.com/rust"], "should include 'fr' article")
	786	+ assert.Assert(t, urls["https://dev.com/cook"], "should include unclassified article")
	787	+ assert.Assert(t, urls["https://dev.com/python"], "should include 'de' article")
695	788	}
696	789
697	790	func TestFeedEmbeddingRecomputedOnDescriptionChange(t *testing.T) {

  }
  func newTestEngine(dbs *db.Store) *Engine {
--	return NewEngine(dbs.SQLDB(), NewMockEmbedder(8), slog.Default())
++	return NewEngine(dbs.SQLDB(), NewMockEmbedder(8), nil, slog.Default())
  }
  func TestComputeFeedSimilarity(t *testing.T) {
  	assert.Equal(t, count, 3, "expected 3 article embeddings")
  }
--func TestArticleRecommendationsWithContentBoost(t *testing.T) {
++func seedArticleRecData(t *testing.T, ctx context.Context, dbs *db.Store) {
++	t.Helper()
++
++	for _, did := range []string{"did:test:alice", "did:test:bob"} {
++		_, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO users (did) VALUES (?)`, did)
++		assert.NilError(t, err)
++	}
++
++	for _, f := range []struct{ url, title string }{
++		{"https://tech.com/feed", "Tech Feed"},
++		{"https://dev.com/feed", "Dev Feed"},
++		{"https://shared.com/feed", "Shared Feed"},
++	} {
++		_, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.feeds (feed_url, title, site_url, feed_type, subscriber_count) VALUES (?, ?, ?, 'rss', 2)`, f.url, f.title, f.url)
++		assert.NilError(t, err)
++	}
++
++	subs := []struct{ user, feed string }{
++		{"did:test:alice", "https://tech.com/feed"},
++		{"did:test:alice", "https://shared.com/feed"},
++		{"did:test:bob", "https://dev.com/feed"},
++		{"did:test:bob", "https://shared.com/feed"},
++	}
++	for _, s := range subs {
++		_, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.subscriptions (user_did, feed_url) VALUES (?, ?)`, s.user, s.feed)
++		assert.NilError(t, err)
++	}
++
++	articles := []struct{ feed, guid, title, summary, url, lang string }{
++		{"https://tech.com/feed", "1", "golang programming tutorial", "learn go programming", "https://tech.com/go", "en"},
++		{"https://dev.com/feed", "2", "rust programming tutorial", "learn rust programming", "https://dev.com/rust", "fr"},
++		{"https://dev.com/feed", "3", "cooking dinner recipes", "easy dinner recipes", "https://dev.com/cook", ""},
++		{"https://dev.com/feed", "4", "python data science", "python for data analysis", "https://dev.com/python", "de"},
++	}
++	for _, a := range articles {
++		_, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.articles (feed_url, guid, title, summary, url, published, language) VALUES (?, ?, ?, ?, ?, datetime('now'), ?)`,
++			a.feed, a.guid, a.title, a.summary, a.url, a.lang)
++		assert.NilError(t, err)
++	}
++
++	likes := []struct{ uri, author, feed, article string }{
++		{"at://alice/like/1", "did:test:alice", "https://tech.com/feed", "https://tech.com/go"},
++		{"at://bob/like/2", "did:test:bob", "https://dev.com/feed", "https://dev.com/rust"},
++		{"at://bob/like/3", "did:test:bob", "https://dev.com/feed", "https://dev.com/cook"},
++		{"at://bob/like/4", "did:test:bob", "https://dev.com/feed", "https://dev.com/python"},
++	}
++	for _, l := range likes {
++		_, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.likes (uri, author_did, feed_url, article_url, created_at) VALUES (?, ?, ?, ?, datetime('now'))`,
++			l.uri, l.author, l.feed, l.article)
++		assert.NilError(t, err)
++	}
++}
++
++func TestArticleRecommendations_LanguageFilter_ShowsUnclassified(t *testing.T) {
  	ctx := context.Background()
  	dbs := setupClusterTestDB(t)
++	seedArticleRecData(t, ctx, dbs)
--	_, err := dbs.SQLDB().ExecContext(ctx, `INSERT INTO users (did) VALUES (?)`, "did:test:alice")
++	engine := newTestEngine(dbs)
--	assert.NilError(t, err)
++	assert.NilError(t, engine.ComputeFeedSimilarity(ctx))
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO users (did) VALUES (?)`, "did:test:bob")
++	assert.NilError(t, engine.ComputeUserSimilarity(ctx))
--	assert.NilError(t, err)
++	assert.NilError(t, engine.ComputeArticleEmbeddings(ctx))
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.feeds (feed_url, title, site_url, feed_type, subscriber_count) VALUES (?, ?, ?, 'rss', 2)`,
++	recs, err := engine.GetArticleRecommendations(ctx, "did:test:alice", []string{"en"}, 10)
--		"https://tech.com/feed", "Tech Feed", "https://tech.com")
--	assert.NilError(t, err)
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.feeds (feed_url, title, site_url, feed_type, subscriber_count) VALUES (?, ?, ?, 'rss', 2)`,
--		"https://dev.com/feed", "Dev Feed", "https://dev.com")
--	assert.NilError(t, err)
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.feeds (feed_url, title, site_url, feed_type, subscriber_count) VALUES (?, ?, ?, 'rss', 2)`,
--		"https://shared.com/feed", "Shared Feed", "https://shared.com")
  	assert.NilError(t, err)
++	assert.Assert(t, len(recs) > 0, "alice should get recommendations with language filter")
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.subscriptions (user_did, feed_url) VALUES (?, ?)`, "did:test:alice", "https://tech.com/feed")
++	urls := make(map[string]bool)
--	assert.NilError(t, err)
++	for _, r := range recs {
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.subscriptions (user_did, feed_url) VALUES (?, ?)`, "did:test:alice", "https://shared.com/feed")
++		urls[r.URL] = true
--	assert.NilError(t, err)
++	}
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.subscriptions (user_did, feed_url) VALUES (?, ?)`, "did:test:bob", "https://dev.com/feed")
++	assert.Assert(t, urls["https://tech.com/go"], "should include article with language 'en'")
--	assert.NilError(t, err)
++	assert.Assert(t, urls["https://dev.com/cook"], "should include article with empty language (unclassified)")
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.subscriptions (user_did, feed_url) VALUES (?, ?)`, "did:test:bob", "https://shared.com/feed")
++	assert.Assert(t, !urls["https://dev.com/rust"], "should exclude article with language 'fr'")
--	assert.NilError(t, err)
++	assert.Assert(t, !urls["https://dev.com/python"], "should exclude article with language 'de'")
++}
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.articles (feed_url, guid, title, summary, url, published) VALUES (?, ?, ?, ?, ?, datetime('now'))`,
++func TestArticleRecommendations_LanguageFilter_MultipleLanguages(t *testing.T) {
--		"https://tech.com/feed", "1", "golang programming tutorial", "learn go programming", "https://tech.com/go")
++	ctx := context.Background()
--	assert.NilError(t, err)
++	dbs := setupClusterTestDB(t)
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.articles (feed_url, guid, title, summary, url, published) VALUES (?, ?, ?, ?, ?, datetime('now'))`,
++	seedArticleRecData(t, ctx, dbs)
--		"https://dev.com/feed", "2", "rust programming tutorial", "learn rust programming", "https://dev.com/rust")
--	assert.NilError(t, err)
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.articles (feed_url, guid, title, summary, url, published) VALUES (?, ?, ?, ?, ?, datetime('now'))`,
--		"https://dev.com/feed", "3", "cooking dinner recipes", "easy dinner recipes", "https://dev.com/cook")
--	assert.NilError(t, err)
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.likes (uri, author_did, feed_url, article_url, created_at) VALUES (?, ?, ?, ?, datetime('now'))`,
++	engine := newTestEngine(dbs)
--		"at://alice/like/1", "did:test:alice", "https://tech.com/feed", "https://tech.com/go")
++	assert.NilError(t, engine.ComputeFeedSimilarity(ctx))
--	assert.NilError(t, err)
++	assert.NilError(t, engine.ComputeUserSimilarity(ctx))
--	_, err = dbs.SQLDB().ExecContext(ctx, `INSERT INTO articles.likes (uri, author_did, feed_url, article_url, created_at) VALUES (?, ?, ?, ?, datetime('now'))`,
++	assert.NilError(t, engine.ComputeArticleEmbeddings(ctx))
--		"at://bob/like/1", "did:test:bob", "https://dev.com/feed", "https://dev.com/rust")
++
++	recs, err := engine.GetArticleRecommendations(ctx, "did:test:alice", []string{"en", "fr"}, 10)
  	assert.NilError(t, err)
++	assert.Assert(t, len(recs) > 0, "alice should get recommendations with multi-language filter")
++
++	urls := make(map[string]bool)
++	for _, r := range recs {
++		urls[r.URL] = true
++	}
++	assert.Assert(t, urls["https://tech.com/go"], "should include article with language 'en'")
++	assert.Assert(t, urls["https://dev.com/rust"], "should include article with language 'fr'")
++	assert.Assert(t, urls["https://dev.com/cook"], "should include article with empty language (unclassified)")
++	assert.Assert(t, !urls["https://dev.com/python"], "should exclude article with language 'de'")
++}
++
++func TestArticleRecommendations_LanguageFilter_NoPreferences(t *testing.T) {
++	ctx := context.Background()
++	dbs := setupClusterTestDB(t)
++	seedArticleRecData(t, ctx, dbs)
  	engine := newTestEngine(dbs)
++	assert.NilError(t, engine.ComputeFeedSimilarity(ctx))
++	assert.NilError(t, engine.ComputeUserSimilarity(ctx))
++	assert.NilError(t, engine.ComputeArticleEmbeddings(ctx))
++
++	recs, err := engine.GetArticleRecommendations(ctx, "did:test:alice", nil, 10)
++	assert.NilError(t, err)
++	assert.Assert(t, len(recs) > 0, "alice should get all recommendations without language filter")
++
++	urls := make(map[string]bool)
++	for _, r := range recs {
++		urls[r.URL] = true
++	}
++	assert.Assert(t, urls["https://tech.com/go"], "should include 'en' article")
++	assert.Assert(t, urls["https://dev.com/rust"], "should include 'fr' article")
++	assert.Assert(t, urls["https://dev.com/cook"], "should include unclassified article")
++	assert.Assert(t, urls["https://dev.com/python"], "should include 'de' article")
++}
++
++func TestArticleRecommendations_LanguageFilter_EmptyPreferences(t *testing.T) {
++	ctx := context.Background()
++	dbs := setupClusterTestDB(t)
++	seedArticleRecData(t, ctx, dbs)
++	engine := newTestEngine(dbs)
  	assert.NilError(t, engine.ComputeFeedSimilarity(ctx))
  	assert.NilError(t, engine.ComputeUserSimilarity(ctx))
  	assert.NilError(t, engine.ComputeArticleEmbeddings(ctx))
--	recs, err := engine.GetArticleRecommendations(ctx, "did:test:alice", 10)
++	recs, err := engine.GetArticleRecommendations(ctx, "did:test:alice", []string{}, 10)
  	assert.NilError(t, err)
--	assert.Assert(t, len(recs) > 0, "alice should get article recommendations")
++	assert.Assert(t, len(recs) > 0, "empty language list should show all articles")
++
++	urls := make(map[string]bool)
++	for _, r := range recs {
++		urls[r.URL] = true
++	}
++	assert.Assert(t, urls["https://tech.com/go"], "should include 'en' article")
++	assert.Assert(t, urls["https://dev.com/rust"], "should include 'fr' article")
++	assert.Assert(t, urls["https://dev.com/cook"], "should include unclassified article")
++	assert.Assert(t, urls["https://dev.com/python"], "should include 'de' article")
  }
  func TestFeedEmbeddingRecomputedOnDescriptionChange(t *testing.T) {

added internal/cluster/llm.go +77 -0 Viewed

new file mode 100644
@@ -0,0 +1,77 @@
	1	+package cluster
	2	+
	3	+import (
	4	+ "context"
	5	+ "fmt"
	6	+ "strings"
	7	+
	8	+ "github.com/openai/openai-go"
	9	+ "github.com/openai/openai-go/option"
	10	+)
	11	+
	12	+type LLMClient struct {
	13	+ client openai.Client
	14	+ model string
	15	+}
	16	+
	17	+type LLMClientConfig struct {
	18	+ BaseURL string
	19	+ APIKey string
	20	+ Model string
	21	+}
	22	+
	23	+func NewLLMClient(cfg LLMClientConfig) *LLMClient {
	24	+ opts := []option.RequestOption{}
	25	+ if cfg.BaseURL != "" {
	26	+ opts = append(opts, option.WithBaseURL(cfg.BaseURL))
	27	+ }
	28	+ if cfg.APIKey != "" {
	29	+ opts = append(opts, option.WithAPIKey(cfg.APIKey))
	30	+ }
	31	+ return &LLMClient{
	32	+ client: openai.NewClient(opts...),
	33	+ model: cfg.Model,
	34	+ }
	35	+}
	36	+
	37	+func (c *LLMClient) DetectLanguages(ctx context.Context, texts []string) ([]string, error) {
	38	+ var b strings.Builder
	39	+ b.WriteString("For each text below, respond with ONLY the ISO 639-1 language code (e.g. en, fr, de, es, pt, it, ru, ja, zh, ko, ar). One code per line, same order as input. If uncertain, respond with 'en'.\n\n")
	40	+ for i, t := range texts {
	41	+ truncated := t
	42	+ if len(truncated) > 500 {
	43	+ truncated = truncated[:500]
	44	+ }
	45	+ fmt.Fprintf(&b, "%d. %s\n", i+1, truncated)
	46	+ }
	47	+
	48	+ resp, err := c.client.Chat.Completions.New(ctx, openai.ChatCompletionNewParams{
	49	+ Model: c.model,
	50	+ Messages: []openai.ChatCompletionMessageParamUnion{
	51	+ openai.UserMessage(b.String()),
	52	+ },
	53	+ Temperature: openai.Float(0.0),
	54	+ })
	55	+ if err != nil {
	56	+ return nil, err
	57	+ }
	58	+
	59	+ content := resp.Choices[0].Message.Content
	60	+ lines := strings.Split(strings.TrimSpace(content), "\n")
	61	+ result := make([]string, len(texts))
	62	+ for i := range result {
	63	+ result[i] = "en"
	64	+ }
	65	+ for i, line := range lines {
	66	+ if i >= len(result) {
	67	+ break
	68	+ }
	69	+ code := strings.TrimSpace(line)
	70	+ code = strings.TrimPrefix(code, fmt.Sprintf("%d.", i+1))
	71	+ code = strings.TrimSpace(code)
	72	+ if code != "" {
	73	+ result[i] = code
	74	+ }
	75	+ }
	76	+ return result, nil
	77	+}

++package cluster
++
++import (
++	"context"
++	"fmt"
++	"strings"
++
++	"github.com/openai/openai-go"
++	"github.com/openai/openai-go/option"
++)
++
++type LLMClient struct {
++	client openai.Client
++	model  string
++}
++
++type LLMClientConfig struct {
++	BaseURL string
++	APIKey  string
++	Model   string
++}
++
++func NewLLMClient(cfg LLMClientConfig) *LLMClient {
++	opts := []option.RequestOption{}
++	if cfg.BaseURL != "" {
++		opts = append(opts, option.WithBaseURL(cfg.BaseURL))
++	}
++	if cfg.APIKey != "" {
++		opts = append(opts, option.WithAPIKey(cfg.APIKey))
++	}
++	return &LLMClient{
++		client: openai.NewClient(opts...),
++		model:  cfg.Model,
++	}
++}
++
++func (c *LLMClient) DetectLanguages(ctx context.Context, texts []string) ([]string, error) {
++	var b strings.Builder
++	b.WriteString("For each text below, respond with ONLY the ISO 639-1 language code (e.g. en, fr, de, es, pt, it, ru, ja, zh, ko, ar). One code per line, same order as input. If uncertain, respond with 'en'.\n\n")
++	for i, t := range texts {
++		truncated := t
++		if len(truncated) > 500 {
++			truncated = truncated[:500]
++		}
++		fmt.Fprintf(&b, "%d. %s\n", i+1, truncated)
++	}
++
++	resp, err := c.client.Chat.Completions.New(ctx, openai.ChatCompletionNewParams{
++		Model: c.model,
++		Messages: []openai.ChatCompletionMessageParamUnion{
++			openai.UserMessage(b.String()),
++		},
++		Temperature: openai.Float(0.0),
++	})
++	if err != nil {
++		return nil, err
++	}
++
++	content := resp.Choices[0].Message.Content
++	lines := strings.Split(strings.TrimSpace(content), "\n")
++	result := make([]string, len(texts))
++	for i := range result {
++		result[i] = "en"
++	}
++	for i, line := range lines {
++		if i >= len(result) {
++			break
++		}
++		code := strings.TrimSpace(line)
++		code = strings.TrimPrefix(code, fmt.Sprintf("%d.", i+1))
++		code = strings.TrimSpace(code)
++		if code != "" {
++			result[i] = code
++		}
++	}
++	return result, nil
++}

modified internal/cluster/scoring.go +25 -7 Viewed

@@ -4,6 +4,7 @@ import (
4	4	"context"
5	5	"database/sql"
6	6	"fmt"
	7	+ "strings"
7	8	)
8	9
9	10	type FeedRecommendation struct {
@@ -102,8 +103,8 @@ func (e *Engine) GetPeopleRecommendations(ctx context.Context, userDID string, l
102	103	// signals (liked by similar users, followed users' feeds), content similarity
103	104	// (embedding KNN against user's liked articles), and recency. Scores are
104	105	// min-max normalized.
105		-func (e Engine) GetArticleRecommendations(ctx context.Context, userDID string, limit int) ([]ArticleRecommendation, error) {
106		- recs, err := e.ComputeArticleRecommendationsOnDemand(ctx, userDID, limit)
	106	+func (e Engine) GetArticleRecommendations(ctx context.Context, userDID string, languages []string, limit int) ([]ArticleRecommendation, error) {
	107	+ recs, err := e.ComputeArticleRecommendationsOnDemand(ctx, userDID, languages, limit)
107	108	if err != nil {
108	109	return nil, err
109	110	}
@@ -389,7 +390,7 @@ func (e *Engine) coldStartFromEmbeddings(ctx context.Context, userDID string, li
389	390	return results, nil
390	391	}
391	392
392		-func (e Engine) ComputeArticleRecommendationsOnDemand(ctx context.Context, userDID string, limit int) ([]ArticleRecommendation, error) {
	393	+func (e Engine) ComputeArticleRecommendationsOnDemand(ctx context.Context, userDID string, languages []string, limit int) ([]ArticleRecommendation, error) {
393	394	w := e.GetWeights(ctx, userDID)
394	395
395	396	conn, err := e.db.Conn(ctx)
@@ -408,7 +409,18 @@ func (e *Engine) ComputeArticleRecommendationsOnDemand(ctx context.Context, user
408	409	}
409	410	}
410	411
411		- rows, err := conn.QueryContext(ctx, `
	412	+ langFilter := ""
	413	+ langArgs := []any{}
	414	+ if len(languages) > 0 {
	415	+ ph := make([]string, len(languages))
	416	+ for i, l := range languages {
	417	+ ph[i] = "?"
	418	+ langArgs = append(langArgs, l)
	419	+ }
	420	+ langFilter = " AND (a.language IN (" + strings.Join(ph, ",") + ") OR a.language = '')"
	421	+ }
	422	+
	423	+ query := fmt.Sprintf(`
412	424	WITH similar_users AS (
413	425	SELECT user_b AS peer, jaccard FROM recs.user_similarity WHERE user_a = ? AND jaccard > 0.15
414	426	UNION ALL
@@ -453,11 +465,17 @@ func (e *Engine) ComputeArticleRecommendationsOnDemand(ctx context.Context, user
453	465	LEFT JOIN social_likes sl ON sl.feed_url = la.feed_url AND sl.article_url = la.article_url
454	466	LEFT JOIN _content_boost cb ON cb.article_id = a.id
455	467	LEFT JOIN articles.read_state rs ON rs.article_id = a.id AND rs.user_did = ?
456		- WHERE COALESCE(rs.is_read, 0) = 0
	468	+ WHERE COALESCE(rs.is_read, 0) = 0%s
457	469	ORDER BY score DESC, (CASE WHEN a.published > 'now' THEN 1 ELSE 0 END), a.published DESC
458	470	LIMIT ?
459		- `, userDID, userDID, userDID, userDID, userDID, userDID,
460		- w.WLike, w.WSocial, w.WContent, userDID, limit)
	471	+ `, langFilter)
	472	+
	473	+ args := []any{userDID, userDID, userDID, userDID, userDID, userDID,
	474	+ w.WLike, w.WSocial, w.WContent, userDID}
	475	+ args = append(args, langArgs...)
	476	+ args = append(args, limit)
	477	+
	478	+ rows, err := conn.QueryContext(ctx, query, args...)
461	479	if err != nil {
462	480	return nil, err
463	481	}

  	"context"
  	"database/sql"
  	"fmt"
++	"strings"
  )
  type FeedRecommendation struct {
  // signals (liked by similar users, followed users' feeds), content similarity
  // (embedding KNN against user's liked articles), and recency. Scores are
  // min-max normalized.
--func (e *Engine) GetArticleRecommendations(ctx context.Context, userDID string, limit int) ([]*ArticleRecommendation, error) {
++func (e *Engine) GetArticleRecommendations(ctx context.Context, userDID string, languages []string, limit int) ([]*ArticleRecommendation, error) {
--	recs, err := e.ComputeArticleRecommendationsOnDemand(ctx, userDID, limit)
++	recs, err := e.ComputeArticleRecommendationsOnDemand(ctx, userDID, languages, limit)
  	if err != nil {
  		return nil, err
  	}
  	return results, nil
  }
--func (e *Engine) ComputeArticleRecommendationsOnDemand(ctx context.Context, userDID string, limit int) ([]*ArticleRecommendation, error) {
++func (e *Engine) ComputeArticleRecommendationsOnDemand(ctx context.Context, userDID string, languages []string, limit int) ([]*ArticleRecommendation, error) {
  	w := e.GetWeights(ctx, userDID)
  	conn, err := e.db.Conn(ctx)
  		}
  	}
--	rows, err := conn.QueryContext(ctx, `
++	langFilter := ""
++	langArgs := []any{}
++	if len(languages) > 0 {
++		ph := make([]string, len(languages))
++		for i, l := range languages {
++			ph[i] = "?"
++			langArgs = append(langArgs, l)
++		}
++		langFilter = " AND (a.language IN (" + strings.Join(ph, ",") + ") OR a.language = '')"
++	}
++
++	query := fmt.Sprintf(`
  		WITH similar_users AS (
  			SELECT user_b AS peer, jaccard FROM recs.user_similarity WHERE user_a = ? AND jaccard > 0.15
  			UNION ALL
  		LEFT JOIN social_likes sl ON sl.feed_url = la.feed_url AND sl.article_url = la.article_url
  		LEFT JOIN _content_boost cb ON cb.article_id = a.id
  		LEFT JOIN articles.read_state rs ON rs.article_id = a.id AND rs.user_did = ?
--		WHERE COALESCE(rs.is_read, 0) = 0
++		WHERE COALESCE(rs.is_read, 0) = 0%s
  		ORDER BY score DESC, (CASE WHEN a.published > 'now' THEN 1 ELSE 0 END), a.published DESC
  		LIMIT ?
--	`, userDID, userDID, userDID, userDID, userDID, userDID,
++	`, langFilter)
--		w.WLike, w.WSocial, w.WContent, userDID, limit)
++
++	args := []any{userDID, userDID, userDID, userDID, userDID, userDID,
++		w.WLike, w.WSocial, w.WContent, userDID}
++	args = append(args, langArgs...)
++	args = append(args, limit)
++
++	rows, err := conn.QueryContext(ctx, query, args...)
  	if err != nil {
  		return nil, err
  	}

modified internal/db/article.go +2 -2 Viewed

@@ -63,8 +63,8 @@ func (s *ArticleStore) BatchUpsertArticles(ctx context.Context, articles []feed.
63	63	defer tx.Rollback()
64	64
65	65	stmt, err := tx.PrepareContext(ctx, `
66		- INSERT INTO articles.articles (feed_url, guid, title, url, author, summary, content, published, updated)
67		- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
	66	+ INSERT INTO articles.articles (feed_url, guid, title, url, author, summary, content, published, updated, language)
	67	+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, '')
68	68	ON CONFLICT(feed_url, guid) DO NOTHING
69	69	`)
70	70	if err != nil {

  	defer tx.Rollback()
  	stmt, err := tx.PrepareContext(ctx, `
--		INSERT INTO articles.articles (feed_url, guid, title, url, author, summary, content, published, updated)
++		INSERT INTO articles.articles (feed_url, guid, title, url, author, summary, content, published, updated, language)
--		VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
++		VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, '')
  		ON CONFLICT(feed_url, guid) DO NOTHING
  	`)
  	if err != nil {

modified internal/db/db.go +8 -0 Viewed

@@ -236,6 +236,12 @@ var usersSchema = []string{
236	236	`CREATE INDEX IF NOT EXISTS idx_follows_uri ON follows(uri)`,
237	237	`CREATE INDEX IF NOT EXISTS idx_follows_followed_at ON follows(followed_at)`,
238	238
	239	+ `CREATE TABLE IF NOT EXISTS user_settings (
	240	+ did TEXT PRIMARY KEY,
	241	+ languages TEXT,
	242	+ updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
	243	+ )`,
	244	+
239	245	`CREATE TABLE IF NOT EXISTS dismissed_recommendations (
240	246	user_did TEXT NOT NULL,
241	247	target_type TEXT NOT NULL CHECK(target_type IN ('feed', 'article', 'person')),
@@ -301,6 +307,7 @@ var articlesSchema = []string{
301	307	published DATETIME,
302	308	updated DATETIME,
303	309	fetched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
	310	+ language TEXT NOT NULL DEFAULT '',
304	311	UNIQUE(feed_url, guid)
305	312	)`,
306	313
@@ -353,6 +360,7 @@ var articlesSchema = []string{
353	360	`CREATE INDEX IF NOT EXISTS articles.idx_likes_article ON likes(feed_url, article_url)`,
354	361	`CREATE INDEX IF NOT EXISTS articles.idx_likes_author ON likes(author_did)`,
355	362	`CREATE INDEX IF NOT EXISTS articles.idx_likes_created_at ON likes(created_at DESC)`,
	363	+ `CREATE INDEX IF NOT EXISTS articles.idx_articles_language ON articles(language)`,
356	364
357	365	`CREATE VIRTUAL TABLE IF NOT EXISTS articles.articles_fts USING fts5(title, summary, content, author, content=articles, content_rowid=id)`,
358	366	`CREATE TRIGGER IF NOT EXISTS articles.articles_ai AFTER INSERT ON articles BEGIN

  	`CREATE INDEX IF NOT EXISTS idx_follows_uri ON follows(uri)`,
  	`CREATE INDEX IF NOT EXISTS idx_follows_followed_at ON follows(followed_at)`,
++	`CREATE TABLE IF NOT EXISTS user_settings (
++		did TEXT PRIMARY KEY,
++		languages TEXT,
++		updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
++	)`,
++
  	`CREATE TABLE IF NOT EXISTS dismissed_recommendations (
  		user_did     TEXT NOT NULL,
  		target_type  TEXT NOT NULL CHECK(target_type IN ('feed', 'article', 'person')),
  		published DATETIME,
  		updated DATETIME,
  		fetched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
++		language TEXT NOT NULL DEFAULT '',
  		UNIQUE(feed_url, guid)
  	)`,
  	`CREATE INDEX IF NOT EXISTS articles.idx_likes_article ON likes(feed_url, article_url)`,
  	`CREATE INDEX IF NOT EXISTS articles.idx_likes_author ON likes(author_did)`,
  	`CREATE INDEX IF NOT EXISTS articles.idx_likes_created_at ON likes(created_at DESC)`,
++	`CREATE INDEX IF NOT EXISTS articles.idx_articles_language ON articles(language)`,
  	`CREATE VIRTUAL TABLE IF NOT EXISTS articles.articles_fts USING fts5(title, summary, content, author, content=articles, content_rowid=id)`,
  	`CREATE TRIGGER IF NOT EXISTS articles.articles_ai AFTER INSERT ON articles BEGIN

modified internal/db/migrations.go +33 -1 Viewed

@@ -13,7 +13,7 @@ func init() {
13	13	}
14	14
15	15	// SchemaVersion must be incremented each time a migration is added to the migrations slice (used so that fresh dbs skip running migrations).
16		-const SchemaVersion = 2
	16	+const SchemaVersion = 3
17	17
18	18	type migration struct {
19	19	id int
@@ -32,6 +32,11 @@ var migrations = []migration{
32	32	name: "feed_type_atproto",
33	33	run: migrateFeedTypeATProto,
34	34	},
	35	+ {
	36	+ id: 3,
	37	+ name: "article_language_user_languages",
	38	+ run: migrateArticleLanguageUserLanguages,
	39	+ },
35	40	}
36	41
37	42	func runMigrations(db *DB) error {
@@ -181,3 +186,30 @@ func migrateAddPersonTargetType(db *DB) error {
181	186
182	187	return nil
183	188	}
	189	+
	190	+func migrateArticleLanguageUserLanguages(db *DB) error {
	191	+ var colCount int
	192	+ err := db.QueryRow("SELECT COUNT(*) FROM pragma_table_info('articles.articles') WHERE name='language'").Scan(&colCount)
	193	+ if err != nil {
	194	+ return fmt.Errorf("check articles.language column: %w", err)
	195	+ }
	196	+ if colCount == 0 {
	197	+ if _, err := db.Exec("ALTER TABLE articles.articles ADD COLUMN language TEXT NOT NULL DEFAULT ''"); err != nil {
	198	+ return fmt.Errorf("add articles.language: %w", err)
	199	+ }
	200	+ if _, err := db.Exec("CREATE INDEX IF NOT EXISTS articles.idx_articles_language ON articles(language)"); err != nil {
	201	+ return fmt.Errorf("create articles.language index: %w", err)
	202	+ }
	203	+ }
	204	+
	205	+ _, err = db.Exec(`CREATE TABLE IF NOT EXISTS user_settings (
	206	+ did TEXT PRIMARY KEY,
	207	+ languages TEXT,
	208	+ updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
	209	+ )`)
	210	+ if err != nil {
	211	+ return fmt.Errorf("create user_settings: %w", err)
	212	+ }
	213	+
	214	+ return nil
	215	+}

  }
  // SchemaVersion must be incremented each time a migration is added to the migrations slice (used so that fresh dbs skip running migrations).
--const SchemaVersion = 2
++const SchemaVersion = 3
  type migration struct {
  	id   int
  		name: "feed_type_atproto",
  		run:  migrateFeedTypeATProto,
  	},
++	{
++		id:   3,
++		name: "article_language_user_languages",
++		run:  migrateArticleLanguageUserLanguages,
++	},
  }
  func runMigrations(db *DB) error {
  	return nil
  }
++
++func migrateArticleLanguageUserLanguages(db *DB) error {
++	var colCount int
++	err := db.QueryRow("SELECT COUNT(*) FROM pragma_table_info('articles.articles') WHERE name='language'").Scan(&colCount)
++	if err != nil {
++		return fmt.Errorf("check articles.language column: %w", err)
++	}
++	if colCount == 0 {
++		if _, err := db.Exec("ALTER TABLE articles.articles ADD COLUMN language TEXT NOT NULL DEFAULT ''"); err != nil {
++			return fmt.Errorf("add articles.language: %w", err)
++		}
++		if _, err := db.Exec("CREATE INDEX IF NOT EXISTS articles.idx_articles_language ON articles(language)"); err != nil {
++			return fmt.Errorf("create articles.language index: %w", err)
++		}
++	}
++
++	_, err = db.Exec(`CREATE TABLE IF NOT EXISTS user_settings (
++		did TEXT PRIMARY KEY,
++		languages TEXT,
++		updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
++	)`)
++	if err != nil {
++		return fmt.Errorf("create user_settings: %w", err)
++	}
++
++	return nil
++}

added internal/db/user_settings.go +48 -0 Viewed

new file mode 100644
@@ -0,0 +1,48 @@
	1	+package db
	2	+
	3	+import (
	4	+ "context"
	5	+ "database/sql"
	6	+ "strings"
	7	+)
	8	+
	9	+type UserSettings struct {
	10	+ DID string
	11	+ Languages []string
	12	+}
	13	+
	14	+func (s UserStore) GetSettings(ctx context.Context, did string) (UserSettings, error) {
	15	+ var langs sql.NullString
	16	+ err := s.db.QueryRowContext(ctx, `SELECT languages FROM user_settings WHERE did = ?`, did).Scan(&langs)
	17	+ if err != nil {
	18	+ if err == sql.ErrNoRows {
	19	+ return &UserSettings{DID: did}, nil
	20	+ }
	21	+ return nil, err
	22	+ }
	23	+ us := &UserSettings{DID: did}
	24	+ if langs.Valid && langs.String != "" {
	25	+ us.Languages = strings.Split(langs.String, ",")
	26	+ }
	27	+ return us, nil
	28	+}
	29	+
	30	+func (s *UserStore) UpdateLanguages(ctx context.Context, did string, languages []string) error {
	31	+ langStr := strings.Join(languages, ",")
	32	+ _, err := s.db.ExecContext(ctx, `
	33	+ INSERT INTO user_settings (did, languages, updated_at) VALUES (?, ?, CURRENT_TIMESTAMP)
	34	+ ON CONFLICT(did) DO UPDATE SET languages = excluded.languages, updated_at = CURRENT_TIMESTAMP
	35	+ `, did, langStr)
	36	+ return err
	37	+}
	38	+
	39	+func (s *UserStore) GetLanguages(ctx context.Context, did string) ([]string, error) {
	40	+ us, err := s.GetSettings(ctx, did)
	41	+ if err != nil {
	42	+ return nil, err
	43	+ }
	44	+ if len(us.Languages) == 0 {
	45	+ return nil, nil
	46	+ }
	47	+ return us.Languages, nil
	48	+}

++package db
++
++import (
++	"context"
++	"database/sql"
++	"strings"
++)
++
++type UserSettings struct {
++	DID       string
++	Languages []string
++}
++
++func (s *UserStore) GetSettings(ctx context.Context, did string) (*UserSettings, error) {
++	var langs sql.NullString
++	err := s.db.QueryRowContext(ctx, `SELECT languages FROM user_settings WHERE did = ?`, did).Scan(&langs)
++	if err != nil {
++		if err == sql.ErrNoRows {
++			return &UserSettings{DID: did}, nil
++		}
++		return nil, err
++	}
++	us := &UserSettings{DID: did}
++	if langs.Valid && langs.String != "" {
++		us.Languages = strings.Split(langs.String, ",")
++	}
++	return us, nil
++}
++
++func (s *UserStore) UpdateLanguages(ctx context.Context, did string, languages []string) error {
++	langStr := strings.Join(languages, ",")
++	_, err := s.db.ExecContext(ctx, `
++		INSERT INTO user_settings (did, languages, updated_at) VALUES (?, ?, CURRENT_TIMESTAMP)
++		ON CONFLICT(did) DO UPDATE SET languages = excluded.languages, updated_at = CURRENT_TIMESTAMP
++	`, did, langStr)
++	return err
++}
++
++func (s *UserStore) GetLanguages(ctx context.Context, did string) ([]string, error) {
++	us, err := s.GetSettings(ctx, did)
++	if err != nil {
++		return nil, err
++	}
++	if len(us.Languages) == 0 {
++		return nil, nil
++	}
++	return us.Languages, nil
++}

modified internal/db/user_test.go +60 -0 Viewed

@@ -19,3 +19,63 @@ func TestGetUser(t *testing.T) {
19	19	assert.NilError(t, err)
20	20	assert.Equal(t, got.DID, "did:test:profile")
21	21	}
	22	+
	23	+func TestUpdateLanguages(t *testing.T) {
	24	+ ctx := context.Background()
	25	+ dbs := setupTestDB(t)
	26	+
	27	+ _, err := dbs.Users.CreateUser(ctx, "did:test:langs")
	28	+ assert.NilError(t, err)
	29	+
	30	+ err = dbs.Users.UpdateLanguages(ctx, "did:test:langs", []string{"en", "fr", "de"})
	31	+ assert.NilError(t, err)
	32	+
	33	+ langs, err := dbs.Users.GetLanguages(ctx, "did:test:langs")
	34	+ assert.NilError(t, err)
	35	+ assert.DeepEqual(t, langs, []string{"en", "fr", "de"})
	36	+}
	37	+
	38	+func TestGetLanguages_Empty(t *testing.T) {
	39	+ ctx := context.Background()
	40	+ dbs := setupTestDB(t)
	41	+
	42	+ _, err := dbs.Users.CreateUser(ctx, "did:test:nolangs")
	43	+ assert.NilError(t, err)
	44	+
	45	+ langs, err := dbs.Users.GetLanguages(ctx, "did:test:nolangs")
	46	+ assert.NilError(t, err)
	47	+ assert.Assert(t, langs == nil)
	48	+}
	49	+
	50	+func TestGetLanguages_Set(t *testing.T) {
	51	+ ctx := context.Background()
	52	+ dbs := setupTestDB(t)
	53	+
	54	+ _, err := dbs.Users.CreateUser(ctx, "did:test:langset")
	55	+ assert.NilError(t, err)
	56	+
	57	+ err = dbs.Users.UpdateLanguages(ctx, "did:test:langset", []string{"en", "ja"})
	58	+ assert.NilError(t, err)
	59	+
	60	+ langs, err := dbs.Users.GetLanguages(ctx, "did:test:langset")
	61	+ assert.NilError(t, err)
	62	+ assert.DeepEqual(t, langs, []string{"en", "ja"})
	63	+}
	64	+
	65	+func TestGetLanguages_ClearToEmpty(t *testing.T) {
	66	+ ctx := context.Background()
	67	+ dbs := setupTestDB(t)
	68	+
	69	+ _, err := dbs.Users.CreateUser(ctx, "did:test:clearlangs")
	70	+ assert.NilError(t, err)
	71	+
	72	+ err = dbs.Users.UpdateLanguages(ctx, "did:test:clearlangs", []string{"en"})
	73	+ assert.NilError(t, err)
	74	+
	75	+ err = dbs.Users.UpdateLanguages(ctx, "did:test:clearlangs", nil)
	76	+ assert.NilError(t, err)
	77	+
	78	+ langs, err := dbs.Users.GetLanguages(ctx, "did:test:clearlangs")
	79	+ assert.NilError(t, err)
	80	+ assert.Assert(t, langs == nil)
	81	+}

  	assert.NilError(t, err)
  	assert.Equal(t, got.DID, "did:test:profile")
  }
++
++func TestUpdateLanguages(t *testing.T) {
++	ctx := context.Background()
++	dbs := setupTestDB(t)
++
++	_, err := dbs.Users.CreateUser(ctx, "did:test:langs")
++	assert.NilError(t, err)
++
++	err = dbs.Users.UpdateLanguages(ctx, "did:test:langs", []string{"en", "fr", "de"})
++	assert.NilError(t, err)
++
++	langs, err := dbs.Users.GetLanguages(ctx, "did:test:langs")
++	assert.NilError(t, err)
++	assert.DeepEqual(t, langs, []string{"en", "fr", "de"})
++}
++
++func TestGetLanguages_Empty(t *testing.T) {
++	ctx := context.Background()
++	dbs := setupTestDB(t)
++
++	_, err := dbs.Users.CreateUser(ctx, "did:test:nolangs")
++	assert.NilError(t, err)
++
++	langs, err := dbs.Users.GetLanguages(ctx, "did:test:nolangs")
++	assert.NilError(t, err)
++	assert.Assert(t, langs == nil)
++}
++
++func TestGetLanguages_Set(t *testing.T) {
++	ctx := context.Background()
++	dbs := setupTestDB(t)
++
++	_, err := dbs.Users.CreateUser(ctx, "did:test:langset")
++	assert.NilError(t, err)
++
++	err = dbs.Users.UpdateLanguages(ctx, "did:test:langset", []string{"en", "ja"})
++	assert.NilError(t, err)
++
++	langs, err := dbs.Users.GetLanguages(ctx, "did:test:langset")
++	assert.NilError(t, err)
++	assert.DeepEqual(t, langs, []string{"en", "ja"})
++}
++
++func TestGetLanguages_ClearToEmpty(t *testing.T) {
++	ctx := context.Background()
++	dbs := setupTestDB(t)
++
++	_, err := dbs.Users.CreateUser(ctx, "did:test:clearlangs")
++	assert.NilError(t, err)
++
++	err = dbs.Users.UpdateLanguages(ctx, "did:test:clearlangs", []string{"en"})
++	assert.NilError(t, err)
++
++	err = dbs.Users.UpdateLanguages(ctx, "did:test:clearlangs", nil)
++	assert.NilError(t, err)
++
++	langs, err := dbs.Users.GetLanguages(ctx, "did:test:clearlangs")
++	assert.NilError(t, err)
++	assert.Assert(t, langs == nil)
++}

added internal/langdetect/langdetect.go +24 -0 Viewed

new file mode 100644
@@ -0,0 +1,24 @@
	1	+package langdetect
	2	+
	3	+type Language struct {
	4	+ Code string
	5	+ Name string
	6	+}
	7	+
	8	+var knownLanguages = []Language{
	9	+ {"en", "English"},
	10	+ {"fr", "French"},
	11	+ {"de", "German"},
	12	+ {"es", "Spanish"},
	13	+ {"pt", "Portuguese"},
	14	+ {"it", "Italian"},
	15	+ {"ru", "Russian"},
	16	+ {"ja", "Japanese"},
	17	+ {"zh", "Chinese"},
	18	+ {"ko", "Korean"},
	19	+ {"ar", "Arabic"},
	20	+}
	21	+
	22	+func KnownLanguages() []Language {
	23	+ return knownLanguages
	24	+}

++package langdetect
++
++type Language struct {
++	Code string
++	Name string
++}
++
++var knownLanguages = []Language{
++	{"en", "English"},
++	{"fr", "French"},
++	{"de", "German"},
++	{"es", "Spanish"},
++	{"pt", "Portuguese"},
++	{"it", "Italian"},
++	{"ru", "Russian"},
++	{"ja", "Japanese"},
++	{"zh", "Chinese"},
++	{"ko", "Korean"},
++	{"ar", "Arabic"},
++}
++
++func KnownLanguages() []Language {
++	return knownLanguages
++}

added internal/langdetect/langdetect_test.go +20 -0 Viewed

new file mode 100644
@@ -0,0 +1,20 @@
	1	+package langdetect
	2	+
	3	+import (
	4	+ "testing"
	5	+
	6	+ "gotest.tools/v3/assert"
	7	+)
	8	+
	9	+func TestKnownLanguages(t *testing.T) {
	10	+ langs := KnownLanguages()
	11	+ assert.Assert(t, len(langs) > 0)
	12	+ found := false
	13	+ for _, l := range langs {
	14	+ if l.Code == "en" {
	15	+ found = true
	16	+ assert.Equal(t, l.Name, "English")
	17	+ }
	18	+ }
	19	+ assert.Assert(t, found)
	20	+}

++package langdetect
++
++import (
++	"testing"
++
++	"gotest.tools/v3/assert"
++)
++
++func TestKnownLanguages(t *testing.T) {
++	langs := KnownLanguages()
++	assert.Assert(t, len(langs) > 0)
++	found := false
++	for _, l := range langs {
++		if l.Code == "en" {
++			found = true
++			assert.Equal(t, l.Name, "English")
++		}
++	}
++	assert.Assert(t, found)
++}

modified internal/server/dashboard_handler.go +6 -1 Viewed

@@ -36,7 +36,12 @@ func (s Server) handleDashboard(w http.ResponseWriter, r http.Request) {
36	36	articles = articles[:page.PageSize]
37	37	}
38	38
39		- articleRecs, err := s.engine.GetArticleRecommendations(ctx, user.DID, 5)
	39	+ userLangs, err := s.dbs.Users.GetLanguages(ctx, user.DID)
	40	+ if err != nil {
	41	+ s.logger.Warn("failed to get user languages", "error", err, "did", user.DID)
	42	+ }
	43	+
	44	+ articleRecs, err := s.engine.GetArticleRecommendations(ctx, user.DID, userLangs, 5)
40	45	if err != nil {
41	46	s.logger.Warn("failed to get article recommendations", "error", err, "did", user.DID)
42	47	}

  		articles = articles[:page.PageSize]
  	}
--	articleRecs, err := s.engine.GetArticleRecommendations(ctx, user.DID, 5)
++	userLangs, err := s.dbs.Users.GetLanguages(ctx, user.DID)
++	if err != nil {
++		s.logger.Warn("failed to get user languages", "error", err, "did", user.DID)
++	}
++
++	articleRecs, err := s.engine.GetArticleRecommendations(ctx, user.DID, userLangs, 5)
  	if err != nil {
  		s.logger.Warn("failed to get article recommendations", "error", err, "did", user.DID)
  	}

modified internal/server/profile_handler.go +12 -7 Viewed

@@ -7,6 +7,7 @@ import (
7	7	"github.com/go-chi/chi/v5"
8	8
9	9	"pkg.rbrt.fr/glean/internal/atproto"
	10	+ "pkg.rbrt.fr/glean/internal/langdetect"
10	11	)
11	12
12	13	func (s Server) handleProfile(w http.ResponseWriter, r http.Request) {
@@ -56,13 +57,17 @@ func (s Server) handleProfile(w http.ResponseWriter, r http.Request) {
56	57
57	58	user := currentUser(r)
58	59
	60	+ userLangs, _ := s.dbs.Users.GetLanguages(ctx, user.DID)
	61	+
59	62	s.render(w, r, "profile.html", map[string]any{
60		- "User": user,
61		- "CurrentUserDID": user.DID,
62		- "ProfileUser": profileUser,
63		- "Subscriptions": subs,
64		- "Annotations": annotations,
65		- "SubscriptionCount": subCount,
66		- "AnnotationCount": len(annotations),
	63	+ "User": user,
	64	+ "CurrentUserDID": user.DID,
	65	+ "ProfileUser": profileUser,
	66	+ "Subscriptions": subs,
	67	+ "Annotations": annotations,
	68	+ "SubscriptionCount": subCount,
	69	+ "AnnotationCount": len(annotations),
	70	+ "UserLanguages": userLangs,
	71	+ "AvailableLanguages": langdetect.KnownLanguages(),
67	72	})
68	73	}

  	"github.com/go-chi/chi/v5"
  	"pkg.rbrt.fr/glean/internal/atproto"
++	"pkg.rbrt.fr/glean/internal/langdetect"
  )
  func (s *Server) handleProfile(w http.ResponseWriter, r *http.Request) {
  	user := currentUser(r)
++	userLangs, _ := s.dbs.Users.GetLanguages(ctx, user.DID)
++
  	s.render(w, r, "profile.html", map[string]any{
--		"User":              user,
++		"User":               user,
--		"CurrentUserDID":    user.DID,
++		"CurrentUserDID":     user.DID,
--		"ProfileUser":       profileUser,
++		"ProfileUser":        profileUser,
--		"Subscriptions":     subs,
++		"Subscriptions":      subs,
--		"Annotations":       annotations,
++		"Annotations":        annotations,
--		"SubscriptionCount": subCount,
++		"SubscriptionCount":  subCount,
--		"AnnotationCount":   len(annotations),
++		"AnnotationCount":    len(annotations),
++		"UserLanguages":      userLangs,
++		"AvailableLanguages": langdetect.KnownLanguages(),
  	})
  }

modified internal/server/server.go +12 -0 Viewed

@@ -203,6 +203,11 @@ func (s *Server) setupRoutes() {
203	203	r.Post("/dismiss-person", s.handleDismissPersonRecommendation)
204	204	})
205	205
	206	+ s.router.Route("/settings", func(r chi.Router) {
	207	+ r.Use(s.requireAuth)
	208	+ r.Post("/languages", s.handleUpdateLanguages)
	209	+ })
	210	+
206	211	s.router.Get("/auth/login", s.handleAuthLogin)
207	212	s.router.Get("/auth/register", s.handleAuthRegister)
208	213	s.router.Get("/auth/resolve", s.handleAuthResolve)
@@ -354,6 +359,13 @@ func (s *Server) loadTemplates() {
354	359	u.RawQuery = q.Encode()
355	360	return u.String()
356	361	},
	362	+ "containsString": func(slice any, s string) bool {
	363	+ sl, ok := slice.([]string)
	364	+ if !ok {
	365	+ return false
	366	+ }
	367	+ return slices.Contains(sl, s)
	368	+ },
357	369	}
358	370
359	371	var err error

  		r.Post("/dismiss-person", s.handleDismissPersonRecommendation)
  	})
++	s.router.Route("/settings", func(r chi.Router) {
++		r.Use(s.requireAuth)
++		r.Post("/languages", s.handleUpdateLanguages)
++	})
++
  	s.router.Get("/auth/login", s.handleAuthLogin)
  	s.router.Get("/auth/register", s.handleAuthRegister)
  	s.router.Get("/auth/resolve", s.handleAuthResolve)
  			u.RawQuery = q.Encode()
  			return u.String()
  		},
++		"containsString": func(slice any, s string) bool {
++			sl, ok := slice.([]string)
++			if !ok {
++				return false
++			}
++			return slices.Contains(sl, s)
++		},
  	}
  	var err error

added internal/server/settings_handler.go +39 -0 Viewed

new file mode 100644
@@ -0,0 +1,39 @@
	1	+package server
	2	+
	3	+import (
	4	+ "net/http"
	5	+ "strings"
	6	+
	7	+ "pkg.rbrt.fr/glean/internal/langdetect"
	8	+)
	9	+
	10	+func (s Server) handleUpdateLanguages(w http.ResponseWriter, r http.Request) {
	11	+ user := currentUser(r)
	12	+
	13	+ if err := r.ParseForm(); err != nil {
	14	+ http.Error(w, err.Error(), http.StatusBadRequest)
	15	+ return
	16	+ }
	17	+
	18	+ valid := make(map[string]bool)
	19	+ for _, known := range langdetect.KnownLanguages() {
	20	+ valid[known.Code] = true
	21	+ }
	22	+
	23	+ var filtered []string
	24	+ for _, l := range r.Form["languages"] {
	25	+ l = strings.TrimSpace(l)
	26	+ if valid[l] {
	27	+ filtered = append(filtered, l)
	28	+ }
	29	+ }
	30	+
	31	+ if err := s.dbs.Users.UpdateLanguages(r.Context(), user.DID, filtered); err != nil {
	32	+ s.logger.Error("failed to update languages", "error", err)
	33	+ http.Error(w, err.Error(), http.StatusInternalServerError)
	34	+ return
	35	+ }
	36	+
	37	+ w.Header().Set("HX-Redirect", "/profile/"+user.DID)
	38	+ w.WriteHeader(http.StatusOK)
	39	+}

++package server
++
++import (
++	"net/http"
++	"strings"
++
++	"pkg.rbrt.fr/glean/internal/langdetect"
++)
++
++func (s *Server) handleUpdateLanguages(w http.ResponseWriter, r *http.Request) {
++	user := currentUser(r)
++
++	if err := r.ParseForm(); err != nil {
++		http.Error(w, err.Error(), http.StatusBadRequest)
++		return
++	}
++
++	valid := make(map[string]bool)
++	for _, known := range langdetect.KnownLanguages() {
++		valid[known.Code] = true
++	}
++
++	var filtered []string
++	for _, l := range r.Form["languages"] {
++		l = strings.TrimSpace(l)
++		if valid[l] {
++			filtered = append(filtered, l)
++		}
++	}
++
++	if err := s.dbs.Users.UpdateLanguages(r.Context(), user.DID, filtered); err != nil {
++		s.logger.Error("failed to update languages", "error", err)
++		http.Error(w, err.Error(), http.StatusInternalServerError)
++		return
++	}
++
++	w.Header().Set("HX-Redirect", "/profile/"+user.DID)
++	w.WriteHeader(http.StatusOK)
++}

modified internal/tmpl/profile.html +33 -0 Viewed

@@ -22,6 +22,39 @@
22	22	</div>
23	23	</div>
24	24
	25	+ {{if eq .ProfileUser.DID .CurrentUserDID}}
	26	+ <h2 class="text-sm font-bold text-spot-text uppercase tracking-wide mb-3">Settings</h2>
	27	+ <div class="bg-spot-surface rounded-xl p-5 mb-6">
	28	+ <form id="lang-form" hx-post="/settings/languages" hx-swap="none" hx-on::after-request="location.reload()">
	29	+ <p class="text-sm font-semibold text-spot-text mb-1">Recommendation languages</p>
	30	+ <p class="text-xs text-spot-secondary mb-4">Filter article recommendations to specific languages. Leave empty to show all.</p>
	31	+ <div class="flex flex-wrap gap-2 mb-4">
	32	+ {{range .AvailableLanguages}}
	33	+ <label class="lang-pill cursor-pointer text-sm px-4 py-1.5 rounded-pill font-bold transition {{if containsString $.UserLanguages .Code}}bg-spot-active-pill-bg text-spot-active-pill-text{{else}}bg-spot-hover text-spot-secondary hover:text-spot-text{{end}}">
	34	+ <input type="checkbox" name="languages" value="{{.Code}}" {{if containsString $.UserLanguages .Code}}checked{{end}} class="hidden">
	35	+ {{.Name}}
	36	+ </label>
	37	+ {{end}}
	38	+ </div>
	39	+ <button type="submit" class="bg-spot-green text-white rounded-pill px-5 py-1.5 text-xs font-bold uppercase tracking-button hover:brightness-110 transition">Save</button>
	40	+ </form>
	41	+ </div>
	42	+ <script>
	43	+ document.querySelectorAll('#lang-form .lang-pill input[type=checkbox]').forEach(function(cb) {
	44	+ cb.addEventListener('change', function() {
	45	+ var pill = this.closest('.lang-pill');
	46	+ if (this.checked) {
	47	+ pill.classList.remove('bg-spot-hover', 'text-spot-secondary', 'hover:text-spot-text');
	48	+ pill.classList.add('bg-spot-active-pill-bg', 'text-spot-active-pill-text');
	49	+ } else {
	50	+ pill.classList.remove('bg-spot-active-pill-bg', 'text-spot-active-pill-text');
	51	+ pill.classList.add('bg-spot-hover', 'text-spot-secondary', 'hover:text-spot-text');
	52	+ }
	53	+ });
	54	+ });
	55	+ </script>
	56	+ {{end}}
	57	+
25	58	<h2 class="text-sm font-bold text-spot-text uppercase tracking-wide mb-3">Feeds</h2>
26	59	<div class="bg-spot-surface rounded-xl divide-y divide-spot-divider mb-6">
27	60	{{range .Subscriptions}}

          </div>
      </div>
++    {{if eq .ProfileUser.DID .CurrentUserDID}}
++    <h2 class="text-sm font-bold text-spot-text uppercase tracking-wide mb-3">Settings</h2>
++    <div class="bg-spot-surface rounded-xl p-5 mb-6">
++        <form id="lang-form" hx-post="/settings/languages" hx-swap="none" hx-on::after-request="location.reload()">
++            <p class="text-sm font-semibold text-spot-text mb-1">Recommendation languages</p>
++            <p class="text-xs text-spot-secondary mb-4">Filter article recommendations to specific languages. Leave empty to show all.</p>
++            <div class="flex flex-wrap gap-2 mb-4">
++                {{range .AvailableLanguages}}
++                <label class="lang-pill cursor-pointer text-sm px-4 py-1.5 rounded-pill font-bold transition {{if containsString $.UserLanguages .Code}}bg-spot-active-pill-bg text-spot-active-pill-text{{else}}bg-spot-hover text-spot-secondary hover:text-spot-text{{end}}">
++                    <input type="checkbox" name="languages" value="{{.Code}}" {{if containsString $.UserLanguages .Code}}checked{{end}} class="hidden">
++                    {{.Name}}
++                </label>
++                {{end}}
++            </div>
++            <button type="submit" class="bg-spot-green text-white rounded-pill px-5 py-1.5 text-xs font-bold uppercase tracking-button hover:brightness-110 transition">Save</button>
++        </form>
++    </div>
++    <script>
++    document.querySelectorAll('#lang-form .lang-pill input[type=checkbox]').forEach(function(cb) {
++        cb.addEventListener('change', function() {
++            var pill = this.closest('.lang-pill');
++            if (this.checked) {
++                pill.classList.remove('bg-spot-hover', 'text-spot-secondary', 'hover:text-spot-text');
++                pill.classList.add('bg-spot-active-pill-bg', 'text-spot-active-pill-text');
++            } else {
++                pill.classList.remove('bg-spot-active-pill-bg', 'text-spot-active-pill-text');
++                pill.classList.add('bg-spot-hover', 'text-spot-secondary', 'hover:text-spot-text');
++            }
++        });
++    });
++    </script>
++    {{end}}
++
      <h2 class="text-sm font-bold text-spot-text uppercase tracking-wide mb-3">Feeds</h2>
      <div class="bg-spot-surface rounded-xl divide-y divide-spot-divider mb-6">
          {{range .Subscriptions}}

modified main.go +10 -1 Viewed

@@ -74,7 +74,16 @@ func main() {
74	74	}
75	75	}
76	76
77		- engine := cluster.NewEngine(dbs.SQLDB(), embedder, logger)
	77	+ var llm *cluster.LLMClient
	78	+ if llmURL := envOr("GLEAN_LLM_BASE_URL", ""); llmURL != "" {
	79	+ llm = cluster.NewLLMClient(cluster.LLMClientConfig{
	80	+ BaseURL: llmURL,
	81	+ APIKey: envOr("GLEAN_LLM_API_KEY", ""),
	82	+ Model: envOr("GLEAN_LLM_MODEL", "gpt-4o-mini"),
	83	+ })
	84	+ }
	85	+
	86	+ engine := cluster.NewEngine(dbs.SQLDB(), embedder, llm, logger)
78	87
79	88	fetcher := feed.NewFetcher(siteFetcher)
80	89	srv := server.New(dbs, clientID, callbackURL, *addr, scheduler, fetcher, engine, logger, []byte(sessionKey))

  		}
  	}
--	engine := cluster.NewEngine(dbs.SQLDB(), embedder, logger)
++	var llm *cluster.LLMClient
++	if llmURL := envOr("GLEAN_LLM_BASE_URL", ""); llmURL != "" {
++		llm = cluster.NewLLMClient(cluster.LLMClientConfig{
++			BaseURL: llmURL,
++			APIKey:  envOr("GLEAN_LLM_API_KEY", ""),
++			Model:   envOr("GLEAN_LLM_MODEL", "gpt-4o-mini"),
++		})
++	}
++
++	engine := cluster.NewEngine(dbs.SQLDB(), embedder, llm, logger)
  	fetcher := feed.NewFetcher(siteFetcher)
  	srv := server.New(dbs, clientID, callbackURL, *addr, scheduler, fetcher, engine, logger, []byte(sessionKey))

modified readme.md +3 -0 Viewed

@@ -69,6 +69,9 @@ Then open `http://localhost:8080`.
69	69	\| `GLEAN_EMBED_API_KEY` \| _(empty)_ \| API key for the embeddings endpoint \|
70	70	\| `GLEAN_EMBED_MODEL` \| `text-embedding-3-small` \| Embedding model name \|
71	71	\| `GLEAN_EMBED_DIMENSION` \| `1536` \| Embedding vector dimension \|
	72	+\| `GLEAN_LLM_BASE_URL` \| _(empty)_ \| LLM API base URL for language detection (see below) \|
	73	+\| `GLEAN_LLM_API_KEY` \| _(empty)_ \| API key for the LLM endpoint \|
	74	+\| `GLEAN_LLM_MODEL` \| `gpt-4o-mini` \| LLM model name \|
72	75
73	76	For production:
74	77

  | `GLEAN_EMBED_API_KEY`        | _(empty)_                  | API key for the embeddings endpoint                        |
  | `GLEAN_EMBED_MODEL`          | `text-embedding-3-small`   | Embedding model name                                       |
  | `GLEAN_EMBED_DIMENSION`      | `1536`                     | Embedding vector dimension                                 |
++| `GLEAN_LLM_BASE_URL`         | _(empty)_                  | LLM API base URL for language detection (see below)        |
++| `GLEAN_LLM_API_KEY`          | _(empty)_                  | API key for the LLM endpoint                               |
++| `GLEAN_LLM_MODEL`            | `gpt-4o-mini`              | LLM model name                                             |
  For production:

Add recommendation language filtersUnverified