Glean - Design Document
1. Overview
Glean is a social RSS reader built on the AT Protocol. It operates as an AppView for the at.glean.* lexicon namespace: it indexes records from Jetstream, serves XRPC query endpoints, and provides the web UI at glean.at.
Users store their RSS feed subscriptions as individual lexicon records on their PDS (one record per feed). Glean's AppView consumes Jetstream, indexes those records, fetches the referenced RSS feeds, and serves both the reader UI and public XRPC APIs for the at.glean.* namespace.
The core idea: your RSS subscriptions are a strong signal about your interests. When enough people expose theirs, you can discover both people (who reads the same things) and content (what similar readers follow that you don't).
2. Stack
| Layer | Technology |
|---|---|
| Backend | Go |
| Database | SQLite (3 files: users, articles, recs via mattn/go-sqlite3 + sqlite-vec for vector search) |
| Frontend | SvelteKit (SSR, adapter-node) + TailwindCSS v4 |
| Auth | AT Protocol OAuth / DID resolution (configurable PLC directory) |
| AT Protocol role | AppView for at.glean.* lexicons |
| Data source | AT Protocol Jetstream → SQLite index |
3. AT Protocol Lexicons
All user data lives on their PDS. The server does not own user data — it indexes and aggregates it.
3.1 at.glean.subscription
A single RSS feed subscription. One record per feed per user. Created automatically when a user subscribes to a feed. During onboarding, existing subscriptions can be bulk-imported from an OPML file — the OPML is parsed and individual subscription records are created.
{
"lexicon": 1,
"id": "at.glean.subscription",
"defs": {
"main": {
"type": "record",
"key": "tid",
"description": "A single RSS feed subscription.",
"record": {
"type": "object",
"required": ["feedUrl"],
"properties": {
"createdAt": { "type": "string", "format": "datetime" },
"feedUrl": { "type": "string" },
"title": { "type": "string" },
"category": { "type": "string" }
}
}
}
}
}
OPML Import/Export
OPML is not part of the lexicon. It is only used as a transport format:
- Import (onboarding): User uploads an OPML file. Glean parses it, validates each feed URL, creates individual
at.glean.subscriptionrecords on the user's PDS, and indexes them locally. - Export (offboarding): Glean reads the user's
at.glean.subscriptionrecords from their PDS and generates an OPML file for download. The user's repository remains the canonical source.
3.2 at.glean.annotation
Reading notes on an article: quote a passage, tag it, rate it, or write a note. A user can have many annotations per article. These are public records on the PDS — users can always share an annotation on Bluesky if they want discussion.
{
"lexicon": 1,
"id": "at.glean.annotation",
"defs": {
"main": {
"type": "record",
"key": "tid",
"description": "Reading note on a specific RSS article.",
"record": {
"type": "object",
"required": ["feedUrl", "articleUrl"],
"properties": {
"createdAt": { "type": "string", "format": "datetime" },
"feedUrl": { "type": "string" },
"articleUrl": { "type": "string" },
"quote": { "type": "string", "maxGraphemes": 5000 },
"note": { "type": "string", "maxGraphemes": 500 },
"tags": {
"type": "array",
"items": { "type": "string", "maxGraphemes": 50 },
"maxLength": 10
},
"rating": { "type": "integer", "minimum": 1, "maximum": 5 }
}
}
}
}
}
3.3 at.glean.like
A user likes an article. The liked feed surfaces popular articles and feeds into discovery. Likes also feed into the recommendation system.
{
"lexicon": 1,
"id": "at.glean.like",
"defs": {
"main": {
"type": "record",
"key": "tid",
"description": "Like an RSS article.",
"record": {
"type": "object",
"required": ["feedUrl", "articleUrl"],
"properties": {
"createdAt": { "type": "string", "format": "datetime" },
"feedUrl": { "type": "string" },
"articleUrl": { "type": "string" }
}
}
}
}
}
3.4 at.margin.note (External)
Glean also indexes records from the at.margin.note lexicon (owned by margin.at). These are displayed in the UI as if they were at.glean.annotation records — margin notes appear alongside glean annotations on article detail pages.
Ingestion (margin.at → Glean)
Margin notes are indexed from both Jetstream and PDS sync, same as glean records. The mapping from margin note to glean annotation:
| Margin note field | Annotation field | Notes |
|---|---|---|
target.source |
article_url |
W3C SpecificResource source URL |
body.value |
note |
Text content of the annotation |
target.selector.exact |
quote |
TextQuoteSelector exact text |
tags |
tags |
Direct mapping |
createdAt |
created_at |
Direct mapping |
| (looked up from articles DB) | feed_url |
Resolved by matching target.source against known article URLs |
When no matching article exists in the local DB, the annotation is stored with an empty feed_url.
Mirroring (Glean → margin.at)
When a user creates an annotation through Glean, two records are written to the user's PDS:
at.glean.annotation— the primary record (canonical URI for the annotation)at.margin.note— mirror for interoperability with margin.at clients
The conversion from glean annotation to margin note uses NewMarginNoteRecord:
| Glean annotation field | Margin note field | Notes |
|---|---|---|
articleUrl |
target.source |
Direct mapping |
quote |
target.selector.exact |
Wrapped in a TextQuoteSelector |
note |
body.value |
body.format = "text/plain" |
tags |
tags |
Direct mapping |
| (constant) | motivation |
Always "commenting" |
The margin note mirror is fire-and-forget — if it fails, the glean annotation still succeeds. The Jetstream consumer will pick up the margin note create event, but handleMarginNote skips it via AnnotationExistsByContent (checks author_did + article_url + quote + note) to prevent duplicate annotations.
3.5 app.skyreader.feed.subscription (External)
Glean also indexes records from the Skyreader lexicon (app.skyreader.feed.subscription). When a user has subscriptions in Skyreader, they are imported as Glean subscriptions during PDS sync and via Jetstream events. This lets users who previously used Skyreader seamlessly transition to Glean without re-subscribing to their feeds.
The mapping from Skyreader subscription to Glean subscription:
| Skyreader field | Glean field | Notes |
|---|---|---|
feedUrl |
feed_url |
Direct mapping |
title |
title |
Direct mapping |
siteUrl |
site_url |
Stored on the feed record |
createdAt |
added_at |
Direct mapping |
| (none) | category |
Empty (Skyreader has no categories) |
If a Glean subscription already exists for the same feed_url, the existing one is kept. If the existing subscription has no URI (was created locally without PDS sync), the Skyreader URI/CID is backfilled.
3.6 app.bsky.graph.follow (External)
Follow relationships are tracked from Bluesky and Tangled follow records. The FollowRecord struct is validated against the lexicon at lexicons/app/bsky/graph/follow.json. The optional via field (a strong ref) is preserved as raw JSON but not used by Glean.
3.7 Lexicon Constants
All collection NSIDs are defined as constants in lexicon.go and used throughout the codebase:
const (
CollectionSubscription = "at.glean.subscription"
CollectionAnnotation = "at.glean.annotation"
CollectionLike = "at.glean.like"
CollectionMarginNote = "at.margin.note"
CollectionSkyreaderSubscription = "app.skyreader.feed.subscription"
CollectionBskyFollow = "app.bsky.graph.follow"
CollectionTangledFollow = "sh.tangled.graph.follow"
)
3.8 AppView Query Lexicons
As an AppView, Glean serves the following XRPC query endpoints. Other AT Protocol applications can call these to access indexed at.glean.* data without implementing their own indexer.
at.glean.listSubscriptions
List subscriptions from a repo, with optional filtering.
Input:
repo: string (DID of the user)
category?: string
limit?: integer (default 50, max 100)
cursor?: string
Output:
cursor?: string
subscriptions: [{ uri, cid, value: at.glean.subscription#main, indexedAt }]
at.glean.listFeedLists
List subscription lists from multiple repos, with optional filtering.
Input:
actors?: string[] (filter by DIDs)
limit?: integer (default 50, max 100)
cursor?: string
Output:
cursor?: string
feeds: [{ did, subscriptionCount, subscriptions: [{ feedUrl, title, category }] }]
at.glean.listAnnotations
List annotations for an article, a feed, or by a user.
Input:
feedUrl?: string
articleUrl?: string
author?: string (DID)
limit?: integer (default 50, max 100)
cursor?: string
Output:
cursor?: string
annotations: [{ uri, cid, author: { did, handle }, value, indexedAt }]
at.glean.listLikes
List liked articles, optionally filtered by user or feed.
Input:
author?: string (DID)
feedUrl?: string
limit?: integer (default 50, max 100)
cursor?: string
Output:
cursor?: string
likes: [{ uri, cid, author: { did, handle }, value: at.glean.like#main, indexedAt }]
at.glean.getTrending
Articles with the most likes, forming the community feed.
Input:
limit?: integer (default 50, max 100)
cursor?: string
since?: string (datetime)
Output:
cursor?: string
articles: [{ feedUrl, articleUrl, title, likeCount, annotations: [...] }]
at.glean.getRecommendations
Get feed recommendations for a user based on clustering.
Input:
repo: string (DID of the user, query parameter)
limit?: integer (default 20, max 50)
Output:
feeds: [{ feedUrl, title, siteUrl, description, subscriberCount, score }]
people: [{ did, handle, displayName, avatar, jaccard, commonFeeds, isFollowed }]
3.9 AppView Jetstream Consumption
Glean subscribes to a Jetstream endpoint (GLEAN_JETSTREAM, default wss://jetstream1.eurosky.network) for all at.glean.* records:
SUBSCRIBE collections: ["at.glean.subscription", "at.glean.annotation", "at.glean.like", "app.bsky.graph.follow", "sh.tangled.graph.follow", "at.margin.note", "app.skyreader.feed.subscription"]
The subscription is scoped with wantedDids to the DIDs of known users
(the users table): events of the wider network are never delivered or
stored. The filter is refreshed on every reconnect, and connections are
rotated periodically so newly signed-up users start streaming.
On each event:
- create: Insert record into local SQLite, update materialized counts
- delete: Tombstone the record (soft delete to preserve foreign key integrity)
- update: Replace the record's CID and value
The AppView does not handle writes. Users write records to their own PDS. Glean only reads them from Jetstream.
4. RSS Reader
Glean is first and foremost an RSS reader. It fetches, parses, and stores articles from RSS, Atom, and JSON feeds so users can read them in a clean interface.
4.1 Feed Fetching
A background scheduler polls subscribed feeds on a configurable tick. Feeds are fetched at most once per cycle regardless of how many users share them.
4.2 Fetch Schedule
The scheduler uses a configurable tick interval with in-flight deduplication:
- Tick interval: The scheduler checks for stale feeds every
GLEAN_FETCH_INTERVAL(default 15 minutes) - Staleness threshold: Feeds not fetched in the last 30 minutes are eligible
- Subscriber filter: Only feeds with
subscriber_count > 0are fetched - In-flight dedup: If a feed is already being fetched (e.g., manual refresh and background scheduler overlap), the second caller waits for the first to complete rather than fetching again
- Error tracking:
error_countincrements on failure, resets to 0 on success. Feeds with high error counts are surfaced as "dead feeds" to the user.
-- Feeds are fetched once regardless of subscriber count
SELECT ... FROM feeds
WHERE subscriber_count > 0
AND (last_fetched_at IS NULL OR last_fetched_at <= :cutoff)
ORDER BY last_fetched_at ASC NULLS FIRST
4.3 Feed Parsing
Go's encoding/xml for RSS and Atom. A simple encoding/json for JSON Feed.
Each parser returns a normalized Feed and a slice of Article structs:
type Article struct {
FeedURL string
GUID string
Title string
URL string
Author string
Content string
Summary string
Published time.Time
Updated time.Time
}
Articles are deduplicated by (feed_url, guid). On upsert, only the article metadata changes — reading state is preserved.
4.4 Article Content
Glean stores article content locally so the reading experience is fast and consistent:
CREATE TABLE articles (
id INTEGER PRIMARY KEY AUTOINCREMENT,
feed_url TEXT NOT NULL,
guid TEXT NOT NULL,
title TEXT NOT NULL DEFAULT '',
url TEXT,
author TEXT,
summary TEXT,
content TEXT,
full_content TEXT,
published DATETIME,
updated DATETIME,
fetched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
language TEXT NOT NULL DEFAULT '',
UNIQUE(feed_url, guid)
);
CREATE INDEX idx_articles_language ON articles(language);
4.5 Read State
Read/unread state is tracked per user per article:
CREATE TABLE read_state (
user_did TEXT NOT NULL,
article_id INTEGER NOT NULL,
is_read BOOLEAN NOT NULL DEFAULT 0,
read_at DATETIME,
PRIMARY KEY (user_did, article_id)
);
CREATE INDEX idx_read_state_unread ON read_state(user_did, is_read) WHERE is_read = 0;
4.6 Reading Experience
The /articles page is the main reading view:
- River of news: Chronological list of all unread articles across all subscriptions
- Feed filter: Narrow to a single feed or category
- Mark as read: Individual or "mark all above" / "mark all"
- Like: Endorse an article (public, synced to Bluesky PDS)
- Open original: Title links to the source article
- Share: Share to Bluesky
- Keyboard navigation:
j/kto navigate,lto like,mto mark read (progressive enhancement via a small<script>block) - Expanded view: User setting that shows full article content inline on the articles page. Articles are automatically marked as read via
IntersectionObserverafter being visible for 3 seconds. YouTube videos are embedded inline. A duplicate like button appears at the bottom of each article. Configurable in profile settings. - Daily digest: An AI-generated summary of unread articles, shown on the dashboard when enabled in profile settings. The LLM receives the titles and summaries of up to 50 unread articles and produces a grouped overview with linked references. A "Read" button marks all digest articles as read. Digests are cached per user for 24 hours and generated via singleflight to avoid duplicate LLM calls.
4.7 Feed Discovery from Content
Beyond the clustering system, Glean also discovers new feeds from article content:
- Auto-discovery: When fetching a feed, parse
<link rel="alternate" type="application/rss+xml">from the feed's site URL to discover related feeds - Feedfavicon: Fetch
favicon.icoor/apple-touch-icon.pngfrom the feed's site URL for display - Dead feed detection: If a feed fails for 7 consecutive fetches, mark it as dead. Notify the user and offer to remove it.
5. System Architecture
Glean is a two-process deployment: a Go binary serving a JSON API (/api/*) plus XRPC endpoints, and a SvelteKit SSR server (adapter-node) that renders the UI and proxies /api/* to Go. The Go binary fills two roles: AppView (indexing at.glean.* records from Jetstream, serving XRPC queries) and RSS reader (fetching and storing feed content). The SvelteKit server owns all page routes, HTML rendering, and SSR data loading.
Jetstream (GLEAN_JETSTREAM)
│ subscribe
▼
Browser ──HTTP──► ┌──────────────────────┐
(SvelteKit UI) │ SvelteKit (glean.at) │ :3000
│ SSR + /api proxy │
└──────────┬───────────┘
│ /api/* (JSON)
▼
┌──────────────────────┐ ┌──────────────────┐
│ Go Server │ ──XRPC──► Other AT apps
│ (glean.at) :8080 │
│ ┌────────────────┐ │ ┌──────────────────┐
│ │ chi Router │ │ │ Feed Scheduler │
│ │ ┌──────────┐ │ │ │ (goroutine) │
│ │ │ Handlers │ │ │ ──sync──►│ Fetcher+Parser │
│ │ │API + XRPC│ │ │ └────────┬─────────┘
│ │ └────┬─────┘ │ │ │
│ └───────┼────────┘ │ RSS/Atom/JSON feeds
│ ┌───────▼────────┐ │ ┌──────────────────┐
│ │ Service Layer │ │ │ Cluster Engine │
│ └───────┬────────┘ │ ────────►│ (periodic cron) │
│ ┌───────▼────────┐ │ └──────────────────┘
│ │ SQLite │ │
│ │ (jetstream idx,│ │
│ │ articles, │ │ PDS writes (on user
│ │ read state, │◄─┼───────── action via UI)
│ │ clustering) │ │
│ └────────────────┘ │
└──────────────────────┘
AppView responsibilities:
• Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like, at.margin.note, app.skyreader.feed.subscription
• Index records into SQLite
• Convert at.margin.note records to annotations (displayed alongside glean.at annotations), skip if duplicate glean annotation exists
• Mirror glean annotations as at.margin.note records on user PDS for interoperability
• Import app.skyreader.feed.subscription records as Glean subscriptions
• Serve XRPC query endpoints (at.glean.listSubscriptions, etc.)
• Write to user PDS on behalf of user (when user acts through UI)
6. Database Schema (SQLite)
Glean uses three separate SQLite database files to reduce write-lock contention. Each is opened with its own connection pool:
| File | Contents | ATTACH alias |
|---|---|---|
<base>_users |
Users, follows, OAuth | main (primary) |
<base>_articles |
Feeds, subscriptions, articles, read state, likes, annotations | articles |
<base>_recs |
Similarity scores, impressions, dismissals, signal weights | recs |
The users connection pool uses a custom SQLite driver with a ConnectHook that ATTACHes the articles and recs databases on every new connection. This allows the cluster engine to run cross-database queries using schema prefixes (articles.subscriptions, recs.user_similarity, main.follows).
Foreign key constraints are not used because SQLite does not support foreign keys across ATTACHed databases. Referential integrity is enforced by the application layer.
6.1 Users (<base>_users)
Profile data (handle, display name, avatar) is resolved on-the-fly via AT Protocol identity resolution rather than stored locally.
CREATE TABLE users (
did TEXT PRIMARY KEY,
indexed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
follows_dirty BOOLEAN NOT NULL DEFAULT 1
);
CREATE TABLE user_settings (
did TEXT PRIMARY KEY,
languages TEXT,
expanded_view BOOLEAN NOT NULL DEFAULT 0,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
6.2 Feed Subscriptions (<base>_articles)
Indexed from at.glean.subscription records on user PDS.
CREATE TABLE subscriptions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
user_did TEXT NOT NULL,
feed_url TEXT NOT NULL,
title TEXT,
category TEXT,
added_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
uri TEXT,
cid TEXT,
UNIQUE(user_did, feed_url)
);
6.3 Feeds (<base>_articles)
Master list of all known RSS feeds.
CREATE TABLE feeds (
feed_url TEXT PRIMARY KEY,
title TEXT,
site_url TEXT,
description TEXT,
feed_type TEXT CHECK(feed_type IN ('rss', 'atom', 'json')),
last_fetched_at DATETIME,
last_error TEXT,
subscriber_count INTEGER NOT NULL DEFAULT 0,
consecutive_empty_fetches INTEGER NOT NULL DEFAULT 0,
error_count INTEGER NOT NULL DEFAULT 0,
favicon_url TEXT
);
6.4 Articles (<base>_articles)
Fetched from RSS feeds. Only fetched for feeds that have local subscribers.
CREATE TABLE articles (
id INTEGER PRIMARY KEY AUTOINCREMENT,
feed_url TEXT NOT NULL,
guid TEXT NOT NULL,
title TEXT NOT NULL DEFAULT '',
url TEXT,
author TEXT,
summary TEXT,
content TEXT,
full_content TEXT,
published DATETIME,
updated DATETIME,
fetched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
language TEXT NOT NULL DEFAULT '',
UNIQUE(feed_url, guid)
);
CREATE INDEX idx_articles_language ON articles(language);
6.5 Read State (<base>_articles)
CREATE TABLE read_state (
user_did TEXT NOT NULL,
article_id INTEGER NOT NULL,
is_read BOOLEAN NOT NULL DEFAULT 0,
read_at DATETIME,
PRIMARY KEY (user_did, article_id)
);
CREATE INDEX idx_read_state_unread ON read_state(user_did, is_read) WHERE is_read = 0;
6.6 Annotations, Likes (<base>_articles)
Local mirror of AT Protocol lexicon records for fast querying.
CREATE TABLE annotations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
uri TEXT NOT NULL UNIQUE,
author_did TEXT NOT NULL,
feed_url TEXT NOT NULL,
article_url TEXT NOT NULL,
quote TEXT,
note TEXT,
tags TEXT,
rating INTEGER,
created_at DATETIME NOT NULL,
cid TEXT
);
CREATE TABLE likes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
uri TEXT NOT NULL UNIQUE,
author_did TEXT NOT NULL,
feed_url TEXT NOT NULL,
article_url TEXT NOT NULL,
created_at DATETIME NOT NULL,
cid TEXT,
UNIQUE(author_did, feed_url, article_url)
);
6.7 Cluster Precomputation (<base>_recs)
Stores precomputed similarity data to avoid recalculating on every request.
CREATE TABLE feed_similarity (
feed_a TEXT NOT NULL,
feed_b TEXT NOT NULL,
jaccard REAL NOT NULL,
computed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (feed_a, feed_b),
CHECK(feed_a < feed_b)
);
CREATE TABLE user_similarity (
user_a TEXT NOT NULL,
user_b TEXT NOT NULL,
jaccard REAL NOT NULL,
common_feeds INTEGER NOT NULL,
common_likes INTEGER NOT NULL DEFAULT 0,
common_tags INTEGER NOT NULL DEFAULT 0,
computed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (user_a, user_b),
CHECK(user_a < user_b)
);
6.8 Follows (<base>_users)
Tracks follow relationships between users (from app.bsky.graph.follow and sh.tangled.graph.follow records).
CREATE TABLE follows (
user_did TEXT NOT NULL,
target_did TEXT NOT NULL,
uri TEXT,
cid TEXT,
followed_at DATETIME,
PRIMARY KEY (user_did, target_did)
);
CREATE INDEX idx_follows_target ON follows(target_did);
CREATE INDEX idx_follows_uri ON follows(uri);
6.9 OAuth Storage (<base>_users)
CREATE TABLE oauth_auth_requests (
state TEXT PRIMARY KEY,
data TEXT NOT NULL
);
CREATE TABLE oauth_sessions (
account_did TEXT NOT NULL,
session_id TEXT NOT NULL,
data TEXT NOT NULL,
PRIMARY KEY (account_did, session_id)
);
7. Recommendations
Glean uses a multi-signal recommendation system that combines subscription overlap, like patterns, social graph distance, and user behavior feedback.
7.1 Signals
| Signal | Source | Weight (default) | Description |
|---|---|---|---|
| Subscription | subscriptions |
1.0 | Jaccard over subscriber sets between similar users |
| Like | likes |
0.5 | Time-decayed like co-occurrence (30-day half-life) |
| Tag | annotations.tags |
0.3 | Jaccard over annotation tag sets |
| Social | follow_distances |
0.7 | Follow distance: 1-hop=1.0, 2-hop=0.3, 3-hop=0.1 |
| Popularity | feeds.subscriber_count |
0.2 | log(1 + subscribers) / log(1 + max) |
| Category | subscriptions.category |
0.4 | Boost feeds matching user's existing categories |
| Content | article_embeddings |
0.4 | Cosine similarity via embedding KNN (requires embedder) |
7.2 Feed Co-occurrence (Jaccard Similarity)
For any two feeds, the similarity is the Jaccard index of their subscriber sets:
J(A, B) = |subscribers(A) ∩ subscribers(B)| / |subscribers(A) ∪ subscribers(B)|
Feed description similarity is also computed via embedding cosine similarity (requires embedder) and added as a boost.
7.3 User Similarity
For any two users, compute Jaccard over their subscription sets, plus like co-occurrence (time-decayed) and tag overlap:
J(U1, U2) = jaccard_subscriptions + 0.3 * jaccard_likes + 0.2 * jaccard_tags + 0.5 * follow_boost
Like overlap uses exponential time decay: EXP(-0.023 * age_days) (30-day half-life).
7.4 Precomputed + On-Demand Scoring
Recommendations are precomputed for all active users on every cron cycle (GLEAN_CLUSTER_INTERVAL, default 1h) and stored in the precomputed_recommendations table
Feed recommendation score (computed in SQL):
score = sub_signal * w_sub
+ like_signal * w_like
+ social_signal * w_social
+ pop_signal * w_pop
+ category_signal * w_category
Where:
sub_signal = SUM(jaccard(target, U))for similar users U subscribed to feedlike_signal = SUM(jaccard(target, U) * time_decay)for likes in that feed by similar userssocial_signal = SUM(distance_weight)from follow_distancespop_signal = log(1 + subscriber_count) / log(1 + max_subscribers)category_signal = 1if feed description matches user's top categories
Article recommendation score:
score = like_signal * w_like
+ social_signal * w_social
+ content_signal * w_content
+ recency_signal * 0.2
Content signal uses embedding vectors: the user's liked article embeddings are averaged into a single interest vector, then a KNN query against the article_embeddings vec0 table finds semantically similar articles. This requires an embedder to be configured; without it, the content signal is 0.
Language filtering: Users can set preferred languages on their profile (stored in the user_settings table as a JSON array of ISO 639-1 codes). When set, article recommendations are filtered to only include articles whose language column matches one of the selected codes or whose language is empty (not yet classified). This ensures users with language preferences still see all recommendations when the LLM hasn't run or hasn't classified certain articles yet. If no languages are set (empty/nil), all articles are shown regardless of language.
7.5 User Feedback (Dismiss)
Users can dismiss recommendations they don't want to see again:
POST /feeds/dismiss— dismiss a feed recommendationPOST /articles/dismiss— dismiss an article recommendation- Dismissals are stored locally in
dismissed_recommendations(not on PDS) - Dismissed items are excluded from all future recommendation queries
- Auto-dismiss: items shown ≥5 times over >5 days without action are auto-dismissed
Recommended articles use the same card template as regular articles, with an additional "Hide" button that dismisses the recommendation. Dismissing only removes that specific article from recommendations — it does not affect the user's likes or signal weights, and similar articles may still be recommended.
7.6 Auto-Tuned Signal Weights
Each user has a row in user_signal_weights with per-signal weights. When a user acts on a recommendation (subscribes, likes), the dominant signal that produced that recommendation is rewarded:
new_weight = MAX(0.1, MIN(3.0, old_weight * (1 + learning_rate * delta)))
learning_rate = 0.1,delta = +1for reward,-1for penalty- Only activates after
minActionsTune = 5positive actions - Defaults are used when no row exists for a user
7.7 Social Graph
Follow distances (1-hop through 3-hop) are computed incrementally. A follows_dirty column on users tracks whose follow graph changed since the last cron run. Only dirty users are reprocessed — their existing rows in follow_distances are deleted and recomputed via BFS, then the dirty flag is cleared.
- 1-hop: direct follows (weight 1.0)
- 2-hop: friends-of-friends (weight 0.3)
- 3-hop: third-degree connections (weight 0.1)
7.8 Diversity & Freshness
After scoring, diversity filtering is applied in Go (not SQL):
- Domain diversity: max 2 feeds from the same domain in results
- Category diversity: max 3 feeds from the same category in results
- This prevents recommendation clustering on a single source
7.9 Cold Start
New users with <5 subscriptions get a fallback strategy:
- Feeds from 1-hop followed users (70% weight)
- Globally popular feeds by subscriber count (30% weight)
7.10 Clustering Engine (Cron)
A background goroutine runs on a configurable schedule (GLEAN_CLUSTER_INTERVAL, default 1h):
- Compute feed embeddings: Embed new feed descriptions via embedding API into
feed_embeddingstable (skipped if no embedder configured) - Compute feed similarity: Batch-update
feed_similaritytable (Jaccard over subscriber sets + embedding cosine similarity) - Compute user similarity: Batch-update
user_similaritytable (subscription Jaccard + time-decayed likes + tags + follow boost) - Compute article embeddings: Embed new articles (
title + summary + content, excludingfull_contentto stay within embedding model token limits) via embedding API intoarticle_embeddingsvec0 table (skipped if no embedder configured) - Detect article languages: Batch-classify article languages via LLM, updating the
languagecolumn (skipped if no LLM configured) - Compute follow distances: Incremental BFS for dirty users (1-hop through 3-hop from
followstable) - Compute signal profiles: Per-user category/tag/like summaries
- Auto-dismiss stale: Dismiss items shown >=5 times over >5 days without action
- Precompute recommendations: For each active user, compute feed, article, and people recommendations and store as JSON in the
precomputed_recommendationstable. This ensures instant load times for all recommendation sections. - DB maintenance: Run
PRAGMA incremental_vacuumon all 3 databases (users, articles, recs) to reclaim freed pages. Incremental auto-vacuum is enabled viaPRAGMA auto_vacuum = INCREMENTALat connection time, so pages freed by impression pruning and other deletions are reclaimed each cycle. - Prune old impressions: Delete
recommendation_impressionsolder than 90 days
Jetstream ingestion and record indexing happen in a separate persistent goroutine (the Jetstream consumer), not in the cron.
7.11 User Interaction Tables (<base>_users)
Per-user interaction state lives in the users database so that real-time writes (impressions, dismissals) never contend with cron batch writes to the recs database.
CREATE TABLE dismissed_recommendations (
user_did TEXT NOT NULL,
target_type TEXT NOT NULL CHECK(target_type IN ('feed', 'article')),
target_id TEXT NOT NULL,
reason TEXT,
dismissed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (user_did, target_type, target_id)
);
CREATE TABLE recommendation_impressions (
user_did TEXT NOT NULL,
target_type TEXT NOT NULL CHECK(target_type IN ('feed', 'article')),
target_id TEXT NOT NULL,
first_shown_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_shown_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
shown_count INTEGER NOT NULL DEFAULT 1,
acted BOOLEAN NOT NULL DEFAULT 0,
PRIMARY KEY (user_did, target_type, target_id)
);
7.12 Computed Recommendation Tables (<base>_recs)
Written exclusively by the cron. Read during recommendation requests (precomputed results served first, on-demand fallback if missing).
CREATE TABLE feed_similarity (
feed_a TEXT NOT NULL,
feed_b TEXT NOT NULL,
jaccard REAL NOT NULL,
computed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (feed_a, feed_b),
CHECK(feed_a < feed_b)
);
CREATE TABLE user_similarity (
user_a TEXT NOT NULL,
user_b TEXT NOT NULL,
jaccard REAL NOT NULL,
common_feeds INTEGER NOT NULL,
common_likes INTEGER NOT NULL DEFAULT 0,
common_tags INTEGER NOT NULL DEFAULT 0,
computed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (user_a, user_b),
CHECK(user_a < user_b)
);
CREATE TABLE follow_distances (
user_a TEXT NOT NULL,
user_b TEXT NOT NULL,
distance INTEGER NOT NULL CHECK(distance IN (1, 2, 3)),
PRIMARY KEY (user_a, user_b)
);
CREATE TABLE user_signal_weights (
user_did TEXT PRIMARY KEY,
w_sub REAL NOT NULL DEFAULT 1.0,
w_like REAL NOT NULL DEFAULT 0.5,
w_tag REAL NOT NULL DEFAULT 0.3,
w_social REAL NOT NULL DEFAULT 0.7,
w_pop REAL NOT NULL DEFAULT 0.2,
w_category REAL NOT NULL DEFAULT 0.4,
w_content REAL NOT NULL DEFAULT 0.4,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE user_signal_profiles (
user_did TEXT PRIMARY KEY,
total_likes INTEGER NOT NULL DEFAULT 0,
total_tags INTEGER NOT NULL DEFAULT 0,
top_categories TEXT,
top_tags TEXT,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE precomputed_recommendations (
user_did TEXT NOT NULL,
rec_type TEXT NOT NULL CHECK(rec_type IN ('feed', 'article', 'person')),
data TEXT NOT NULL,
computed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (user_did, rec_type)
);
7.13 Embeddings (recommended)
When GLEAN_EMBED_BASE_URL is configured, article text and feed descriptions are embedded into vectors stored in sqlite-vec virtual tables (recs.feed_embeddings, recs.article_embeddings). The vec0 extension provides native KNN vector search via WHERE embedding MATCH ? AND k = ?, replacing Go-side cosine similarity for large-scale lookups. Without embeddings, recommendations rely only on subscription overlap, like patterns, and social graph — no content-based signals.
The embedder uses the official github.com/openai/openai-go SDK with option.WithBaseURL(), so any OpenAI-compatible /v1/embeddings endpoint works (OpenAI, Gemini, Ollama, local inference servers).
7.14 LLM Client (optional)
When GLEAN_LLM_BASE_URL is configured, an LLM client is available for text classification and summarization tasks. It uses the same github.com/openai/openai-go SDK pointed at any OpenAI-compatible /v1/chat/completions endpoint.
Language detection: The cron job calls DetectLanguages in batches of up to 100 articles per request. For each article, the title and summary (truncated to 500 characters) are sent with a prompt asking for ISO 639-1 codes. Results are written to articles.language. Articles that are already classified (non-empty language) are skipped. An empty response from the LLM defaults to 'unknown' rather than a specific language, ensuring unclassifiable articles aren't miscategorized.
Without an LLM, all articles remain at the default empty language value and language-based filtering is unavailable. The daily digest feature also requires an LLM to generate summaries — without it, the digest is unavailable.
vec0 tables are created dynamically at startup with the configured dimension (GLEAN_EMBED_DIMENSION, default 1536):
CREATE VIRTUAL TABLE recs.feed_embeddings USING vec0(
feed_url TEXT PRIMARY KEY,
embedding float[1536]
);
CREATE VIRTUAL TABLE recs.article_embeddings USING vec0(
article_id INTEGER PRIMARY KEY,
embedding float[1536]
);
Since vec0 virtual tables cannot hold metadata columns, a side table tracks the source text for re-embedding on description changes:
CREATE TABLE recs.feed_embedding_meta (
feed_url TEXT PRIMARY KEY,
source_text TEXT NOT NULL DEFAULT ''
);
During cron, ComputeArticleEmbeddings embeds new articles in batches using title + summary + content (the scraped full_content is excluded to stay within model token limits — most embedding models cap at ~8k tokens). Text is truncated to 8000 characters as a safety net. Batches are capped at 100 inputs per API call. ComputeFeedEmbeddings embeds feed descriptions (title || description) and re-embeds when the source text changes (detected via feed_embedding_meta). During on-demand article recommendations, the user's liked article embeddings are averaged into an interest vector, then a vec0 KNN query finds the top-200 most semantically similar articles. For cold-start users (<5 subscriptions), their subscribed feed embeddings are averaged and a KNN query finds similar feeds.
8. HTTP API
The Go server exposes a JSON API under /api/*. The SvelteKit frontend (see web/) owns all HTML page routes and consumes these endpoints via web/src/lib/api.ts; web/src/hooks.server.ts proxies /api/* to Go, forwarding cookies and headers. All state-changing requests require a double-submit CSRF token (X-CSRF-Token header or csrf_token form field) matching the glean_csrf cookie.
8.1 JSON endpoints
All endpoints below return JSON. Those marked 🔒 require authentication (session cookie). Form fields are sent application/x-www-form-urlencoded; file uploads are multipart/form-data.
| Route | Method | Auth | Description |
|---|---|---|---|
/api/me |
GET | Current user, CSRF token, feature flags (has_llm, client_id) |
|
/api/dashboard |
GET | 🔒 | Dashboard: article recs, unread articles, trending, digest flag |
/api/feeds |
GET | 🔒 | Subscriptions (paginated), categories, dead feeds |
/api/feeds/add |
POST | 🔒 | Add a feed URL (fields: feed_url, category) |
/api/feeds/edit |
POST | 🔒 | Edit a subscription category |
/api/feeds/opml/upload |
POST | 🔒 | Bulk-import subscriptions from OPML (field: opml file) |
/api/feeds/opml/download |
GET | 🔒 | Export subscriptions as OPML |
/api/feeds/refresh |
POST | 🔒 | Refresh all subscribed feeds |
/api/feeds/retry |
POST | 🔒 | Retry a failed feed |
/api/feeds/list |
GET | 🔒 | Flat subscription list (optional ?category=) |
/api/feeds/clear |
POST | 🔒 | Remove all subscriptions |
/api/articles |
GET | 🔒 | Read articles (paginated; ?feed=, ?status=, ?q=, ?sort=, ?category=) |
/api/articles/new-count |
GET | 🔒 | Count of articles newer than ?since= (unix seconds) — banner polling |
/api/articles/{id} |
GET | 🔒 | Article detail + annotations (marks read as a side effect) |
/api/articles/{id}/read |
POST | 🔒 | Mark article read |
/api/articles/{id}/unread |
POST | 🔒 | Mark article unread |
/api/articles/{id}/like |
POST | 🔒 | Toggle like (writes/deletes on PDS) |
/api/articles/{id}/fetch-content |
POST | 🔒 | Scrape full article content from the source URL |
/api/articles/mark-all-read |
POST | 🔒 | Mark all (or ?feed=-scoped) articles read |
/api/trending |
GET | Trending articles (?scope=for-me requires auth) |
|
/api/profile/{did} |
GET | 🔒 | Public profile: feeds, annotations, languages (resolves handles) |
/api/library |
GET | 🔒 | Liked articles + annotations (paginated) |
/api/library/create |
POST | 🔒 | Create annotation (writes at.glean.annotation + at.margin.note mirror) |
/api/library/{id}/delete |
POST | 🔒 | Delete annotation + mirror |
/api/recs/articles |
GET | 🔒 | Article recommendations |
/api/recs/feeds |
GET | 🔒 | Feed recommendations |
/api/recs/people |
GET | 🔒 | People recommendations (followed + discover) |
/api/recs/dismiss-feed |
POST | 🔒 | Dismiss a feed recommendation |
/api/recs/dismiss-article |
POST | 🔒 | Dismiss an article recommendation |
/api/recs/dismiss-person |
POST | 🔒 | Dismiss a person recommendation |
/api/settings/languages/{code} |
POST | 🔒 | Toggle a preferred recommendation language |
/api/settings/expanded-view |
POST | 🔒 | Toggle expanded article view |
/api/settings/digest-enabled |
POST | 🔒 | Toggle daily digest |
/api/digest |
GET | 🔒 | Daily digest (LLM summary of unread articles); 204 when none/unavailable |
/api/digest/mark-read |
POST | 🔒 | Mark digest articles read (field: repeated ids) |
/api/auth/login |
GET | Whether OAuth is enabled (oauth_enabled) |
|
/api/auth/register |
GET | Start registration via Eurosky ({redirect}) |
|
/api/auth/actors |
GET | Handle typeahead (?q=) → {actors} |
|
/api/auth/start |
POST | 🔒 | Start OAuth flow (field: handle) → {redirect} |
/api/auth/callback |
GET | OAuth callback (browser redirect, not JSON) | |
/api/auth/logout |
POST | 🔒 | End session → {redirect} |
/api/oauth/client-metadata |
GET | OAuth client metadata document | |
/api/sitemap |
GET | Sitemap entries (JSON) | |
/api/stats |
GET | Prometheus metrics parsed into {metrics} |
|
/metrics |
GET | Raw Prometheus exposition |
XRPC query endpoints (public, no /api prefix):
| Route | Description |
|---|---|
/xrpc/at.glean.listSubscriptions |
List a user's subscriptions |
/xrpc/at.glean.listAnnotations |
List annotations |
/xrpc/at.glean.listLikes |
List likes |
/xrpc/at.glean.getTrending |
Trending articles |
/xrpc/at.glean.getRecommendations |
Recommendations |
/xrpc/at.glean.listFeedLists |
Feed lists |
8.2 Frontend routes
All HTML page routes live in the SvelteKit app (web/src/routes/), not on the Go server. Each route has a +page.server.ts load function that calls the JSON API via endpointsFor(event.fetch) and a +page.svelte rendering component. The layout (+layout.svelte) renders the shared chrome and mounts the new-articles banner.
| Path | Load source |
|---|---|
/ |
+page (landing) |
/dashboard |
/api/dashboard |
/articles |
/api/articles |
/articles/[id] |
/api/articles/{id} |
/feeds |
/api/feeds |
/library |
/api/library |
/trending |
/api/trending |
/profile/[did] |
/api/profile/{did} |
/stats |
/api/stats |
/terms |
(static content) |
/auth/login |
/api/auth/login |
/sitemap.xml |
/api/sitemap |
9. Project Structure
glean/
├── main.go # Entry point, wire everything
├── go.mod / go.sum
├── Dockerfile # Multi-stage: builds web/ then Go, runs both
├── Makefile # Targets: build, dev-api, dev-web, web-build, test, ...
├── lexicons/
│ └── at/
│ ├── glean/ # Glean lexicon JSON schemas (subscription, annotation, like)
│ └── margin/ # External: at.margin.note W3C Web Annotation schema
│ └── app/bsky/graph/ # External: app.bsky.graph.follow schema
├── internal/
│ ├── atproto/
│ │ ├── auth.go # DID resolution, OAuth flow
│ │ ├── client.go # XRPC client (write to user PDS)
│ │ ├── collectiondir.go # Collection directory backfill (startup)
│ │ ├── jetstream.go # Subscribe to Jetstream via official client
│ │ ├── stream_handler.go # Stream event → DB handler
│ │ ├── lexicon.go # Lexicon record types (at.glean.*, maintained by hand)
│ │ ├── lexicon_external.go # External lexicon record types (FollowRecord, MarginNoteRecord, SkyreaderSubscriptionRecord)
│ │ ├── lexicon_test.go # Test: Go structs match lexicon JSON schemas
│ │ ├── sync.go # PDS record reconciliation
│ │ └── xrpc.go # XRPC query handlers (AppView endpoints)
│ ├── db/
│ │ ├── db.go # SQLite connection with ATTACH for cross-database queries
│ │ ├── user.go # User queries
│ │ ├── feed.go # Feed + subscription queries
│ │ ├── article.go # Article queries
│ │ ├── social.go # Like, annotation queries
│ │ ├── follow.go # Follow queries
│ │ ├── oauth_store.go # OAuth session storage
│ │ ├── user_settings.go # User settings queries
│ │ └── store.go # FeedStore adapter for scheduler
│ ├── feed/
│ │ ├── parser.go # RSS/Atom/RDF/JSON feed parser
│ │ ├── fetcher.go # Scheduler with dedup + Fetcher
│ │ ├── discover.go # Feed auto-discovery from URLs
│ │ └── opml.go # OPML import/export
│ ├── httpclient/
│ │ └── httpclient.go # Shared HTTP transport, retry logic, User-Agent
│ ├── scraper/
│ │ └── scraper.go # Full article content scraper
│ ├── metrics/
│ │ └── metrics.go # Prometheus metrics definitions
│ ├── ml/
│ │ ├── embed.go # Embedder interface + OpenAI-compatible implementation + vector helpers
│ │ ├── langdetect.go # Known-language table + detection
│ │ └── llm.go # TextModel interface + OpenAI-compatible LLM implementation
│ ├── cluster/
│ │ ├── jaccard.go # Jaccard similarity computation
│ │ ├── article.go # Article + feed embedding computation, vec0 KNN content boost, language detection
│ │ ├── scoring.go # Feed + people + article recommendation queries (precomputed + on-demand fallback)
│ │ ├── precompute.go # Precompute all user recommendations (run by cron 4x/day)
│ │ ├── social.go # Incremental follow-distance computation (1-3 hop, dirty-flag)
│ │ ├── weights.go # Bandit-style signal weight auto-tuning
│ │ ├── diversity.go # Post-query domain/category diversity filtering
│ │ └── cron.go # Background recomputation scheduler
│ ├── feedback/
│ │ └── feedback.go # Dismiss + impression tracking service
│ ├── server/
│ │ ├── server.go # HTTP server, chi router setup (/api/* routes)
│ │ ├── api.go # JSON helpers (writeJSON), DTOs, null helpers
│ │ ├── api_helpers.go # URL validation helpers
│ │ ├── auth_handler.go # OAuth login/callback/register/logout
│ │ ├── feeds_handler.go # Feed management handlers
│ │ ├── articles_handler.go # Article reading handlers
│ │ ├── annotations_handler.go # Annotation + library handlers
│ │ ├── dashboard_handler.go # Dashboard + recommendation handlers
│ │ ├── recs_handler.go # Recommendation dismiss handlers
│ │ ├── trending_handler.go # Trending handler
│ │ ├── profile_handler.go # Public profile handler
│ │ ├── settings_handler.go # User settings (language preferences, digest/expanded toggle)
│ │ ├── digest_handler.go # Daily digest handler (LLM summary, mark-read)
│ │ ├── stats_handler.go # Stats handler (Prometheus metrics → JSON)
│ │ ├── sitemap_handler.go # Sitemap handler
│ │ ├── index_handler.go # /api/me + auth-login meta handler
│ │ ├── sync_handlers.go # Periodic sync + collection-dir backfill
│ │ ├── sanitize.go # HTML sanitization for article content
│ │ ├── pagination.go # Pagination helpers
│ │ ├── middleware.go # Auth, logging, CSRF, CORS middleware
│ │ └── session.go # Session management
├── web/ # SvelteKit (SSR) frontend, adapter-node
│ ├── src/
│ │ ├── app.css # Tailwind v4 entry + design tokens (@theme, [data-theme], @utility)
│ │ ├── app.html # HTML shell
│ │ ├── app.d.ts # Locals types (user, csrfToken)
│ │ ├── hooks.server.ts # /api/* proxy to Go + per-request user load
│ │ ├── lib/
│ │ │ ├── api.ts # Typed endpoint client (endpoints / endpointsFor(fetch))
│ │ │ ├── types.ts # API response types
│ │ │ ├── format.ts # Date/HTML/youtube formatting helpers
│ │ │ └── components/ # Svelte components (ArticleCard, FeedItem, Icon, ...)
│ │ └── routes/ # File-based routes (+page.server.ts load + +page.svelte)
│ │ ├── +layout.svelte # Chrome (nav, footer, dialogs, new-articles banner)
│ │ ├── articles/[id]/ # Article detail
│ │ ├── dashboard/ # Dashboard
│ │ ├── feeds/ # Feed management
│ │ ├── library/ # Liked + annotations
│ │ ├── trending/ # Trending
│ │ ├── profile/[did]/ # Public profile
│ │ ├── stats/ # Metrics
│ │ ├── auth/login/ # Login
│ │ ├── terms/ # Terms (static)
│ │ └── sitemap.xml/ # Sitemap (+server.ts)
│ ├── static/ # Favicons, manifest, banner
│ ├── svelte.config.js # adapter-node
│ ├── vite.config.ts # dev server on :3000
│ └── package.json # bun; svelte, @sveltejs/kit, @tailwindcss/vite
└── docs/
├── specs.md # Technical specification (this document)
└── design.md # Design system
10. Auth Flow
Session cookies are HMAC-signed using GLEAN_SESSION_KEY (required, must be set to a random string). The server refuses to start without it.
DID resolution uses a configurable PLC directory (GLEAN_PLC_URL, defaults to https://plc.eurosky.network). The identity directory is initialized once at startup via InitIdentity() with a caching layer (250k entries, 24h TTL).
- User visits
/, clicks "Sign in with Bluesky" (or any AT Proto PDS) - Server redirects to AT Protocol OAuth authorization endpoint
- User authorizes on their PDS
- PDS redirects back with authorization code
- Server exchanges code for access token + refresh token
- Server resolves DID and creates a session (cookie with encrypted DID)
- On each request, middleware decrypts session, loads user from DB (or creates if new)
The server never stores the user's AT Protocol password. It stores the session tokens for XRPC calls to the user's PDS (to read/write their lexicon records).
11. Data Flow
11.1 User Imports OPML
Browser ──POST /feeds/opml/upload──► Server
│
├─► Parse OPML, extract feed URLs
├─► Fetch each feed, validate + store in `feeds` table
├─► For each feed, create an `at.glean.subscription` record
│ via XRPC write to user's PDS
├─► Insert subscriptions in local `subscriptions` table
└─◄ Redirect to `/feeds`
11.2 Reading the Feed
Browser ──GET /articles──► Server
│
├─► Query articles for user's subscriptions
├─► Render article-list.html partial
└─◄ Return HTML fragment (htmx)
11.3 Recommendations
Cron (every 1h) ──► Cluster Engine
│
├─► Compute feed similarity
├─► Compute user similarity
├─► Compute article embeddings (if embedder configured)
├─► Detect article languages (if LLM configured)
├─► Compute follow distances
├─► Compute signal profiles
└─► Auto-dismiss stale recommendations
Browser ──GET /dashboard──► Server
│
├─► Read precomputed recommendations from DB (instant)
│ (fallback to on-demand if missing)
├─► Fetch feed metadata
└─◄ Render recommendation cards (htmx)
12. Key Design Decisions
12.1 Why use lexicon records for feed subscriptions?
- User sovereignty: Each feed subscription lives as a record on the user's PDS. They can export them, move PDS, or revoke access at any time.
- Interoperability: Any AT Protocol app can read the lexicon and integrate with Glean data.
- No lock-in: If Glean shuts down, the user's data is intact on their PDS.
- OPML as transport only: OPML is a common interchange format used solely for import (onboarding from existing readers) and export (offboarding). The canonical representation is always the lexicon — individual
at.glean.subscriptionrecords, one per feed.
12.2 Why SQLite?
- Single-binary deployment, no external database dependency
- More than sufficient for the expected scale (tens of thousands of users)
- Go's
database/sqlinterface makes it easy to swap later if needed - Matches the project's philosophy of simplicity
12.3 Prometheus Metrics
Glean exposes a /metrics endpoint for monitoring. Key metrics:
glean_feeds_fetched_total— Total feed fetch attempts (counter)glean_feeds_fetched_last_timestamp_seconds— Unix timestamp of last feed fetch (gauge)glean_feed_fetch_duration_seconds— Histogram of feed fetch latencyglean_articles_upserted_total— Counter of articles stored from feedsglean_jetstream_events_total— Jetstream events labeled by collection and actionglean_jetstream_errors_total— Jetstream handler errorsglean_jetstream_reconnects_total— Jetstream reconnection countglean_http_requests_total— HTTP request counts labeled by method, path, and statusglean_http_request_duration_seconds— HTTP request duration labeled by method and pathglean_pds_sync_runs_total/glean_pds_sync_errors_total— PDS sync runs and errorsglean_cluster_runs_total/glean_cluster_duration_seconds— Recommendation engine runs and timing
12.4 Why htmx?
- No JavaScript build pipeline
- Server renders everything — simpler mental model
- Progressive enhancement works naturally
- Perfect fit for a read-centric application
- TailwindCSS handles styling without writing custom CSS
12.5 AppView Architecture
Glean operates as an AT Protocol AppView. This means:
- Read path: All
at.glean.*data is consumed from Jetstream, not by polling individual PDS instances. The Jetstream consumer runs as a persistent goroutine, upserting records into SQLite as they arrive. - Write path: Users write records to their own PDS (via standard AT Protocol
com.atproto.repo.createRecord/deleteRecord). Glean never stores user data directly — it only indexes what Jetstream delivers. - Query path: Other AT Protocol apps can query Glean's XRPC endpoints to access indexed data (subscriptions, annotations, likes, recommendations) without building their own indexer.
- Trade-off: Article content (fetched from RSS feeds) is local-only and not part of the AT Protocol layer. Only individual feed subscription records (
at.glean.subscription) live on the PDS.
12.6 Privacy Model
All PDS records are public. There is no notion of private data on the AT Protocol — everything stored on the repo is visible. Users should be aware that annotations, subscriptions, and likes are all public records.
13. Future Considerations
- Email digest: Periodic email with top articles from subscribed feeds
- Digest personalization: Feed the user's liked topics and reading patterns into the digest prompt for more targeted summaries
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 |
|