rickub
nandi/gleanpublic⑂ Fork 0
⑂ e230487
Branches
claude/modal-container-min-76527a main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/glean.git
git clone ssh://git@rickub.com/nandi/glean.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Support margin notes and add external lexicons in repoUnverified

Julien Robert committed 2026-04-21T10:56:24+02:00 Browse files
e230487 parent: a33920b
modified docs/specs.md +124 -65
@@ -120,7 +120,24 @@ A user likes an article. The liked feed surfaces popular articles and feeds into
120120 }
121121 ```
122122
123-### 3.4 AppView Query Lexicons
123+### 3.4 `at.margin.note` (External)
124+
125+Glean also indexes records from the `at.margin.note` lexicon (owned by [margin.at](https://margin.at)). These are displayed in the UI as if they were `at.glean.annotation` records — margin notes appear alongside glean annotations on article detail pages.
126+
127+The mapping from margin note to glean annotation:
128+
129+| Margin note field | Annotation field | Notes |
130+| ------------------------------ | ---------------- | --------------------------------------------------------------- |
131+| `target.source` | `article_url` | W3C SpecificResource source URL |
132+| `body.value` | `note` | Text content of the annotation |
133+| `target.selector.exact` | `quote` | TextQuoteSelector exact text |
134+| `tags` | `tags` | Direct mapping |
135+| `createdAt` | `created_at` | Direct mapping |
136+| _(looked up from articles DB)_ | `feed_url` | Resolved by matching `target.source` against known article URLs |
137+
138+When no matching article exists in the local DB, the annotation is stored with an empty `feed_url`. Margin notes are indexed from both Jetstream and PDS sync, same as glean records.
139+
140+### 3.5 AppView Query Lexicons
124141
125142 As an AppView, Glean serves the following XRPC query endpoints. Other AT Protocol applications can call these to access indexed `at.glean.*` data without implementing their own indexer.
126143
@@ -209,7 +226,7 @@ Get feed recommendations for a user based on clustering.
209226
210227 ```
211228 Input:
212- repo: string (DID of the user)
229+ repo: string (DID of the user, query parameter)
213230 limit?: integer (default 20, max 50)
214231
215232 Output:
@@ -217,12 +234,12 @@ Output:
217234 people: [{ did, handle, displayName, avatar, jaccard, commonFeeds }]
218235 ```
219236
220-### 3.5 AppView Jetstream Consumption
237+### 3.6 AppView Jetstream Consumption
221238
222239 Glean subscribes to a Jetstream endpoint (`GLEAN_JETSTREAM`, default `wss://jetstream2.fr.hose.cam`) for all `at.glean.*` records:
223240
224241 ```
225-SUBSCRIBE collections: ["at.glean.subscription", "at.glean.annotation", "at.glean.like"]
242+SUBSCRIBE collections: ["at.glean.subscription", "at.glean.annotation", "at.glean.like", "app.bsky.graph.follow", "sh.tangled.graph.follow", "at.margin.note"]
226243 ```
227244
228245 On each event:
@@ -299,25 +316,16 @@ Go's `encoding/xml` for RSS and Atom. A simple `encoding/json` for JSON Feed.
299316 Each parser returns a normalized `Feed` and a slice of `Article` structs:
300317
301318 ```go
302-type Feed struct {
303- URL string
304- Title string
305- SiteURL string
306- Description string
307- Type string // "rss", "atom", "json"
308- ETag string
309- LastModified string
310-}
311-
312319 type Article struct {
313- GUID string
314- Title string
315- URL string
316- Author string
317- Content string
318- Summary string
319- Published time.Time
320- Updated time.Time
320+ FeedURL string
321+ GUID string
322+ Title string
323+ URL string
324+ Author string
325+ Content string
326+ Summary string
327+ Published time.Time
328+ Updated time.Time
321329 }
322330 ```
323331
@@ -337,14 +345,18 @@ CREATE TABLE articles (
337345 author TEXT,
338346 summary TEXT,
339347 content TEXT,
348+ full_content TEXT,
340349 published DATETIME,
341350 updated DATETIME,
342351 fetched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
343352 UNIQUE(feed_url, guid)
344353 );
354+
355+CREATE INDEX idx_articles_feed ON articles(feed_url);
356+CREATE INDEX idx_articles_published ON articles(published DESC);
345357 ```
346358
347-Content is stored as raw HTML from the feed's `<content:encoded>`, `<summary>`, or JSON Feed `content_html`. The server renders it in a sanitized view (strip `<script>`, `<iframe>`, etc.).
359+Content is stored as raw HTML from the feed's `<content:encoded>`, `<summary>`, or JSON Feed `content_html`. `full_content` stores scraped article content fetched from the original URL. The server renders it in a sanitized view (strip `<script>`, `<iframe>`, etc.).
348360
349361 ### 4.5 Read State
350362
@@ -415,9 +427,10 @@ Glean runs as a single Go binary that fills three roles: **AppView** (indexing `
415427 │ └─────────────────┘ │ └──────────────────┘
416428 └──────────────────────┘
417429
418- AppView responsibilities:
419- • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like
420- • Index records into SQLite
430+ AppView responsibilities:
431+ • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like, at.margin.note
432+ • Index records into SQLite
433+ • Convert at.margin.note records to annotations (displayed alongside glean.at annotations)
421434 • Serve XRPC query endpoints (at.glean.listSubscriptions, etc.)
422435 • Host the web UI at glean.at
423436 • Write to user PDS on behalf of user (when user acts through UI)
@@ -515,13 +528,10 @@ CREATE TABLE read_state (
515528 article_id INTEGER NOT NULL REFERENCES articles(id),
516529 is_read BOOLEAN NOT NULL DEFAULT 0,
517530 read_at DATETIME,
518- is_starred BOOLEAN NOT NULL DEFAULT 0,
519- starred_at DATETIME,
520531 PRIMARY KEY (user_did, article_id)
521532 );
522533
523534 CREATE INDEX idx_read_state_unread ON read_state(user_did, is_read) WHERE is_read = 0;
524-CREATE INDEX idx_read_state_starred ON read_state(user_did, is_starred) WHERE is_starred = 1;
525535 ```
526536
527537 ### 6.6 Annotations, Likes
@@ -580,7 +590,39 @@ CREATE TABLE user_similarity (
580590 );
581591 ```
582592
583-## 7. Clustering & Recommendations
593+### 6.8 Follows
594+
595+Tracks follow relationships between users (from `app.bsky.graph.follow` and `sh.tangled.graph.follow` records).
596+
597+```sql
598+CREATE TABLE follows (
599+ user_did TEXT NOT NULL REFERENCES users(did),
600+ target_did TEXT NOT NULL,
601+ uri TEXT,
602+ cid TEXT,
603+ followed_at DATETIME,
604+ PRIMARY KEY (user_did, target_did)
605+);
606+
607+CREATE INDEX idx_follows_user ON follows(user_did);
608+CREATE INDEX idx_follows_target ON follows(target_did);
609+```
610+
611+### 6.9 OAuth Storage
612+
613+```sql
614+CREATE TABLE oauth_auth_requests (
615+ state TEXT PRIMARY KEY,
616+ data TEXT NOT NULL
617+);
618+
619+CREATE TABLE oauth_sessions (
620+ account_did TEXT NOT NULL,
621+ session_id TEXT NOT NULL,
622+ data TEXT NOT NULL,
623+ PRIMARY KEY (account_did, session_id)
624+);
625+```
584626
585627 Glean has two complementary recommendation signals:
586628
@@ -626,11 +668,9 @@ score(feed) = Σ J(target, U) for each user U subscribed to feed
626668 4. Return top N articles as recommendations
627669
628670 ```
629-score(article) = Σ 1/logN(likers(article)) for each user U who liked it
671+score(article) = Σ J(target, U) for each similar user U who liked the article
630672 ```
631673
632-The `1/logN` weighting avoids over-recommending articles from very large feeds.
633-
634674 **People recommendations (to follow on Bluesky):**
635675
636676 1. Compute user similarity for all pairs
@@ -657,10 +697,12 @@ For larger scale, move to MinHash + LSH (banded hashing) to approximate Jaccard
657697
658698 A background goroutine runs on a configurable schedule (`GLEAN_CLUSTER_INTERVAL`, default 6h):
659699
660-1. **Jetstream ingestion**: Subscribe to Jetstream for `at.glean.*` records
661-2. **Index new records**: Parse lexicon records, upsert into SQLite
662-3. **Compute similarities**: Batch-update the `feed_similarity`, `user_similarity`, and `article_co_like` tables
663-4. **Generate recommendations**: Materialize top recommendations per user into cache tables
700+1. **Compute feed similarity**: Batch-update the `feed_similarity` table (Jaccard over subscriber sets)
701+2. **Compute user similarity**: Batch-update the `user_similarity` table (Jaccard over subscription sets, boosted by follow relationships)
702+3. **Generate feed recommendations**: Materialize top feed recommendations per user into `user_feed_recommendations`
703+4. **Generate article recommendations**: Materialize top article recommendations per user into `user_article_recommendations`
704+
705+Jetstream ingestion and record indexing happen in a separate persistent goroutine (the Jetstream consumer), not in the cron.
664706
665707 ```sql
666708 CREATE TABLE user_feed_recommendations (
@@ -687,28 +729,31 @@ The server renders HTML fragments that htmx swaps into the page. No JSON API nee
687729
688730 ### 8.1 Pages
689731
690-| Route | Method | Description |
691-| ------------------------- | ------ | -------------------------------------------------------- |
692-| `/` | GET | Landing page / auth redirect |
693-| `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar |
694-| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |
695-| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions |
696-| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |
697-| `/feeds/add` | POST | Add a single feed URL |
698-| `/feeds/remove` | DELETE | Remove a feed |
699-| `/feeds/refresh` | POST | Refresh all subscribed feeds |
700-| `/feeds/clear` | POST | Clear all subscriptions |
701-| `/articles` | GET | Read articles (paginated, filterable by feed) |
702-| `/articles/{id}` | GET | Article detail view |
703-| `/articles/{id}/read` | POST | Mark article as read |
704-| `/articles/{id}/unread` | POST | Mark article as unread |
705-| `/articles/{id}/like` | POST | Like an article |
706-| `/articles/mark-all-read` | POST | Mark all articles as read |
707-| `/trending` | GET | Community feed: articles ranked by likes |
708-| `/library` | GET | Liked articles and annotations |
709-| `/library/create` | POST | Create annotation on an article |
710-| `/library/{id}/delete` | POST | Delete an annotation |
711-| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |
732+| Route | Method | Description |
733+| ------------------------------ | ------ | -------------------------------------------------------- |
734+| `/` | GET | Landing page / auth redirect |
735+| `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar |
736+| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |
737+| `/feeds/list` | GET | Feed list fragment (htmx partial) |
738+| `/feeds/discover-url` | GET | Discover feed URL from a website |
739+| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions |
740+| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |
741+| `/feeds/add` | POST | Add a single feed URL |
742+| `/feeds/remove` | DELETE | Remove a feed |
743+| `/feeds/refresh` | POST | Refresh all subscribed feeds |
744+| `/feeds/clear` | POST | Clear all subscriptions |
745+| `/articles` | GET | Read articles (paginated, filterable by feed) |
746+| `/articles/{id}` | GET | Article detail view |
747+| `/articles/{id}/read` | POST | Mark article as read |
748+| `/articles/{id}/unread` | POST | Mark article as unread |
749+| `/articles/{id}/like` | POST | Like an article |
750+| `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL |
751+| `/articles/mark-all-read` | POST | Mark all articles as read |
752+| `/trending` | GET | Community feed: articles ranked by likes |
753+| `/library` | GET | Liked articles and annotations |
754+| `/library/create` | POST | Create annotation on an article |
755+| `/library/{id}/delete` | POST | Delete an annotation |
756+| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |
712757
713758 ### 8.2 htmx Patterns
714759
@@ -726,12 +771,19 @@ glean/
726771 ├── go.sum
727772 ├── Dockerfile
728773 ├── Makefile
774+├── lexicons/
775+│ └── at/
776+│ ├── glean/ # Glean lexicon JSON schemas (subscription, annotation, like)
777+│ └── margin/ # External lexicon schemas (note.json for at.margin.note)
729778 ├── internal/
730779 │ ├── atproto/
731780 │ │ ├── auth.go # DID resolution, OAuth flow
732781 │ │ ├── client.go # XRPC client (write to user PDS)
733782 │ │ ├── jetstream.go # Subscribe to Jetstream via official client
734783 │ │ ├── stream_handler.go # Stream event → DB handler
784+│ │ ├── lexicon.go # Lexicon record types (at.glean.*, maintained by hand)
785+│ │ ├── lexicon_external.go # External lexicon record types (FollowRecord, MarginNoteRecord)
786+│ │ ├── lexicon_test.go # Test: Go structs match lexicon JSON schemas
735787 │ │ ├── sync.go # PDS record reconciliation
736788 │ │ └── xrpc.go # XRPC query handlers (AppView endpoints)
737789 │ ├── db/
@@ -740,6 +792,7 @@ glean/
740792 │ │ ├── feed.go # Feed + subscription queries
741793 │ │ ├── article.go # Article queries
742794 │ │ ├── social.go # Like, annotation queries
795+│ │ ├── follow.go # Follow queries
743796 │ │ ├── cluster.go # Similarity + recommendation queries
744797 │ │ ├── oauth_store.go # OAuth session storage
745798 │ │ └── store.go # FeedStore adapter for scheduler
@@ -748,21 +801,25 @@ glean/
748801 │ │ ├── fetcher.go # Scheduler with dedup + Fetcher
749802 │ │ ├── discover.go # Feed auto-discovery from URLs
750803 │ │ └── opml.go # OPML import/export
804+│ ├── scraper/
805+│ │ └── scraper.go # Full article content scraper
751806 │ ├── metrics/
752807 │ │ └── metrics.go # Prometheus metrics definitions
753808 │ ├── cluster/
754809 │ │ ├── jaccard.go # Jaccard similarity computation
755-│ │ ├── recommender.go # Feed + people recommendation logic
810+│ │ ├── recommender.go # Feed + people recommendation queries
756811 │ │ └── cron.go # Background recomputation scheduler
757812 │ ├── server/
758813 │ │ ├── server.go # HTTP server, router setup
759814 │ │ ├── auth_handler.go # OAuth login/callback
760815 │ │ ├── feeds_handler.go # Feed management handlers
761816 │ │ ├── articles_handler.go # Article reading handlers
817+│ │ ├── annotations_handler.go # Annotation handlers
762818 │ │ ├── dashboard_handler.go # Dashboard handler
763819 │ │ ├── trending_handler.go # Trending handler
764-│ │ ├── library_handler.go # Library (likes + annotations)
820+│ │ ├── index_handler.go # Landing page handler
765821 │ │ ├── profile_handler.go # Public profile handler
822+│ │ ├── pagination.go # Pagination helpers
766823 │ │ ├── middleware.go # Auth, logging, CSRF middleware
767824 │ │ └── session.go # Session management
768825 │ ├── sanitize/
@@ -770,7 +827,6 @@ glean/
770827 │ └── tmpl/
771828 │ ├── base.html # Base template with htmx + Tailwind
772829 │ ├── index.html # Landing page
773-│ ├── login.html # Login page
774830 │ ├── dashboard.html # Dashboard
775831 │ ├── feeds.html # Feed management
776832 │ ├── articles.html # Article listing
@@ -865,10 +921,13 @@ Glean exposes a `/metrics` endpoint for monitoring. Key metrics:
865921 - **`glean_feed_fetch_duration_seconds`** — Histogram of feed fetch latency
866922 - **`glean_articles_upserted_total`** — Counter of articles stored from feeds
867923 - **`glean_jetstream_events_total`** — Jetstream events labeled by collection and action
924+- **`glean_jetstream_errors_total`** — Jetstream handler errors
925+- **`glean_jetstream_reconnects_total`** — Jetstream reconnection count
868926 - **`glean_http_requests_total`** — HTTP request counts labeled by method, path, and status
927+- **`glean_pds_sync_runs_total`** / **`glean_pds_sync_errors_total`** — PDS sync runs and errors
869928 - **`glean_cluster_runs_total`** / **`glean_cluster_duration_seconds`** — Recommendation engine runs and timing
870929
871-### 12.3 Why htmx?
930+### 12.4 Why htmx?
872931
873932 - No JavaScript build pipeline
874933 - Server renders everything — simpler mental model
@@ -876,7 +935,7 @@ Glean exposes a `/metrics` endpoint for monitoring. Key metrics:
876935 - Perfect fit for a read-centric application
877936 - TailwindCSS handles styling without writing custom CSS
878937
879-### 12.4 AppView Architecture
938+### 12.5 AppView Architecture
880939
881940 Glean operates as an AT Protocol AppView. This means:
882941
@@ -885,7 +944,7 @@ Glean operates as an AT Protocol AppView. This means:
885944 - **Query path**: Other AT Protocol apps can query Glean's XRPC endpoints to access indexed data (subscriptions, annotations, likes, recommendations) without building their own indexer.
886945 - **Trade-off**: Article content (fetched from RSS feeds) is local-only and not part of the AT Protocol layer. Only individual feed subscription records (`at.glean.subscription`) live on the PDS.
887946
888-### 12.5 Privacy Model
947+### 12.6 Privacy Model
889948
890949 All PDS records are public. There is no notion of private data on the AT Protocol — everything stored on the repo is visible. Users should be aware that annotations, subscriptions, and likes are all public records.
891950
@@ -120,7 +120,24 @@ A user likes an article. The liked feed surfaces popular articles and feeds into
120 }120 }
121 ```121 ```
122 122
123-### 3.4 AppView Query Lexicons123+### 3.4 `at.margin.note` (External)
124+
125+Glean also indexes records from the `at.margin.note` lexicon (owned by [margin.at](https://margin.at)). These are displayed in the UI as if they were `at.glean.annotation` records — margin notes appear alongside glean annotations on article detail pages.
126+
127+The mapping from margin note to glean annotation:
128+
129+| Margin note field | Annotation field | Notes |
130+| ------------------------------ | ---------------- | --------------------------------------------------------------- |
131+| `target.source` | `article_url` | W3C SpecificResource source URL |
132+| `body.value` | `note` | Text content of the annotation |
133+| `target.selector.exact` | `quote` | TextQuoteSelector exact text |
134+| `tags` | `tags` | Direct mapping |
135+| `createdAt` | `created_at` | Direct mapping |
136+| _(looked up from articles DB)_ | `feed_url` | Resolved by matching `target.source` against known article URLs |
137+
138+When no matching article exists in the local DB, the annotation is stored with an empty `feed_url`. Margin notes are indexed from both Jetstream and PDS sync, same as glean records.
139+
140+### 3.5 AppView Query Lexicons
124 141
125 As an AppView, Glean serves the following XRPC query endpoints. Other AT Protocol applications can call these to access indexed `at.glean.*` data without implementing their own indexer.142 As an AppView, Glean serves the following XRPC query endpoints. Other AT Protocol applications can call these to access indexed `at.glean.*` data without implementing their own indexer.
126 143
@@ -209,7 +226,7 @@ Get feed recommendations for a user based on clustering.
209 226
210 ```227 ```
211 Input:228 Input:
212- repo: string (DID of the user)229+ repo: string (DID of the user, query parameter)
213 limit?: integer (default 20, max 50)230 limit?: integer (default 20, max 50)
214 231
215 Output:232 Output:
@@ -217,12 +234,12 @@ Output:
217 people: [{ did, handle, displayName, avatar, jaccard, commonFeeds }]234 people: [{ did, handle, displayName, avatar, jaccard, commonFeeds }]
218 ```235 ```
219 236
220-### 3.5 AppView Jetstream Consumption237+### 3.6 AppView Jetstream Consumption
221 238
222 Glean subscribes to a Jetstream endpoint (`GLEAN_JETSTREAM`, default `wss://jetstream2.fr.hose.cam`) for all `at.glean.*` records:239 Glean subscribes to a Jetstream endpoint (`GLEAN_JETSTREAM`, default `wss://jetstream2.fr.hose.cam`) for all `at.glean.*` records:
223 240
224 ```241 ```
225-SUBSCRIBE collections: ["at.glean.subscription", "at.glean.annotation", "at.glean.like"]242+SUBSCRIBE collections: ["at.glean.subscription", "at.glean.annotation", "at.glean.like", "app.bsky.graph.follow", "sh.tangled.graph.follow", "at.margin.note"]
226 ```243 ```
227 244
228 On each event:245 On each event:
@@ -299,25 +316,16 @@ Go's `encoding/xml` for RSS and Atom. A simple `encoding/json` for JSON Feed.
299 Each parser returns a normalized `Feed` and a slice of `Article` structs:316 Each parser returns a normalized `Feed` and a slice of `Article` structs:
300 317
301 ```go318 ```go
302-type Feed struct {
303- URL string
304- Title string
305- SiteURL string
306- Description string
307- Type string // "rss", "atom", "json"
308- ETag string
309- LastModified string
310-}
311-
312 type Article struct {319 type Article struct {
313- GUID string320+ FeedURL string
314- Title string321+ GUID string
315- URL string322+ Title string
316- Author string323+ URL string
317- Content string324+ Author string
318- Summary string325+ Content string
319- Published time.Time326+ Summary string
320- Updated time.Time327+ Published time.Time
328+ Updated time.Time
321 }329 }
322 ```330 ```
323 331
@@ -337,14 +345,18 @@ CREATE TABLE articles (
337 author TEXT,345 author TEXT,
338 summary TEXT,346 summary TEXT,
339 content TEXT,347 content TEXT,
348+ full_content TEXT,
340 published DATETIME,349 published DATETIME,
341 updated DATETIME,350 updated DATETIME,
342 fetched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,351 fetched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
343 UNIQUE(feed_url, guid)352 UNIQUE(feed_url, guid)
344 );353 );
354+
355+CREATE INDEX idx_articles_feed ON articles(feed_url);
356+CREATE INDEX idx_articles_published ON articles(published DESC);
345 ```357 ```
346 358
347-Content is stored as raw HTML from the feed's `<content:encoded>`, `<summary>`, or JSON Feed `content_html`. The server renders it in a sanitized view (strip `<script>`, `<iframe>`, etc.).359+Content is stored as raw HTML from the feed's `<content:encoded>`, `<summary>`, or JSON Feed `content_html`. `full_content` stores scraped article content fetched from the original URL. The server renders it in a sanitized view (strip `<script>`, `<iframe>`, etc.).
348 360
349 ### 4.5 Read State361 ### 4.5 Read State
350 362
@@ -415,9 +427,10 @@ Glean runs as a single Go binary that fills three roles: **AppView** (indexing `
415 │ └─────────────────┘ │ └──────────────────┘427 │ └─────────────────┘ │ └──────────────────┘
416 └──────────────────────┘428 └──────────────────────┘
417 429
418- AppView responsibilities:430+ AppView responsibilities:
419- • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like431+ • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like, at.margin.note
420- • Index records into SQLite432+ • Index records into SQLite
433+ • Convert at.margin.note records to annotations (displayed alongside glean.at annotations)
421 • Serve XRPC query endpoints (at.glean.listSubscriptions, etc.)434 • Serve XRPC query endpoints (at.glean.listSubscriptions, etc.)
422 • Host the web UI at glean.at435 • Host the web UI at glean.at
423 • Write to user PDS on behalf of user (when user acts through UI)436 • Write to user PDS on behalf of user (when user acts through UI)
@@ -515,13 +528,10 @@ CREATE TABLE read_state (
515 article_id INTEGER NOT NULL REFERENCES articles(id),528 article_id INTEGER NOT NULL REFERENCES articles(id),
516 is_read BOOLEAN NOT NULL DEFAULT 0,529 is_read BOOLEAN NOT NULL DEFAULT 0,
517 read_at DATETIME,530 read_at DATETIME,
518- is_starred BOOLEAN NOT NULL DEFAULT 0,
519- starred_at DATETIME,
520 PRIMARY KEY (user_did, article_id)531 PRIMARY KEY (user_did, article_id)
521 );532 );
522 533
523 CREATE INDEX idx_read_state_unread ON read_state(user_did, is_read) WHERE is_read = 0;534 CREATE INDEX idx_read_state_unread ON read_state(user_did, is_read) WHERE is_read = 0;
524-CREATE INDEX idx_read_state_starred ON read_state(user_did, is_starred) WHERE is_starred = 1;
525 ```535 ```
526 536
527 ### 6.6 Annotations, Likes537 ### 6.6 Annotations, Likes
@@ -580,7 +590,39 @@ CREATE TABLE user_similarity (
580 );590 );
581 ```591 ```
582 592
583-## 7. Clustering & Recommendations593+### 6.8 Follows
594+
595+Tracks follow relationships between users (from `app.bsky.graph.follow` and `sh.tangled.graph.follow` records).
596+
597+```sql
598+CREATE TABLE follows (
599+ user_did TEXT NOT NULL REFERENCES users(did),
600+ target_did TEXT NOT NULL,
601+ uri TEXT,
602+ cid TEXT,
603+ followed_at DATETIME,
604+ PRIMARY KEY (user_did, target_did)
605+);
606+
607+CREATE INDEX idx_follows_user ON follows(user_did);
608+CREATE INDEX idx_follows_target ON follows(target_did);
609+```
610+
611+### 6.9 OAuth Storage
612+
613+```sql
614+CREATE TABLE oauth_auth_requests (
615+ state TEXT PRIMARY KEY,
616+ data TEXT NOT NULL
617+);
618+
619+CREATE TABLE oauth_sessions (
620+ account_did TEXT NOT NULL,
621+ session_id TEXT NOT NULL,
622+ data TEXT NOT NULL,
623+ PRIMARY KEY (account_did, session_id)
624+);
625+```
584 626
585 Glean has two complementary recommendation signals:627 Glean has two complementary recommendation signals:
586 628
@@ -626,11 +668,9 @@ score(feed) = Σ J(target, U) for each user U subscribed to feed
626 4. Return top N articles as recommendations668 4. Return top N articles as recommendations
627 669
628 ```670 ```
629-score(article) = Σ 1/logN(likers(article)) for each user U who liked it671+score(article) = Σ J(target, U) for each similar user U who liked the article
630 ```672 ```
631 673
632-The `1/logN` weighting avoids over-recommending articles from very large feeds.
633-
634 **People recommendations (to follow on Bluesky):**674 **People recommendations (to follow on Bluesky):**
635 675
636 1. Compute user similarity for all pairs676 1. Compute user similarity for all pairs
@@ -657,10 +697,12 @@ For larger scale, move to MinHash + LSH (banded hashing) to approximate Jaccard
657 697
658 A background goroutine runs on a configurable schedule (`GLEAN_CLUSTER_INTERVAL`, default 6h):698 A background goroutine runs on a configurable schedule (`GLEAN_CLUSTER_INTERVAL`, default 6h):
659 699
660-1. **Jetstream ingestion**: Subscribe to Jetstream for `at.glean.*` records700+1. **Compute feed similarity**: Batch-update the `feed_similarity` table (Jaccard over subscriber sets)
661-2. **Index new records**: Parse lexicon records, upsert into SQLite701+2. **Compute user similarity**: Batch-update the `user_similarity` table (Jaccard over subscription sets, boosted by follow relationships)
662-3. **Compute similarities**: Batch-update the `feed_similarity`, `user_similarity`, and `article_co_like` tables702+3. **Generate feed recommendations**: Materialize top feed recommendations per user into `user_feed_recommendations`
663-4. **Generate recommendations**: Materialize top recommendations per user into cache tables703+4. **Generate article recommendations**: Materialize top article recommendations per user into `user_article_recommendations`
704+
705+Jetstream ingestion and record indexing happen in a separate persistent goroutine (the Jetstream consumer), not in the cron.
664 706
665 ```sql707 ```sql
666 CREATE TABLE user_feed_recommendations (708 CREATE TABLE user_feed_recommendations (
@@ -687,28 +729,31 @@ The server renders HTML fragments that htmx swaps into the page. No JSON API nee
687 729
688 ### 8.1 Pages730 ### 8.1 Pages
689 731
690-| Route | Method | Description |732+| Route | Method | Description |
691-| ------------------------- | ------ | -------------------------------------------------------- |733+| ------------------------------ | ------ | -------------------------------------------------------- |
692-| `/` | GET | Landing page / auth redirect |734+| `/` | GET | Landing page / auth redirect |
693-| `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar |735+| `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar |
694-| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |736+| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |
695-| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions |737+| `/feeds/list` | GET | Feed list fragment (htmx partial) |
696-| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |738+| `/feeds/discover-url` | GET | Discover feed URL from a website |
697-| `/feeds/add` | POST | Add a single feed URL |739+| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions |
698-| `/feeds/remove` | DELETE | Remove a feed |740+| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |
699-| `/feeds/refresh` | POST | Refresh all subscribed feeds |741+| `/feeds/add` | POST | Add a single feed URL |
700-| `/feeds/clear` | POST | Clear all subscriptions |742+| `/feeds/remove` | DELETE | Remove a feed |
701-| `/articles` | GET | Read articles (paginated, filterable by feed) |743+| `/feeds/refresh` | POST | Refresh all subscribed feeds |
702-| `/articles/{id}` | GET | Article detail view |744+| `/feeds/clear` | POST | Clear all subscriptions |
703-| `/articles/{id}/read` | POST | Mark article as read |745+| `/articles` | GET | Read articles (paginated, filterable by feed) |
704-| `/articles/{id}/unread` | POST | Mark article as unread |746+| `/articles/{id}` | GET | Article detail view |
705-| `/articles/{id}/like` | POST | Like an article |747+| `/articles/{id}/read` | POST | Mark article as read |
706-| `/articles/mark-all-read` | POST | Mark all articles as read |748+| `/articles/{id}/unread` | POST | Mark article as unread |
707-| `/trending` | GET | Community feed: articles ranked by likes |749+| `/articles/{id}/like` | POST | Like an article |
708-| `/library` | GET | Liked articles and annotations |750+| `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL |
709-| `/library/create` | POST | Create annotation on an article |751+| `/articles/mark-all-read` | POST | Mark all articles as read |
710-| `/library/{id}/delete` | POST | Delete an annotation |752+| `/trending` | GET | Community feed: articles ranked by likes |
711-| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |753+| `/library` | GET | Liked articles and annotations |
754+| `/library/create` | POST | Create annotation on an article |
755+| `/library/{id}/delete` | POST | Delete an annotation |
756+| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |
712 757
713 ### 8.2 htmx Patterns758 ### 8.2 htmx Patterns
714 759
@@ -726,12 +771,19 @@ glean/
726 ├── go.sum771 ├── go.sum
727 ├── Dockerfile772 ├── Dockerfile
728 ├── Makefile773 ├── Makefile
774+├── lexicons/
775+│ └── at/
776+│ ├── glean/ # Glean lexicon JSON schemas (subscription, annotation, like)
777+│ └── margin/ # External lexicon schemas (note.json for at.margin.note)
729 ├── internal/778 ├── internal/
730 │ ├── atproto/779 │ ├── atproto/
731 │ │ ├── auth.go # DID resolution, OAuth flow780 │ │ ├── auth.go # DID resolution, OAuth flow
732 │ │ ├── client.go # XRPC client (write to user PDS)781 │ │ ├── client.go # XRPC client (write to user PDS)
733 │ │ ├── jetstream.go # Subscribe to Jetstream via official client782 │ │ ├── jetstream.go # Subscribe to Jetstream via official client
734 │ │ ├── stream_handler.go # Stream event → DB handler783 │ │ ├── stream_handler.go # Stream event → DB handler
784+│ │ ├── lexicon.go # Lexicon record types (at.glean.*, maintained by hand)
785+│ │ ├── lexicon_external.go # External lexicon record types (FollowRecord, MarginNoteRecord)
786+│ │ ├── lexicon_test.go # Test: Go structs match lexicon JSON schemas
735 │ │ ├── sync.go # PDS record reconciliation787 │ │ ├── sync.go # PDS record reconciliation
736 │ │ └── xrpc.go # XRPC query handlers (AppView endpoints)788 │ │ └── xrpc.go # XRPC query handlers (AppView endpoints)
737 │ ├── db/789 │ ├── db/
@@ -740,6 +792,7 @@ glean/
740 │ │ ├── feed.go # Feed + subscription queries792 │ │ ├── feed.go # Feed + subscription queries
741 │ │ ├── article.go # Article queries793 │ │ ├── article.go # Article queries
742 │ │ ├── social.go # Like, annotation queries794 │ │ ├── social.go # Like, annotation queries
795+│ │ ├── follow.go # Follow queries
743 │ │ ├── cluster.go # Similarity + recommendation queries796 │ │ ├── cluster.go # Similarity + recommendation queries
744 │ │ ├── oauth_store.go # OAuth session storage797 │ │ ├── oauth_store.go # OAuth session storage
745 │ │ └── store.go # FeedStore adapter for scheduler798 │ │ └── store.go # FeedStore adapter for scheduler
@@ -748,21 +801,25 @@ glean/
748 │ │ ├── fetcher.go # Scheduler with dedup + Fetcher801 │ │ ├── fetcher.go # Scheduler with dedup + Fetcher
749 │ │ ├── discover.go # Feed auto-discovery from URLs802 │ │ ├── discover.go # Feed auto-discovery from URLs
750 │ │ └── opml.go # OPML import/export803 │ │ └── opml.go # OPML import/export
804+│ ├── scraper/
805+│ │ └── scraper.go # Full article content scraper
751 │ ├── metrics/806 │ ├── metrics/
752 │ │ └── metrics.go # Prometheus metrics definitions807 │ │ └── metrics.go # Prometheus metrics definitions
753 │ ├── cluster/808 │ ├── cluster/
754 │ │ ├── jaccard.go # Jaccard similarity computation809 │ │ ├── jaccard.go # Jaccard similarity computation
755-│ │ ├── recommender.go # Feed + people recommendation logic810+│ │ ├── recommender.go # Feed + people recommendation queries
756 │ │ └── cron.go # Background recomputation scheduler811 │ │ └── cron.go # Background recomputation scheduler
757 │ ├── server/812 │ ├── server/
758 │ │ ├── server.go # HTTP server, router setup813 │ │ ├── server.go # HTTP server, router setup
759 │ │ ├── auth_handler.go # OAuth login/callback814 │ │ ├── auth_handler.go # OAuth login/callback
760 │ │ ├── feeds_handler.go # Feed management handlers815 │ │ ├── feeds_handler.go # Feed management handlers
761 │ │ ├── articles_handler.go # Article reading handlers816 │ │ ├── articles_handler.go # Article reading handlers
817+│ │ ├── annotations_handler.go # Annotation handlers
762 │ │ ├── dashboard_handler.go # Dashboard handler818 │ │ ├── dashboard_handler.go # Dashboard handler
763 │ │ ├── trending_handler.go # Trending handler819 │ │ ├── trending_handler.go # Trending handler
764-│ │ ├── library_handler.go # Library (likes + annotations)820+│ │ ├── index_handler.go # Landing page handler
765 │ │ ├── profile_handler.go # Public profile handler821 │ │ ├── profile_handler.go # Public profile handler
822+│ │ ├── pagination.go # Pagination helpers
766 │ │ ├── middleware.go # Auth, logging, CSRF middleware823 │ │ ├── middleware.go # Auth, logging, CSRF middleware
767 │ │ └── session.go # Session management824 │ │ └── session.go # Session management
768 │ ├── sanitize/825 │ ├── sanitize/
@@ -770,7 +827,6 @@ glean/
770 │ └── tmpl/827 │ └── tmpl/
771 │ ├── base.html # Base template with htmx + Tailwind828 │ ├── base.html # Base template with htmx + Tailwind
772 │ ├── index.html # Landing page829 │ ├── index.html # Landing page
773-│ ├── login.html # Login page
774 │ ├── dashboard.html # Dashboard830 │ ├── dashboard.html # Dashboard
775 │ ├── feeds.html # Feed management831 │ ├── feeds.html # Feed management
776 │ ├── articles.html # Article listing832 │ ├── articles.html # Article listing
@@ -865,10 +921,13 @@ Glean exposes a `/metrics` endpoint for monitoring. Key metrics:
865 - **`glean_feed_fetch_duration_seconds`** — Histogram of feed fetch latency921 - **`glean_feed_fetch_duration_seconds`** — Histogram of feed fetch latency
866 - **`glean_articles_upserted_total`** — Counter of articles stored from feeds922 - **`glean_articles_upserted_total`** — Counter of articles stored from feeds
867 - **`glean_jetstream_events_total`** — Jetstream events labeled by collection and action923 - **`glean_jetstream_events_total`** — Jetstream events labeled by collection and action
924+- **`glean_jetstream_errors_total`** — Jetstream handler errors
925+- **`glean_jetstream_reconnects_total`** — Jetstream reconnection count
868 - **`glean_http_requests_total`** — HTTP request counts labeled by method, path, and status926 - **`glean_http_requests_total`** — HTTP request counts labeled by method, path, and status
927+- **`glean_pds_sync_runs_total`** / **`glean_pds_sync_errors_total`** — PDS sync runs and errors
869 - **`glean_cluster_runs_total`** / **`glean_cluster_duration_seconds`** — Recommendation engine runs and timing928 - **`glean_cluster_runs_total`** / **`glean_cluster_duration_seconds`** — Recommendation engine runs and timing
870 929
871-### 12.3 Why htmx?930+### 12.4 Why htmx?
872 931
873 - No JavaScript build pipeline932 - No JavaScript build pipeline
874 - Server renders everything — simpler mental model933 - Server renders everything — simpler mental model
@@ -876,7 +935,7 @@ Glean exposes a `/metrics` endpoint for monitoring. Key metrics:
876 - Perfect fit for a read-centric application935 - Perfect fit for a read-centric application
877 - TailwindCSS handles styling without writing custom CSS936 - TailwindCSS handles styling without writing custom CSS
878 937
879-### 12.4 AppView Architecture938+### 12.5 AppView Architecture
880 939
881 Glean operates as an AT Protocol AppView. This means:940 Glean operates as an AT Protocol AppView. This means:
882 941
@@ -885,7 +944,7 @@ Glean operates as an AT Protocol AppView. This means:
885 - **Query path**: Other AT Protocol apps can query Glean's XRPC endpoints to access indexed data (subscriptions, annotations, likes, recommendations) without building their own indexer.944 - **Query path**: Other AT Protocol apps can query Glean's XRPC endpoints to access indexed data (subscriptions, annotations, likes, recommendations) without building their own indexer.
886 - **Trade-off**: Article content (fetched from RSS feeds) is local-only and not part of the AT Protocol layer. Only individual feed subscription records (`at.glean.subscription`) live on the PDS.945 - **Trade-off**: Article content (fetched from RSS feeds) is local-only and not part of the AT Protocol layer. Only individual feed subscription records (`at.glean.subscription`) live on the PDS.
887 946
888-### 12.5 Privacy Model947+### 12.6 Privacy Model
889 948
890 All PDS records are public. There is no notion of private data on the AT Protocol — everything stored on the repo is visible. Users should be aware that annotations, subscriptions, and likes are all public records.949 All PDS records are public. There is no notion of private data on the AT Protocol — everything stored on the repo is visible. Users should be aware that annotations, subscriptions, and likes are all public records.
891 950
modified internal/atproto/auth.go +4 -2
@@ -8,8 +8,10 @@ import (
88 "github.com/bluesky-social/indigo/atproto/syntax"
99 )
1010
11-type DIDDocument = identity.DIDDocument
12-type Identity = identity.Identity
11+type (
12+ DIDDocument = identity.DIDDocument
13+ Identity = identity.Identity
14+)
1315
1416 func ResolveHandle(ctx context.Context, handle string) (string, error) {
1517 h, err := syntax.ParseHandle(handle)
@@ -8,8 +8,10 @@ import (
8 "github.com/bluesky-social/indigo/atproto/syntax"8 "github.com/bluesky-social/indigo/atproto/syntax"
9 )9 )
10 10
11-type DIDDocument = identity.DIDDocument11+type (
12-type Identity = identity.Identity12+ DIDDocument = identity.DIDDocument
13+ Identity = identity.Identity
14+)
13 15
14 func ResolveHandle(ctx context.Context, handle string) (string, error) {16 func ResolveHandle(ctx context.Context, handle string) (string, error) {
15 h, err := syntax.ParseHandle(handle)17 h, err := syntax.ParseHandle(handle)
modified internal/atproto/jetstream.go +6 -5
@@ -94,11 +94,12 @@ func NewJetstreamConsumer(jetstreamURL string, handler EventHandler, logger *slo
9494 "User-Agent": "glean/1.0",
9595 },
9696 WantedCollections: []string{
97- "at.glean.subscription",
98- "at.glean.annotation",
99- "at.glean.like",
100- "app.bsky.graph.follow",
101- "sh.tangled.graph.follow",
97+ CollectionSubscription,
98+ CollectionAnnotation,
99+ CollectionLike,
100+ CollectionBskyFollow,
101+ CollectionTangledFollow,
102+ CollectionMarginNote,
102103 },
103104 }
104105
@@ -94,11 +94,12 @@ func NewJetstreamConsumer(jetstreamURL string, handler EventHandler, logger *slo
94 "User-Agent": "glean/1.0",94 "User-Agent": "glean/1.0",
95 },95 },
96 WantedCollections: []string{96 WantedCollections: []string{
97- "at.glean.subscription",97+ CollectionSubscription,
98- "at.glean.annotation",98+ CollectionAnnotation,
99- "at.glean.like",99+ CollectionLike,
100- "app.bsky.graph.follow",100+ CollectionBskyFollow,
101- "sh.tangled.graph.follow",101+ CollectionTangledFollow,
102+ CollectionMarginNote,
102 },103 },
103 }104 }
104 105
modified internal/atproto/lexicon.go +9 -5
@@ -7,6 +7,15 @@ import (
77 "time"
88 )
99
10+const (
11+ CollectionSubscription = "at.glean.subscription"
12+ CollectionAnnotation = "at.glean.annotation"
13+ CollectionLike = "at.glean.like"
14+ CollectionMarginNote = "at.margin.note"
15+ CollectionBskyFollow = "app.bsky.graph.follow"
16+ CollectionTangledFollow = "sh.tangled.graph.follow"
17+)
18+
1019 type SubscriptionRecord struct {
1120 CreatedAt string `json:"createdAt"`
1221 FeedURL string `json:"feedUrl"`
@@ -30,11 +39,6 @@ type LikeRecord struct {
3039 ArticleURL string `json:"articleUrl"`
3140 }
3241
33-type FollowRecord struct {
34- Subject string `json:"subject"`
35- CreatedAt string `json:"createdAt"`
36-}
37-
3842 type Record struct {
3943 URI string
4044 CID string
@@ -7,6 +7,15 @@ import (
7 "time"7 "time"
8 )8 )
9 9
10+const (
11+ CollectionSubscription = "at.glean.subscription"
12+ CollectionAnnotation = "at.glean.annotation"
13+ CollectionLike = "at.glean.like"
14+ CollectionMarginNote = "at.margin.note"
15+ CollectionBskyFollow = "app.bsky.graph.follow"
16+ CollectionTangledFollow = "sh.tangled.graph.follow"
17+)
18+
10 type SubscriptionRecord struct {19 type SubscriptionRecord struct {
11 CreatedAt string `json:"createdAt"`20 CreatedAt string `json:"createdAt"`
12 FeedURL string `json:"feedUrl"`21 FeedURL string `json:"feedUrl"`
@@ -30,11 +39,6 @@ type LikeRecord struct {
30 ArticleURL string `json:"articleUrl"`39 ArticleURL string `json:"articleUrl"`
31 }40 }
32 41
33-type FollowRecord struct {
34- Subject string `json:"subject"`
35- CreatedAt string `json:"createdAt"`
36-}
37-
38 type Record struct {42 type Record struct {
39 URI string43 URI string
40 CID string44 CID string
added internal/atproto/lexicon_external.go +74 -0
new file mode 100644
@@ -0,0 +1,74 @@
1+// External lexicon record types (not owned by glean.at).
2+// These are maintained by hand to match the upstream lexicon schemas.
3+// See lexicon_test.go for the test ensuring these stay in sync with lexicons/.
4+package atproto
5+
6+import "encoding/json"
7+
8+type FollowRecord struct {
9+ Subject string `json:"subject"`
10+ CreatedAt string `json:"createdAt"`
11+ Via json.RawMessage `json:"via,omitempty"`
12+}
13+
14+type MarginNoteRecord struct {
15+ Body *MarginNoteBody `json:"body,omitempty"`
16+ Color string `json:"color,omitempty"`
17+ CreatedAt string `json:"createdAt"`
18+ Facets json.RawMessage `json:"facets,omitempty"`
19+ Generator *MarginNoteGenerator `json:"generator,omitempty"`
20+ Labels json.RawMessage `json:"labels,omitempty"`
21+ ModifiedAt string `json:"modifiedAt,omitempty"`
22+ Motivation string `json:"motivation"`
23+ Rights string `json:"rights,omitempty"`
24+ Tags []string `json:"tags,omitempty"`
25+ Target MarginNoteTarget `json:"target"`
26+}
27+
28+type MarginNoteBody struct {
29+ Format string `json:"format,omitempty"`
30+ URI string `json:"uri,omitempty"`
31+ Value string `json:"value,omitempty"`
32+}
33+
34+type MarginNoteGenerator struct {
35+ Homepage string `json:"homepage,omitempty"`
36+ ID string `json:"id,omitempty"`
37+ Name string `json:"name,omitempty"`
38+}
39+
40+type MarginNoteSelector struct {
41+ ConformsTo string `json:"conformsTo,omitempty"`
42+ End int `json:"end,omitempty"`
43+ Exact string `json:"exact,omitempty"`
44+ Prefix string `json:"prefix,omitempty"`
45+ Start int `json:"start,omitempty"`
46+ Suffix string `json:"suffix,omitempty"`
47+ Type string `json:"type"`
48+ Value string `json:"value,omitempty"`
49+}
50+
51+type MarginNoteTarget struct {
52+ Selector *MarginNoteSelector `json:"selector,omitempty"`
53+ Source string `json:"source"`
54+ SourceHash string `json:"sourceHash,omitempty"`
55+ State *MarginNoteTimeState `json:"state,omitempty"`
56+ Title string `json:"title,omitempty"`
57+}
58+
59+type MarginNoteTimeState struct {
60+ Cached string `json:"cached,omitempty"`
61+ SourceDate string `json:"sourceDate,omitempty"`
62+}
63+
64+func (r MarginNoteRecord) ToAnnotation() (articleURL, quote, note string, tags []string) {
65+ articleURL = r.Target.Source
66+ if r.Body != nil {
67+ note = r.Body.Value
68+ }
69+ if r.Target.Selector != nil && r.Target.Selector.Exact != "" {
70+ quote = r.Target.Selector.Exact
71+ }
72+ tags = r.Tags
73+ return
74+}
new file mode 100644
@@ -0,0 +1,74 @@
1+// External lexicon record types (not owned by glean.at).
2+// These are maintained by hand to match the upstream lexicon schemas.
3+// See lexicon_test.go for the test ensuring these stay in sync with lexicons/.
4+package atproto
5+
6+import "encoding/json"
7+
8+type FollowRecord struct {
9+ Subject string `json:"subject"`
10+ CreatedAt string `json:"createdAt"`
11+ Via json.RawMessage `json:"via,omitempty"`
12+}
13+
14+type MarginNoteRecord struct {
15+ Body *MarginNoteBody `json:"body,omitempty"`
16+ Color string `json:"color,omitempty"`
17+ CreatedAt string `json:"createdAt"`
18+ Facets json.RawMessage `json:"facets,omitempty"`
19+ Generator *MarginNoteGenerator `json:"generator,omitempty"`
20+ Labels json.RawMessage `json:"labels,omitempty"`
21+ ModifiedAt string `json:"modifiedAt,omitempty"`
22+ Motivation string `json:"motivation"`
23+ Rights string `json:"rights,omitempty"`
24+ Tags []string `json:"tags,omitempty"`
25+ Target MarginNoteTarget `json:"target"`
26+}
27+
28+type MarginNoteBody struct {
29+ Format string `json:"format,omitempty"`
30+ URI string `json:"uri,omitempty"`
31+ Value string `json:"value,omitempty"`
32+}
33+
34+type MarginNoteGenerator struct {
35+ Homepage string `json:"homepage,omitempty"`
36+ ID string `json:"id,omitempty"`
37+ Name string `json:"name,omitempty"`
38+}
39+
40+type MarginNoteSelector struct {
41+ ConformsTo string `json:"conformsTo,omitempty"`
42+ End int `json:"end,omitempty"`
43+ Exact string `json:"exact,omitempty"`
44+ Prefix string `json:"prefix,omitempty"`
45+ Start int `json:"start,omitempty"`
46+ Suffix string `json:"suffix,omitempty"`
47+ Type string `json:"type"`
48+ Value string `json:"value,omitempty"`
49+}
50+
51+type MarginNoteTarget struct {
52+ Selector *MarginNoteSelector `json:"selector,omitempty"`
53+ Source string `json:"source"`
54+ SourceHash string `json:"sourceHash,omitempty"`
55+ State *MarginNoteTimeState `json:"state,omitempty"`
56+ Title string `json:"title,omitempty"`
57+}
58+
59+type MarginNoteTimeState struct {
60+ Cached string `json:"cached,omitempty"`
61+ SourceDate string `json:"sourceDate,omitempty"`
62+}
63+
64+func (r MarginNoteRecord) ToAnnotation() (articleURL, quote, note string, tags []string) {
65+ articleURL = r.Target.Source
66+ if r.Body != nil {
67+ note = r.Body.Value
68+ }
69+ if r.Target.Selector != nil && r.Target.Selector.Exact != "" {
70+ quote = r.Target.Selector.Exact
71+ }
72+ tags = r.Tags
73+ return
74+}
modified internal/atproto/lexicon_test.go +23 -6
@@ -15,9 +15,13 @@ func lexiconPath(filename string) string {
1515 return filepath.Join("..", "..", "lexicons", "at", "glean", filename)
1616 }
1717
18-func readLexiconProperties(t *testing.T, filename string) map[string]any {
18+func lexiconPathFromRoot(relPath string) string {
19+ return filepath.Join("..", "..", "lexicons", relPath)
20+}
21+
22+func readLexiconPropertiesFromPath(t *testing.T, path string) map[string]any {
1923 t.Helper()
20- data, err := os.ReadFile(lexiconPath(filename))
24+ data, err := os.ReadFile(path)
2125 assert.NilError(t, err)
2226
2327 var schema struct {
@@ -30,13 +34,18 @@ func readLexiconProperties(t *testing.T, filename string) map[string]any {
3034 } `json:"defs"`
3135 }
3236 assert.NilError(t, json.Unmarshal(data, &schema))
33- assert.Assert(t, len(schema.Defs.Main.Record.Properties) > 0, "no properties found in %s", filename)
37+ assert.Assert(t, len(schema.Defs.Main.Record.Properties) > 0, "no properties found in %s", path)
3438 return schema.Defs.Main.Record.Properties
3539 }
3640
3741 func assertStructMatchesLexicon[T any](t *testing.T, filename string) {
3842 t.Helper()
39- properties := readLexiconProperties(t, filename)
43+ assertStructMatchesLexiconPath[T](t, lexiconPath(filename))
44+}
45+
46+func assertStructMatchesLexiconPath[T any](t *testing.T, path string) {
47+ t.Helper()
48+ properties := readLexiconPropertiesFromPath(t, path)
4049
4150 var zero T
4251 typ := reflect.TypeOf(zero)
@@ -52,12 +61,12 @@ func assertStructMatchesLexicon[T any](t *testing.T, filename string) {
5261 }
5362
5463 for prop := range properties {
55- assert.Assert(t, jsonTags[prop], "lexicon property %q missing from %s (lexicon file: %s)", prop, typ.Name(), filename)
64+ assert.Assert(t, jsonTags[prop], "lexicon property %q missing from %s (lexicon file: %s)", prop, typ.Name(), path)
5665 }
5766
5867 for name := range jsonTags {
5968 _, exists := properties[name]
60- assert.Assert(t, exists, "Go field %q in %s missing from lexicon %s", name, typ.Name(), filename)
69+ assert.Assert(t, exists, "Go field %q in %s missing from lexicon %s", name, typ.Name(), path)
6170 }
6271 }
6372
@@ -72,3 +81,11 @@ func TestAnnotationRecordMatchesLexicon(t *testing.T) {
7281 func TestLikeRecordMatchesLexicon(t *testing.T) {
7382 assertStructMatchesLexicon[LikeRecord](t, "like.json")
7483 }
84+
85+func TestFollowRecordMatchesLexicon(t *testing.T) {
86+ assertStructMatchesLexiconPath[FollowRecord](t, lexiconPathFromRoot("app/bsky/graph/follow.json"))
87+}
88+
89+func TestMarginNoteRecordMatchesLexicon(t *testing.T) {
90+ assertStructMatchesLexiconPath[MarginNoteRecord](t, lexiconPathFromRoot("at/margin/note.json"))
91+}
@@ -15,9 +15,13 @@ func lexiconPath(filename string) string {
15 return filepath.Join("..", "..", "lexicons", "at", "glean", filename)15 return filepath.Join("..", "..", "lexicons", "at", "glean", filename)
16 }16 }
17 17
18-func readLexiconProperties(t *testing.T, filename string) map[string]any {18+func lexiconPathFromRoot(relPath string) string {
19+ return filepath.Join("..", "..", "lexicons", relPath)
20+}
21+
22+func readLexiconPropertiesFromPath(t *testing.T, path string) map[string]any {
19 t.Helper()23 t.Helper()
20- data, err := os.ReadFile(lexiconPath(filename))24+ data, err := os.ReadFile(path)
21 assert.NilError(t, err)25 assert.NilError(t, err)
22 26
23 var schema struct {27 var schema struct {
@@ -30,13 +34,18 @@ func readLexiconProperties(t *testing.T, filename string) map[string]any {
30 } `json:"defs"`34 } `json:"defs"`
31 }35 }
32 assert.NilError(t, json.Unmarshal(data, &schema))36 assert.NilError(t, json.Unmarshal(data, &schema))
33- assert.Assert(t, len(schema.Defs.Main.Record.Properties) > 0, "no properties found in %s", filename)37+ assert.Assert(t, len(schema.Defs.Main.Record.Properties) > 0, "no properties found in %s", path)
34 return schema.Defs.Main.Record.Properties38 return schema.Defs.Main.Record.Properties
35 }39 }
36 40
37 func assertStructMatchesLexicon[T any](t *testing.T, filename string) {41 func assertStructMatchesLexicon[T any](t *testing.T, filename string) {
38 t.Helper()42 t.Helper()
39- properties := readLexiconProperties(t, filename)43+ assertStructMatchesLexiconPath[T](t, lexiconPath(filename))
44+}
45+
46+func assertStructMatchesLexiconPath[T any](t *testing.T, path string) {
47+ t.Helper()
48+ properties := readLexiconPropertiesFromPath(t, path)
40 49
41 var zero T50 var zero T
42 typ := reflect.TypeOf(zero)51 typ := reflect.TypeOf(zero)
@@ -52,12 +61,12 @@ func assertStructMatchesLexicon[T any](t *testing.T, filename string) {
52 }61 }
53 62
54 for prop := range properties {63 for prop := range properties {
55- assert.Assert(t, jsonTags[prop], "lexicon property %q missing from %s (lexicon file: %s)", prop, typ.Name(), filename)64+ assert.Assert(t, jsonTags[prop], "lexicon property %q missing from %s (lexicon file: %s)", prop, typ.Name(), path)
56 }65 }
57 66
58 for name := range jsonTags {67 for name := range jsonTags {
59 _, exists := properties[name]68 _, exists := properties[name]
60- assert.Assert(t, exists, "Go field %q in %s missing from lexicon %s", name, typ.Name(), filename)69+ assert.Assert(t, exists, "Go field %q in %s missing from lexicon %s", name, typ.Name(), path)
61 }70 }
62 }71 }
63 72
@@ -72,3 +81,11 @@ func TestAnnotationRecordMatchesLexicon(t *testing.T) {
72 func TestLikeRecordMatchesLexicon(t *testing.T) {81 func TestLikeRecordMatchesLexicon(t *testing.T) {
73 assertStructMatchesLexicon[LikeRecord](t, "like.json")82 assertStructMatchesLexicon[LikeRecord](t, "like.json")
74 }83 }
84+
85+func TestFollowRecordMatchesLexicon(t *testing.T) {
86+ assertStructMatchesLexiconPath[FollowRecord](t, lexiconPathFromRoot("app/bsky/graph/follow.json"))
87+}
88+
89+func TestMarginNoteRecordMatchesLexicon(t *testing.T) {
90+ assertStructMatchesLexiconPath[MarginNoteRecord](t, lexiconPathFromRoot("at/margin/note.json"))
91+}
modified internal/atproto/stream_handler.go +49 -4
@@ -21,13 +21,15 @@ func NewStreamDBHandler(database *db.DB, logger *slog.Logger) *StreamDBHandler {
2121
2222 func (h *StreamDBHandler) Handle(ctx context.Context, event *Event) error {
2323 switch event.Collection {
24- case "at.glean.subscription":
24+ case CollectionSubscription:
2525 return h.handleSubscription(ctx, event)
26- case "at.glean.like":
26+ case CollectionLike:
2727 return h.handleLike(ctx, event)
28- case "at.glean.annotation":
28+ case CollectionAnnotation:
2929 return h.handleAnnotation(ctx, event)
30- case "app.bsky.graph.follow", "sh.tangled.graph.follow":
30+ case CollectionMarginNote:
31+ return h.handleMarginNote(ctx, event)
32+ case CollectionBskyFollow, CollectionTangledFollow:
3133 return h.handleFollow(ctx, event)
3234 }
3335 return nil
@@ -155,3 +157,46 @@ func (h *StreamDBHandler) handleFollow(ctx context.Context, event *Event) error
155157 }
156158 return nil
157159 }
160+
161+func (h *StreamDBHandler) handleMarginNote(ctx context.Context, event *Event) error {
162+ switch event.Type {
163+ case "create", "update":
164+ var rec MarginNoteRecord
165+ if err := json.Unmarshal(event.Value, &rec); err != nil {
166+ return err
167+ }
168+
169+ articleURL, quote, note, tags := rec.ToAnnotation()
170+ if articleURL == "" {
171+ return nil
172+ }
173+
174+ feedURL := h.resolveFeedURL(ctx, articleURL)
175+
176+ t, _ := time.Parse(time.RFC3339, rec.CreatedAt)
177+ a := &db.Annotation{
178+ URI: event.URI,
179+ AuthorDID: event.DID,
180+ FeedURL: feedURL,
181+ ArticleURL: articleURL,
182+ Quote: db.NullStr(quote),
183+ Note: db.NullStr(note),
184+ Tags: db.NullStrTags(tags),
185+ CreatedAt: sql.NullTime{Time: t, Valid: true},
186+ CID: sql.NullString{String: event.CID, Valid: event.CID != ""},
187+ }
188+ return h.db.CreateAnnotation(ctx, a)
189+
190+ case "delete":
191+ return h.db.DeleteAnnotation(ctx, event.URI)
192+ }
193+ return nil
194+}
195+
196+func (h *StreamDBHandler) resolveFeedURL(ctx context.Context, articleURL string) string {
197+ article, err := h.db.GetArticleByURL(ctx, articleURL)
198+ if err != nil {
199+ return ""
200+ }
201+ return article.FeedURL
202+}
@@ -21,13 +21,15 @@ func NewStreamDBHandler(database *db.DB, logger *slog.Logger) *StreamDBHandler {
21 21
22 func (h *StreamDBHandler) Handle(ctx context.Context, event *Event) error {22 func (h *StreamDBHandler) Handle(ctx context.Context, event *Event) error {
23 switch event.Collection {23 switch event.Collection {
24- case "at.glean.subscription":24+ case CollectionSubscription:
25 return h.handleSubscription(ctx, event)25 return h.handleSubscription(ctx, event)
26- case "at.glean.like":26+ case CollectionLike:
27 return h.handleLike(ctx, event)27 return h.handleLike(ctx, event)
28- case "at.glean.annotation":28+ case CollectionAnnotation:
29 return h.handleAnnotation(ctx, event)29 return h.handleAnnotation(ctx, event)
30- case "app.bsky.graph.follow", "sh.tangled.graph.follow":30+ case CollectionMarginNote:
31+ return h.handleMarginNote(ctx, event)
32+ case CollectionBskyFollow, CollectionTangledFollow:
31 return h.handleFollow(ctx, event)33 return h.handleFollow(ctx, event)
32 }34 }
33 return nil35 return nil
@@ -155,3 +157,46 @@ func (h *StreamDBHandler) handleFollow(ctx context.Context, event *Event) error
155 }157 }
156 return nil158 return nil
157 }159 }
160+
161+func (h *StreamDBHandler) handleMarginNote(ctx context.Context, event *Event) error {
162+ switch event.Type {
163+ case "create", "update":
164+ var rec MarginNoteRecord
165+ if err := json.Unmarshal(event.Value, &rec); err != nil {
166+ return err
167+ }
168+
169+ articleURL, quote, note, tags := rec.ToAnnotation()
170+ if articleURL == "" {
171+ return nil
172+ }
173+
174+ feedURL := h.resolveFeedURL(ctx, articleURL)
175+
176+ t, _ := time.Parse(time.RFC3339, rec.CreatedAt)
177+ a := &db.Annotation{
178+ URI: event.URI,
179+ AuthorDID: event.DID,
180+ FeedURL: feedURL,
181+ ArticleURL: articleURL,
182+ Quote: db.NullStr(quote),
183+ Note: db.NullStr(note),
184+ Tags: db.NullStrTags(tags),
185+ CreatedAt: sql.NullTime{Time: t, Valid: true},
186+ CID: sql.NullString{String: event.CID, Valid: event.CID != ""},
187+ }
188+ return h.db.CreateAnnotation(ctx, a)
189+
190+ case "delete":
191+ return h.db.DeleteAnnotation(ctx, event.URI)
192+ }
193+ return nil
194+}
195+
196+func (h *StreamDBHandler) resolveFeedURL(ctx context.Context, articleURL string) string {
197+ article, err := h.db.GetArticleByURL(ctx, articleURL)
198+ if err != nil {
199+ return ""
200+ }
201+ return article.FeedURL
202+}
modified internal/atproto/sync.go +46 -4
@@ -33,15 +33,18 @@ func NewSync(database *db.DB, client *Client, logger *slog.Logger) *Sync {
3333 func (s *Sync) Run(ctx context.Context, userDID string) error {
3434 s.logger.Info("syncing from PDS", "did", userDID)
3535
36- if err := s.syncCollection(ctx, userDID, "at.glean.subscription", s.reconcileSubscription); err != nil {
36+ if err := s.syncCollection(ctx, userDID, CollectionSubscription, s.reconcileSubscription); err != nil {
3737 s.logger.Error("sync subscriptions failed", "error", err, "did", userDID)
3838 }
39- if err := s.syncCollection(ctx, userDID, "at.glean.like", s.reconcileLike); err != nil {
39+ if err := s.syncCollection(ctx, userDID, CollectionLike, s.reconcileLike); err != nil {
4040 s.logger.Error("sync likes failed", "error", err, "did", userDID)
4141 }
42- if err := s.syncCollection(ctx, userDID, "at.glean.annotation", s.reconcileAnnotation); err != nil {
42+ if err := s.syncCollection(ctx, userDID, CollectionAnnotation, s.reconcileAnnotation); err != nil {
4343 s.logger.Error("sync annotations failed", "error", err, "did", userDID)
4444 }
45+ if err := s.syncCollection(ctx, userDID, CollectionMarginNote, s.reconcileMarginNote); err != nil {
46+ s.logger.Error("sync margin notes failed", "error", err, "did", userDID)
47+ }
4548 if err := s.syncFollows(ctx, userDID); err != nil {
4649 s.logger.Error("sync follows failed", "error", err, "did", userDID)
4750 }
@@ -163,10 +166,49 @@ func (s *Sync) reconcileAnnotation(ctx context.Context, userDID, uri, cid string
163166 return s.db.CreateAnnotation(ctx, a)
164167 }
165168
169+func (s *Sync) reconcileMarginNote(ctx context.Context, userDID, uri, cid string, value json.RawMessage) error {
170+ var rec MarginNoteRecord
171+ if err := json.Unmarshal(value, &rec); err != nil {
172+ return err
173+ }
174+
175+ articleURL, quote, note, tags := rec.ToAnnotation()
176+ if articleURL == "" {
177+ return nil
178+ }
179+
180+ var existing []*db.Annotation
181+ existing, _ = s.db.ListAnnotations(ctx, "", articleURL, userDID, 100, 0)
182+ for _, a := range existing {
183+ if a.URI == uri {
184+ return nil
185+ }
186+ }
187+
188+ feedURL := ""
189+ if article, err := s.db.GetArticleByURL(ctx, articleURL); err == nil {
190+ feedURL = article.FeedURL
191+ }
192+
193+ t, _ := time.Parse(time.RFC3339, rec.CreatedAt)
194+ a := &db.Annotation{
195+ URI: uri,
196+ AuthorDID: userDID,
197+ FeedURL: feedURL,
198+ ArticleURL: articleURL,
199+ Quote: db.NullStr(quote),
200+ Note: db.NullStr(note),
201+ Tags: db.NullStrTags(tags),
202+ CreatedAt: db.NullTime(t),
203+ CID: db.NullStr(cid),
204+ }
205+ return s.db.CreateAnnotation(ctx, a)
206+}
207+
166208 func (s *Sync) syncFollows(ctx context.Context, userDID string) error {
167209 activeFollows := make(map[string]db.Follow)
168210
169- for _, collection := range []string{"app.bsky.graph.follow", "sh.tangled.graph.follow"} {
211+ for _, collection := range []string{CollectionBskyFollow, CollectionTangledFollow} {
170212 cursor := ""
171213 for {
172214 records, next, err := s.client.ListRecords(ctx, userDID, collection, 100, cursor)
@@ -33,15 +33,18 @@ func NewSync(database *db.DB, client *Client, logger *slog.Logger) *Sync {
33 func (s *Sync) Run(ctx context.Context, userDID string) error {33 func (s *Sync) Run(ctx context.Context, userDID string) error {
34 s.logger.Info("syncing from PDS", "did", userDID)34 s.logger.Info("syncing from PDS", "did", userDID)
35 35
36- if err := s.syncCollection(ctx, userDID, "at.glean.subscription", s.reconcileSubscription); err != nil {36+ if err := s.syncCollection(ctx, userDID, CollectionSubscription, s.reconcileSubscription); err != nil {
37 s.logger.Error("sync subscriptions failed", "error", err, "did", userDID)37 s.logger.Error("sync subscriptions failed", "error", err, "did", userDID)
38 }38 }
39- if err := s.syncCollection(ctx, userDID, "at.glean.like", s.reconcileLike); err != nil {39+ if err := s.syncCollection(ctx, userDID, CollectionLike, s.reconcileLike); err != nil {
40 s.logger.Error("sync likes failed", "error", err, "did", userDID)40 s.logger.Error("sync likes failed", "error", err, "did", userDID)
41 }41 }
42- if err := s.syncCollection(ctx, userDID, "at.glean.annotation", s.reconcileAnnotation); err != nil {42+ if err := s.syncCollection(ctx, userDID, CollectionAnnotation, s.reconcileAnnotation); err != nil {
43 s.logger.Error("sync annotations failed", "error", err, "did", userDID)43 s.logger.Error("sync annotations failed", "error", err, "did", userDID)
44 }44 }
45+ if err := s.syncCollection(ctx, userDID, CollectionMarginNote, s.reconcileMarginNote); err != nil {
46+ s.logger.Error("sync margin notes failed", "error", err, "did", userDID)
47+ }
45 if err := s.syncFollows(ctx, userDID); err != nil {48 if err := s.syncFollows(ctx, userDID); err != nil {
46 s.logger.Error("sync follows failed", "error", err, "did", userDID)49 s.logger.Error("sync follows failed", "error", err, "did", userDID)
47 }50 }
@@ -163,10 +166,49 @@ func (s *Sync) reconcileAnnotation(ctx context.Context, userDID, uri, cid string
163 return s.db.CreateAnnotation(ctx, a)166 return s.db.CreateAnnotation(ctx, a)
164 }167 }
165 168
169+func (s *Sync) reconcileMarginNote(ctx context.Context, userDID, uri, cid string, value json.RawMessage) error {
170+ var rec MarginNoteRecord
171+ if err := json.Unmarshal(value, &rec); err != nil {
172+ return err
173+ }
174+
175+ articleURL, quote, note, tags := rec.ToAnnotation()
176+ if articleURL == "" {
177+ return nil
178+ }
179+
180+ var existing []*db.Annotation
181+ existing, _ = s.db.ListAnnotations(ctx, "", articleURL, userDID, 100, 0)
182+ for _, a := range existing {
183+ if a.URI == uri {
184+ return nil
185+ }
186+ }
187+
188+ feedURL := ""
189+ if article, err := s.db.GetArticleByURL(ctx, articleURL); err == nil {
190+ feedURL = article.FeedURL
191+ }
192+
193+ t, _ := time.Parse(time.RFC3339, rec.CreatedAt)
194+ a := &db.Annotation{
195+ URI: uri,
196+ AuthorDID: userDID,
197+ FeedURL: feedURL,
198+ ArticleURL: articleURL,
199+ Quote: db.NullStr(quote),
200+ Note: db.NullStr(note),
201+ Tags: db.NullStrTags(tags),
202+ CreatedAt: db.NullTime(t),
203+ CID: db.NullStr(cid),
204+ }
205+ return s.db.CreateAnnotation(ctx, a)
206+}
207+
166 func (s *Sync) syncFollows(ctx context.Context, userDID string) error {208 func (s *Sync) syncFollows(ctx context.Context, userDID string) error {
167 activeFollows := make(map[string]db.Follow)209 activeFollows := make(map[string]db.Follow)
168 210
169- for _, collection := range []string{"app.bsky.graph.follow", "sh.tangled.graph.follow"} {211+ for _, collection := range []string{CollectionBskyFollow, CollectionTangledFollow} {
170 cursor := ""212 cursor := ""
171 for {213 for {
172 records, next, err := s.client.ListRecords(ctx, userDID, collection, 100, cursor)214 records, next, err := s.client.ListRecords(ctx, userDID, collection, 100, cursor)
modified internal/atproto/xrpc.go +6 -3
@@ -61,7 +61,7 @@ func (h *XRPCHandler) ListSubscriptions(w http.ResponseWriter, r *http.Request)
6161 }
6262
6363 sv := SubscriptionView{
64- URI: fmtATURI(repo, "at.glean.subscription", strconv.Itoa(id)),
64+ URI: fmtATURI(repo, CollectionSubscription, strconv.Itoa(id)),
6565 Value: SubscriptionRecord{
6666 CreatedAt: addedAt.String,
6767 FeedURL: feedURL,
@@ -300,8 +300,11 @@ func (h *XRPCHandler) GetTrending(w http.ResponseWriter, r *http.Request) {
300300 }
301301
302302 func (h *XRPCHandler) GetRecommendations(w http.ResponseWriter, r *http.Request) {
303- repo := chi.URLParam(r, "repo")
304- limit := parseIntParam(r, "limit", 10)
303+ repo := r.URL.Query().Get("repo")
304+ limit := parseIntParam(r, "limit", 20)
305+ if limit > 50 {
306+ limit = 50
307+ }
305308
306309 feedRows, err := h.db.QueryContext(r.Context(), `
307310 SELECT r.feed_url, f.title, f.site_url, f.description, f.subscriber_count, r.score
@@ -61,7 +61,7 @@ func (h *XRPCHandler) ListSubscriptions(w http.ResponseWriter, r *http.Request)
61 }61 }
62 62
63 sv := SubscriptionView{63 sv := SubscriptionView{
64- URI: fmtATURI(repo, "at.glean.subscription", strconv.Itoa(id)),64+ URI: fmtATURI(repo, CollectionSubscription, strconv.Itoa(id)),
65 Value: SubscriptionRecord{65 Value: SubscriptionRecord{
66 CreatedAt: addedAt.String,66 CreatedAt: addedAt.String,
67 FeedURL: feedURL,67 FeedURL: feedURL,
@@ -300,8 +300,11 @@ func (h *XRPCHandler) GetTrending(w http.ResponseWriter, r *http.Request) {
300 }300 }
301 301
302 func (h *XRPCHandler) GetRecommendations(w http.ResponseWriter, r *http.Request) {302 func (h *XRPCHandler) GetRecommendations(w http.ResponseWriter, r *http.Request) {
303- repo := chi.URLParam(r, "repo")303+ repo := r.URL.Query().Get("repo")
304- limit := parseIntParam(r, "limit", 10)304+ limit := parseIntParam(r, "limit", 20)
305+ if limit > 50 {
306+ limit = 50
307+ }
305 308
306 feedRows, err := h.db.QueryContext(r.Context(), `309 feedRows, err := h.db.QueryContext(r.Context(), `
307 SELECT r.feed_url, f.title, f.site_url, f.description, f.subscriber_count, r.score310 SELECT r.feed_url, f.title, f.site_url, f.description, f.subscriber_count, r.score
modified internal/db/article.go +14 -0
@@ -265,3 +265,17 @@ func (db *DB) UpdateArticleFullContent(ctx context.Context, id int64, fullConten
265265 `, fullContent, id)
266266 return err
267267 }
268+
269+func (db *DB) GetArticleByURL(ctx context.Context, url string) (*Article, error) {
270+ a := &Article{}
271+ err := db.QueryRowContext(ctx, `
272+ SELECT id, feed_url, guid, title, url, author, summary, content, full_content, published, updated, fetched_at
273+ FROM articles WHERE url = ?
274+ LIMIT 1
275+ `, url).Scan(&a.ID, &a.FeedURL, &a.GUID, &a.Title, &a.URL, &a.Author,
276+ &a.Summary, &a.Content, &a.FullContent, &a.Published, &a.Updated, &a.FetchedAt)
277+ if err != nil {
278+ return nil, err
279+ }
280+ return a, nil
281+}
@@ -265,3 +265,17 @@ func (db *DB) UpdateArticleFullContent(ctx context.Context, id int64, fullConten
265 `, fullContent, id)265 `, fullContent, id)
266 return err266 return err
267 }267 }
268+
269+func (db *DB) GetArticleByURL(ctx context.Context, url string) (*Article, error) {
270+ a := &Article{}
271+ err := db.QueryRowContext(ctx, `
272+ SELECT id, feed_url, guid, title, url, author, summary, content, full_content, published, updated, fetched_at
273+ FROM articles WHERE url = ?
274+ LIMIT 1
275+ `, url).Scan(&a.ID, &a.FeedURL, &a.GUID, &a.Title, &a.URL, &a.Author,
276+ &a.Summary, &a.Content, &a.FullContent, &a.Published, &a.Updated, &a.FetchedAt)
277+ if err != nil {
278+ return nil, err
279+ }
280+ return a, nil
281+}
modified internal/server/annotations_handler.go +1 -1
@@ -89,7 +89,7 @@ func (s *Server) handleCreateAnnotation(w http.ResponseWriter, r *http.Request)
8989 Note: a.Note.String,
9090 Rating: int(a.Rating.Int64),
9191 }
92- uri, cid, err := client.CreateRecord(r.Context(), user.DID, "at.glean.annotation", record)
92+ uri, cid, err := client.CreateRecord(r.Context(), user.DID, atproto.CollectionAnnotation, record)
9393 if err != nil {
9494 s.logger.Error("failed to write annotation to PDS", "error", err)
9595 http.Error(w, "failed to write annotation to PDS: "+err.Error(), http.StatusBadGateway)
@@ -89,7 +89,7 @@ func (s *Server) handleCreateAnnotation(w http.ResponseWriter, r *http.Request)
89 Note: a.Note.String,89 Note: a.Note.String,
90 Rating: int(a.Rating.Int64),90 Rating: int(a.Rating.Int64),
91 }91 }
92- uri, cid, err := client.CreateRecord(r.Context(), user.DID, "at.glean.annotation", record)92+ uri, cid, err := client.CreateRecord(r.Context(), user.DID, atproto.CollectionAnnotation, record)
93 if err != nil {93 if err != nil {
94 s.logger.Error("failed to write annotation to PDS", "error", err)94 s.logger.Error("failed to write annotation to PDS", "error", err)
95 http.Error(w, "failed to write annotation to PDS: "+err.Error(), http.StatusBadGateway)95 http.Error(w, "failed to write annotation to PDS: "+err.Error(), http.StatusBadGateway)
modified internal/server/articles_handler.go +1 -1
@@ -186,7 +186,7 @@ func (s *Server) handleLikeArticle(w http.ResponseWriter, r *http.Request) {
186186 }
187187
188188 if client := s.pdsClientForUser(r); client != nil {
189- uri, _, err := client.CreateRecord(r.Context(), user.DID, "at.glean.like", likeRecord)
189+ uri, _, err := client.CreateRecord(r.Context(), user.DID, atproto.CollectionLike, likeRecord)
190190 if err != nil {
191191 s.logger.Error("failed to write like to PDS", "error", err)
192192 http.Error(w, "failed to write like to PDS: "+err.Error(), http.StatusBadGateway)
@@ -186,7 +186,7 @@ func (s *Server) handleLikeArticle(w http.ResponseWriter, r *http.Request) {
186 }186 }
187 187
188 if client := s.pdsClientForUser(r); client != nil {188 if client := s.pdsClientForUser(r); client != nil {
189- uri, _, err := client.CreateRecord(r.Context(), user.DID, "at.glean.like", likeRecord)189+ uri, _, err := client.CreateRecord(r.Context(), user.DID, atproto.CollectionLike, likeRecord)
190 if err != nil {190 if err != nil {
191 s.logger.Error("failed to write like to PDS", "error", err)191 s.logger.Error("failed to write like to PDS", "error", err)
192 http.Error(w, "failed to write like to PDS: "+err.Error(), http.StatusBadGateway)192 http.Error(w, "failed to write like to PDS: "+err.Error(), http.StatusBadGateway)
modified internal/server/feeds_handler.go +2 -2
@@ -108,7 +108,7 @@ func (s *Server) handleAddFeed(w http.ResponseWriter, r *http.Request) {
108108 Title: feedTitle,
109109 Category: category,
110110 }
111- uri, cid, err := client.CreateRecord(r.Context(), user.DID, "at.glean.subscription", record)
111+ uri, cid, err := client.CreateRecord(r.Context(), user.DID, atproto.CollectionSubscription, record)
112112 if err != nil {
113113 s.logger.Error("failed to write subscription to PDS", "error", err)
114114 http.Error(w, "failed to write subscription to PDS: "+err.Error(), http.StatusBadGateway)
@@ -228,7 +228,7 @@ func (s *Server) handleOPMLUpload(w http.ResponseWriter, r *http.Request) {
228228 Title: fu.Title,
229229 Category: fu.Category,
230230 }
231- uri, cid, err := client.CreateRecord(r.Context(), user.DID, "at.glean.subscription", record)
231+ uri, cid, err := client.CreateRecord(r.Context(), user.DID, atproto.CollectionSubscription, record)
232232 if err != nil {
233233 s.logger.Error("failed to write subscription to PDS", "error", err, "url", fu.URL)
234234 continue
@@ -108,7 +108,7 @@ func (s *Server) handleAddFeed(w http.ResponseWriter, r *http.Request) {
108 Title: feedTitle,108 Title: feedTitle,
109 Category: category,109 Category: category,
110 }110 }
111- uri, cid, err := client.CreateRecord(r.Context(), user.DID, "at.glean.subscription", record)111+ uri, cid, err := client.CreateRecord(r.Context(), user.DID, atproto.CollectionSubscription, record)
112 if err != nil {112 if err != nil {
113 s.logger.Error("failed to write subscription to PDS", "error", err)113 s.logger.Error("failed to write subscription to PDS", "error", err)
114 http.Error(w, "failed to write subscription to PDS: "+err.Error(), http.StatusBadGateway)114 http.Error(w, "failed to write subscription to PDS: "+err.Error(), http.StatusBadGateway)
@@ -228,7 +228,7 @@ func (s *Server) handleOPMLUpload(w http.ResponseWriter, r *http.Request) {
228 Title: fu.Title,228 Title: fu.Title,
229 Category: fu.Category,229 Category: fu.Category,
230 }230 }
231- uri, cid, err := client.CreateRecord(r.Context(), user.DID, "at.glean.subscription", record)231+ uri, cid, err := client.CreateRecord(r.Context(), user.DID, atproto.CollectionSubscription, record)
232 if err != nil {232 if err != nil {
233 s.logger.Error("failed to write subscription to PDS", "error", err, "url", fu.URL)233 s.logger.Error("failed to write subscription to PDS", "error", err, "url", fu.URL)
234 continue234 continue
added lexicons/app/bsky/graph/follow.json +32 -0
new file mode 100644
@@ -0,0 +1,32 @@
1+{
2+ "defs": {
3+ "main": {
4+ "description": "Record declaring a social 'follow' relationship of another account. Duplicate follows will be ignored by the AppView.",
5+ "key": "tid",
6+ "record": {
7+ "properties": {
8+ "createdAt": {
9+ "format": "datetime",
10+ "type": "string"
11+ },
12+ "subject": {
13+ "format": "did",
14+ "type": "string"
15+ },
16+ "via": {
17+ "ref": "com.atproto.repo.strongRef",
18+ "type": "ref"
19+ }
20+ },
21+ "required": [
22+ "subject",
23+ "createdAt"
24+ ],
25+ "type": "object"
26+ },
27+ "type": "record"
28+ }
29+ },
30+ "id": "app.bsky.graph.follow",
31+ "lexicon": 1
32+}
new file mode 100644
@@ -0,0 +1,32 @@
1+{
2+ "defs": {
3+ "main": {
4+ "description": "Record declaring a social 'follow' relationship of another account. Duplicate follows will be ignored by the AppView.",
5+ "key": "tid",
6+ "record": {
7+ "properties": {
8+ "createdAt": {
9+ "format": "datetime",
10+ "type": "string"
11+ },
12+ "subject": {
13+ "format": "did",
14+ "type": "string"
15+ },
16+ "via": {
17+ "ref": "com.atproto.repo.strongRef",
18+ "type": "ref"
19+ }
20+ },
21+ "required": [
22+ "subject",
23+ "createdAt"
24+ ],
25+ "type": "object"
26+ },
27+ "type": "record"
28+ }
29+ },
30+ "id": "app.bsky.graph.follow",
31+ "lexicon": 1
32+}
added lexicons/at/margin/note.json +243 -0
new file mode 100644
@@ -0,0 +1,243 @@
1+{
2+ "defs": {
3+ "body": {
4+ "description": "Annotation body - the content of the annotation",
5+ "properties": {
6+ "format": {
7+ "default": "text/plain",
8+ "description": "MIME type of the body content",
9+ "type": "string"
10+ },
11+ "uri": {
12+ "description": "Reference to external body content",
13+ "format": "uri",
14+ "type": "string"
15+ },
16+ "value": {
17+ "description": "Text content of the annotation. For bookmarks, this is the description.",
18+ "maxGraphemes": 3000,
19+ "maxLength": 10000,
20+ "type": "string"
21+ }
22+ },
23+ "type": "object"
24+ },
25+ "generator": {
26+ "description": "The client/agent that created this record",
27+ "properties": {
28+ "homepage": {
29+ "format": "uri",
30+ "type": "string"
31+ },
32+ "id": {
33+ "format": "uri",
34+ "type": "string"
35+ },
36+ "name": {
37+ "type": "string"
38+ }
39+ },
40+ "type": "object"
41+ },
42+ "main": {
43+ "description": "A W3C-compliant web annotation stored on the AT Protocol",
44+ "key": "tid",
45+ "record": {
46+ "properties": {
47+ "body": {
48+ "description": "The annotation content (text or reference). For bookmarks, use body.value for the description.",
49+ "ref": "#body",
50+ "type": "ref"
51+ },
52+ "color": {
53+ "description": "Highlight color tint",
54+ "maxLength": 20,
55+ "type": "string"
56+ },
57+ "createdAt": {
58+ "format": "datetime",
59+ "type": "string"
60+ },
61+ "facets": {
62+ "description": "Rich text facets (e.g. mentions, links)",
63+ "items": {
64+ "ref": "app.bsky.richtext.facet",
65+ "type": "ref"
66+ },
67+ "type": "array"
68+ },
69+ "generator": {
70+ "description": "The client/agent that created this record",
71+ "ref": "#generator",
72+ "type": "ref"
73+ },
74+ "labels": {
75+ "description": "Self-applied content labels for this annotation",
76+ "ref": "com.atproto.label.defs#selfLabels",
77+ "type": "ref"
78+ },
79+ "modifiedAt": {
80+ "description": "When this record was last modified",
81+ "format": "datetime",
82+ "type": "string"
83+ },
84+ "motivation": {
85+ "description": "W3C motivation for the annotation",
86+ "knownValues": [
87+ "commenting",
88+ "highlighting",
89+ "bookmarking",
90+ "tagging",
91+ "describing",
92+ "linking",
93+ "replying",
94+ "editing",
95+ "questioning",
96+ "assessing"
97+ ],
98+ "type": "string"
99+ },
100+ "rights": {
101+ "description": "License URI (e.g., https://creativecommons.org/licenses/by/4.0/)",
102+ "format": "uri",
103+ "type": "string"
104+ },
105+ "tags": {
106+ "description": "Tags for categorization",
107+ "items": {
108+ "maxGraphemes": 32,
109+ "maxLength": 64,
110+ "type": "string"
111+ },
112+ "maxLength": 10,
113+ "type": "array"
114+ },
115+ "target": {
116+ "description": "The resource being annotated with optional selector",
117+ "ref": "#target",
118+ "type": "ref"
119+ }
120+ },
121+ "required": [
122+ "motivation",
123+ "target",
124+ "createdAt"
125+ ],
126+ "type": "object"
127+ },
128+ "type": "record"
129+ },
130+ "selector": {
131+ "description": "W3C Web Annotation Selector. The 'type' field discriminates the selector kind using W3C type names (e.g. TextQuoteSelector). This follows W3C conventions, not ATProto union $type.",
132+ "properties": {
133+ "conformsTo": {
134+ "description": "FragmentSelector: URI of the specification the fragment conforms to",
135+ "format": "uri",
136+ "type": "string"
137+ },
138+ "end": {
139+ "description": "TextPositionSelector: end character offset (exclusive)",
140+ "minimum": 0,
141+ "type": "integer"
142+ },
143+ "exact": {
144+ "description": "TextQuoteSelector: the exact text being selected",
145+ "maxGraphemes": 1500,
146+ "maxLength": 5000,
147+ "type": "string"
148+ },
149+ "prefix": {
150+ "description": "TextQuoteSelector: text immediately before the selection, for disambiguation",
151+ "maxGraphemes": 150,
152+ "maxLength": 500,
153+ "type": "string"
154+ },
155+ "start": {
156+ "description": "TextPositionSelector: start character offset (inclusive)",
157+ "minimum": 0,
158+ "type": "integer"
159+ },
160+ "suffix": {
161+ "description": "TextQuoteSelector: text immediately after the selection, for disambiguation",
162+ "maxGraphemes": 150,
163+ "maxLength": 500,
164+ "type": "string"
165+ },
166+ "type": {
167+ "description": "W3C selector type identifier",
168+ "knownValues": [
169+ "TextQuoteSelector",
170+ "TextPositionSelector",
171+ "CssSelector",
172+ "XPathSelector",
173+ "FragmentSelector",
174+ "RangeSelector"
175+ ],
176+ "type": "string"
177+ },
178+ "value": {
179+ "description": "CssSelector/XPathSelector/FragmentSelector: the selector expression or fragment value",
180+ "maxLength": 2000,
181+ "type": "string"
182+ }
183+ },
184+ "required": [
185+ "type"
186+ ],
187+ "type": "object"
188+ },
189+ "target": {
190+ "description": "W3C SpecificResource - the target with optional selector",
191+ "properties": {
192+ "selector": {
193+ "description": "W3C Selector to identify the annotated segment. Uses W3C 'type' field (not ATProto $type) per the Web Annotation Data Model.",
194+ "ref": "#selector",
195+ "type": "ref"
196+ },
197+ "source": {
198+ "description": "The URL being annotated",
199+ "format": "uri",
200+ "type": "string"
201+ },
202+ "sourceHash": {
203+ "description": "SHA256 hash of normalized URL for indexing",
204+ "type": "string"
205+ },
206+ "state": {
207+ "description": "State of the resource at annotation time",
208+ "ref": "#timeState",
209+ "type": "ref"
210+ },
211+ "title": {
212+ "description": "Page title at time of annotation",
213+ "maxLength": 500,
214+ "type": "string"
215+ }
216+ },
217+ "required": [
218+ "source"
219+ ],
220+ "type": "object"
221+ },
222+ "timeState": {
223+ "description": "W3C TimeState - record when content was captured",
224+ "properties": {
225+ "cached": {
226+ "description": "URL to cached/archived version",
227+ "format": "uri",
228+ "type": "string"
229+ },
230+ "sourceDate": {
231+ "description": "When the source was accessed",
232+ "format": "datetime",
233+ "type": "string"
234+ }
235+ },
236+ "type": "object"
237+ }
238+ },
239+ "description": "W3C Web Annotation Data Model compliant unified note record for ATProto",
240+ "id": "at.margin.note",
241+ "lexicon": 1,
242+ "revision": 3
243+}
new file mode 100644
@@ -0,0 +1,243 @@
1+{
2+ "defs": {
3+ "body": {
4+ "description": "Annotation body - the content of the annotation",
5+ "properties": {
6+ "format": {
7+ "default": "text/plain",
8+ "description": "MIME type of the body content",
9+ "type": "string"
10+ },
11+ "uri": {
12+ "description": "Reference to external body content",
13+ "format": "uri",
14+ "type": "string"
15+ },
16+ "value": {
17+ "description": "Text content of the annotation. For bookmarks, this is the description.",
18+ "maxGraphemes": 3000,
19+ "maxLength": 10000,
20+ "type": "string"
21+ }
22+ },
23+ "type": "object"
24+ },
25+ "generator": {
26+ "description": "The client/agent that created this record",
27+ "properties": {
28+ "homepage": {
29+ "format": "uri",
30+ "type": "string"
31+ },
32+ "id": {
33+ "format": "uri",
34+ "type": "string"
35+ },
36+ "name": {
37+ "type": "string"
38+ }
39+ },
40+ "type": "object"
41+ },
42+ "main": {
43+ "description": "A W3C-compliant web annotation stored on the AT Protocol",
44+ "key": "tid",
45+ "record": {
46+ "properties": {
47+ "body": {
48+ "description": "The annotation content (text or reference). For bookmarks, use body.value for the description.",
49+ "ref": "#body",
50+ "type": "ref"
51+ },
52+ "color": {
53+ "description": "Highlight color tint",
54+ "maxLength": 20,
55+ "type": "string"
56+ },
57+ "createdAt": {
58+ "format": "datetime",
59+ "type": "string"
60+ },
61+ "facets": {
62+ "description": "Rich text facets (e.g. mentions, links)",
63+ "items": {
64+ "ref": "app.bsky.richtext.facet",
65+ "type": "ref"
66+ },
67+ "type": "array"
68+ },
69+ "generator": {
70+ "description": "The client/agent that created this record",
71+ "ref": "#generator",
72+ "type": "ref"
73+ },
74+ "labels": {
75+ "description": "Self-applied content labels for this annotation",
76+ "ref": "com.atproto.label.defs#selfLabels",
77+ "type": "ref"
78+ },
79+ "modifiedAt": {
80+ "description": "When this record was last modified",
81+ "format": "datetime",
82+ "type": "string"
83+ },
84+ "motivation": {
85+ "description": "W3C motivation for the annotation",
86+ "knownValues": [
87+ "commenting",
88+ "highlighting",
89+ "bookmarking",
90+ "tagging",
91+ "describing",
92+ "linking",
93+ "replying",
94+ "editing",
95+ "questioning",
96+ "assessing"
97+ ],
98+ "type": "string"
99+ },
100+ "rights": {
101+ "description": "License URI (e.g., https://creativecommons.org/licenses/by/4.0/)",
102+ "format": "uri",
103+ "type": "string"
104+ },
105+ "tags": {
106+ "description": "Tags for categorization",
107+ "items": {
108+ "maxGraphemes": 32,
109+ "maxLength": 64,
110+ "type": "string"
111+ },
112+ "maxLength": 10,
113+ "type": "array"
114+ },
115+ "target": {
116+ "description": "The resource being annotated with optional selector",
117+ "ref": "#target",
118+ "type": "ref"
119+ }
120+ },
121+ "required": [
122+ "motivation",
123+ "target",
124+ "createdAt"
125+ ],
126+ "type": "object"
127+ },
128+ "type": "record"
129+ },
130+ "selector": {
131+ "description": "W3C Web Annotation Selector. The 'type' field discriminates the selector kind using W3C type names (e.g. TextQuoteSelector). This follows W3C conventions, not ATProto union $type.",
132+ "properties": {
133+ "conformsTo": {
134+ "description": "FragmentSelector: URI of the specification the fragment conforms to",
135+ "format": "uri",
136+ "type": "string"
137+ },
138+ "end": {
139+ "description": "TextPositionSelector: end character offset (exclusive)",
140+ "minimum": 0,
141+ "type": "integer"
142+ },
143+ "exact": {
144+ "description": "TextQuoteSelector: the exact text being selected",
145+ "maxGraphemes": 1500,
146+ "maxLength": 5000,
147+ "type": "string"
148+ },
149+ "prefix": {
150+ "description": "TextQuoteSelector: text immediately before the selection, for disambiguation",
151+ "maxGraphemes": 150,
152+ "maxLength": 500,
153+ "type": "string"
154+ },
155+ "start": {
156+ "description": "TextPositionSelector: start character offset (inclusive)",
157+ "minimum": 0,
158+ "type": "integer"
159+ },
160+ "suffix": {
161+ "description": "TextQuoteSelector: text immediately after the selection, for disambiguation",
162+ "maxGraphemes": 150,
163+ "maxLength": 500,
164+ "type": "string"
165+ },
166+ "type": {
167+ "description": "W3C selector type identifier",
168+ "knownValues": [
169+ "TextQuoteSelector",
170+ "TextPositionSelector",
171+ "CssSelector",
172+ "XPathSelector",
173+ "FragmentSelector",
174+ "RangeSelector"
175+ ],
176+ "type": "string"
177+ },
178+ "value": {
179+ "description": "CssSelector/XPathSelector/FragmentSelector: the selector expression or fragment value",
180+ "maxLength": 2000,
181+ "type": "string"
182+ }
183+ },
184+ "required": [
185+ "type"
186+ ],
187+ "type": "object"
188+ },
189+ "target": {
190+ "description": "W3C SpecificResource - the target with optional selector",
191+ "properties": {
192+ "selector": {
193+ "description": "W3C Selector to identify the annotated segment. Uses W3C 'type' field (not ATProto $type) per the Web Annotation Data Model.",
194+ "ref": "#selector",
195+ "type": "ref"
196+ },
197+ "source": {
198+ "description": "The URL being annotated",
199+ "format": "uri",
200+ "type": "string"
201+ },
202+ "sourceHash": {
203+ "description": "SHA256 hash of normalized URL for indexing",
204+ "type": "string"
205+ },
206+ "state": {
207+ "description": "State of the resource at annotation time",
208+ "ref": "#timeState",
209+ "type": "ref"
210+ },
211+ "title": {
212+ "description": "Page title at time of annotation",
213+ "maxLength": 500,
214+ "type": "string"
215+ }
216+ },
217+ "required": [
218+ "source"
219+ ],
220+ "type": "object"
221+ },
222+ "timeState": {
223+ "description": "W3C TimeState - record when content was captured",
224+ "properties": {
225+ "cached": {
226+ "description": "URL to cached/archived version",
227+ "format": "uri",
228+ "type": "string"
229+ },
230+ "sourceDate": {
231+ "description": "When the source was accessed",
232+ "format": "datetime",
233+ "type": "string"
234+ }
235+ },
236+ "type": "object"
237+ }
238+ },
239+ "description": "W3C Web Annotation Data Model compliant unified note record for ATProto",
240+ "id": "at.margin.note",
241+ "lexicon": 1,
242+ "revision": 3
243+}