rickub
nandi/gleanpublic⑂ Fork 0
⑂ 29d5292
Branches
claude/modal-container-min-76527a main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/glean.git
git clone ssh://git@rickub.com/nandi/glean.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Add margin note synchronization (bidirectional, was previously one way)Unverified

Julien Robert committed 2026-05-12T10:26:12+02:00 Browse files
29d5292 parent: ad10774
modified docs/specs.md +64 -42
@@ -124,7 +124,9 @@ A user likes an article. The liked feed surfaces popular articles and feeds into
124124
125125 Glean also indexes records from the `at.margin.note` lexicon (owned by [margin.at](https://margin.at)). These are displayed in the UI as if they were `at.glean.annotation` records — margin notes appear alongside glean annotations on article detail pages.
126126
127-The mapping from margin note to glean annotation:
127+#### Ingestion (margin.at → Glean)
128+
129+Margin notes are indexed from both Jetstream and PDS sync, same as glean records. The mapping from margin note to glean annotation:
128130
129131 | Margin note field | Annotation field | Notes |
130132 | ------------------------------ | ---------------- | --------------------------------------------------------------- |
@@ -135,7 +137,26 @@ The mapping from margin note to glean annotation:
135137 | `createdAt` | `created_at` | Direct mapping |
136138 | _(looked up from articles DB)_ | `feed_url` | Resolved by matching `target.source` against known article URLs |
137139
138-When no matching article exists in the local DB, the annotation is stored with an empty `feed_url`. Margin notes are indexed from both Jetstream and PDS sync, same as glean records.
140+When no matching article exists in the local DB, the annotation is stored with an empty `feed_url`.
141+
142+#### Mirroring (Glean → margin.at)
143+
144+When a user creates an annotation through Glean, two records are written to the user's PDS:
145+
146+1. **`at.glean.annotation`** — the primary record (canonical URI for the annotation)
147+2. **`at.margin.note`** — mirror for interoperability with margin.at clients
148+
149+The conversion from glean annotation to margin note uses `NewMarginNoteRecord`:
150+
151+| Glean annotation field | Margin note field | Notes |
152+| ---------------------- | ----------------------- | -------------------------------- |
153+| `articleUrl` | `target.source` | Direct mapping |
154+| `quote` | `target.selector.exact` | Wrapped in a `TextQuoteSelector` |
155+| `note` | `body.value` | `body.format` = `"text/plain"` |
156+| `tags` | `tags` | Direct mapping |
157+| _(constant)_ | `motivation` | Always `"commenting"` |
158+
159+The margin note mirror is fire-and-forget — if it fails, the glean annotation still succeeds. The Jetstream consumer will pick up the margin note create event, but `handleMarginNote` skips it via `AnnotationExistsByContent` (checks `author_did` + `article_url` + `quote` + `note`) to prevent duplicate annotations.
139160
140161 ### 3.5 `app.skyreader.feed.subscription` (External)
141162
@@ -433,11 +454,12 @@ Glean runs as a single Go binary that fills three roles: **AppView** (indexing `
433454 │ └─────────────────┘ │ └──────────────────┘
434455 └──────────────────────┘
435456
436- AppView responsibilities:
437- • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like, at.margin.note, app.skyreader.feed.subscription
438- • Index records into SQLite
439- • Convert at.margin.note records to annotations (displayed alongside glean.at annotations)
440- • Import app.skyreader.feed.subscription records as Glean subscriptions
457+ AppView responsibilities:
458+ • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like, at.margin.note, app.skyreader.feed.subscription
459+ • Index records into SQLite
460+ • Convert at.margin.note records to annotations (displayed alongside glean.at annotations), skip if duplicate glean annotation exists
461+ • Mirror glean annotations as at.margin.note records on user PDS for interoperability
462+ • Import app.skyreader.feed.subscription records as Glean subscriptions
441463 • Serve XRPC query endpoints (at.glean.listSubscriptions, etc.)
442464 • Host the web UI at glean.at
443465 • Write to user PDS on behalf of user (when user acts through UI)
@@ -904,42 +926,42 @@ The server renders HTML fragments that htmx swaps into the page. No JSON API nee
904926
905927 ### 8.1 Pages
906928
907-| Route | Method | Description |
908-| ------------------------------ | ------ | ------------------------------------------------------------------- |
909-| `/` | GET | Landing page / auth redirect |
929+| Route | Method | Description |
930+| ------------------------------ | ------ | ---------------------------------------------------------------------- |
931+| `/` | GET | Landing page / auth redirect |
910932 | `/dashboard` | GET | Main dashboard: article recs, unread articles, trending, people, feeds |
911-| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |
912-| `/feeds/list` | GET | Feed list fragment (htmx partial) |
913-| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions (redirects to /feeds) |
914-| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |
915-| `/feeds/add` | POST | Add a single feed URL |
916-| `/feeds/remove` | DELETE | Remove a feed |
917-| `/feeds/refresh` | POST | Refresh all subscribed feeds |
918-| `/feeds/retry` | POST | Retry a failed feed |
919-| `/feeds/clear` | POST | Clear all subscriptions |
920-| `/feeds/dismiss` | POST | Dismiss a feed recommendation |
921-| `/articles` | GET | Read articles (paginated, filterable by feed) |
922-| `/articles/new-count` | GET | Get count of new articles (for badge updates) |
923-| `/articles/{id}` | GET | Article detail view |
924-| `/articles/{id}/read` | POST | Mark article as read |
925-| `/articles/{id}/unread` | POST | Mark article as unread |
926-| `/articles/{id}/like` | POST | Like an article |
927-| `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL |
928-| `/articles/mark-all-read` | POST | Mark all articles as read |
929-| `/articles/dismiss` | POST | Dismiss an article recommendation |
930-| `/trending` | GET | Community feed: articles ranked by likes (public) |
931-| `/library` | GET | Liked articles and annotations |
932-| `/library/create` | POST | Create annotation on an article |
933-| `/library/{id}/delete` | POST | Delete an annotation |
934-| `/stats` | GET | Application metrics and performance data (Prometheus, public) |
935-| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |
936-| `/settings/languages` | POST | Save preferred recommendation languages (htmx, requires auth) |
937-| `/auth/login` | GET | Login page |
938-| `/auth/register` | GET | Register with Eurosky (OAuth flow with hardcoded PDS) |
939-| `/auth/resolve` | GET | Resolve handle to DID |
940-| `/auth/start` | POST | Start OAuth authorization flow |
941-| `/auth/callback` | GET | OAuth callback |
942-| `/terms` | GET | Terms of service |
933+| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |
934+| `/feeds/list` | GET | Feed list fragment (htmx partial) |
935+| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions (redirects to /feeds) |
936+| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |
937+| `/feeds/add` | POST | Add a single feed URL |
938+| `/feeds/remove` | DELETE | Remove a feed |
939+| `/feeds/refresh` | POST | Refresh all subscribed feeds |
940+| `/feeds/retry` | POST | Retry a failed feed |
941+| `/feeds/clear` | POST | Clear all subscriptions |
942+| `/feeds/dismiss` | POST | Dismiss a feed recommendation |
943+| `/articles` | GET | Read articles (paginated, filterable by feed) |
944+| `/articles/new-count` | GET | Get count of new articles (for badge updates) |
945+| `/articles/{id}` | GET | Article detail view |
946+| `/articles/{id}/read` | POST | Mark article as read |
947+| `/articles/{id}/unread` | POST | Mark article as unread |
948+| `/articles/{id}/like` | POST | Like an article |
949+| `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL |
950+| `/articles/mark-all-read` | POST | Mark all articles as read |
951+| `/articles/dismiss` | POST | Dismiss an article recommendation |
952+| `/trending` | GET | Community feed: articles ranked by likes (public) |
953+| `/library` | GET | Liked articles and annotations |
954+| `/library/create` | POST | Create annotation on an article |
955+| `/library/{id}/delete` | POST | Delete an annotation |
956+| `/stats` | GET | Application metrics and performance data (Prometheus, public) |
957+| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |
958+| `/settings/languages` | POST | Save preferred recommendation languages (htmx, requires auth) |
959+| `/auth/login` | GET | Login page |
960+| `/auth/register` | GET | Register with Eurosky (OAuth flow with hardcoded PDS) |
961+| `/auth/resolve` | GET | Resolve handle to DID |
962+| `/auth/start` | POST | Start OAuth authorization flow |
963+| `/auth/callback` | GET | OAuth callback |
964+| `/terms` | GET | Terms of service |
943965
944966 ### 8.2 htmx Patterns
945967
@@ -124,7 +124,9 @@ A user likes an article. The liked feed surfaces popular articles and feeds into
124 124
125 Glean also indexes records from the `at.margin.note` lexicon (owned by [margin.at](https://margin.at)). These are displayed in the UI as if they were `at.glean.annotation` records — margin notes appear alongside glean annotations on article detail pages.125 Glean also indexes records from the `at.margin.note` lexicon (owned by [margin.at](https://margin.at)). These are displayed in the UI as if they were `at.glean.annotation` records — margin notes appear alongside glean annotations on article detail pages.
126 126
127-The mapping from margin note to glean annotation:127+#### Ingestion (margin.at → Glean)
128+
129+Margin notes are indexed from both Jetstream and PDS sync, same as glean records. The mapping from margin note to glean annotation:
128 130
129 | Margin note field | Annotation field | Notes |131 | Margin note field | Annotation field | Notes |
130 | ------------------------------ | ---------------- | --------------------------------------------------------------- |132 | ------------------------------ | ---------------- | --------------------------------------------------------------- |
@@ -135,7 +137,26 @@ The mapping from margin note to glean annotation:
135 | `createdAt` | `created_at` | Direct mapping |137 | `createdAt` | `created_at` | Direct mapping |
136 | _(looked up from articles DB)_ | `feed_url` | Resolved by matching `target.source` against known article URLs |138 | _(looked up from articles DB)_ | `feed_url` | Resolved by matching `target.source` against known article URLs |
137 139
138-When no matching article exists in the local DB, the annotation is stored with an empty `feed_url`. Margin notes are indexed from both Jetstream and PDS sync, same as glean records.140+When no matching article exists in the local DB, the annotation is stored with an empty `feed_url`.
141+
142+#### Mirroring (Glean → margin.at)
143+
144+When a user creates an annotation through Glean, two records are written to the user's PDS:
145+
146+1. **`at.glean.annotation`** — the primary record (canonical URI for the annotation)
147+2. **`at.margin.note`** — mirror for interoperability with margin.at clients
148+
149+The conversion from glean annotation to margin note uses `NewMarginNoteRecord`:
150+
151+| Glean annotation field | Margin note field | Notes |
152+| ---------------------- | ----------------------- | -------------------------------- |
153+| `articleUrl` | `target.source` | Direct mapping |
154+| `quote` | `target.selector.exact` | Wrapped in a `TextQuoteSelector` |
155+| `note` | `body.value` | `body.format` = `"text/plain"` |
156+| `tags` | `tags` | Direct mapping |
157+| _(constant)_ | `motivation` | Always `"commenting"` |
158+
159+The margin note mirror is fire-and-forget — if it fails, the glean annotation still succeeds. The Jetstream consumer will pick up the margin note create event, but `handleMarginNote` skips it via `AnnotationExistsByContent` (checks `author_did` + `article_url` + `quote` + `note`) to prevent duplicate annotations.
139 160
140 ### 3.5 `app.skyreader.feed.subscription` (External)161 ### 3.5 `app.skyreader.feed.subscription` (External)
141 162
@@ -433,11 +454,12 @@ Glean runs as a single Go binary that fills three roles: **AppView** (indexing `
433 │ └─────────────────┘ │ └──────────────────┘454 │ └─────────────────┘ │ └──────────────────┘
434 └──────────────────────┘455 └──────────────────────┘
435 456
436- AppView responsibilities:457+ AppView responsibilities:
437- • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like, at.margin.note, app.skyreader.feed.subscription458+ • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like, at.margin.note, app.skyreader.feed.subscription
438- • Index records into SQLite459+ • Index records into SQLite
439- • Convert at.margin.note records to annotations (displayed alongside glean.at annotations)460+ • Convert at.margin.note records to annotations (displayed alongside glean.at annotations), skip if duplicate glean annotation exists
440- • Import app.skyreader.feed.subscription records as Glean subscriptions461+ • Mirror glean annotations as at.margin.note records on user PDS for interoperability
462+ • Import app.skyreader.feed.subscription records as Glean subscriptions
441 • Serve XRPC query endpoints (at.glean.listSubscriptions, etc.)463 • Serve XRPC query endpoints (at.glean.listSubscriptions, etc.)
442 • Host the web UI at glean.at464 • Host the web UI at glean.at
443 • Write to user PDS on behalf of user (when user acts through UI)465 • Write to user PDS on behalf of user (when user acts through UI)
@@ -904,42 +926,42 @@ The server renders HTML fragments that htmx swaps into the page. No JSON API nee
904 926
905 ### 8.1 Pages927 ### 8.1 Pages
906 928
907-| Route | Method | Description |929+| Route | Method | Description |
908-| ------------------------------ | ------ | ------------------------------------------------------------------- |930+| ------------------------------ | ------ | ---------------------------------------------------------------------- |
909-| `/` | GET | Landing page / auth redirect |931+| `/` | GET | Landing page / auth redirect |
910 | `/dashboard` | GET | Main dashboard: article recs, unread articles, trending, people, feeds |932 | `/dashboard` | GET | Main dashboard: article recs, unread articles, trending, people, feeds |
911-| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |933+| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |
912-| `/feeds/list` | GET | Feed list fragment (htmx partial) |934+| `/feeds/list` | GET | Feed list fragment (htmx partial) |
913-| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions (redirects to /feeds) |935+| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions (redirects to /feeds) |
914-| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |936+| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |
915-| `/feeds/add` | POST | Add a single feed URL |937+| `/feeds/add` | POST | Add a single feed URL |
916-| `/feeds/remove` | DELETE | Remove a feed |938+| `/feeds/remove` | DELETE | Remove a feed |
917-| `/feeds/refresh` | POST | Refresh all subscribed feeds |939+| `/feeds/refresh` | POST | Refresh all subscribed feeds |
918-| `/feeds/retry` | POST | Retry a failed feed |940+| `/feeds/retry` | POST | Retry a failed feed |
919-| `/feeds/clear` | POST | Clear all subscriptions |941+| `/feeds/clear` | POST | Clear all subscriptions |
920-| `/feeds/dismiss` | POST | Dismiss a feed recommendation |942+| `/feeds/dismiss` | POST | Dismiss a feed recommendation |
921-| `/articles` | GET | Read articles (paginated, filterable by feed) |943+| `/articles` | GET | Read articles (paginated, filterable by feed) |
922-| `/articles/new-count` | GET | Get count of new articles (for badge updates) |944+| `/articles/new-count` | GET | Get count of new articles (for badge updates) |
923-| `/articles/{id}` | GET | Article detail view |945+| `/articles/{id}` | GET | Article detail view |
924-| `/articles/{id}/read` | POST | Mark article as read |946+| `/articles/{id}/read` | POST | Mark article as read |
925-| `/articles/{id}/unread` | POST | Mark article as unread |947+| `/articles/{id}/unread` | POST | Mark article as unread |
926-| `/articles/{id}/like` | POST | Like an article |948+| `/articles/{id}/like` | POST | Like an article |
927-| `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL |949+| `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL |
928-| `/articles/mark-all-read` | POST | Mark all articles as read |950+| `/articles/mark-all-read` | POST | Mark all articles as read |
929-| `/articles/dismiss` | POST | Dismiss an article recommendation |951+| `/articles/dismiss` | POST | Dismiss an article recommendation |
930-| `/trending` | GET | Community feed: articles ranked by likes (public) |952+| `/trending` | GET | Community feed: articles ranked by likes (public) |
931-| `/library` | GET | Liked articles and annotations |953+| `/library` | GET | Liked articles and annotations |
932-| `/library/create` | POST | Create annotation on an article |954+| `/library/create` | POST | Create annotation on an article |
933-| `/library/{id}/delete` | POST | Delete an annotation |955+| `/library/{id}/delete` | POST | Delete an annotation |
934-| `/stats` | GET | Application metrics and performance data (Prometheus, public) |956+| `/stats` | GET | Application metrics and performance data (Prometheus, public) |
935-| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |957+| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |
936-| `/settings/languages` | POST | Save preferred recommendation languages (htmx, requires auth) |958+| `/settings/languages` | POST | Save preferred recommendation languages (htmx, requires auth) |
937-| `/auth/login` | GET | Login page |959+| `/auth/login` | GET | Login page |
938-| `/auth/register` | GET | Register with Eurosky (OAuth flow with hardcoded PDS) |960+| `/auth/register` | GET | Register with Eurosky (OAuth flow with hardcoded PDS) |
939-| `/auth/resolve` | GET | Resolve handle to DID |961+| `/auth/resolve` | GET | Resolve handle to DID |
940-| `/auth/start` | POST | Start OAuth authorization flow |962+| `/auth/start` | POST | Start OAuth authorization flow |
941-| `/auth/callback` | GET | OAuth callback |963+| `/auth/callback` | GET | OAuth callback |
942-| `/terms` | GET | Terms of service |964+| `/terms` | GET | Terms of service |
943 965
944 ### 8.2 htmx Patterns966 ### 8.2 htmx Patterns
945 967
modified internal/atproto/lexicon_external.go +28 -0
@@ -78,6 +78,34 @@ type MarginNoteTimeState struct {
7878 SourceDate string `json:"sourceDate,omitempty"`
7979 }
8080
81+func NewMarginNoteRecord(articleURL, quote, note string, tags []string, createdAt string) MarginNoteRecord {
82+ rec := MarginNoteRecord{
83+ Body: nil,
84+ CreatedAt: createdAt,
85+ Motivation: "commenting",
86+ Tags: tags,
87+ Target: MarginNoteTarget{
88+ Source: articleURL,
89+ },
90+ }
91+
92+ if quote != "" {
93+ rec.Target.Selector = &MarginNoteSelector{
94+ Type: "TextQuoteSelector",
95+ Exact: quote,
96+ }
97+ }
98+
99+ if note != "" {
100+ rec.Body = &MarginNoteBody{
101+ Format: "text/plain",
102+ Value: note,
103+ }
104+ }
105+
106+ return rec
107+}
108+
81109 func (r MarginNoteRecord) ToAnnotation() (articleURL, quote, note string, tags []string) {
82110 articleURL = r.Target.Source
83111 if r.Body != nil {
@@ -78,6 +78,34 @@ type MarginNoteTimeState struct {
78 SourceDate string `json:"sourceDate,omitempty"`78 SourceDate string `json:"sourceDate,omitempty"`
79 }79 }
80 80
81+func NewMarginNoteRecord(articleURL, quote, note string, tags []string, createdAt string) MarginNoteRecord {
82+ rec := MarginNoteRecord{
83+ Body: nil,
84+ CreatedAt: createdAt,
85+ Motivation: "commenting",
86+ Tags: tags,
87+ Target: MarginNoteTarget{
88+ Source: articleURL,
89+ },
90+ }
91+
92+ if quote != "" {
93+ rec.Target.Selector = &MarginNoteSelector{
94+ Type: "TextQuoteSelector",
95+ Exact: quote,
96+ }
97+ }
98+
99+ if note != "" {
100+ rec.Body = &MarginNoteBody{
101+ Format: "text/plain",
102+ Value: note,
103+ }
104+ }
105+
106+ return rec
107+}
108+
81 func (r MarginNoteRecord) ToAnnotation() (articleURL, quote, note string, tags []string) {109 func (r MarginNoteRecord) ToAnnotation() (articleURL, quote, note string, tags []string) {
82 articleURL = r.Target.Source110 articleURL = r.Target.Source
83 if r.Body != nil {111 if r.Body != nil {
modified internal/atproto/stream_handler.go +7 -0
@@ -195,6 +195,13 @@ func (h *StreamDBHandler) handleMarginNote(ctx context.Context, event *Event) er
195195 if err != nil || a == nil {
196196 return err
197197 }
198+ exists, err := h.articles.AnnotationExistsByContent(ctx, a.AuthorDID, a.ArticleURL, a.Quote.String, a.Note.String)
199+ if err != nil {
200+ return err
201+ }
202+ if exists {
203+ return nil
204+ }
198205 return h.articles.CreateAnnotation(ctx, a)
199206
200207 case actionUpdate:
@@ -195,6 +195,13 @@ func (h *StreamDBHandler) handleMarginNote(ctx context.Context, event *Event) er
195 if err != nil || a == nil {195 if err != nil || a == nil {
196 return err196 return err
197 }197 }
198+ exists, err := h.articles.AnnotationExistsByContent(ctx, a.AuthorDID, a.ArticleURL, a.Quote.String, a.Note.String)
199+ if err != nil {
200+ return err
201+ }
202+ if exists {
203+ return nil
204+ }
198 return h.articles.CreateAnnotation(ctx, a)205 return h.articles.CreateAnnotation(ctx, a)
199 206
200 case actionUpdate:207 case actionUpdate:
added internal/atproto/stream_handler_test.go +172 -0
new file mode 100644
@@ -0,0 +1,172 @@
1+package atproto
2+
3+import (
4+ "context"
5+ "encoding/json"
6+ "log/slog"
7+ "os"
8+ "testing"
9+ "time"
10+
11+ "gotest.tools/v3/assert"
12+
13+ "pkg.rbrt.fr/glean/internal/db"
14+ "pkg.rbrt.fr/glean/internal/feed"
15+)
16+
17+func setupStreamTestDB(t *testing.T) *db.Store {
18+ t.Helper()
19+ f, err := os.CreateTemp("", "glean-stream-test-*.db")
20+ assert.NilError(t, err)
21+ assert.NilError(t, f.Close())
22+ path := f.Name()
23+ t.Cleanup(func() {
24+ for _, suffix := range []string{"", "_users", "_users-shm", "_users-wal", "_articles", "_articles-shm", "_articles-wal", "_recs", "_recs-shm", "_recs-wal"} {
25+ _ = os.Remove(path + suffix)
26+ }
27+ })
28+ dbs, err := db.Open(path)
29+ assert.NilError(t, err)
30+ t.Cleanup(func() { _ = dbs.Close() })
31+ return dbs
32+}
33+
34+func TestHandleMarginNote_SkipsDuplicateFromGleanAnnotation(t *testing.T) {
35+ ctx := context.Background()
36+ dbs := setupStreamTestDB(t)
37+ handler := NewStreamDBHandler(dbs.Articles, dbs.Users, slog.Default())
38+
39+ now := time.Now().Format(time.RFC3339)
40+
41+ _ = dbs.Articles.UpsertFeed(ctx, &db.Feed{FeedURL: "https://a.com/feed"})
42+
43+ err := dbs.Articles.CreateAnnotation(ctx, &db.Annotation{
44+ URI: "at://did:test:u1/at.glean.annotation/rkey1", AuthorDID: "did:test:u1",
45+ FeedURL: "https://a.com/feed", ArticleURL: "https://a.com/1",
46+ Quote: db.NullStr("selected text"), Note: db.NullStr("my note"),
47+ CreatedAt: db.NullTime(time.Now()),
48+ })
49+ assert.NilError(t, err)
50+
51+ marginRec := NewMarginNoteRecord("https://a.com/1", "selected text", "my note", nil, now)
52+ value, err := json.Marshal(marginRec)
53+ assert.NilError(t, err)
54+
55+ event := &Event{
56+ Type: actionCreate,
57+ DID: "did:test:u1",
58+ Collection: CollectionMarginNote,
59+ RKey: "margin-rkey1",
60+ URI: "at://did:test:u1/at.margin.note/margin-rkey1",
61+ CID: "cid-margin1",
62+ Value: value,
63+ }
64+
65+ err = handler.Handle(ctx, event)
66+ assert.NilError(t, err)
67+
68+ exists, err := dbs.Articles.AnnotationExists(ctx, "at://did:test:u1/at.margin.note/margin-rkey1")
69+ assert.NilError(t, err)
70+ assert.Equal(t, exists, false, "margin note should not create a duplicate annotation")
71+}
72+
73+func TestHandleMarginNote_CreatesWhenNoDuplicate(t *testing.T) {
74+ ctx := context.Background()
75+ dbs := setupStreamTestDB(t)
76+ handler := NewStreamDBHandler(dbs.Articles, dbs.Users, slog.Default())
77+
78+ now := time.Now().Format(time.RFC3339)
79+
80+ _ = dbs.Articles.UpsertFeed(ctx, &db.Feed{FeedURL: "https://a.com/feed"})
81+ _ = dbs.Articles.BatchUpsertArticles(ctx, []feed.Article{{
82+ FeedURL: "https://a.com/feed", URL: "https://a.com/1", Title: "Test",
83+ }})
84+
85+ marginRec := NewMarginNoteRecord("https://a.com/1", "some quote", "some note", nil, now)
86+ value, err := json.Marshal(marginRec)
87+ assert.NilError(t, err)
88+
89+ event := &Event{
90+ Type: actionCreate,
91+ DID: "did:test:u1",
92+ Collection: CollectionMarginNote,
93+ RKey: "margin-rkey2",
94+ URI: "at://did:test:u1/at.margin.note/margin-rkey2",
95+ CID: "cid-margin2",
96+ Value: value,
97+ }
98+
99+ err = handler.Handle(ctx, event)
100+ assert.NilError(t, err)
101+
102+ exists, err := dbs.Articles.AnnotationExists(ctx, "at://did:test:u1/at.margin.note/margin-rkey2")
103+ assert.NilError(t, err)
104+ assert.Equal(t, exists, true, "margin note should be stored when no duplicate exists")
105+}
106+
107+func TestHandleMarginNote_SkipsWhenSameContentDifferentURI(t *testing.T) {
108+ ctx := context.Background()
109+ dbs := setupStreamTestDB(t)
110+ handler := NewStreamDBHandler(dbs.Articles, dbs.Users, slog.Default())
111+
112+ now := time.Now().Format(time.RFC3339)
113+
114+ _ = dbs.Articles.UpsertFeed(ctx, &db.Feed{FeedURL: "https://a.com/feed"})
115+
116+ err := dbs.Articles.CreateAnnotation(ctx, &db.Annotation{
117+ URI: "at://did:test:u1/at.glean.annotation/rkey3", AuthorDID: "did:test:u1",
118+ FeedURL: "https://a.com/feed", ArticleURL: "https://a.com/1",
119+ Note: db.NullStr("identical note"),
120+ CreatedAt: db.NullTime(time.Now()),
121+ })
122+ assert.NilError(t, err)
123+
124+ marginRec := NewMarginNoteRecord("https://a.com/1", "", "identical note", nil, now)
125+ value, err := json.Marshal(marginRec)
126+ assert.NilError(t, err)
127+
128+ event := &Event{
129+ Type: actionCreate,
130+ DID: "did:test:u1",
131+ Collection: CollectionMarginNote,
132+ RKey: "margin-rkey3",
133+ URI: "at://did:test:u1/at.margin.note/margin-rkey3",
134+ CID: "cid-margin3",
135+ Value: value,
136+ }
137+
138+ err = handler.Handle(ctx, event)
139+ assert.NilError(t, err)
140+
141+ var count int
142+ err = dbs.SQLDB().QueryRow(`SELECT COUNT(*) FROM articles.annotations WHERE article_url = 'https://a.com/1'`).Scan(&count)
143+ assert.NilError(t, err)
144+ assert.Equal(t, count, 1)
145+}
146+
147+func TestHandleMarginNote_SkipsEmptyArticleURL(t *testing.T) {
148+ ctx := context.Background()
149+ dbs := setupStreamTestDB(t)
150+ handler := NewStreamDBHandler(dbs.Articles, dbs.Users, slog.Default())
151+
152+ marginRec := NewMarginNoteRecord("", "quote", "note", nil, time.Now().Format(time.RFC3339))
153+ value, err := json.Marshal(marginRec)
154+ assert.NilError(t, err)
155+
156+ event := &Event{
157+ Type: actionCreate,
158+ DID: "did:test:u1",
159+ Collection: CollectionMarginNote,
160+ RKey: "margin-rkey4",
161+ URI: "at://did:test:u1/at.margin.note/margin-rkey4",
162+ CID: "cid-margin4",
163+ Value: value,
164+ }
165+
166+ err = handler.Handle(ctx, event)
167+ assert.NilError(t, err)
168+
169+ var count int
170+ _ = dbs.SQLDB().QueryRow(`SELECT COUNT(*) FROM articles.annotations`).Scan(&count)
171+ assert.Equal(t, count, 0)
172+}
new file mode 100644
@@ -0,0 +1,172 @@
1+package atproto
2+
3+import (
4+ "context"
5+ "encoding/json"
6+ "log/slog"
7+ "os"
8+ "testing"
9+ "time"
10+
11+ "gotest.tools/v3/assert"
12+
13+ "pkg.rbrt.fr/glean/internal/db"
14+ "pkg.rbrt.fr/glean/internal/feed"
15+)
16+
17+func setupStreamTestDB(t *testing.T) *db.Store {
18+ t.Helper()
19+ f, err := os.CreateTemp("", "glean-stream-test-*.db")
20+ assert.NilError(t, err)
21+ assert.NilError(t, f.Close())
22+ path := f.Name()
23+ t.Cleanup(func() {
24+ for _, suffix := range []string{"", "_users", "_users-shm", "_users-wal", "_articles", "_articles-shm", "_articles-wal", "_recs", "_recs-shm", "_recs-wal"} {
25+ _ = os.Remove(path + suffix)
26+ }
27+ })
28+ dbs, err := db.Open(path)
29+ assert.NilError(t, err)
30+ t.Cleanup(func() { _ = dbs.Close() })
31+ return dbs
32+}
33+
34+func TestHandleMarginNote_SkipsDuplicateFromGleanAnnotation(t *testing.T) {
35+ ctx := context.Background()
36+ dbs := setupStreamTestDB(t)
37+ handler := NewStreamDBHandler(dbs.Articles, dbs.Users, slog.Default())
38+
39+ now := time.Now().Format(time.RFC3339)
40+
41+ _ = dbs.Articles.UpsertFeed(ctx, &db.Feed{FeedURL: "https://a.com/feed"})
42+
43+ err := dbs.Articles.CreateAnnotation(ctx, &db.Annotation{
44+ URI: "at://did:test:u1/at.glean.annotation/rkey1", AuthorDID: "did:test:u1",
45+ FeedURL: "https://a.com/feed", ArticleURL: "https://a.com/1",
46+ Quote: db.NullStr("selected text"), Note: db.NullStr("my note"),
47+ CreatedAt: db.NullTime(time.Now()),
48+ })
49+ assert.NilError(t, err)
50+
51+ marginRec := NewMarginNoteRecord("https://a.com/1", "selected text", "my note", nil, now)
52+ value, err := json.Marshal(marginRec)
53+ assert.NilError(t, err)
54+
55+ event := &Event{
56+ Type: actionCreate,
57+ DID: "did:test:u1",
58+ Collection: CollectionMarginNote,
59+ RKey: "margin-rkey1",
60+ URI: "at://did:test:u1/at.margin.note/margin-rkey1",
61+ CID: "cid-margin1",
62+ Value: value,
63+ }
64+
65+ err = handler.Handle(ctx, event)
66+ assert.NilError(t, err)
67+
68+ exists, err := dbs.Articles.AnnotationExists(ctx, "at://did:test:u1/at.margin.note/margin-rkey1")
69+ assert.NilError(t, err)
70+ assert.Equal(t, exists, false, "margin note should not create a duplicate annotation")
71+}
72+
73+func TestHandleMarginNote_CreatesWhenNoDuplicate(t *testing.T) {
74+ ctx := context.Background()
75+ dbs := setupStreamTestDB(t)
76+ handler := NewStreamDBHandler(dbs.Articles, dbs.Users, slog.Default())
77+
78+ now := time.Now().Format(time.RFC3339)
79+
80+ _ = dbs.Articles.UpsertFeed(ctx, &db.Feed{FeedURL: "https://a.com/feed"})
81+ _ = dbs.Articles.BatchUpsertArticles(ctx, []feed.Article{{
82+ FeedURL: "https://a.com/feed", URL: "https://a.com/1", Title: "Test",
83+ }})
84+
85+ marginRec := NewMarginNoteRecord("https://a.com/1", "some quote", "some note", nil, now)
86+ value, err := json.Marshal(marginRec)
87+ assert.NilError(t, err)
88+
89+ event := &Event{
90+ Type: actionCreate,
91+ DID: "did:test:u1",
92+ Collection: CollectionMarginNote,
93+ RKey: "margin-rkey2",
94+ URI: "at://did:test:u1/at.margin.note/margin-rkey2",
95+ CID: "cid-margin2",
96+ Value: value,
97+ }
98+
99+ err = handler.Handle(ctx, event)
100+ assert.NilError(t, err)
101+
102+ exists, err := dbs.Articles.AnnotationExists(ctx, "at://did:test:u1/at.margin.note/margin-rkey2")
103+ assert.NilError(t, err)
104+ assert.Equal(t, exists, true, "margin note should be stored when no duplicate exists")
105+}
106+
107+func TestHandleMarginNote_SkipsWhenSameContentDifferentURI(t *testing.T) {
108+ ctx := context.Background()
109+ dbs := setupStreamTestDB(t)
110+ handler := NewStreamDBHandler(dbs.Articles, dbs.Users, slog.Default())
111+
112+ now := time.Now().Format(time.RFC3339)
113+
114+ _ = dbs.Articles.UpsertFeed(ctx, &db.Feed{FeedURL: "https://a.com/feed"})
115+
116+ err := dbs.Articles.CreateAnnotation(ctx, &db.Annotation{
117+ URI: "at://did:test:u1/at.glean.annotation/rkey3", AuthorDID: "did:test:u1",
118+ FeedURL: "https://a.com/feed", ArticleURL: "https://a.com/1",
119+ Note: db.NullStr("identical note"),
120+ CreatedAt: db.NullTime(time.Now()),
121+ })
122+ assert.NilError(t, err)
123+
124+ marginRec := NewMarginNoteRecord("https://a.com/1", "", "identical note", nil, now)
125+ value, err := json.Marshal(marginRec)
126+ assert.NilError(t, err)
127+
128+ event := &Event{
129+ Type: actionCreate,
130+ DID: "did:test:u1",
131+ Collection: CollectionMarginNote,
132+ RKey: "margin-rkey3",
133+ URI: "at://did:test:u1/at.margin.note/margin-rkey3",
134+ CID: "cid-margin3",
135+ Value: value,
136+ }
137+
138+ err = handler.Handle(ctx, event)
139+ assert.NilError(t, err)
140+
141+ var count int
142+ err = dbs.SQLDB().QueryRow(`SELECT COUNT(*) FROM articles.annotations WHERE article_url = 'https://a.com/1'`).Scan(&count)
143+ assert.NilError(t, err)
144+ assert.Equal(t, count, 1)
145+}
146+
147+func TestHandleMarginNote_SkipsEmptyArticleURL(t *testing.T) {
148+ ctx := context.Background()
149+ dbs := setupStreamTestDB(t)
150+ handler := NewStreamDBHandler(dbs.Articles, dbs.Users, slog.Default())
151+
152+ marginRec := NewMarginNoteRecord("", "quote", "note", nil, time.Now().Format(time.RFC3339))
153+ value, err := json.Marshal(marginRec)
154+ assert.NilError(t, err)
155+
156+ event := &Event{
157+ Type: actionCreate,
158+ DID: "did:test:u1",
159+ Collection: CollectionMarginNote,
160+ RKey: "margin-rkey4",
161+ URI: "at://did:test:u1/at.margin.note/margin-rkey4",
162+ CID: "cid-margin4",
163+ Value: value,
164+ }
165+
166+ err = handler.Handle(ctx, event)
167+ assert.NilError(t, err)
168+
169+ var count int
170+ _ = dbs.SQLDB().QueryRow(`SELECT COUNT(*) FROM articles.annotations`).Scan(&count)
171+ assert.Equal(t, count, 0)
172+}
modified internal/db/batch_test.go +55 -0
@@ -558,3 +558,58 @@ func TestDeleteOrphanedAnnotations_RemovesOrphaned(t *testing.T) {
558558 assert.NilError(t, err)
559559 assert.Equal(t, exists, false)
560560 }
561+
562+func TestAnnotationExistsByContent_FindsExactMatch(t *testing.T) {
563+ ctx := context.Background()
564+ dbs := setupTestDB(t)
565+
566+ now := NullTime(time.Now())
567+ err := dbs.Articles.CreateAnnotation(ctx, &Annotation{
568+ URI: "at://glean:ann1", AuthorDID: "did:test:u1", FeedURL: "https://a.com/feed",
569+ ArticleURL: "https://a.com/1", Quote: NullStr("selected text"), Note: NullStr("my note"),
570+ CreatedAt: now,
571+ })
572+ assert.NilError(t, err)
573+
574+ exists, err := dbs.Articles.AnnotationExistsByContent(ctx, "did:test:u1", "https://a.com/1", "selected text", "my note")
575+ assert.NilError(t, err)
576+ assert.Equal(t, exists, true)
577+}
578+
579+func TestAnnotationExistsByContent_NoMatch(t *testing.T) {
580+ ctx := context.Background()
581+ dbs := setupTestDB(t)
582+
583+ now := NullTime(time.Now())
584+ err := dbs.Articles.CreateAnnotation(ctx, &Annotation{
585+ URI: "at://glean:ann1", AuthorDID: "did:test:u1", FeedURL: "https://a.com/feed",
586+ ArticleURL: "https://a.com/1", Quote: NullStr("selected text"), Note: NullStr("my note"),
587+ CreatedAt: now,
588+ })
589+ assert.NilError(t, err)
590+
591+ exists, err := dbs.Articles.AnnotationExistsByContent(ctx, "did:test:u1", "https://a.com/1", "different quote", "my note")
592+ assert.NilError(t, err)
593+ assert.Equal(t, exists, false)
594+}
595+
596+func TestAnnotationExistsByContent_EmptyFields(t *testing.T) {
597+ ctx := context.Background()
598+ dbs := setupTestDB(t)
599+
600+ now := NullTime(time.Now())
601+ err := dbs.Articles.CreateAnnotation(ctx, &Annotation{
602+ URI: "at://glean:ann1", AuthorDID: "did:test:u1", FeedURL: "https://a.com/feed",
603+ ArticleURL: "https://a.com/1", Note: NullStr("just a note"),
604+ CreatedAt: now,
605+ })
606+ assert.NilError(t, err)
607+
608+ exists, err := dbs.Articles.AnnotationExistsByContent(ctx, "did:test:u1", "https://a.com/1", "", "just a note")
609+ assert.NilError(t, err)
610+ assert.Equal(t, exists, true)
611+
612+ exists, err = dbs.Articles.AnnotationExistsByContent(ctx, "did:test:u1", "https://a.com/1", "", "")
613+ assert.NilError(t, err)
614+ assert.Equal(t, exists, false)
615+}
@@ -558,3 +558,58 @@ func TestDeleteOrphanedAnnotations_RemovesOrphaned(t *testing.T) {
558 assert.NilError(t, err)558 assert.NilError(t, err)
559 assert.Equal(t, exists, false)559 assert.Equal(t, exists, false)
560 }560 }
561+
562+func TestAnnotationExistsByContent_FindsExactMatch(t *testing.T) {
563+ ctx := context.Background()
564+ dbs := setupTestDB(t)
565+
566+ now := NullTime(time.Now())
567+ err := dbs.Articles.CreateAnnotation(ctx, &Annotation{
568+ URI: "at://glean:ann1", AuthorDID: "did:test:u1", FeedURL: "https://a.com/feed",
569+ ArticleURL: "https://a.com/1", Quote: NullStr("selected text"), Note: NullStr("my note"),
570+ CreatedAt: now,
571+ })
572+ assert.NilError(t, err)
573+
574+ exists, err := dbs.Articles.AnnotationExistsByContent(ctx, "did:test:u1", "https://a.com/1", "selected text", "my note")
575+ assert.NilError(t, err)
576+ assert.Equal(t, exists, true)
577+}
578+
579+func TestAnnotationExistsByContent_NoMatch(t *testing.T) {
580+ ctx := context.Background()
581+ dbs := setupTestDB(t)
582+
583+ now := NullTime(time.Now())
584+ err := dbs.Articles.CreateAnnotation(ctx, &Annotation{
585+ URI: "at://glean:ann1", AuthorDID: "did:test:u1", FeedURL: "https://a.com/feed",
586+ ArticleURL: "https://a.com/1", Quote: NullStr("selected text"), Note: NullStr("my note"),
587+ CreatedAt: now,
588+ })
589+ assert.NilError(t, err)
590+
591+ exists, err := dbs.Articles.AnnotationExistsByContent(ctx, "did:test:u1", "https://a.com/1", "different quote", "my note")
592+ assert.NilError(t, err)
593+ assert.Equal(t, exists, false)
594+}
595+
596+func TestAnnotationExistsByContent_EmptyFields(t *testing.T) {
597+ ctx := context.Background()
598+ dbs := setupTestDB(t)
599+
600+ now := NullTime(time.Now())
601+ err := dbs.Articles.CreateAnnotation(ctx, &Annotation{
602+ URI: "at://glean:ann1", AuthorDID: "did:test:u1", FeedURL: "https://a.com/feed",
603+ ArticleURL: "https://a.com/1", Note: NullStr("just a note"),
604+ CreatedAt: now,
605+ })
606+ assert.NilError(t, err)
607+
608+ exists, err := dbs.Articles.AnnotationExistsByContent(ctx, "did:test:u1", "https://a.com/1", "", "just a note")
609+ assert.NilError(t, err)
610+ assert.Equal(t, exists, true)
611+
612+ exists, err = dbs.Articles.AnnotationExistsByContent(ctx, "did:test:u1", "https://a.com/1", "", "")
613+ assert.NilError(t, err)
614+ assert.Equal(t, exists, false)
615+}
modified internal/db/social.go +15 -0
@@ -96,6 +96,21 @@ func (s *ArticleStore) AnnotationExists(ctx context.Context, uri string) (bool,
9696 return true, nil
9797 }
9898
99+func (s *ArticleStore) AnnotationExistsByContent(ctx context.Context, authorDID, articleURL, quote, note string) (bool, error) {
100+ var exists int
101+ err := s.db.QueryRowContext(ctx, `
102+ SELECT 1 FROM articles.annotations
103+ WHERE author_did = ? AND article_url = ? AND COALESCE(quote, '') = COALESCE(?, '') AND COALESCE(note, '') = COALESCE(?, '')
104+ `, authorDID, articleURL, quote, note).Scan(&exists)
105+ if err == sql.ErrNoRows {
106+ return false, nil
107+ }
108+ if err != nil {
109+ return false, err
110+ }
111+ return true, nil
112+}
113+
99114 func (s *ArticleStore) ListAnnotations(ctx context.Context, feedURL, articleURL, authorDID string, limit, offset int) ([]*Annotation, error) {
100115 var conds []string
101116 var args []any
@@ -96,6 +96,21 @@ func (s *ArticleStore) AnnotationExists(ctx context.Context, uri string) (bool,
96 return true, nil96 return true, nil
97 }97 }
98 98
99+func (s *ArticleStore) AnnotationExistsByContent(ctx context.Context, authorDID, articleURL, quote, note string) (bool, error) {
100+ var exists int
101+ err := s.db.QueryRowContext(ctx, `
102+ SELECT 1 FROM articles.annotations
103+ WHERE author_did = ? AND article_url = ? AND COALESCE(quote, '') = COALESCE(?, '') AND COALESCE(note, '') = COALESCE(?, '')
104+ `, authorDID, articleURL, quote, note).Scan(&exists)
105+ if err == sql.ErrNoRows {
106+ return false, nil
107+ }
108+ if err != nil {
109+ return false, err
110+ }
111+ return true, nil
112+}
113+
99 func (s *ArticleStore) ListAnnotations(ctx context.Context, feedURL, articleURL, authorDID string, limit, offset int) ([]*Annotation, error) {114 func (s *ArticleStore) ListAnnotations(ctx context.Context, feedURL, articleURL, authorDID string, limit, offset int) ([]*Annotation, error) {
100 var conds []string115 var conds []string
101 var args []any116 var args []any
modified internal/server/annotations_handler.go +11 -0
@@ -139,6 +139,17 @@ func (s *Server) handleCreateAnnotation(w http.ResponseWriter, r *http.Request)
139139 }
140140 a.URI = uri
141141 a.CID = sql.NullString{String: cid, Valid: true}
142+
143+ marginRec := atproto.NewMarginNoteRecord(
144+ a.ArticleURL,
145+ a.Quote.String,
146+ a.Note.String,
147+ tags,
148+ time.Now().Format(time.RFC3339),
149+ )
150+ if _, _, err := client.CreateRecord(ctx, user.DID, atproto.CollectionMarginNote, marginRec); err != nil {
151+ s.logger.Error("failed to write margin note to PDS", "error", err)
152+ }
142153 } else {
143154 a.URI = fmt.Sprintf("glean:annotation:%d", time.Now().UnixNano())
144155 }
@@ -139,6 +139,17 @@ func (s *Server) handleCreateAnnotation(w http.ResponseWriter, r *http.Request)
139 }139 }
140 a.URI = uri140 a.URI = uri
141 a.CID = sql.NullString{String: cid, Valid: true}141 a.CID = sql.NullString{String: cid, Valid: true}
142+
143+ marginRec := atproto.NewMarginNoteRecord(
144+ a.ArticleURL,
145+ a.Quote.String,
146+ a.Note.String,
147+ tags,
148+ time.Now().Format(time.RFC3339),
149+ )
150+ if _, _, err := client.CreateRecord(ctx, user.DID, atproto.CollectionMarginNote, marginRec); err != nil {
151+ s.logger.Error("failed to write margin note to PDS", "error", err)
152+ }
142 } else {153 } else {
143 a.URI = fmt.Sprintf("glean:annotation:%d", time.Now().UnixNano())154 a.URI = fmt.Sprintf("glean:annotation:%d", time.Now().UnixNano())
144 }155 }
modified internal/server/server.go +1 -0
@@ -43,6 +43,7 @@ var oauthScopes = []string{
4343 fmt.Sprintf("repo:%s", atproto.CollectionSubscription),
4444 fmt.Sprintf("repo:%s", atproto.CollectionLike),
4545 fmt.Sprintf("repo:%s", atproto.CollectionAnnotation),
46+ fmt.Sprintf("repo:%s", atproto.CollectionMarginNote),
4647
4748 "rpc:at.glean.listSubscriptions?aud=*",
4849 "rpc:at.glean.listLikes?aud=*",
@@ -43,6 +43,7 @@ var oauthScopes = []string{
43 fmt.Sprintf("repo:%s", atproto.CollectionSubscription),43 fmt.Sprintf("repo:%s", atproto.CollectionSubscription),
44 fmt.Sprintf("repo:%s", atproto.CollectionLike),44 fmt.Sprintf("repo:%s", atproto.CollectionLike),
45 fmt.Sprintf("repo:%s", atproto.CollectionAnnotation),45 fmt.Sprintf("repo:%s", atproto.CollectionAnnotation),
46+ fmt.Sprintf("repo:%s", atproto.CollectionMarginNote),
46 47
47 "rpc:at.glean.listSubscriptions?aud=*",48 "rpc:at.glean.listSubscriptions?aud=*",
48 "rpc:at.glean.listLikes?aud=*",49 "rpc:at.glean.listLikes?aud=*",