rickub
nandi/gleanpublic⑂ Fork 0
⑂ c951c57
Branches
claude/modal-container-min-76527a main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/glean.git
git clone ssh://git@rickub.com/nandi/glean.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Add Skyreader subscription supportUnverified

Julien Robert committed 2026-04-21T22:13:38+02:00 Browse files
c951c57 parent: eed7f2e
modified docs/specs.md +34 -16
@@ -137,26 +137,43 @@ The mapping from margin note to glean annotation:
137137
138138 When no matching article exists in the local DB, the annotation is stored with an empty `feed_url`. Margin notes are indexed from both Jetstream and PDS sync, same as glean records.
139139
140-### 3.5 `app.bsky.graph.follow` (External)
140+### 3.5 `app.skyreader.feed.subscription` (External)
141+
142+Glean also indexes records from the Skyreader lexicon (`app.skyreader.feed.subscription`). When a user has subscriptions in Skyreader, they are imported as Glean subscriptions during PDS sync and via Jetstream events. This lets users who previously used Skyreader seamlessly transition to Glean without re-subscribing to their feeds.
143+
144+The mapping from Skyreader subscription to Glean subscription:
145+
146+| Skyreader field | Glean field | Notes |
147+| --------------- | ------------- | ------------------------------------ |
148+| `feedUrl` | `feed_url` | Direct mapping |
149+| `title` | `title` | Direct mapping |
150+| `siteUrl` | `site_url` | Stored on the feed record |
151+| `createdAt` | `added_at` | Direct mapping |
152+| _(none)_ | `category` | Empty (Skyreader has no categories) |
153+
154+If a Glean subscription already exists for the same `feed_url`, the existing one is kept. If the existing subscription has no URI (was created locally without PDS sync), the Skyreader URI/CID is backfilled.
155+
156+### 3.6 `app.bsky.graph.follow` (External)
141157
142158 Follow relationships are tracked from Bluesky and Tangled follow records. The `FollowRecord` struct is validated against the lexicon at `lexicons/app/bsky/graph/follow.json`. The optional `via` field (a strong ref) is preserved as raw JSON but not used by Glean.
143159
144-### 3.6 Lexicon Constants
160+### 3.7 Lexicon Constants
145161
146162 All collection NSIDs are defined as constants in `lexicon.go` and used throughout the codebase:
147163
148164 ```go
149165 const (
150- CollectionSubscription = "at.glean.subscription"
151- CollectionAnnotation = "at.glean.annotation"
152- CollectionLike = "at.glean.like"
153- CollectionMarginNote = "at.margin.note"
154- CollectionBskyFollow = "app.bsky.graph.follow"
155- CollectionTangledFollow = "sh.tangled.graph.follow"
166+ CollectionSubscription = "at.glean.subscription"
167+ CollectionAnnotation = "at.glean.annotation"
168+ CollectionLike = "at.glean.like"
169+ CollectionMarginNote = "at.margin.note"
170+ CollectionSkyreaderSubscription = "app.skyreader.feed.subscription"
171+ CollectionBskyFollow = "app.bsky.graph.follow"
172+ CollectionTangledFollow = "sh.tangled.graph.follow"
156173 )
157174 ```
158175
159-### 3.7 AppView Query Lexicons
176+### 3.8 AppView Query Lexicons
160177
161178 As an AppView, Glean serves the following XRPC query endpoints. Other AT Protocol applications can call these to access indexed `at.glean.*` data without implementing their own indexer.
162179
@@ -253,12 +270,12 @@ Output:
253270 people: [{ did, handle, displayName, avatar, jaccard, commonFeeds }]
254271 ```
255272
256-### 3.8 AppView Jetstream Consumption
273+### 3.9 AppView Jetstream Consumption
257274
258275 Glean subscribes to a Jetstream endpoint (`GLEAN_JETSTREAM`, default `wss://jetstream.glean.at`) for all `at.glean.*` records:
259276
260277 ```
261-SUBSCRIBE collections: ["at.glean.subscription", "at.glean.annotation", "at.glean.like", "app.bsky.graph.follow", "sh.tangled.graph.follow", "at.margin.note"]
278+SUBSCRIBE collections: ["at.glean.subscription", "at.glean.annotation", "at.glean.like", "app.bsky.graph.follow", "sh.tangled.graph.follow", "at.margin.note", "app.skyreader.feed.subscription"]
262279 ```
263280
264281 On each event:
@@ -446,10 +463,11 @@ Glean runs as a single Go binary that fills three roles: **AppView** (indexing `
446463 │ └─────────────────┘ │ └──────────────────┘
447464 └──────────────────────┘
448465
449- AppView responsibilities:
450- • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like, at.margin.note
451- • Index records into SQLite
452- • Convert at.margin.note records to annotations (displayed alongside glean.at annotations)
466+ AppView responsibilities:
467+ • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like, at.margin.note, app.skyreader.feed.subscription
468+ • Index records into SQLite
469+ • Convert at.margin.note records to annotations (displayed alongside glean.at annotations)
470+ • Import app.skyreader.feed.subscription records as Glean subscriptions
453471 • Serve XRPC query endpoints (at.glean.listSubscriptions, etc.)
454472 • Host the web UI at glean.at
455473 • Write to user PDS on behalf of user (when user acts through UI)
@@ -803,7 +821,7 @@ glean/
803821 │ │ ├── jetstream.go # Subscribe to Jetstream via official client
804822 │ │ ├── stream_handler.go # Stream event → DB handler
805823 │ │ ├── lexicon.go # Lexicon record types (at.glean.*, maintained by hand)
806-│ │ ├── lexicon_external.go # External lexicon record types (FollowRecord, MarginNoteRecord)
824+│ │ ├── lexicon_external.go # External lexicon record types (FollowRecord, MarginNoteRecord, SkyreaderSubscriptionRecord)
807825 │ │ ├── lexicon_test.go # Test: Go structs match lexicon JSON schemas
808826 │ │ ├── sync.go # PDS record reconciliation
809827 │ │ └── xrpc.go # XRPC query handlers (AppView endpoints)
@@ -137,26 +137,43 @@ The mapping from margin note to glean annotation:
137 137
138 When no matching article exists in the local DB, the annotation is stored with an empty `feed_url`. Margin notes are indexed from both Jetstream and PDS sync, same as glean records.138 When no matching article exists in the local DB, the annotation is stored with an empty `feed_url`. Margin notes are indexed from both Jetstream and PDS sync, same as glean records.
139 139
140-### 3.5 `app.bsky.graph.follow` (External)140+### 3.5 `app.skyreader.feed.subscription` (External)
141+
142+Glean also indexes records from the Skyreader lexicon (`app.skyreader.feed.subscription`). When a user has subscriptions in Skyreader, they are imported as Glean subscriptions during PDS sync and via Jetstream events. This lets users who previously used Skyreader seamlessly transition to Glean without re-subscribing to their feeds.
143+
144+The mapping from Skyreader subscription to Glean subscription:
145+
146+| Skyreader field | Glean field | Notes |
147+| --------------- | ------------- | ------------------------------------ |
148+| `feedUrl` | `feed_url` | Direct mapping |
149+| `title` | `title` | Direct mapping |
150+| `siteUrl` | `site_url` | Stored on the feed record |
151+| `createdAt` | `added_at` | Direct mapping |
152+| _(none)_ | `category` | Empty (Skyreader has no categories) |
153+
154+If a Glean subscription already exists for the same `feed_url`, the existing one is kept. If the existing subscription has no URI (was created locally without PDS sync), the Skyreader URI/CID is backfilled.
155+
156+### 3.6 `app.bsky.graph.follow` (External)
141 157
142 Follow relationships are tracked from Bluesky and Tangled follow records. The `FollowRecord` struct is validated against the lexicon at `lexicons/app/bsky/graph/follow.json`. The optional `via` field (a strong ref) is preserved as raw JSON but not used by Glean.158 Follow relationships are tracked from Bluesky and Tangled follow records. The `FollowRecord` struct is validated against the lexicon at `lexicons/app/bsky/graph/follow.json`. The optional `via` field (a strong ref) is preserved as raw JSON but not used by Glean.
143 159
144-### 3.6 Lexicon Constants160+### 3.7 Lexicon Constants
145 161
146 All collection NSIDs are defined as constants in `lexicon.go` and used throughout the codebase:162 All collection NSIDs are defined as constants in `lexicon.go` and used throughout the codebase:
147 163
148 ```go164 ```go
149 const (165 const (
150- CollectionSubscription = "at.glean.subscription"166+ CollectionSubscription = "at.glean.subscription"
151- CollectionAnnotation = "at.glean.annotation"167+ CollectionAnnotation = "at.glean.annotation"
152- CollectionLike = "at.glean.like"168+ CollectionLike = "at.glean.like"
153- CollectionMarginNote = "at.margin.note"169+ CollectionMarginNote = "at.margin.note"
154- CollectionBskyFollow = "app.bsky.graph.follow"170+ CollectionSkyreaderSubscription = "app.skyreader.feed.subscription"
155- CollectionTangledFollow = "sh.tangled.graph.follow"171+ CollectionBskyFollow = "app.bsky.graph.follow"
172+ CollectionTangledFollow = "sh.tangled.graph.follow"
156 )173 )
157 ```174 ```
158 175
159-### 3.7 AppView Query Lexicons176+### 3.8 AppView Query Lexicons
160 177
161 As an AppView, Glean serves the following XRPC query endpoints. Other AT Protocol applications can call these to access indexed `at.glean.*` data without implementing their own indexer.178 As an AppView, Glean serves the following XRPC query endpoints. Other AT Protocol applications can call these to access indexed `at.glean.*` data without implementing their own indexer.
162 179
@@ -253,12 +270,12 @@ Output:
253 people: [{ did, handle, displayName, avatar, jaccard, commonFeeds }]270 people: [{ did, handle, displayName, avatar, jaccard, commonFeeds }]
254 ```271 ```
255 272
256-### 3.8 AppView Jetstream Consumption273+### 3.9 AppView Jetstream Consumption
257 274
258 Glean subscribes to a Jetstream endpoint (`GLEAN_JETSTREAM`, default `wss://jetstream.glean.at`) for all `at.glean.*` records:275 Glean subscribes to a Jetstream endpoint (`GLEAN_JETSTREAM`, default `wss://jetstream.glean.at`) for all `at.glean.*` records:
259 276
260 ```277 ```
261-SUBSCRIBE collections: ["at.glean.subscription", "at.glean.annotation", "at.glean.like", "app.bsky.graph.follow", "sh.tangled.graph.follow", "at.margin.note"]278+SUBSCRIBE collections: ["at.glean.subscription", "at.glean.annotation", "at.glean.like", "app.bsky.graph.follow", "sh.tangled.graph.follow", "at.margin.note", "app.skyreader.feed.subscription"]
262 ```279 ```
263 280
264 On each event:281 On each event:
@@ -446,10 +463,11 @@ Glean runs as a single Go binary that fills three roles: **AppView** (indexing `
446 │ └─────────────────┘ │ └──────────────────┘463 │ └─────────────────┘ │ └──────────────────┘
447 └──────────────────────┘464 └──────────────────────┘
448 465
449- AppView responsibilities:466+ AppView responsibilities:
450- • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like, at.margin.note467+ • Subscribe to Jetstream for at.glean.subscription, at.glean.annotation, at.glean.like, at.margin.note, app.skyreader.feed.subscription
451- • Index records into SQLite468+ • Index records into SQLite
452- • Convert at.margin.note records to annotations (displayed alongside glean.at annotations)469+ • Convert at.margin.note records to annotations (displayed alongside glean.at annotations)
470+ • Import app.skyreader.feed.subscription records as Glean subscriptions
453 • Serve XRPC query endpoints (at.glean.listSubscriptions, etc.)471 • Serve XRPC query endpoints (at.glean.listSubscriptions, etc.)
454 • Host the web UI at glean.at472 • Host the web UI at glean.at
455 • Write to user PDS on behalf of user (when user acts through UI)473 • Write to user PDS on behalf of user (when user acts through UI)
@@ -803,7 +821,7 @@ glean/
803 │ │ ├── jetstream.go # Subscribe to Jetstream via official client821 │ │ ├── jetstream.go # Subscribe to Jetstream via official client
804 │ │ ├── stream_handler.go # Stream event → DB handler822 │ │ ├── stream_handler.go # Stream event → DB handler
805 │ │ ├── lexicon.go # Lexicon record types (at.glean.*, maintained by hand)823 │ │ ├── lexicon.go # Lexicon record types (at.glean.*, maintained by hand)
806-│ │ ├── lexicon_external.go # External lexicon record types (FollowRecord, MarginNoteRecord)824+│ │ ├── lexicon_external.go # External lexicon record types (FollowRecord, MarginNoteRecord, SkyreaderSubscriptionRecord)
807 │ │ ├── lexicon_test.go # Test: Go structs match lexicon JSON schemas825 │ │ ├── lexicon_test.go # Test: Go structs match lexicon JSON schemas
808 │ │ ├── sync.go # PDS record reconciliation826 │ │ ├── sync.go # PDS record reconciliation
809 │ │ └── xrpc.go # XRPC query handlers (AppView endpoints)827 │ │ └── xrpc.go # XRPC query handlers (AppView endpoints)
modified internal/atproto/auth.go +4 -4
@@ -28,9 +28,9 @@ func InitIdentity(plcURL string) {
2828 HTTPClient: http.Client{
2929 Timeout: 10 * time.Second,
3030 Transport: &http.Transport{
31- Proxy: http.ProxyFromEnvironment,
32- IdleConnTimeout: 1000 * time.Millisecond,
33- MaxIdleConns: 100,
31+ Proxy: http.ProxyFromEnvironment,
32+ IdleConnTimeout: 1000 * time.Millisecond,
33+ MaxIdleConns: 100,
3434 },
3535 },
3636 Resolver: net.Resolver{
@@ -39,7 +39,7 @@ func InitIdentity(plcURL string) {
3939 return d.DialContext(ctx, network, address)
4040 },
4141 },
42- TryAuthoritativeDNS: true,
42+ TryAuthoritativeDNS: true,
4343 SkipDNSDomainSuffixes: []string{".bsky.social"},
4444 UserAgent: "glean/1.0",
4545 }
@@ -28,9 +28,9 @@ func InitIdentity(plcURL string) {
28 HTTPClient: http.Client{28 HTTPClient: http.Client{
29 Timeout: 10 * time.Second,29 Timeout: 10 * time.Second,
30 Transport: &http.Transport{30 Transport: &http.Transport{
31- Proxy: http.ProxyFromEnvironment,31+ Proxy: http.ProxyFromEnvironment,
32- IdleConnTimeout: 1000 * time.Millisecond,32+ IdleConnTimeout: 1000 * time.Millisecond,
33- MaxIdleConns: 100,33+ MaxIdleConns: 100,
34 },34 },
35 },35 },
36 Resolver: net.Resolver{36 Resolver: net.Resolver{
@@ -39,7 +39,7 @@ func InitIdentity(plcURL string) {
39 return d.DialContext(ctx, network, address)39 return d.DialContext(ctx, network, address)
40 },40 },
41 },41 },
42- TryAuthoritativeDNS: true,42+ TryAuthoritativeDNS: true,
43 SkipDNSDomainSuffixes: []string{".bsky.social"},43 SkipDNSDomainSuffixes: []string{".bsky.social"},
44 UserAgent: "glean/1.0",44 UserAgent: "glean/1.0",
45 }45 }
modified internal/atproto/collectiondir_test.go +2 -2
@@ -12,7 +12,7 @@ import (
1212
1313 func TestFetchSubscriberDIDs(t *testing.T) {
1414 page1 := struct {
15- Repos []struct {
15+ Repos []struct {
1616 DID string `json:"did"`
1717 } `json:"repos"`
1818 Cursor string `json:"cursor"`
@@ -26,7 +26,7 @@ func TestFetchSubscriberDIDs(t *testing.T) {
2626 Cursor: "nextpage",
2727 }
2828 page2 := struct {
29- Repos []struct {
29+ Repos []struct {
3030 DID string `json:"did"`
3131 } `json:"repos"`
3232 Cursor string `json:"cursor"`
@@ -12,7 +12,7 @@ import (
12 12
13 func TestFetchSubscriberDIDs(t *testing.T) {13 func TestFetchSubscriberDIDs(t *testing.T) {
14 page1 := struct {14 page1 := struct {
15- Repos []struct {15+ Repos []struct {
16 DID string `json:"did"`16 DID string `json:"did"`
17 } `json:"repos"`17 } `json:"repos"`
18 Cursor string `json:"cursor"`18 Cursor string `json:"cursor"`
@@ -26,7 +26,7 @@ func TestFetchSubscriberDIDs(t *testing.T) {
26 Cursor: "nextpage",26 Cursor: "nextpage",
27 }27 }
28 page2 := struct {28 page2 := struct {
29- Repos []struct {29+ Repos []struct {
30 DID string `json:"did"`30 DID string `json:"did"`
31 } `json:"repos"`31 } `json:"repos"`
32 Cursor string `json:"cursor"`32 Cursor string `json:"cursor"`
modified internal/atproto/lexicon.go +7 -6
@@ -8,12 +8,13 @@ import (
88 )
99
1010 const (
11- CollectionSubscription = "at.glean.subscription"
12- CollectionAnnotation = "at.glean.annotation"
13- CollectionLike = "at.glean.like"
14- CollectionMarginNote = "at.margin.note"
15- CollectionBskyFollow = "app.bsky.graph.follow"
16- CollectionTangledFollow = "sh.tangled.graph.follow"
11+ CollectionSubscription = "at.glean.subscription"
12+ CollectionAnnotation = "at.glean.annotation"
13+ CollectionLike = "at.glean.like"
14+ CollectionMarginNote = "at.margin.note"
15+ CollectionSkyreaderSubscription = "app.skyreader.feed.subscription"
16+ CollectionBskyFollow = "app.bsky.graph.follow"
17+ CollectionTangledFollow = "sh.tangled.graph.follow"
1718 )
1819
1920 type SubscriptionRecord struct {
@@ -8,12 +8,13 @@ import (
8 )8 )
9 9
10 const (10 const (
11- CollectionSubscription = "at.glean.subscription"11+ CollectionSubscription = "at.glean.subscription"
12- CollectionAnnotation = "at.glean.annotation"12+ CollectionAnnotation = "at.glean.annotation"
13- CollectionLike = "at.glean.like"13+ CollectionLike = "at.glean.like"
14- CollectionMarginNote = "at.margin.note"14+ CollectionMarginNote = "at.margin.note"
15- CollectionBskyFollow = "app.bsky.graph.follow"15+ CollectionSkyreaderSubscription = "app.skyreader.feed.subscription"
16- CollectionTangledFollow = "sh.tangled.graph.follow"16+ CollectionBskyFollow = "app.bsky.graph.follow"
17+ CollectionTangledFollow = "sh.tangled.graph.follow"
17 )18 )
18 19
19 type SubscriptionRecord struct {20 type SubscriptionRecord struct {
modified internal/atproto/lexicon_external.go +13 -5
@@ -11,6 +11,14 @@ type FollowRecord struct {
1111 Via json.RawMessage `json:"via,omitempty"`
1212 }
1313
14+// Note: app.skyreader.feed.subscription is not published, so thre is no tests that verifies this is correct. (ref: https://tangled.org/julien.rbrt.fr/glean/issues/3#comment-5760)
15+type SkyreaderSubscriptionRecord struct {
16+ CreatedAt string `json:"createdAt"`
17+ FeedURL string `json:"feedUrl"`
18+ Title string `json:"title,omitempty"`
19+ SiteURL string `json:"siteUrl,omitempty"`
20+}
21+
1422 type MarginNoteRecord struct {
1523 Body *MarginNoteBody `json:"body,omitempty"`
1624 Color string `json:"color,omitempty"`
@@ -49,11 +57,11 @@ type MarginNoteSelector struct {
4957 }
5058
5159 type MarginNoteTarget struct {
52- Selector *MarginNoteSelector `json:"selector,omitempty"`
53- Source string `json:"source"`
54- SourceHash string `json:"sourceHash,omitempty"`
55- State *MarginNoteTimeState `json:"state,omitempty"`
56- Title string `json:"title,omitempty"`
60+ Selector *MarginNoteSelector `json:"selector,omitempty"`
61+ Source string `json:"source"`
62+ SourceHash string `json:"sourceHash,omitempty"`
63+ State *MarginNoteTimeState `json:"state,omitempty"`
64+ Title string `json:"title,omitempty"`
5765 }
5866
5967 type MarginNoteTimeState struct {
@@ -11,6 +11,14 @@ type FollowRecord struct {
11 Via json.RawMessage `json:"via,omitempty"`11 Via json.RawMessage `json:"via,omitempty"`
12 }12 }
13 13
14+// Note: app.skyreader.feed.subscription is not published, so thre is no tests that verifies this is correct. (ref: https://tangled.org/julien.rbrt.fr/glean/issues/3#comment-5760)
15+type SkyreaderSubscriptionRecord struct {
16+ CreatedAt string `json:"createdAt"`
17+ FeedURL string `json:"feedUrl"`
18+ Title string `json:"title,omitempty"`
19+ SiteURL string `json:"siteUrl,omitempty"`
20+}
21+
14 type MarginNoteRecord struct {22 type MarginNoteRecord struct {
15 Body *MarginNoteBody `json:"body,omitempty"`23 Body *MarginNoteBody `json:"body,omitempty"`
16 Color string `json:"color,omitempty"`24 Color string `json:"color,omitempty"`
@@ -49,11 +57,11 @@ type MarginNoteSelector struct {
49 }57 }
50 58
51 type MarginNoteTarget struct {59 type MarginNoteTarget struct {
52- Selector *MarginNoteSelector `json:"selector,omitempty"`60+ Selector *MarginNoteSelector `json:"selector,omitempty"`
53- Source string `json:"source"`61+ Source string `json:"source"`
54- SourceHash string `json:"sourceHash,omitempty"`62+ SourceHash string `json:"sourceHash,omitempty"`
55- State *MarginNoteTimeState `json:"state,omitempty"`63+ State *MarginNoteTimeState `json:"state,omitempty"`
56- Title string `json:"title,omitempty"`64+ Title string `json:"title,omitempty"`
57 }65 }
58 66
59 type MarginNoteTimeState struct {67 type MarginNoteTimeState struct {
modified internal/atproto/stream_handler.go +39 -0
@@ -24,6 +24,8 @@ func (h *StreamDBHandler) Handle(ctx context.Context, event *Event) error {
2424 switch event.Collection {
2525 case CollectionSubscription:
2626 return h.handleSubscription(ctx, event)
27+ case CollectionSkyreaderSubscription:
28+ return h.handleSkyreaderSubscription(ctx, event)
2729 case CollectionLike:
2830 return h.handleLike(ctx, event)
2931 case CollectionAnnotation:
@@ -201,6 +203,43 @@ func (h *StreamDBHandler) handleMarginNote(ctx context.Context, event *Event) er
201203 return nil
202204 }
203205
206+func (h *StreamDBHandler) handleSkyreaderSubscription(ctx context.Context, event *Event) error {
207+ switch event.Type {
208+ case "create", "update":
209+ var rec SkyreaderSubscriptionRecord
210+ if err := json.Unmarshal(event.Value, &rec); err != nil {
211+ return err
212+ }
213+ if rec.FeedURL == "" {
214+ return nil
215+ }
216+
217+ existing, err := h.db.GetSubscription(ctx, event.DID, rec.FeedURL)
218+ if err == nil && existing != nil {
219+ if !existing.URI.Valid || existing.URI.String == "" {
220+ return h.db.UpdateSubscriptionURI(ctx, event.DID, rec.FeedURL, event.URI, event.CID)
221+ }
222+ return nil
223+ }
224+
225+ f := &db.Feed{FeedURL: rec.FeedURL, Title: db.NullStr(rec.Title), SiteURL: db.NullStr(rec.SiteURL)}
226+ _ = h.db.UpsertFeed(ctx, f)
227+ err = h.db.CreateSubscription(ctx, event.DID, rec.FeedURL, rec.Title, "", event.URI, event.CID)
228+ if errors.Is(err, db.ErrDuplicateSubscription) {
229+ return nil
230+ }
231+ return err
232+
233+ case "delete":
234+ // TODO: I actually don't think we should delete an subscription on Glean if deleted from Skyreader
235+ // sub, err := h.db.GetSubscriptionByURI(ctx, event.DID, event.URI)
236+ // if err == nil && sub != nil {
237+ // return h.db.DeleteSubscription(ctx, event.DID, sub.FeedURL)
238+ // }
239+ }
240+ return nil
241+}
242+
204243 func (h *StreamDBHandler) resolveFeedURL(ctx context.Context, articleURL string) string {
205244 article, err := h.db.GetArticleByURL(ctx, articleURL)
206245 if err != nil {
@@ -24,6 +24,8 @@ func (h *StreamDBHandler) Handle(ctx context.Context, event *Event) error {
24 switch event.Collection {24 switch event.Collection {
25 case CollectionSubscription:25 case CollectionSubscription:
26 return h.handleSubscription(ctx, event)26 return h.handleSubscription(ctx, event)
27+ case CollectionSkyreaderSubscription:
28+ return h.handleSkyreaderSubscription(ctx, event)
27 case CollectionLike:29 case CollectionLike:
28 return h.handleLike(ctx, event)30 return h.handleLike(ctx, event)
29 case CollectionAnnotation:31 case CollectionAnnotation:
@@ -201,6 +203,43 @@ func (h *StreamDBHandler) handleMarginNote(ctx context.Context, event *Event) er
201 return nil203 return nil
202 }204 }
203 205
206+func (h *StreamDBHandler) handleSkyreaderSubscription(ctx context.Context, event *Event) error {
207+ switch event.Type {
208+ case "create", "update":
209+ var rec SkyreaderSubscriptionRecord
210+ if err := json.Unmarshal(event.Value, &rec); err != nil {
211+ return err
212+ }
213+ if rec.FeedURL == "" {
214+ return nil
215+ }
216+
217+ existing, err := h.db.GetSubscription(ctx, event.DID, rec.FeedURL)
218+ if err == nil && existing != nil {
219+ if !existing.URI.Valid || existing.URI.String == "" {
220+ return h.db.UpdateSubscriptionURI(ctx, event.DID, rec.FeedURL, event.URI, event.CID)
221+ }
222+ return nil
223+ }
224+
225+ f := &db.Feed{FeedURL: rec.FeedURL, Title: db.NullStr(rec.Title), SiteURL: db.NullStr(rec.SiteURL)}
226+ _ = h.db.UpsertFeed(ctx, f)
227+ err = h.db.CreateSubscription(ctx, event.DID, rec.FeedURL, rec.Title, "", event.URI, event.CID)
228+ if errors.Is(err, db.ErrDuplicateSubscription) {
229+ return nil
230+ }
231+ return err
232+
233+ case "delete":
234+ // TODO: I actually don't think we should delete an subscription on Glean if deleted from Skyreader
235+ // sub, err := h.db.GetSubscriptionByURI(ctx, event.DID, event.URI)
236+ // if err == nil && sub != nil {
237+ // return h.db.DeleteSubscription(ctx, event.DID, sub.FeedURL)
238+ // }
239+ }
240+ return nil
241+}
242+
204 func (h *StreamDBHandler) resolveFeedURL(ctx context.Context, articleURL string) string {243 func (h *StreamDBHandler) resolveFeedURL(ctx context.Context, articleURL string) string {
205 article, err := h.db.GetArticleByURL(ctx, articleURL)244 article, err := h.db.GetArticleByURL(ctx, articleURL)
206 if err != nil {245 if err != nil {
modified internal/atproto/sync.go +31 -0
@@ -37,6 +37,9 @@ func (s *Sync) Run(ctx context.Context, userDID string) error {
3737 if err := s.syncCollection(ctx, userDID, CollectionSubscription, s.reconcileSubscription); err != nil {
3838 s.logger.Error("sync subscriptions failed", "error", err, "did", userDID)
3939 }
40+ if err := s.syncCollection(ctx, userDID, CollectionSkyreaderSubscription, s.reconcileSkyreaderSubscription); err != nil {
41+ s.logger.Error("sync skyreader subscriptions failed", "error", err, "did", userDID)
42+ }
4043 if err := s.syncCollection(ctx, userDID, CollectionLike, s.reconcileLike); err != nil {
4144 s.logger.Error("sync likes failed", "error", err, "did", userDID)
4245 }
@@ -105,6 +108,34 @@ func (s *Sync) reconcileSubscription(ctx context.Context, userDID, uri, cid stri
105108 return err
106109 }
107110
111+func (s *Sync) reconcileSkyreaderSubscription(ctx context.Context, userDID, uri, cid string, value json.RawMessage) error {
112+ var rec SkyreaderSubscriptionRecord
113+ if err := json.Unmarshal(value, &rec); err != nil {
114+ return err
115+ }
116+
117+ if rec.FeedURL == "" {
118+ return nil
119+ }
120+
121+ f := &db.Feed{FeedURL: rec.FeedURL, Title: db.NullStr(rec.Title), SiteURL: db.NullStr(rec.SiteURL)}
122+ _ = s.db.UpsertFeed(ctx, f)
123+
124+ existing, err := s.db.GetSubscription(ctx, userDID, rec.FeedURL)
125+ if err == nil && existing != nil {
126+ if !existing.URI.Valid || existing.URI.String == "" {
127+ return s.db.UpdateSubscriptionURI(ctx, userDID, rec.FeedURL, uri, cid)
128+ }
129+ return nil
130+ }
131+
132+ err = s.db.CreateSubscription(ctx, userDID, rec.FeedURL, rec.Title, "", uri, cid)
133+ if errors.Is(err, db.ErrDuplicateSubscription) {
134+ return nil
135+ }
136+ return err
137+}
138+
108139 func (s *Sync) reconcileLike(ctx context.Context, userDID, uri, cid string, value json.RawMessage) error {
109140 var rec LikeRecord
110141 if err := json.Unmarshal(value, &rec); err != nil {
@@ -37,6 +37,9 @@ func (s *Sync) Run(ctx context.Context, userDID string) error {
37 if err := s.syncCollection(ctx, userDID, CollectionSubscription, s.reconcileSubscription); err != nil {37 if err := s.syncCollection(ctx, userDID, CollectionSubscription, s.reconcileSubscription); err != nil {
38 s.logger.Error("sync subscriptions failed", "error", err, "did", userDID)38 s.logger.Error("sync subscriptions failed", "error", err, "did", userDID)
39 }39 }
40+ if err := s.syncCollection(ctx, userDID, CollectionSkyreaderSubscription, s.reconcileSkyreaderSubscription); err != nil {
41+ s.logger.Error("sync skyreader subscriptions failed", "error", err, "did", userDID)
42+ }
40 if err := s.syncCollection(ctx, userDID, CollectionLike, s.reconcileLike); err != nil {43 if err := s.syncCollection(ctx, userDID, CollectionLike, s.reconcileLike); err != nil {
41 s.logger.Error("sync likes failed", "error", err, "did", userDID)44 s.logger.Error("sync likes failed", "error", err, "did", userDID)
42 }45 }
@@ -105,6 +108,34 @@ func (s *Sync) reconcileSubscription(ctx context.Context, userDID, uri, cid stri
105 return err108 return err
106 }109 }
107 110
111+func (s *Sync) reconcileSkyreaderSubscription(ctx context.Context, userDID, uri, cid string, value json.RawMessage) error {
112+ var rec SkyreaderSubscriptionRecord
113+ if err := json.Unmarshal(value, &rec); err != nil {
114+ return err
115+ }
116+
117+ if rec.FeedURL == "" {
118+ return nil
119+ }
120+
121+ f := &db.Feed{FeedURL: rec.FeedURL, Title: db.NullStr(rec.Title), SiteURL: db.NullStr(rec.SiteURL)}
122+ _ = s.db.UpsertFeed(ctx, f)
123+
124+ existing, err := s.db.GetSubscription(ctx, userDID, rec.FeedURL)
125+ if err == nil && existing != nil {
126+ if !existing.URI.Valid || existing.URI.String == "" {
127+ return s.db.UpdateSubscriptionURI(ctx, userDID, rec.FeedURL, uri, cid)
128+ }
129+ return nil
130+ }
131+
132+ err = s.db.CreateSubscription(ctx, userDID, rec.FeedURL, rec.Title, "", uri, cid)
133+ if errors.Is(err, db.ErrDuplicateSubscription) {
134+ return nil
135+ }
136+ return err
137+}
138+
108 func (s *Sync) reconcileLike(ctx context.Context, userDID, uri, cid string, value json.RawMessage) error {139 func (s *Sync) reconcileLike(ctx context.Context, userDID, uri, cid string, value json.RawMessage) error {
109 var rec LikeRecord140 var rec LikeRecord
110 if err := json.Unmarshal(value, &rec); err != nil {141 if err := json.Unmarshal(value, &rec); err != nil {
modified internal/server/feeds_handler.go +5 -5
@@ -128,11 +128,11 @@ func (s *Server) handleAddFeed(w http.ResponseWriter, r *http.Request) {
128128 return
129129 }
130130 s.render(w, r, "feed-item.html", map[string]any{
131- "User": user,
132- "FeedURL": sub.FeedURL,
133- "FeedTitle": sub.FeedTitle,
134- "Category": sub.Category,
135- "FaviconURL": sub.FaviconURL,
131+ "User": user,
132+ "FeedURL": sub.FeedURL,
133+ "FeedTitle": sub.FeedTitle,
134+ "Category": sub.Category,
135+ "FaviconURL": sub.FaviconURL,
136136 "UnreadCount": sub.UnreadCount,
137137 })
138138 }
@@ -128,11 +128,11 @@ func (s *Server) handleAddFeed(w http.ResponseWriter, r *http.Request) {
128 return128 return
129 }129 }
130 s.render(w, r, "feed-item.html", map[string]any{130 s.render(w, r, "feed-item.html", map[string]any{
131- "User": user,131+ "User": user,
132- "FeedURL": sub.FeedURL,132+ "FeedURL": sub.FeedURL,
133- "FeedTitle": sub.FeedTitle,133+ "FeedTitle": sub.FeedTitle,
134- "Category": sub.Category,134+ "Category": sub.Category,
135- "FaviconURL": sub.FaviconURL,135+ "FaviconURL": sub.FaviconURL,
136 "UnreadCount": sub.UnreadCount,136 "UnreadCount": sub.UnreadCount,
137 })137 })
138 }138 }