rickub
nandi/gleanpublic⑂ Fork 0
⑂ af6a931
Branches
main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/glean.git
git clone ssh://git@rickub.com/nandi/glean.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Improve docsUnverified

Julien Robert committed 2026-04-24T02:29:40+02:00 Browse files
af6a931 parent: 8943a72
modified .env.example +3 -3
@@ -3,9 +3,9 @@ GLEAN_DB=glean.db
33 GLEAN_SESSION_KEY=change-me-to-a-random-string
44 GLEAN_JETSTREAM=wss://jetstream.glean.at
55 GLEAN_PLC_URL=https://didplc.glean.at
6-GLEAN_SYNC_INTERVAL=10m
7-GLEAN_CLUSTER_INTERVAL=15m
8-GLEAN_FETCH_INTERVAL=5m
6+GLEAN_SYNC_INTERVAL=30m
7+GLEAN_CLUSTER_INTERVAL=60m
8+GLEAN_FETCH_INTERVAL=15m
99 GLEAN_COLLECTION_DIR_URL=https://lightrail.microcosm.blue/xrpc/com.atproto.sync.listReposByCollection?collection=at.glean.subscription
1010 GLEAN_BACKFILL_CONCURRENCY=5
1111 # Leave empty for localhost OAuth (development)
@@ -3,9 +3,9 @@ GLEAN_DB=glean.db
3 GLEAN_SESSION_KEY=change-me-to-a-random-string3 GLEAN_SESSION_KEY=change-me-to-a-random-string
4 GLEAN_JETSTREAM=wss://jetstream.glean.at4 GLEAN_JETSTREAM=wss://jetstream.glean.at
5 GLEAN_PLC_URL=https://didplc.glean.at5 GLEAN_PLC_URL=https://didplc.glean.at
6-GLEAN_SYNC_INTERVAL=10m6+GLEAN_SYNC_INTERVAL=30m
7-GLEAN_CLUSTER_INTERVAL=15m7+GLEAN_CLUSTER_INTERVAL=60m
8-GLEAN_FETCH_INTERVAL=5m8+GLEAN_FETCH_INTERVAL=15m
9 GLEAN_COLLECTION_DIR_URL=https://lightrail.microcosm.blue/xrpc/com.atproto.sync.listReposByCollection?collection=at.glean.subscription9 GLEAN_COLLECTION_DIR_URL=https://lightrail.microcosm.blue/xrpc/com.atproto.sync.listReposByCollection?collection=at.glean.subscription
10 GLEAN_BACKFILL_CONCURRENCY=510 GLEAN_BACKFILL_CONCURRENCY=5
11 # Leave empty for localhost OAuth (development)11 # Leave empty for localhost OAuth (development)
modified docs/design.md +1 -2
@@ -167,6 +167,5 @@ All custom colors live under the `spot` namespace in `tailwind.config.js`. CSS v
167167 | Article Detail | `max-w-3xl` centered | Content, like/share/read buttons, annotations |
168168 | Feeds | 2/3 + 1/3 grid | Feed list with categories + add/import sidebar, refresh button |
169169 | Trending | Full-width list | Like/annotation counts on each article |
170-| Discover | Mixed grid | Recommendations + people + browse all |
171-| Annotations | Full-width list | Filter by article URL, load more |
170+| Library | Full-width list | Liked articles and annotations |
172171 | Profile | `max-w-2xl` centered | Avatar, stats, feeds, annotations |
@@ -167,6 +167,5 @@ All custom colors live under the `spot` namespace in `tailwind.config.js`. CSS v
167 | Article Detail | `max-w-3xl` centered | Content, like/share/read buttons, annotations |167 | Article Detail | `max-w-3xl` centered | Content, like/share/read buttons, annotations |
168 | Feeds | 2/3 + 1/3 grid | Feed list with categories + add/import sidebar, refresh button |168 | Feeds | 2/3 + 1/3 grid | Feed list with categories + add/import sidebar, refresh button |
169 | Trending | Full-width list | Like/annotation counts on each article |169 | Trending | Full-width list | Like/annotation counts on each article |
170-| Discover | Mixed grid | Recommendations + people + browse all |170+| Library | Full-width list | Liked articles and annotations |
171-| Annotations | Full-width list | Filter by article URL, load more |
172 | Profile | `max-w-2xl` centered | Avatar, stats, feeds, annotations |171 | Profile | `max-w-2xl` centered | Avatar, stats, feeds, annotations |
modified docs/specs.md +48 -50
@@ -298,7 +298,7 @@ A background scheduler polls subscribed feeds on a configurable tick. Feeds are
298298
299299 The scheduler uses a configurable tick interval with in-flight deduplication:
300300
301-- **Tick interval**: The scheduler checks for stale feeds every `GLEAN_FETCH_INTERVAL` (default 5 minutes)
301+- **Tick interval**: The scheduler checks for stale feeds every `GLEAN_FETCH_INTERVAL` (default 15 minutes)
302302 - **Staleness threshold**: Feeds not fetched in the last 30 minutes are eligible
303303 - **Subscriber filter**: Only feeds with `subscriber_count > 0` are fetched
304304 - **In-flight dedup**: If a feed is already being fetched (e.g., manual refresh and background scheduler overlap), the second caller waits for the first to complete rather than fetching again
@@ -395,7 +395,7 @@ Beyond the clustering system, Glean also discovers new feeds from article conten
395395
396396 - **Auto-discovery**: When fetching a feed, parse `<link rel="alternate" type="application/rss+xml">` from the feed's site URL to discover related feeds
397397 - **Feedfavicon**: Fetch `favicon.ico` or `/apple-touch-icon.png` from the feed's site URL for display
398-- **Dead feed detection**: If a feed fails for 7 consecutive fetches (14 days at base interval), mark it as dead. Notify the user and offer to remove it.
398+- **Dead feed detection**: If a feed fails for 7 consecutive fetches, mark it as dead. Notify the user and offer to remove it.
399399
400400 ## 5. System Architecture
401401
@@ -668,7 +668,7 @@ Feed description text similarity is also computed (word overlap after stopword r
668668 For any two users, compute Jaccard over their subscription sets, plus like co-occurrence (time-decayed) and tag overlap:
669669
670670 ```
671-J(U1, U2) = jaccard_subscriptions + 0.3 * jaccard_likes + 0.2 * jaccard_tags + follow_boost
671+J(U1, U2) = jaccard_subscriptions + 0.3 * jaccard_likes + 0.2 * jaccard_tags + 0.5 * follow_boost
672672 ```
673673
674674 Like overlap uses exponential time decay: `EXP(-0.023 * age_days)` (30-day half-life).
@@ -818,34 +818,34 @@ The server renders HTML fragments that htmx swaps into the page. No JSON API nee
818818
819819 ### 8.1 Pages
820820
821-| Route | Method | Description |
822-| ------------------------------ | ------ | -------------------------------------------------------- |
823-| `/` | GET | Landing page / auth redirect |
824-| `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar |
825-| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |
826-| `/feeds/list` | GET | Feed list fragment (htmx partial) |
827-| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions |
828-| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |
829-| `/feeds/add` | POST | Add a single feed URL |
830-| `/feeds/remove` | DELETE | Remove a feed |
831-| `/feeds/refresh` | POST | Refresh all subscribed feeds |
832-| `/feeds/retry` | POST | Retry a failed feed |
833-| `/feeds/clear` | POST | Clear all subscriptions |
834-| `/feeds/dismiss` | POST | Dismiss a feed recommendation |
835-| `/articles` | GET | Read articles (paginated, filterable by feed) |
836-| `/articles/new-count` | GET | Get count of new articles (for badge updates) |
837-| `/articles/{id}` | GET | Article detail view |
838-| `/articles/{id}/read` | POST | Mark article as read |
839-| `/articles/{id}/unread` | POST | Mark article as unread |
840-| `/articles/{id}/like` | POST | Like an article |
841-| `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL |
842-| `/articles/mark-all-read` | POST | Mark all articles as read |
843-| `/articles/dismiss` | POST | Dismiss an article recommendation |
844-| `/trending` | GET | Community feed: articles ranked by likes |
845-| `/library` | GET | Liked articles and annotations |
846-| `/library/create` | POST | Create annotation on an article |
847-| `/library/{id}/delete` | POST | Delete an annotation |
848-| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |
821+| Route | Method | Description |
822+| ------------------------------ | ------ | ------------------------------------------------------------------- |
823+| `/` | GET | Landing page / auth redirect |
824+| `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar |
825+| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |
826+| `/feeds/list` | GET | Feed list fragment (htmx partial) |
827+| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions (redirects to /feeds) |
828+| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |
829+| `/feeds/add` | POST | Add a single feed URL |
830+| `/feeds/remove` | DELETE | Remove a feed |
831+| `/feeds/refresh` | POST | Refresh all subscribed feeds |
832+| `/feeds/retry` | POST | Retry a failed feed |
833+| `/feeds/clear` | POST | Clear all subscriptions |
834+| `/feeds/dismiss` | POST | Dismiss a feed recommendation |
835+| `/articles` | GET | Read articles (paginated, filterable by feed) |
836+| `/articles/new-count` | GET | Get count of new articles (for badge updates) |
837+| `/articles/{id}` | GET | Article detail view |
838+| `/articles/{id}/read` | POST | Mark article as read |
839+| `/articles/{id}/unread` | POST | Mark article as unread |
840+| `/articles/{id}/like` | POST | Like an article |
841+| `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL |
842+| `/articles/mark-all-read` | POST | Mark all articles as read |
843+| `/articles/dismiss` | POST | Dismiss an article recommendation |
844+| `/trending` | GET | Community feed: articles ranked by likes |
845+| `/library` | GET | Liked articles and annotations |
846+| `/library/create` | POST | Create annotation on an article |
847+| `/library/{id}/delete` | POST | Delete an annotation |
848+| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |
849849
850850 ### 8.2 htmx Patterns
851851
@@ -881,14 +881,12 @@ glean/
881881 │ │ ├── sync.go # PDS record reconciliation
882882 │ │ └── xrpc.go # XRPC query handlers (AppView endpoints)
883883 │ ├── db/
884-│ │ ├── db.go # SQLite connection, single-DB schema
885-│ │ ├── multi.go # Multi-DB setup with ATTACH for cross-database queries
884+│ │ ├── db.go # SQLite connection with ATTACH for cross-database queries
886885 │ │ ├── user.go # User queries
887886 │ │ ├── feed.go # Feed + subscription queries
888887 │ │ ├── article.go # Article queries
889888 │ │ ├── social.go # Like, annotation queries
890889 │ │ ├── follow.go # Follow queries
891-│ │ ├── cluster.go # Similarity + recommendation queries
892890 │ │ ├── oauth_store.go # OAuth session storage
893891 │ │ └── store.go # FeedStore adapter for scheduler
894892 │ ├── feed/
@@ -904,8 +902,7 @@ glean/
904902 │ │ └── metrics.go # Prometheus metrics definitions
905903 │ ├── cluster/
906904 │ │ ├── jaccard.go # Jaccard similarity computation
907-│ │ ├── recommender.go # Feed + people recommendation queries (on-demand)
908-│ │ ├── scoring.go # Multi-signal composite scoring queries
905+│ │ ├── scoring.go # Feed + people + article recommendation queries (on-demand)
909906 │ │ ├── social.go # Follow-distance computation (1-2 hop)
910907 │ │ ├── dismiss.go # Dismiss + impression tracking
911908 │ │ ├── weights.go # Bandit-style signal weight auto-tuning
@@ -976,8 +973,8 @@ Browser ──POST /feeds/opml/upload──► Server
976973 ├─► Fetch each feed, validate + store in `feeds` table
977974 ├─► For each feed, create an `at.glean.subscription` record
978975 │ via XRPC write to user's PDS
979- ├─► Insert subscriptions in local `subscriptions` table
980- └─◄ Return updated feed list fragment (htmx)
976+ ├─► Insert subscriptions in local `subscriptions` table
977+ └─◄ Redirect to `/feeds`
981978 ```
982979
983980 ### 11.2 Reading the Feed
@@ -993,17 +990,19 @@ Browser ──GET /articles──► Server
993990 ### 11.3 Recommendations
994991
995992 ```
996-Cron (every 6h) ──► Cluster Engine
997- │
998- ├─► SELECT user similarity pairs
999- ├─► Compute recommendation scores
1000- └─► INSERT into user_feed_recommendations
1001-
1002-Browser ──GET /discover/feeds──► Server
1003- │
1004- ├─► SELECT from user_feed_recommendations
1005- ├─► Fetch feed metadata
1006- └─◄ Render recommendation cards (htmx)
993+Cron (every 10m) ──► Cluster Engine
994+ │
995+ ├─► Compute feed similarity
996+ ├─► Compute user similarity
997+ ├─► Compute follow distances
998+ ├─► Compute signal profiles
999+ └─► Auto-dismiss stale recommendations
1000+
1001+Browser ──GET /dashboard──► Server
1002+ │
1003+ ├─► Compute recommendations on-demand
1004+ ├─► Fetch feed metadata
1005+ └─◄ Render recommendation cards (htmx)
10071006 ```
10081007
10091008 ## 12. Key Design Decisions
@@ -1035,7 +1034,6 @@ Glean exposes a `/metrics` endpoint for monitoring. Key metrics:
10351034 - **`glean_jetstream_reconnects_total`** — Jetstream reconnection count
10361035 - **`glean_http_requests_total`** — HTTP request counts labeled by method, path, and status
10371036 - **`glean_http_request_duration_seconds`** — HTTP request duration labeled by method and path
1038-- **`glean_users_active_total`** — Number of users with active sessions
10391037 - **`glean_pds_sync_runs_total`** / **`glean_pds_sync_errors_total`** — PDS sync runs and errors
10401038 - **`glean_cluster_runs_total`** / **`glean_cluster_duration_seconds`** — Recommendation engine runs and timing
10411039
@@ -298,7 +298,7 @@ A background scheduler polls subscribed feeds on a configurable tick. Feeds are
298 298
299 The scheduler uses a configurable tick interval with in-flight deduplication:299 The scheduler uses a configurable tick interval with in-flight deduplication:
300 300
301-- **Tick interval**: The scheduler checks for stale feeds every `GLEAN_FETCH_INTERVAL` (default 5 minutes)301+- **Tick interval**: The scheduler checks for stale feeds every `GLEAN_FETCH_INTERVAL` (default 15 minutes)
302 - **Staleness threshold**: Feeds not fetched in the last 30 minutes are eligible302 - **Staleness threshold**: Feeds not fetched in the last 30 minutes are eligible
303 - **Subscriber filter**: Only feeds with `subscriber_count > 0` are fetched303 - **Subscriber filter**: Only feeds with `subscriber_count > 0` are fetched
304 - **In-flight dedup**: If a feed is already being fetched (e.g., manual refresh and background scheduler overlap), the second caller waits for the first to complete rather than fetching again304 - **In-flight dedup**: If a feed is already being fetched (e.g., manual refresh and background scheduler overlap), the second caller waits for the first to complete rather than fetching again
@@ -395,7 +395,7 @@ Beyond the clustering system, Glean also discovers new feeds from article conten
395 395
396 - **Auto-discovery**: When fetching a feed, parse `<link rel="alternate" type="application/rss+xml">` from the feed's site URL to discover related feeds396 - **Auto-discovery**: When fetching a feed, parse `<link rel="alternate" type="application/rss+xml">` from the feed's site URL to discover related feeds
397 - **Feedfavicon**: Fetch `favicon.ico` or `/apple-touch-icon.png` from the feed's site URL for display397 - **Feedfavicon**: Fetch `favicon.ico` or `/apple-touch-icon.png` from the feed's site URL for display
398-- **Dead feed detection**: If a feed fails for 7 consecutive fetches (14 days at base interval), mark it as dead. Notify the user and offer to remove it.398+- **Dead feed detection**: If a feed fails for 7 consecutive fetches, mark it as dead. Notify the user and offer to remove it.
399 399
400 ## 5. System Architecture400 ## 5. System Architecture
401 401
@@ -668,7 +668,7 @@ Feed description text similarity is also computed (word overlap after stopword r
668 For any two users, compute Jaccard over their subscription sets, plus like co-occurrence (time-decayed) and tag overlap:668 For any two users, compute Jaccard over their subscription sets, plus like co-occurrence (time-decayed) and tag overlap:
669 669
670 ```670 ```
671-J(U1, U2) = jaccard_subscriptions + 0.3 * jaccard_likes + 0.2 * jaccard_tags + follow_boost671+J(U1, U2) = jaccard_subscriptions + 0.3 * jaccard_likes + 0.2 * jaccard_tags + 0.5 * follow_boost
672 ```672 ```
673 673
674 Like overlap uses exponential time decay: `EXP(-0.023 * age_days)` (30-day half-life).674 Like overlap uses exponential time decay: `EXP(-0.023 * age_days)` (30-day half-life).
@@ -818,34 +818,34 @@ The server renders HTML fragments that htmx swaps into the page. No JSON API nee
818 818
819 ### 8.1 Pages819 ### 8.1 Pages
820 820
821-| Route | Method | Description |821+| Route | Method | Description |
822-| ------------------------------ | ------ | -------------------------------------------------------- |822+| ------------------------------ | ------ | ------------------------------------------------------------------- |
823-| `/` | GET | Landing page / auth redirect |823+| `/` | GET | Landing page / auth redirect |
824-| `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar |824+| `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar |
825-| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |825+| `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) |
826-| `/feeds/list` | GET | Feed list fragment (htmx partial) |826+| `/feeds/list` | GET | Feed list fragment (htmx partial) |
827-| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions |827+| `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions (redirects to /feeds) |
828-| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |828+| `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) |
829-| `/feeds/add` | POST | Add a single feed URL |829+| `/feeds/add` | POST | Add a single feed URL |
830-| `/feeds/remove` | DELETE | Remove a feed |830+| `/feeds/remove` | DELETE | Remove a feed |
831-| `/feeds/refresh` | POST | Refresh all subscribed feeds |831+| `/feeds/refresh` | POST | Refresh all subscribed feeds |
832-| `/feeds/retry` | POST | Retry a failed feed |832+| `/feeds/retry` | POST | Retry a failed feed |
833-| `/feeds/clear` | POST | Clear all subscriptions |833+| `/feeds/clear` | POST | Clear all subscriptions |
834-| `/feeds/dismiss` | POST | Dismiss a feed recommendation |834+| `/feeds/dismiss` | POST | Dismiss a feed recommendation |
835-| `/articles` | GET | Read articles (paginated, filterable by feed) |835+| `/articles` | GET | Read articles (paginated, filterable by feed) |
836-| `/articles/new-count` | GET | Get count of new articles (for badge updates) |836+| `/articles/new-count` | GET | Get count of new articles (for badge updates) |
837-| `/articles/{id}` | GET | Article detail view |837+| `/articles/{id}` | GET | Article detail view |
838-| `/articles/{id}/read` | POST | Mark article as read |838+| `/articles/{id}/read` | POST | Mark article as read |
839-| `/articles/{id}/unread` | POST | Mark article as unread |839+| `/articles/{id}/unread` | POST | Mark article as unread |
840-| `/articles/{id}/like` | POST | Like an article |840+| `/articles/{id}/like` | POST | Like an article |
841-| `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL |841+| `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL |
842-| `/articles/mark-all-read` | POST | Mark all articles as read |842+| `/articles/mark-all-read` | POST | Mark all articles as read |
843-| `/articles/dismiss` | POST | Dismiss an article recommendation |843+| `/articles/dismiss` | POST | Dismiss an article recommendation |
844-| `/trending` | GET | Community feed: articles ranked by likes |844+| `/trending` | GET | Community feed: articles ranked by likes |
845-| `/library` | GET | Liked articles and annotations |845+| `/library` | GET | Liked articles and annotations |
846-| `/library/create` | POST | Create annotation on an article |846+| `/library/create` | POST | Create annotation on an article |
847-| `/library/{id}/delete` | POST | Delete an annotation |847+| `/library/{id}/delete` | POST | Delete an annotation |
848-| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |848+| `/profile/{did}` | GET | Public profile: their feeds, likes, annotations |
849 849
850 ### 8.2 htmx Patterns850 ### 8.2 htmx Patterns
851 851
@@ -881,14 +881,12 @@ glean/
881 │ │ ├── sync.go # PDS record reconciliation881 │ │ ├── sync.go # PDS record reconciliation
882 │ │ └── xrpc.go # XRPC query handlers (AppView endpoints)882 │ │ └── xrpc.go # XRPC query handlers (AppView endpoints)
883 │ ├── db/883 │ ├── db/
884-│ │ ├── db.go # SQLite connection, single-DB schema884+│ │ ├── db.go # SQLite connection with ATTACH for cross-database queries
885-│ │ ├── multi.go # Multi-DB setup with ATTACH for cross-database queries
886 │ │ ├── user.go # User queries885 │ │ ├── user.go # User queries
887 │ │ ├── feed.go # Feed + subscription queries886 │ │ ├── feed.go # Feed + subscription queries
888 │ │ ├── article.go # Article queries887 │ │ ├── article.go # Article queries
889 │ │ ├── social.go # Like, annotation queries888 │ │ ├── social.go # Like, annotation queries
890 │ │ ├── follow.go # Follow queries889 │ │ ├── follow.go # Follow queries
891-│ │ ├── cluster.go # Similarity + recommendation queries
892 │ │ ├── oauth_store.go # OAuth session storage890 │ │ ├── oauth_store.go # OAuth session storage
893 │ │ └── store.go # FeedStore adapter for scheduler891 │ │ └── store.go # FeedStore adapter for scheduler
894 │ ├── feed/892 │ ├── feed/
@@ -904,8 +902,7 @@ glean/
904 │ │ └── metrics.go # Prometheus metrics definitions902 │ │ └── metrics.go # Prometheus metrics definitions
905 │ ├── cluster/903 │ ├── cluster/
906 │ │ ├── jaccard.go # Jaccard similarity computation904 │ │ ├── jaccard.go # Jaccard similarity computation
907-│ │ ├── recommender.go # Feed + people recommendation queries (on-demand)905+│ │ ├── scoring.go # Feed + people + article recommendation queries (on-demand)
908-│ │ ├── scoring.go # Multi-signal composite scoring queries
909 │ │ ├── social.go # Follow-distance computation (1-2 hop)906 │ │ ├── social.go # Follow-distance computation (1-2 hop)
910 │ │ ├── dismiss.go # Dismiss + impression tracking907 │ │ ├── dismiss.go # Dismiss + impression tracking
911 │ │ ├── weights.go # Bandit-style signal weight auto-tuning908 │ │ ├── weights.go # Bandit-style signal weight auto-tuning
@@ -976,8 +973,8 @@ Browser ──POST /feeds/opml/upload──► Server
976 ├─► Fetch each feed, validate + store in `feeds` table973 ├─► Fetch each feed, validate + store in `feeds` table
977 ├─► For each feed, create an `at.glean.subscription` record974 ├─► For each feed, create an `at.glean.subscription` record
978 │ via XRPC write to user's PDS975 │ via XRPC write to user's PDS
979- ├─► Insert subscriptions in local `subscriptions` table976+ ├─► Insert subscriptions in local `subscriptions` table
980- └─◄ Return updated feed list fragment (htmx)977+ └─◄ Redirect to `/feeds`
981 ```978 ```
982 979
983 ### 11.2 Reading the Feed980 ### 11.2 Reading the Feed
@@ -993,17 +990,19 @@ Browser ──GET /articles──► Server
993 ### 11.3 Recommendations990 ### 11.3 Recommendations
994 991
995 ```992 ```
996-Cron (every 6h) ──► Cluster Engine993+Cron (every 10m) ──► Cluster Engine
997- │994+ │
998- ├─► SELECT user similarity pairs995+ ├─► Compute feed similarity
999- ├─► Compute recommendation scores996+ ├─► Compute user similarity
1000- └─► INSERT into user_feed_recommendations997+ ├─► Compute follow distances
1001-998+ ├─► Compute signal profiles
1002-Browser ──GET /discover/feeds──► Server999+ └─► Auto-dismiss stale recommendations
1003- │1000+
1004- ├─► SELECT from user_feed_recommendations1001+Browser ──GET /dashboard──► Server
1005- ├─► Fetch feed metadata1002+ │
1006- └─◄ Render recommendation cards (htmx)1003+ ├─► Compute recommendations on-demand
1004+ ├─► Fetch feed metadata
1005+ └─◄ Render recommendation cards (htmx)
1007 ```1006 ```
1008 1007
1009 ## 12. Key Design Decisions1008 ## 12. Key Design Decisions
@@ -1035,7 +1034,6 @@ Glean exposes a `/metrics` endpoint for monitoring. Key metrics:
1035 - **`glean_jetstream_reconnects_total`** — Jetstream reconnection count1034 - **`glean_jetstream_reconnects_total`** — Jetstream reconnection count
1036 - **`glean_http_requests_total`** — HTTP request counts labeled by method, path, and status1035 - **`glean_http_requests_total`** — HTTP request counts labeled by method, path, and status
1037 - **`glean_http_request_duration_seconds`** — HTTP request duration labeled by method and path1036 - **`glean_http_request_duration_seconds`** — HTTP request duration labeled by method and path
1038-- **`glean_users_active_total`** — Number of users with active sessions
1039 - **`glean_pds_sync_runs_total`** / **`glean_pds_sync_errors_total`** — PDS sync runs and errors1037 - **`glean_pds_sync_runs_total`** / **`glean_pds_sync_errors_total`** — PDS sync runs and errors
1040 - **`glean_cluster_runs_total`** / **`glean_cluster_duration_seconds`** — Recommendation engine runs and timing1038 - **`glean_cluster_runs_total`** / **`glean_cluster_duration_seconds`** — Recommendation engine runs and timing
1041 1039
modified main.go +3 -3
@@ -23,9 +23,9 @@ func main() {
2323 addr := flag.String("addr", envOr("GLEAN_ADDR", ":8080"), "listen address")
2424 dbPath := flag.String("db", envOr("GLEAN_DB", "glean.db"), "database path")
2525 jetstreamURL := flag.String("jetstream", envOr("GLEAN_JETSTREAM", "wss://jetstream.glean.at"), "Jetstream URL")
26- syncInterval := flag.Duration("sync-interval", envDuration("GLEAN_SYNC_INTERVAL", 1*time.Hour), "PDS sync interval")
27- clusterInterval := flag.Duration("cluster-interval", envDuration("GLEAN_CLUSTER_INTERVAL", 10*time.Minute), "cluster recomputation interval")
28- fetchInterval := flag.Duration("fetch-interval", envDuration("GLEAN_FETCH_INTERVAL", 5*time.Minute), "feed fetch tick interval")
26+ syncInterval := flag.Duration("sync-interval", envDuration("GLEAN_SYNC_INTERVAL", 30*time.Minute), "PDS sync interval")
27+ clusterInterval := flag.Duration("cluster-interval", envDuration("GLEAN_CLUSTER_INTERVAL", 1*time.Hour), "cluster recomputation interval")
28+ fetchInterval := flag.Duration("fetch-interval", envDuration("GLEAN_FETCH_INTERVAL", 15*time.Minute), "feed fetch tick interval")
2929 collectionDirURL := flag.String("collection-dir", envOr("GLEAN_COLLECTION_DIR_URL", ""), "collection directory URL for startup backfill")
3030 backfillConcurrency := flag.Int("backfill-concurrency", envInt("GLEAN_BACKFILL_CONCURRENCY", 5), "max concurrent backfill workers")
3131 sessionKey := envOr("GLEAN_SESSION_KEY", "")
@@ -23,9 +23,9 @@ func main() {
23 addr := flag.String("addr", envOr("GLEAN_ADDR", ":8080"), "listen address")23 addr := flag.String("addr", envOr("GLEAN_ADDR", ":8080"), "listen address")
24 dbPath := flag.String("db", envOr("GLEAN_DB", "glean.db"), "database path")24 dbPath := flag.String("db", envOr("GLEAN_DB", "glean.db"), "database path")
25 jetstreamURL := flag.String("jetstream", envOr("GLEAN_JETSTREAM", "wss://jetstream.glean.at"), "Jetstream URL")25 jetstreamURL := flag.String("jetstream", envOr("GLEAN_JETSTREAM", "wss://jetstream.glean.at"), "Jetstream URL")
26- syncInterval := flag.Duration("sync-interval", envDuration("GLEAN_SYNC_INTERVAL", 1*time.Hour), "PDS sync interval")26+ syncInterval := flag.Duration("sync-interval", envDuration("GLEAN_SYNC_INTERVAL", 30*time.Minute), "PDS sync interval")
27- clusterInterval := flag.Duration("cluster-interval", envDuration("GLEAN_CLUSTER_INTERVAL", 10*time.Minute), "cluster recomputation interval")27+ clusterInterval := flag.Duration("cluster-interval", envDuration("GLEAN_CLUSTER_INTERVAL", 1*time.Hour), "cluster recomputation interval")
28- fetchInterval := flag.Duration("fetch-interval", envDuration("GLEAN_FETCH_INTERVAL", 5*time.Minute), "feed fetch tick interval")28+ fetchInterval := flag.Duration("fetch-interval", envDuration("GLEAN_FETCH_INTERVAL", 15*time.Minute), "feed fetch tick interval")
29 collectionDirURL := flag.String("collection-dir", envOr("GLEAN_COLLECTION_DIR_URL", ""), "collection directory URL for startup backfill")29 collectionDirURL := flag.String("collection-dir", envOr("GLEAN_COLLECTION_DIR_URL", ""), "collection directory URL for startup backfill")
30 backfillConcurrency := flag.Int("backfill-concurrency", envInt("GLEAN_BACKFILL_CONCURRENCY", 5), "max concurrent backfill workers")30 backfillConcurrency := flag.Int("backfill-concurrency", envInt("GLEAN_BACKFILL_CONCURRENCY", 5), "max concurrent backfill workers")
31 sessionKey := envOr("GLEAN_SESSION_KEY", "")31 sessionKey := envOr("GLEAN_SESSION_KEY", "")
modified readme.md +14 -14
@@ -37,20 +37,20 @@ Then open `http://localhost:8080`.
3737
3838 ## Configuration
3939
40-| Variable | Default | What it does |
41-| -------------------------- | -------------------------- | --------------------------------------------------------- |
42-| `GLEAN_SESSION_KEY` | _(required)_ | Secret key for signing session cookies (any random string) |
43-| `GLEAN_ADDR` | `:8080` | Listen address |
44-| `GLEAN_DB` | `glean.db` | SQLite base path (`_users`, `_articles`, `_recs` suffixes) |
45-| `GLEAN_JETSTREAM` | `wss://jetstream.glean.at` | Jetstream WebSocket URL |
46-| `GLEAN_SYNC_INTERVAL` | `1h` | PDS sync interval (Go duration: `30m`, `2h30m`, etc.) |
47-| `GLEAN_CLUSTER_INTERVAL` | `10m` | Cluster recomputation interval (Go duration) |
48-| `GLEAN_FETCH_INTERVAL` | `5m` | Feed fetch scheduler tick interval (Go duration) |
49-| `GLEAN_COLLECTION_DIR_URL` | _(empty)_ | Collection directory URL for startup backfill |
50-| `GLEAN_BACKFILL_CONCURRENCY` | `5` | Max concurrent backfill workers |
51-| `GLEAN_PLC_URL` | `https://didplc.glean.at` | PLC directory URL for DID resolution |
52-| `GLEAN_OAUTH_CLIENT_ID` | _(empty)_ | OAuth client metadata URL (leave empty for localhost dev) |
53-| `GLEAN_OAUTH_REDIRECT_URL` | _(empty)_ | OAuth redirect URL (leave empty for localhost dev) |
40+| Variable | Default | What it does |
41+| ---------------------------- | -------------------------- | ---------------------------------------------------------- |
42+| `GLEAN_SESSION_KEY` | _(required)_ | Secret key for signing session cookies (any random string) |
43+| `GLEAN_ADDR` | `:8080` | Listen address |
44+| `GLEAN_DB` | `glean.db` | SQLite base path (`_users`, `_articles`, `_recs` suffixes) |
45+| `GLEAN_JETSTREAM` | `wss://jetstream.glean.at` | Jetstream WebSocket URL |
46+| `GLEAN_SYNC_INTERVAL` | `30m` | PDS sync interval (Go duration: `30m`, `2h30m`, etc.) |
47+| `GLEAN_CLUSTER_INTERVAL` | `1h` | Cluster recomputation interval (Go duration) |
48+| `GLEAN_FETCH_INTERVAL` | `15m` | Feed fetch scheduler tick interval (Go duration) |
49+| `GLEAN_COLLECTION_DIR_URL` | _(empty)_ | Collection directory URL for startup backfill |
50+| `GLEAN_BACKFILL_CONCURRENCY` | `5` | Max concurrent backfill workers |
51+| `GLEAN_PLC_URL` | `https://didplc.glean.at` | PLC directory URL for DID resolution |
52+| `GLEAN_OAUTH_CLIENT_ID` | _(empty)_ | OAuth client metadata URL (leave empty for localhost dev) |
53+| `GLEAN_OAUTH_REDIRECT_URL` | _(empty)_ | OAuth redirect URL (leave empty for localhost dev) |
5454
5555 For production:
5656
@@ -37,20 +37,20 @@ Then open `http://localhost:8080`.
37 37
38 ## Configuration38 ## Configuration
39 39
40-| Variable | Default | What it does |40+| Variable | Default | What it does |
41-| -------------------------- | -------------------------- | --------------------------------------------------------- |41+| ---------------------------- | -------------------------- | ---------------------------------------------------------- |
42-| `GLEAN_SESSION_KEY` | _(required)_ | Secret key for signing session cookies (any random string) |42+| `GLEAN_SESSION_KEY` | _(required)_ | Secret key for signing session cookies (any random string) |
43-| `GLEAN_ADDR` | `:8080` | Listen address |43+| `GLEAN_ADDR` | `:8080` | Listen address |
44-| `GLEAN_DB` | `glean.db` | SQLite base path (`_users`, `_articles`, `_recs` suffixes) |44+| `GLEAN_DB` | `glean.db` | SQLite base path (`_users`, `_articles`, `_recs` suffixes) |
45-| `GLEAN_JETSTREAM` | `wss://jetstream.glean.at` | Jetstream WebSocket URL |45+| `GLEAN_JETSTREAM` | `wss://jetstream.glean.at` | Jetstream WebSocket URL |
46-| `GLEAN_SYNC_INTERVAL` | `1h` | PDS sync interval (Go duration: `30m`, `2h30m`, etc.) |46+| `GLEAN_SYNC_INTERVAL` | `30m` | PDS sync interval (Go duration: `30m`, `2h30m`, etc.) |
47-| `GLEAN_CLUSTER_INTERVAL` | `10m` | Cluster recomputation interval (Go duration) |47+| `GLEAN_CLUSTER_INTERVAL` | `1h` | Cluster recomputation interval (Go duration) |
48-| `GLEAN_FETCH_INTERVAL` | `5m` | Feed fetch scheduler tick interval (Go duration) |48+| `GLEAN_FETCH_INTERVAL` | `15m` | Feed fetch scheduler tick interval (Go duration) |
49-| `GLEAN_COLLECTION_DIR_URL` | _(empty)_ | Collection directory URL for startup backfill |49+| `GLEAN_COLLECTION_DIR_URL` | _(empty)_ | Collection directory URL for startup backfill |
50-| `GLEAN_BACKFILL_CONCURRENCY` | `5` | Max concurrent backfill workers |50+| `GLEAN_BACKFILL_CONCURRENCY` | `5` | Max concurrent backfill workers |
51-| `GLEAN_PLC_URL` | `https://didplc.glean.at` | PLC directory URL for DID resolution |51+| `GLEAN_PLC_URL` | `https://didplc.glean.at` | PLC directory URL for DID resolution |
52-| `GLEAN_OAUTH_CLIENT_ID` | _(empty)_ | OAuth client metadata URL (leave empty for localhost dev) |52+| `GLEAN_OAUTH_CLIENT_ID` | _(empty)_ | OAuth client metadata URL (leave empty for localhost dev) |
53-| `GLEAN_OAUTH_REDIRECT_URL` | _(empty)_ | OAuth redirect URL (leave empty for localhost dev) |53+| `GLEAN_OAUTH_REDIRECT_URL` | _(empty)_ | OAuth redirect URL (leave empty for localhost dev) |
54 54
55 For production:55 For production:
56 56