# ajet-chat — Team Messaging Platform

## Overview
Clojure monorepo for a team messaging app with multiple clients.
PostgreSQL everywhere (dev + prod) via next.jdbc + HoneySQL.

## Architecture
```
ajet-chat/
├── shared/      — Common DB layer, schemas, protocols, EventBus (NATS), API client SDK
├── auth-gw/     — Auth gateway: Clojure edge proxy, authn/authz, custom DB reads/writes
├── api/         — Stateless REST API: all reads & writes, publishes to event bus
├── web-sm/      — Web session manager: Hiccup + Datastar SSE for browsers
├── tui-sm/      — TUI session manager: SSE for terminal clients
├── cli/         — Terminal client: CLI args (one-off) + TUI (interactive), babashka/bbin
└── mobile/      — Mobile client + session manager (deferred)
```

Modules reference each other via `:local/root` in deps.edn.

## Unified Deps
All server modules (auth-gw, api, web-sm, tui-sm):
- org.clojure/clojure 1.12.0
- http-kit/http-kit 2.8.0
- metosin/reitit 0.7.2
- ring/ring-core 1.13.0
- ajet/chat-shared (local) — EventBus (NATS via io.nats/jnats), schemas, protocols

PG access (api + auth-gw only):
- next.jdbc + HoneySQL + PostgreSQL driver (via shared/)
- migratus (for DB migrations)

Additional per-module:
- web-sm: hiccup 2.0.0-RC4, dev.data-star.clojure/sdk 1.0.0-RC5
- cli: babashka, clojure-tui (local)

**Dependency matrix:**
```
              PG    NATS   API(HTTP)
API           ✓     pub    —
Auth GW       ✓     —      —
Web SM        —     sub    ✓ (internal)
TUI SM        —     sub    ✓ (internal)
CLI           —     —      ✓ (external, via Auth GW)
```

## System Architecture
```
  Internet
     │
     ▼
  ┌──────────────────────────────────────────────────────┐
  │  nginx (TLS termination, prod only)                  │
  └──────────────────────┬───────────────────────────────┘
                         │
  ┌──────────────────────▼───────────────────────────────┐
  │              Auth Gateway (Clojure, http-kit)         │
  │  • Session token + API token validation (DB reads)    │
  │  • Session creation on login (DB writes)              │
  │  • Rate limiting, CORS, audit logging                 │
  │  • Routes authed requests to internal services        │
  │  • SSE connection auth                                │
  └──┬──────────────┬───────────────┬──────────┬─────────┘
     │              │               │          │
     │ /api/*       │ / (web)       │ /ws/tui  │ /ws/mobile
     ▼              ▼               ▼          ▼
  ┌───────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐
  │  API  │  │ Web SM   │  │ TUI SM   │  │Mobile SM │
  │(REST) │  │(Datastar │  │(SSE)     │  │(future)  │
  │       │  │ SSE)     │  │          │  │          │
  └──┬────┘  └──┬──┬────┘  └──┬──┬────┘  └──┬──┬────┘
     │  │       │  │          │  │          │  │
     │  │       │ browsers    │ terminals   │ phones
     │  │       │             │             │
     │  │ pub   │ sub         │ sub         │ sub
     │  ▼       ▼             ▼             ▼
     │  ┌────────────────────────────────────────────┐
     │  │                   NATS                      │
     │  │   pub/sub event bus · community-scoped      │
     │  │   subjects · JetStream for replay           │
     │  └────────────────────────────────────────────┘
     │ r/w
     ▼
  ┌──────────────────────────────────────────────────────┐
  │                     PostgreSQL                        │
  │   data store · full-text search                       │
  │   ◄── Auth GW: session reads/writes                   │
  └──────────────────────────────────────────────────────┘
```

### Auth Gateway
Clojure-based edge proxy (http-kit). All external traffic enters here.
- Custom DB reads: session token + API token validation
- Custom DB writes: session creation, rate limit counters, audit log
- Authenticates HTTPS and SSE connections
- Proxies to internal services after authentication
- In prod: sits behind nginx for TLS termination

### Auth Strategy
- **OAuth for v1** — all user login via OAuth providers (GitHub, Gitea)
  - Auth GW handles OAuth callback, creates/links user, creates session
  - No local passwords in v1
  - Email-based session auth for local-only accounts planned for later (requires email infrastructure)
- **Session tokens** for interactive users (web, TUI, mobile)
  - 32 random bytes (256 bits), encoded as base64url, stored as bcrypt hash in DB
  - Rolling expiry: TTL extends on each authenticated request
  - Cookie for web (`HttpOnly`, `Secure`, `SameSite=Strict`), `Authorization: Bearer` header for CLI/mobile
- **API tokens** for programmatic access
  - Long-lived, scoped tokens for bots, integrations, scripts
  - Created and managed via admin UI in each client
- **Incoming webhooks** for external service integration
  - Each webhook is bound to a specific channel + has a unique URL + secret token
  - External services POST JSON to `/hook/<webhook-id>/<token>`
  - Auth GW validates the webhook token (no session needed)
  - API creates the message in the bound channel, broadcasts via EventBus
  - Webhook messages display with a bot name/avatar (configurable per webhook)
  - Use cases: git push notifications, CI/CD status, server alerts, monitoring
  - Managed via slash commands:
    `/webhook create #channel <name>`, `/webhook list`, `/webhook revoke <id>`
- **Slash commands** — universal interaction model across all clients
  - Typed in the message input (web, TUI) or as CLI args
  - API parses and executes, returns result or side-effect
  - Regular user commands: `/join #channel`, `/leave`, `/topic new topic`, `/msg @user`, etc.
  - Admin commands (role-gated: owner/admin only):
    `/ban @user [reason]`, `/kick @user`, `/mute @user [duration]`,
    `/token create <name> [scopes]`, `/token revoke <id>`,
    `/role @user admin|member`, `/purge #channel [count]`
  - `/help` lists available commands (filtered by user's role)
  - `/help <command>` shows detailed usage for a specific command
  - All backed by the same API routes — clients just send the command string,
    API handles parsing, permission checks, and execution

### First-User Bootstrap
On first server setup (no communities exist in DB):
1. Auth GW login page shows a community creation form alongside OAuth buttons
2. First user authenticates via OAuth, creates the initial community, becomes owner
3. After initial community exists, new users must be invited (invite-only)

### Service Ports (Dev)
Hardcoded per service for now. Config files / env vars can be layered later.
- Auth GW generates a unique `X-Trace-Id` on each inbound request; all downstream services propagate it

### API
Pure stateless REST (http-kit + reitit). Only service (besides Auth GW) with a PG connection.
All data reads and writes. Publishes events to NATS after writes.
No live connections. Scales trivially.

### Session Managers
Each manages live connections for its client type (http-kit + reitit):
- **Web SM**: serves Hiccup pages, holds Datastar SSE connections, pushes HTML fragments
- **TUI SM**: holds SSE connections for terminal clients (HTTP POSTs for client→server signals)
- **Mobile SM**: same pattern for mobile (future)

**Data flow for session managers:**
- **All data reads** (channel list, message history, user info, full message after notification): via API
- **Live events** (new messages, typing, presence): via NATS subscription
- **Ephemeral writes** (typing indicators): SM publishes directly to NATS — no API round-trip

SMs call the API directly (bypass Auth GW) and propagate the user's token
from the original request. The API validates the token the same way regardless
of caller — no separate service tokens needed.
All data reads go through the API — keeps auth and permission checks centralized.
Independently scalable.

**SM subscription model:**
When a user connects, their SM subscribes to:
1. Community event subject for their active community (`chat.events.{community-id}`)
2. All their DM channel IDs for DM events (`chat.dm.{channel-id}`)
3. Per-user notification subject (`chat.notifications.{user-id}`)

### EventBus — NATS
Dedicated pub/sub via [NATS](https://nats.io). Lightweight (~20MB), single binary,
starts in milliseconds. Runs as a Docker container alongside Postgres and MinIO.

```clojure
(defprotocol EventBus
  (publish! [this topic event])
  (subscribe! [this topic handler])
  (unsubscribe! [this topic handler]))
```

- API publishes to NATS after DB writes (e.g., `chat.events.{community-id}`)
- Session managers subscribe to NATS for real-time events — no PG connection needed
- SMs publish ephemeral events (typing) directly to NATS — no API round-trip
- SMs fetch full data from the API when needed (e.g., full message after receiving a notification ID)
- NATS subjects are community-scoped — natural sharding boundary
- JetStream (NATS persistence layer) enables replay for SSE reconnection (`Last-Event-ID`)
- Clojure client: `io.nats/jnats` (official Java client)

**NATS subject hierarchy:**
```
chat.events.{community-id}              — channel message CRUD, channel events, reactions
chat.dm.{channel-id}               — DM message events (global, no community)
chat.typing.{community-id}.{channel-id} — channel typing indicators
chat.typing.dm.{channel-id}        — DM typing indicators
chat.presence.{community-id}            — heartbeat/presence events
chat.notifications.{user-id}            — per-user notification delivery
```

**Event envelope (hybrid fat events):**
```json
{
  "type":         "message.created",
  "community_id": "uuid or null",
  "channel_id":   "uuid",
  "actor_id":     "uuid",
  "timestamp":    "ISO-8601",
  "data":         { /* enough to render the common case */ }
}
```
- `data` includes body, author ID, timestamp — enough to render without a follow-up API call
- Does NOT include deeply nested or volatile data (e.g. full author profile — SMs cache separately)
- Schema evolves additively only: add fields, never remove. Consumers ignore unknown fields.

## Data Model
Slack-like with planned voice channel support.

### Core Entities
```
Community
 └── Channel (text / voice*, public / private)
      └── Message
           ├── Thread reply (1 level deep)
           └── Reaction (emoji per user)

Conversation (global, not community-scoped)
 └── Message (same table, via shared container)

User
 └── CommunityMembership (user <> community, role)
 └── ChannelMembership (user <> channel)
 └── ChannelMembership (user <> channel, includes DMs)

APIUser (bot/integration accounts, managed via admin UI in each client)
```
*voice channels: planned, not in v1

### DMs — Global, Not Community-Scoped
DMs are just channels with `community_id = NULL` and type `dm` or `group_dm`.
No extra tables needed — `channels`, `channel_members`, and `messages` all work as-is.

- **DM**: 1:1 channel between two users (`type = 'dm'`, `community_id = NULL`)
- **Group DM**: channel with 2+ users (`type = 'group_dm'`, `community_id = NULL`)
- Starting a DM auto-creates a channel if one doesn't exist
- DM history persists across communities — same channel everywhere
- UI shows one community at a time; DM sidebar is always visible (global)

### URL Routing
- **Community-scoped routes**: `/c/<slug>/channels/...` — channels, messages, members
- **Global DM routes**: `/dm/...` — DM channels, DM messages
- **Other global routes**: `/api/users/...`, `/api/notifications/...`
- UI shows one community at a time; DM sidebar is always visible (global)
- Users can be members of multiple communities

### API Pagination
Cursor-based pagination using entity UUIDs as cursors:
- Forward: `?after=<uuid>&limit=N`
- Backward: `?before=<uuid>&limit=N`
- Applies to messages, channels, notifications, and any other list endpoints

### Messages
- **Body**: Markdown (rendered to HTML in web, terminal escape codes in CLI)
- **Attachments**: file/image uploads stored in MinIO (S3-compatible)
- **Pings/mentions**: `@user`, `@channel`, `@here` — parsed from markdown, stored as structured refs
- **Threads**: replies link to a parent message. 1 level deep (no nested threads)
- **Reactions**: emoji reactions per message. Any unicode emoji, one per user per emoji.
  - `PUT /api/messages/:id/reactions/:emoji` — toggle on (idempotent)
  - `DELETE /api/messages/:id/reactions/:emoji` — toggle off
  - API publishes reaction events to EventBus; SMs push live updates to clients
  - When fetching messages, reactions are returned aggregated: `[{emoji, count, reacted_by_me}]`

### Key Tables
All `id` columns are UUIDs (generated server-side, `java.util.UUID/randomUUID`).
```
communities          (id:uuid, name, slug, created_at)
users                (id:uuid, username, display_name, email, avatar_url, created_at)
oauth_accounts       (id:uuid, user_id:uuid, provider [github/gitea/...], provider_user_id, created_at)
api_users            (id:uuid, name, community_id:uuid, scopes, created_at)
sessions             (id:uuid, user_id:uuid, token_hash, expires_at, created_at)
api_tokens           (id:uuid, api_user_id:uuid, token_hash, scopes, expires_at, created_at)

channels             (id:uuid, community_id:uuid?, name, type [text/voice/dm/group_dm], visibility [public/private], topic, created_at)

community_members    (community_id:uuid, user_id:uuid, role [owner/admin/member], nickname, avatar_url)
channel_members      (channel_id:uuid, user_id:uuid, joined_at)
messages             (id:uuid, channel_id:uuid, user_id:uuid, parent_id:uuid [null=top-level], body_md, created_at, edited_at)
attachments          (id:uuid, message_id:uuid, filename, content_type, size_bytes, storage_key)
webhooks             (id:uuid, community_id:uuid, channel_id:uuid, name, avatar_url, token_hash, created_by:uuid, created_at)
reactions            (message_id:uuid, user_id:uuid, emoji:text, created_at)  PK: (message_id, user_id, emoji)
mentions             (id:uuid, message_id:uuid, target_type [user/channel/here], target_id:uuid)
notifications        (id:uuid, user_id:uuid, type [mention/dm/thread_reply/invite/system], source_id:uuid, read:bool, created_at)
```

Note: DMs are channels with `community_id = NULL`. Community channels always have
`community_id` set. Queries for community channels filter by `community_id`;
DM queries filter by `type IN ('dm', 'group_dm') AND community_id IS NULL`.

### Presence (Heartbeats)
- Clients send `POST /api/heartbeat` every 60 seconds
- API returns 200 immediately, writes async (fire-and-forget)
- Updates `last_seen_at` on user record (or dedicated presence table)
- User is "online" if last heartbeat < 2 min ago, "offline" otherwise
- Typing indicators: live via NATS, not batched, not persisted
  - Client sends typing event to their session manager (via HTTP POST)
  - SM publishes to NATS (`chat.typing.{community}.{channel}`) — does not go through API
  - `typing:start` — sent on keypress, delivered immediately to relevant clients (same channel/DM)
  - `typing:stop` — sent explicitly when user clears their input box or sends a message
  - Auto-expire: if no new keypress for 15 seconds, client stops sending `typing:start`;
    receiving clients timeout and hide the indicator after 15s of silence
- **Session manager presence delivery:**
  - SMs collect heartbeat events from NATS into a buffer
  - Once per minute, flush a batched presence update to each connected client
  - Filter per client: only include users relevant to that client
    (same channels, active DMs — not the entire community)
  - Presence update is a diff: only users whose status changed (online→offline or vice versa)
  - Clients never receive raw heartbeat events — only batched, filtered presence snapshots

### Notifications
Persisted per-user. Syncs across all clients — unread on web shows unread on CLI and TUI.
- **Triggers** (API creates notification rows on):
  - @mention (user, channel, here)
  - DM received
  - Thread reply to a message you authored or participated in
  - Channel invite
  - System events (role change, ban, etc.)
- `notifications` table stores one row per user per event, with `read:bool`
- `source_id` points to the originating entity (message, channel, etc.) based on `type`
- **Read/unread state**: clients mark notifications read via API (`POST /api/notifications/read`)
- **Real-time**: EventBus publishes notification events; session managers deliver immediately
- **Query**: `GET /api/notifications?unread=true` — any client can fetch current unread count/list
- `mentions` table remains for structured @mention data; `notifications` is the user-facing delivery layer

## Database Strategy
- **PostgreSQL everywhere** (dev + prod). Dev Postgres runs in Docker.
- Shared DB layer in `shared/` uses next.jdbc + HoneySQL
- Avoid raw SQL strings; use HoneySQL maps everywhere
- Schema migrations: **migratus** (SQL migration files, works with next.jdbc)
- PostgreSQL is a pure data store — no pub/sub responsibility

## File Storage
S3-compatible object storage via **MinIO**. Same S3 API everywhere — portable to AWS later.

- **Dev:** MinIO container (alongside Postgres in docker-compose.dev.yml)
- **Prod self-hosted:** MinIO container in Docker Compose alongside other services
- **Prod AWS (future):** swap endpoint URL to real S3, zero code changes
- API handles uploads: validates, stores in MinIO, writes `attachments` row with `storage_key`
- `storage_key` format: `<community-id>/<channel-id>/<message-id>/<filename>`
- Clojure S3 client: amazonica or cognitect aws-api (both work with MinIO via endpoint override)
- Avatars, webhook bot icons also stored in MinIO

## Development Workflow
REPL-driven development. Infrastructure (Postgres, MinIO) runs in Docker.
Clojure services run locally via nREPL — not in containers.

- **REPL-first**: start services from the REPL, hot-reload code, eval as you go
- **CIDER middleware**: nrepl, cider-nrepl, refactor-nrepl in `:dev` alias
- **Docker for infra only**: `docker-compose.dev.yml` runs Postgres + MinIO + NATS
- **EventBus**: NATS — same as prod, no mock/shim
- **Dev without auth gateway**: services run directly, auth bypassed or mocked

### REPL Options
1. **Single REPL** — start one JVM with all modules loaded. Good for early dev.
   `clj -A:dev:api:web-sm:tui-sm:auth-gw` — all services in one process, shared Postgres + NATS.
2. **Multiple REPLs** — one per service. Better isolation, closer to prod topology.
   Each connects its own nREPL on a different port.

### Dev Aliases
```
clj -M:dev:api        # API service + CIDER nREPL
clj -M:dev:web-sm     # Web SM + CIDER nREPL
clj -M:dev:tui-sm     # TUI SM + CIDER nREPL
clj -M:dev:auth-gw    # Auth gateway + CIDER nREPL
```

Each `:dev` alias includes nrepl + cider-nrepl + refactor-nrepl.
Services expose `(start!)` / `(stop!)` / `(reset!)` functions for REPL control.

## Testing

Three tiers, escalating in scope and infrastructure.

### Unit Tests — no external deps
Fast, pure-function tests. No Docker, no DB, no HTTP.
- Test runner: **Kaocha** (`:test/unit` alias)
- What's tested: validation, parsing, formatting, HoneySQL query builders,
  markdown processing, permission logic, slash command parsing, data transforms
- DB layer tested via protocol mocks/stubs — verify queries are built correctly
  without executing them
- Run: `clj -M:test/unit` — takes seconds, runs in CI on every push

### Integration Tests — dockerized infra
Test service internals against real Postgres + MinIO + NATS.
- `docker-compose.test.yml` spins up Postgres + MinIO + NATS with test-specific
  ports and a fresh DB (no data carried between runs)
- Tests run on the host JVM, connecting to Docker infra
- What's tested: DB migrations, repository functions (actual SQL round-trips),
  EventBus (real NATS pub/sub), file upload/download, API route handlers
  with a real DB behind them
- Each test namespace gets a transaction that rolls back — tests don't leak state
- Run: `docker compose -f docker-compose.test.yml up -d && clj -M:test/integration`
- CI: Docker services started as job services, tests run in the pipeline

### E2E Tests — full stack in Docker
Production-like topology. Everything containerized.
- `docker-compose.test.yml` with additional service containers:
  nginx + auth-gw + api + web-sm + tui-sm (built from uberjars)
- Tests run from **outside** the stack as a client would
- What's tested: full request lifecycle (auth → API → DB → event → SM → client),
  SSE connections, webhook delivery, multi-user scenarios,
  cross-service event propagation
- HTTP client for API + SM tests (SSE consumer for streaming)
- Playwright for web UI tests (page loads, live updates, reactions, typing indicators)
- Run: `docker compose -f docker-compose.test.yml --profile e2e up -d && clj -M:test/e2e`

### Test Aliases
```
clj -M:test/unit          # unit tests — no Docker needed
clj -M:test/integration   # integration — requires docker-compose.test.yml
clj -M:test/e2e           # end-to-end — requires full stack in Docker
clj -M:test/all           # all tiers sequentially
```

### Docker Compose Files
```
docker-compose.dev.yml    # dev: Postgres + MinIO + NATS
docker-compose.test.yml   # test: Postgres + MinIO + NATS (fresh DB per run)
                          #   --profile e2e adds all service containers
docker-compose.yml        # prod: full stack
```

## Deployment (Prod)
- **Docker Compose** for production
- Each service as a container: auth-gw, api, web-sm, tui-sm
- PostgreSQL + MinIO + NATS + nginx as infrastructure containers
- docker-compose.yml defines the full topology
- Uberjars built per service

## Key Decisions Made
- [x] Monorepo with :local/root deps
- [x] http-kit + reitit + Ring for all server modules (no Pedestal)
- [x] Hiccup + Datastar for web frontend (no ClojureScript)
- [x] Datastar Clojure SDK (dev.data-star.clojure/sdk) with http-kit adapter
- [x] Single `ajet-chat` executable: CLI mode (one-off) + TUI mode (interactive)
- [x] clojure-tui for TUI rendering, babashka/bbin for distribution
- [x] PostgreSQL everywhere (dev + prod), no SQLite
- [x] Persistent writes through API, event bus for real-time fan-out
- [x] EventBus: NATS — dedicated pub/sub, community-scoped subjects, JetStream for replay
- [x] API is stateless REST — no live connections
- [x] Session managers per client type — each independently scalable
- [x] Session managers: NATS for events, API for all data reads — no direct PG connection
- [x] SMs publish ephemeral events (typing) directly to NATS — bypasses API for latency
- [x] Auth gateway in Clojure (http-kit) — custom DB reads/writes for auth
- [x] Auth: OAuth-only for v1 (GitHub, Gitea), email-based local auth later
- [x] Auth: session tokens + API tokens for programmatic access
- [x] Slash commands as universal interaction model (regular + admin, role-gated)
- [x] No separate admin module — admin is just elevated slash commands
- [x] Incoming webhooks for external integrations (git, CI, monitoring)
- [x] MinIO for file/attachment storage (S3-compatible, portable to AWS)
- [x] Presence via heartbeats: 1 min intervals, async DB write, instant 200 response
- [x] Slack-like data model: communities > channels > messages > threads (1 deep)
- [x] Global DMs: channels with `community_id = NULL`, type `dm`/`group_dm` — no extra tables
- [x] Multi-community: users can join multiple communities, UI shows one at a time
- [x] Path-based community routing: `/c/<slug>/...` for community, `/dm/...` for global DMs
- [x] Voice channels: LiveKit SFU, TUI opens browser for audio (not v1)
- [x] Search: PG tsvector + GIN index, Meilisearch upgrade path
- [x] Messages: markdown + file uploads + @mentions/pings
- [x] Mobile: deferred, focus on web + CLI first
- [x] nginx for TLS termination in prod
- [x] Migratus for DB migrations
- [x] Docker Compose for prod; docker-compose.dev.yml for dev infra (Postgres + MinIO + NATS)
- [x] REPL-driven dev: nrepl + cider-nrepl + refactor-nrepl
- [x] Clojure services run locally via REPL, not in containers
- [x] Three-tier testing: unit (no deps) → integration (Docker infra) → E2E (full stack)
- [x] Kaocha test runner, separate aliases per tier
- [x] E2E: Playwright for web, HTTP/SSE client for API/SM

## Planned (not v1)

### Voice Channels — LiveKit SFU
- Self-hosted LiveKit (Go binary, Docker container) as the SFU
- Web/mobile: full WebRTC audio via LiveKit JS/native SDKs
- TUI: voice channel state display (who's in, mute status) + opens browser link for audio
- LiveKit Java SDK for Clojure interop (room management, token generation)

### Search — PostgreSQL Full-Text Search
- PostgreSQL tsvector + GIN index (dev and prod)
- Zero extra infrastructure
- Supports stemming, ranking, phrase search
- Upgrade path: swap in Meilisearch behind the same API endpoint if needed later