Manifold – Architecture & Module Map

1. What this system does

• Purpose: Manifold is an AI workflow automation platform that lets operators define specialist agents, orchestrate tool-backed agents/workflows, observe every run, manage shared assets (projects/playground), and offer the surface via CLIs, HTTP/SSE APIs, Kafka orchestrations, and a web UI.
• Key capabilities / responsibilities:
  • Autonomous chat-style agents that can route to specialists, orchestrate tool calls (local CLI, web search/fetch, patch updates, RAG helpers, TTS, Kafka, MCP), and optionally execute deterministic WARPP workflows.
  • Long-lived persistent services (notably agentd) exposing REST/SSE surfaces, authentication, project/playground management, RAG ingestion, MCP/OAuth hooks, Whisper STT, and observability/metrics.
  • Kafka-based orchestrator and helper CLI tooling (agent, orchestrator, whisper-go, adk) covering single runs, workflow consumption, STT support, and future developer helpers.
  • SPA frontend (web/agentd-ui) that drives the in-browser control plane, communicating with agentd’s HTTP endpoints.

2. High-level architecture

Manifold centers on a shared internal/agent.Engine that orchestrates LLM + tool loops. Each runtime (CLI, HTTP, Kafka) loads the same config/observability/LLM/providers/tool registry but swaps how intent is injected and how output is surfaced. The big picture combines:

• Configuration & observability (internal/config, internal/observability, config.yaml, example.env) – every binary loads these to understand LLM providers, tool toggles, databases, OAuth hooks, Kafka/Redis endpoints, and telemetry settings.
• Agent + tool layer (internal/agent, internal/tools, internal/llm, internal/specialists, internal/mcpclient) – orchestrates message construction, LLM requests, tool dispatch/schema exposure, specialist routing (DB+YAML), and MCP tool registration.
• Persistence & workflow helpers (internal/persistence, internal/rag, internal/playground, internal/projects, internal/warpp) – provide storage for specialists, playground experiments, projects, WARPP workflows, RAG ingestion, and data retrieval.
• Transports: CLI (cmd/agent), HTTP daemon (cmd/agentd → internal/agentd), Kafka worker (cmd/orchestrator → internal/orchestrator), Whisper STT CLI (cmd/whisper-go), and placeholder cmd/adk.
• Frontend & docs: web/agentd-ui SPA served via agentd’s static handler or deployed separately; documentation/examples/deployments live under docs/, examples/, deploy/.

Every major service eventually delegates to the same internal/agent.Engine and shared tool stack, but they differ on intent sources (CLI args vs HTTP requests vs Kafka messages) and sinks (stdout, HTTP response, Kafka response).

3. Modules and packages

• ./cmd/
  • Purpose: Hosts Go binaries that bootstrap each runtime (agent CLI, agent daemon, Kafka orchestrator, Whisper STT CLI, and placeholder adk).
  • Key entries:
    • cmd/agent/main.go – single LLM/agent run with tool registry, specialist seeding, optional WARPP mode, and CLI output.
    • cmd/agentd/main.go → internal/agentd.Run() – HTTP/SSE server that composes tools, WARPP, specialists, playground, projects, auth, and MCP.
    • cmd/orchestrator/main.go – Kafka WARPP worker with deduping via Redis and response publishing.
    • cmd/whisper-go/main.go – speech-to-text CLI around vendored whisper.cpp.
    • cmd/adk/ – currently empty placeholder (TODO: define tooling).
  • Interactions: Each binary calls into internal/config, internal/observability, internal/llm, internal/tools, internal/specialists, internal/persistence, and internal/warpp.
• ./internal/
  • ./internal/agent/ – Core reasoning loop (engine.go, steps.go), message formatting, memory summarization, WARPP helpers, and shared prompts. Depends on internal/llm/internal/tools.
  • ./internal/agentd/ – HTTP server bootstrap, router, and handlers (chat, tools, playground, projects, auth, STT, specialists, WARPP, metrics). Ties to all other packages plus internal/webui.
  • ./internal/orchestrator/ – Kafka consumer/dedupe logic, WARPP adapter, response publishing, and helper tools under cmd/orchestrator/tools.
  • ./internal/warpp/ – Workflow loader/runner (guards, personalization, step publisher). Used by agent CLI WARPP mode, agentd workflows, and Kafka orchestrator.
  • ./internal/tools/ – Registry plus concrete tools (cli, web, patchtool, kafka, tts, specialists, llmtool, textsplitter, rag, warpptool, utility, multi_tool_use, etc.). Tools register per-process and expose tool schemas consumed by the engine.
  • ./internal/llm/ – Provider abstraction (OpenAI/Anthropic/Google/local), streaming/tracing support, logging hooks, and factories referenced by every runtime.
  • ./internal/specialists/ – Registry/router for specialist agents (DB-backed or YAML). Populates tool registries and supports route overrides via config.
  • ./internal/mcpclient/ – MCP server registration and tool bridging (Model Context Protocol servers register tools into the same registry).
  • ./internal/persistence/ – Interfaces plus platform implementations (memory/Postgres) for chat history, playground, projects, WARPP, MCP, specialists, etc. databases.Manager constructs stores used by agentd & orchestrator.
  • ./internal/rag/ – Chunking, embeddings, ingestion, retrieval, and ingestion service used by RAG tools and persistence.
  • ./internal/playground/ – Prompt/dataset/experiment storage and evaluation orchestration; agentd exposes APIs for editing/running experiments.
  • ./internal/projects/ – Filesystem-backed project artifacts service; integrated with agentd APIs.
  • ./internal/auth/ – OIDC/OAuth providers, session middleware, RBAC helpers for agentd routes.
  • ./internal/config/ – Config loader structs referencing config.yaml, config.yaml.example, example.env to unify every runtime.
  • ./internal/observability/ – Logger bootstrap, OTel tracer, HTTP client wrappers, payload filtering. Initialized by every binary.
  • ./internal/webui/ – Static asset handler, dev proxy, and optional auth guard used by agentd for / routes.

• ./web/agentd-ui/
  • Vite-based TypeScript SPA (Svelte/Vue-like) that calls agentd HTTP APIs for chat, tools, projects, playground, experiments, WARPP, etc. Built assets live under web/agentd-ui/dist. Agentd serves them via internal/webui.
• ./docs/, ./examples/, ./deploy/, ./configs/, ./scripts/, ./tools/, ./external/, ./dist/:
  • Provide onboarding/architecture guides (docs/*), workflow samples (examples/workflows), deployment scaffolding (docker-compose, vector_db_local, configs), helper scripts, vendor code (external/whisper.cpp), and build artifacts.

4. Entrypoints & startup flow

• ./cmd/agent/main.go – single-run CLI
  1. internal/config.Load() reads YAML/env to configure LLM, tools, specialists, WARPP, logging, observability, and timeouts.
  2. Initializes logging/OTel via internal/observability, builds HTTP client (with extra headers), configures global LLM logging, and seeds specialists from internal/persistence/databases (reflecting DB overrides).
  3. Builds the LLM provider via internal/llm/providers, registers tools (CLI exec, web search/fetch, patch, text splitter, textbox, TTS, llm_transform, specialists, MCP registrations), and wraps them per cfg.EnableTools/cfg.ToolAllowList.
  4. Connects to MCP servers to register MCP tools.
  5. If -warpp is set, loads WARPP workflows (DB-backed), personalizes intent, and executes via internal/warpp.Runner. Otherwise, optionally routes to a named specialist and ends.
  6. Constructs internal/agent.Engine with prompts/summary rules and runs Engine.Run, respecting AgentRunTimeoutSeconds, printing final response.
• ./cmd/agentd/main.go → internal/agentd.Run() – HTTP daemon
  1. Loads config/env, initializes logger/OTel/tracing, and builds shared HTTP client + LLM provider (plus summary provider).
  2. Constructs database manager (internal/persistence/databases.NewManager) for chat, projects, playground, specialists, WARPP, MCP, etc., and creates specialist registry overlays.
  3. Registers tools (CLI/web search/fetch, patch, text splitter, textbox, TTS, Kafka, RAG ingest/retrieve, llm_transform, image tools, specialists, agent-wrappers, multi-tool). Applies global tool filters/allow lists to mirror cfg.EnableTools.
  4. Registers MCP clients, loads WARPP workflows (FS/DB), and prepares internal/warpp.Runner.
  5. Builds higher-level services: summary helpers, playground service (internal/playground), project service, RAG ingestion/state, Whisper STT loader, auth provider, metrics reporter.
  6. Wires HTTP routes (router.go, handlers_*): chat, tools, specialists, playground, projects, WARPP, MCP OAuth, audio/STT, metrics, auth, static SPA assets (via internal/webui), and SSE endpoints for progress.
  7. Listens on configured port (default :32180) serving API + SPA.
• ./cmd/orchestrator/main.go – Kafka WARPP worker
  1. Loads Kafka brokers/topics, Redis dedupe settings, worker counts, timeouts, and logging config.
  2. Initializes Redis dedupe store, Kafka producer/admin, observability, HTTP client, LLM provider, and tool registry (CLI/web/patch/TTS/llm_transform/specialists/Kafka send). Applies tool filters per config.
  3. Registers MCP servers, loads WARPP workflows via internal/warpp, wraps runner with internal/orchestrator.NewWarppAdapter.
  4. Ensures Kafka topics exist (command/response/DLQ), then starts internal/orchestrator.StartKafkaConsumer.
  5. Workers loop: read CommandEnvelope, dedupe (Redis), map reply topic, execute workflows with the WARPP adapter (streaming steps via publisher), and publish success/error ResponseEnvelope plus DLQs when needed.

Manifold – Runtime & Environments

1. Runtime overview

manifold is a long-lived service (cmd/agentd) that exposes a web UI on port 32180 and orchestrates agent workflows (agents, orchestrator, orchestrated prompts, workflows). The platform ships with supporting CLI utilities (e.g., scripts/dev.sh for fmt/vet/build checks, scripts/pre-commit hook) plus additional command binaries under cmd/orchestrator, cmd/agent, etc. The main runtime (agentd) runs as a Go server backed by the configuration loader (internal/config/loader.go), which merges .env, config.yaml, and defaults before wiring OpenAI/Google/Anthropic providers, databases, and metrics exporters.

2. Environments

• local (default) – development workstation. Uses docker-compose.yml to spin up the Go service, Postgres (pg-manifold), and optional observability stack (ClickHouse + OTEL collector). Configuration files (config.yaml, .env) are tailored for localhost connections, non-HTTPS auth (cookieSecure=false), and demo credentials. This environment is what the quickstart (QUICKSTART.md) walks through; it expects the host to run Docker, Node.js 20, pnpm, and a Chromium browser.
• dev / custom – implied by configuration defaults (obs.environment defaults to dev, LlmClient.Provider can be set via env). Switchable by setting ENVIRONMENT, OTEL_EXPORTER_OTLP_ENDPOINT, or alternative config files (SPECIALISTS_CONFIG). This allows pointing at staging/cloud endpoints (e.g., production Kafka brokers or ClickHouse DSNs) without code changes. Feature flags/toggles such as ENABLE_TOOLS/SPECIALISTS_DISABLED and AUTH.* can be changed per environment via .env overrides.

3. Configuration

• Primary config files: config.yaml (active) and config.yaml.example (template). The loader (internal/config/loader.go) first merges environment variables (via godotenv.Overload() reading .env), then overlays the YAML file(s), and finally applies hard defaults. .env is expected at repo root and is mounted into the Docker container (docker-compose.yml, service manifold).
• Environment variables override YAML for every major area: OpenAI/GPT settings (OPENAI_API_KEY, OPENAI_MODEL, LLM_PROVIDER), embedding service (EMBED_*), databases (DATABASE_URL, SEARCH_DSN, etc.), Kafka topics, observability exports (OTEL_SERVICE_NAME, CLICKHOUSE_*), TTS, tools toggles (ENABLE_TOOLS, SPECIALISTS_DISABLED), and runtime controls (timeout, log levels). Load() ensures required settings (OPENAI_API_KEY, WORKDIR) are present and resolves WORKDIR to an absolute path.
• Specialist definitions (agents/tools) can live in the same config.yaml or an alternate file pointed to by SPECIALISTS_CONFIG; environment variables such as SPECIALISTS_DISABLED or provider-specific vars (e.g., GOOGLE_LLM_API_KEY) take precedence.

4. How to run locally

Prerequisites

1. Docker – required to build containers, start Postgres, Redis, optional ClickHouse/OTEL.
2. Node 20 – required for the UI (web/agentd-ui). Use nvm use 20.
3. pnpm – install global or via package manager to install frontend deps.
4. Chrome/Chromium – needed for web tools (Playwright MCP).
5. GNU Make + Go toolchain – scripts/dev.sh, make fmt, go build run from repo root.

Steps

cd ~/path/to/manifold                 # repo root (“./manifold” in prompt)
cp example.env .env                   # create env file (contains OPENAI_API_KEY placeholder)
cp config.yaml.example config.yaml    # copy default configuration
# edit .env and config.yaml to provide OPENAI_API_KEY, WORKDIR, and any overrides
# e.g., replace OPENAI_API_KEY in .env with a real key; set WORKDIR to repo root

nvm use 20
cd web/agentd-ui
pnpm install
cd ../..                              # back to repo root

touch manifold.log                    # optional, runtime writes to log
docker compose up -d manifold pg-manifold

Manifold – Data & Storage

1. Data stores overview

• PostgreSQL (primary relational store)
  • Purpose: Long-lived persistence for authentication/RBAC, specialists registry, chat sessions/messages, MCP server configurations, WARPP workflows, playground prompts/datasets/exercises, and shared search/vector/graph backends (documents/chunks, embeddings, nodes/edges).
  • Configuration: internal/config/config.go exposes Config.Databases → DBConfig struct with DefaultDSN plus per-subsystem Search, Vector, Graph, and Chat configs. internal/persistence/databases/factory.go reads Databases.DBConfig and routes each subsystem to Postgres or an in-memory fallback.
• In-memory stores (development/testing fallback)
  • Purpose: Provide zero-dependency persistence when Postgres is absent for chat, specialists, MCP, WARPP, and the search/vector/graph backends, keeping interfaces consistent for the rest of the app.

2. Core data models / entities

Authentication / RBAC (internal/auth/store.go)

Playgrounds (internal/persistence/databases/playground_store.go)

Entity Relationships

3. Migrations and schema management

• Location: Every persistence module bootstraps its schema under internal/persistence/databases/ (e.g., auth/store.go, specialists_store.go, chat_store_postgres.go, playground_store.go).
• How applied: databases.Manager.Init instantiates each store and invokes Init/InitSchema methods that execute CREATE TABLE IF NOT EXISTS statements.
• Ordering/versioning: Schema updates happen at runtime; there’s no explicit migration history. Each initializer runs in code order during startup.

Manifold – Interfaces & Workflows

1. Inbound interfaces overview

2. HTTP / RPC APIs

• Agentd surface (internal/agentd)
  • GET /healthz & GET /readyz – basic liveness/readiness endpoints.
  • /auth/* & /api/me – exposed when auth enabled.
  • /api/projects & /api/projects/{id} – CRUD/listing backed by internal/projects.Service.
  • /api/runs, /api/chat/sessions – chat session management.
  • /agent/run + /agent/vision, /api/prompt – agent execution entrypoints.
  • /api/mcp/* – MCP server listing and OAuth endpoints.
• Playground API (internal/httpapi)
  • Prompts (GET list /api/v1/playground/prompts, POST create, GET/DELETE by ID).
  • Datasets and Experiments (/api/v1/playground/experiments, /runs).
• CLI-to-internal APIs
  • CLI agent runs call into internal/agent.Engine directly but expose consistent configuration via cmd/agent/main.go.

3. Key workflows

Agentd HTTP request → agent engine run

Triggered via POST /agent/run. The handler extracts context, builds prompt/history attributes, and calls agent.Engine.Run, which iterates the LLM reasoning loop.

Kafka command → WARPP workflow execution

Messages posted to the command topic are consumed by the orchestrator, deduped via Redis, and executed via the WARPP runner.