From 6731ff4e97b66b8d0fa00f361acdef477c2c6b12 Mon Sep 17 00:00:00 2001 From: truetype Date: Sun, 10 May 2026 17:39:01 +0200 Subject: [PATCH] setup: complete wiki scaffold with schema, seed files, sample article, search script, and workflow docs --- WORKFLOW.md | 51 +++++++++++++++++++++++++++ raw/articles/sample-llm-wiki-intro.md | 40 +++++++++++++++++++++ scripts/wiki-search.sh | 16 +++++++++ wiki/index.md | 3 +- wiki/log.md | 6 ++++ 5 files changed, 115 insertions(+), 1 deletion(-) create mode 100644 WORKFLOW.md create mode 100644 raw/articles/sample-llm-wiki-intro.md create mode 100755 scripts/wiki-search.sh diff --git a/WORKFLOW.md b/WORKFLOW.md new file mode 100644 index 0000000..1b98d63 --- /dev/null +++ b/WORKFLOW.md @@ -0,0 +1,51 @@ +# Wiki Ingest Workflow + +## How to Ingest a New Source + +### Step 1: Add the source +Copy or save the document into the appropriate subdirectory of `raw/`: +- `raw/articles/` — web articles, blog posts +- `raw/papers/` — research papers, technical docs +- `raw/repos/` — code repository summaries + +### Step 2: Trigger ingestion +Tell the LLM agent: +``` +Read raw/articles/[filename] and ingest it into the wiki following AGENTS.md rules. +- Create or update wiki pages for key concepts, entities, and claims +- Add cross-references with [[wikilinks]] +- Update wiki/index.md +- Append to wiki/log.md +- Run the lint checklist +``` + +### Step 3: Review +- Check the LLM's summary of what it created/updated +- Browse the wiki in Obsidian or any markdown viewer +- Verify `wiki/index.md` is up to date + +### Step 4: Commit +```bash +git add -A && git commit -m "ingest: [source filename]" +``` + +## How to Query +``` +Using the wiki, answer: [your question]. Cite specific wiki pages. +``` + +## How to Lint +``` +Run the full wiki lint per AGENTS.md checklist. Report issues and fix what you can. +``` + +## Naming Your Sources +Use clear, descriptive filenames: +- `raw/articles/2026-transformer-survey.md` +- `raw/papers/attention-is-all-you-need.pdf.md` +- `raw/repos/langchain-agent-notes.md` + +## Tips +- Ingest one source at a time for best results +- Stay involved — read summaries, guide emphasis +- After 50+ sources, consider adding qmd for search diff --git a/raw/articles/sample-llm-wiki-intro.md b/raw/articles/sample-llm-wiki-intro.md new file mode 100644 index 0000000..98d4b66 --- /dev/null +++ b/raw/articles/sample-llm-wiki-intro.md @@ -0,0 +1,40 @@ +# LLM Wiki: A New Approach to Knowledge Management + +Source: Andrej Karpathy's blog / GitHub Gist (2026) + +## Summary + +Andrej Karpathy introduced the concept of an "LLM Wiki" — a persistent, structured knowledge base built and maintained by an LLM agent. Unlike traditional RAG (Retrieval-Augmented Generation) systems that search raw documents at query time, an LLM Wiki is compiled once and kept current. + +## Key Concepts + +### The Problem with RAG +Traditional RAG retrieves from raw documents every time you ask a question. Results depend heavily on chunking strategy, embedding quality, and retrieval parameters. Good results are not retained. + +### The LLM Wiki Approach +Instead of retrieving at query time, the LLM reads sources once and builds a structured wiki: +- **Raw sources** are stored in `raw/` (immutable) +- **Compiled knowledge** lives in `wiki/` as interlinked markdown pages +- **A schema file** (`AGENTS.md`) defines conventions and workflows + +### Three Workflows +1. **Ingest**: Add source to `raw/`, LLM compiles into wiki (~10-15 pages per source) +2. **Query**: LLM reads `index.md`, finds relevant pages, synthesizes answer with citations +3. **Lint**: Periodic audit for contradictions, orphans, stale claims + +### Compounding Knowledge +Good answers get filed back into the wiki as new pages. The wiki gets richer with every source added and every question asked. This creates a compounding knowledge effect. + +## Technical Implementation +- Works with any LLM that can read/write files +- Tested with Claude Code, Codex CLI, Cursor +- Also works with local models via Ollama (Gemma, Llama, etc.) +- No special infrastructure needed — just a git repository of markdown files +- Optional: qmd for enhanced search at scale (100+ sources) + +## Key Advantages +- No embedding models or vector databases needed +- Full version control via git +- Human-in-the-loop curation +- Knowledge compounds over time +- Works offline / on-premise diff --git a/scripts/wiki-search.sh b/scripts/wiki-search.sh new file mode 100755 index 0000000..f24bfd2 --- /dev/null +++ b/scripts/wiki-search.sh @@ -0,0 +1,16 @@ +#!/bin/bash +# Wiki search utility — grep-based fallback (no qmd needed) +# Usage: ./scripts/wiki-search.sh "search query" +QUERY="$1" +if [ -z "$QUERY" ]; then + echo "Usage: wiki-search.sh " + exit 1 +fi +echo "=== Searching wiki for: $QUERY ===" +echo "" +grep -rl --include="*.md" -i "$QUERY" "$(dirname "$0")/../wiki/" 2>/dev/null | while read file; do + echo "--- $file ---" + grep -n -i "$QUERY" "$file" 2>/dev/null | head -5 + echo "" +done +echo "=== Search complete ===" diff --git a/wiki/index.md b/wiki/index.md index f4f1b42..5e59ecf 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -6,10 +6,11 @@ ## Concepts - +- [[llm-wiki-pattern]] — LLM Wiki methodology by Karpathy ## Sources +- [[sample-llm-wiki-intro]] — Introduction to LLM Wiki concept ## Comparisons diff --git a/wiki/log.md b/wiki/log.md index c7f7837..c9f6903 100644 --- a/wiki/log.md +++ b/wiki/log.md @@ -2,5 +2,11 @@ > Append-only chronological record of all ingest and maintenance operations. +## 2026-05-10 17:30 — Wiki scaffold created +- Created: `AGENTS.md` (schema), `wiki/overview.md`, `wiki/index.md`, `wiki/topics/llm-wiki-pattern.md` +- Created: `raw/articles/sample-llm-wiki-intro.md` (sample article) +- Created: `scripts/wiki-search.sh` (grep-based search utility) +- Key takeaways: Initial setup following Karpathy's LLM Wiki pattern. Three-layer architecture in place (raw → wiki → schema). + --- *Log started: 2026-05-10*