setup: complete wiki scaffold with schema, seed files, sample article, search script, and workflow docs
This commit is contained in:
1 parent
125f74c563
commit
6731ff4e97
5 files changed
+115
-1
No files matched your search
+51
@@ -0,0 +1,51 @@
|
||||
# Wiki Ingest Workflow
|
||||
|
||||
## How to Ingest a New Source
|
||||
|
||||
### Step 1: Add the source
|
||||
Copy or save the document into the appropriate subdirectory of `raw/`:
|
||||
- `raw/articles/` — web articles, blog posts
|
||||
- `raw/papers/` — research papers, technical docs
|
||||
- `raw/repos/` — code repository summaries
|
||||
|
||||
### Step 2: Trigger ingestion
|
||||
Tell the LLM agent:
|
||||
```
|
||||
Read raw/articles/[filename] and ingest it into the wiki following AGENTS.md rules.
|
||||
- Create or update wiki pages for key concepts, entities, and claims
|
||||
- Add cross-references with [[wikilinks]]
|
||||
- Update wiki/index.md
|
||||
- Append to wiki/log.md
|
||||
- Run the lint checklist
|
||||
```
|
||||
|
||||
### Step 3: Review
|
||||
- Check the LLM's summary of what it created/updated
|
||||
- Browse the wiki in Obsidian or any markdown viewer
|
||||
- Verify `wiki/index.md` is up to date
|
||||
|
||||
### Step 4: Commit
|
||||
```bash
|
||||
git add -A && git commit -m "ingest: [source filename]"
|
||||
```
|
||||
|
||||
## How to Query
|
||||
```
|
||||
Using the wiki, answer: [your question]. Cite specific wiki pages.
|
||||
```
|
||||
|
||||
## How to Lint
|
||||
```
|
||||
Run the full wiki lint per AGENTS.md checklist. Report issues and fix what you can.
|
||||
```
|
||||
|
||||
## Naming Your Sources
|
||||
Use clear, descriptive filenames:
|
||||
- `raw/articles/2026-transformer-survey.md`
|
||||
- `raw/papers/attention-is-all-you-need.pdf.md`
|
||||
- `raw/repos/langchain-agent-notes.md`
|
||||
|
||||
## Tips
|
||||
- Ingest one source at a time for best results
|
||||
- Stay involved — read summaries, guide emphasis
|
||||
- After 50+ sources, consider adding qmd for search
|
||||
@@ -0,0 +1,40 @@
|
||||
# LLM Wiki: A New Approach to Knowledge Management
|
||||
|
||||
Source: Andrej Karpathy's blog / GitHub Gist (2026)
|
||||
|
||||
## Summary
|
||||
|
||||
Andrej Karpathy introduced the concept of an "LLM Wiki" — a persistent, structured knowledge base built and maintained by an LLM agent. Unlike traditional RAG (Retrieval-Augmented Generation) systems that search raw documents at query time, an LLM Wiki is compiled once and kept current.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### The Problem with RAG
|
||||
Traditional RAG retrieves from raw documents every time you ask a question. Results depend heavily on chunking strategy, embedding quality, and retrieval parameters. Good results are not retained.
|
||||
|
||||
### The LLM Wiki Approach
|
||||
Instead of retrieving at query time, the LLM reads sources once and builds a structured wiki:
|
||||
- **Raw sources** are stored in `raw/` (immutable)
|
||||
- **Compiled knowledge** lives in `wiki/` as interlinked markdown pages
|
||||
- **A schema file** (`AGENTS.md`) defines conventions and workflows
|
||||
|
||||
### Three Workflows
|
||||
1. **Ingest**: Add source to `raw/`, LLM compiles into wiki (~10-15 pages per source)
|
||||
2. **Query**: LLM reads `index.md`, finds relevant pages, synthesizes answer with citations
|
||||
3. **Lint**: Periodic audit for contradictions, orphans, stale claims
|
||||
|
||||
### Compounding Knowledge
|
||||
Good answers get filed back into the wiki as new pages. The wiki gets richer with every source added and every question asked. This creates a compounding knowledge effect.
|
||||
|
||||
## Technical Implementation
|
||||
- Works with any LLM that can read/write files
|
||||
- Tested with Claude Code, Codex CLI, Cursor
|
||||
- Also works with local models via Ollama (Gemma, Llama, etc.)
|
||||
- No special infrastructure needed — just a git repository of markdown files
|
||||
- Optional: qmd for enhanced search at scale (100+ sources)
|
||||
|
||||
## Key Advantages
|
||||
- No embedding models or vector databases needed
|
||||
- Full version control via git
|
||||
- Human-in-the-loop curation
|
||||
- Knowledge compounds over time
|
||||
- Works offline / on-premise
|
||||
Executable
+16
@@ -0,0 +1,16 @@
|
||||
#!/bin/bash
|
||||
# Wiki search utility — grep-based fallback (no qmd needed)
|
||||
# Usage: ./scripts/wiki-search.sh "search query"
|
||||
QUERY="$1"
|
||||
if [ -z "$QUERY" ]; then
|
||||
echo "Usage: wiki-search.sh <query>"
|
||||
exit 1
|
||||
fi
|
||||
echo "=== Searching wiki for: $QUERY ==="
|
||||
echo ""
|
||||
grep -rl --include="*.md" -i "$QUERY" "$(dirname "$0")/../wiki/" 2>/dev/null | while read file; do
|
||||
echo "--- $file ---"
|
||||
grep -n -i "$QUERY" "$file" 2>/dev/null | head -5
|
||||
echo ""
|
||||
done
|
||||
echo "=== Search complete ==="
|
||||
+2
-1
@@ -6,10 +6,11 @@
|
||||
<!-- pages about people, organizations, tools -->
|
||||
|
||||
## Concepts
|
||||
<!-- pages about ideas, techniques, theories -->
|
||||
- [[llm-wiki-pattern]] — LLM Wiki methodology by Karpathy
|
||||
|
||||
## Sources
|
||||
<!-- pages summarizing original documents, papers, articles -->
|
||||
- [[sample-llm-wiki-intro]] — Introduction to LLM Wiki concept
|
||||
|
||||
## Comparisons
|
||||
<!-- pages comparing tools, approaches, or ideas -->
|
||||
|
||||
@@ -2,5 +2,11 @@
|
||||
|
||||
> Append-only chronological record of all ingest and maintenance operations.
|
||||
|
||||
## 2026-05-10 17:30 — Wiki scaffold created
|
||||
- Created: `AGENTS.md` (schema), `wiki/overview.md`, `wiki/index.md`, `wiki/topics/llm-wiki-pattern.md`
|
||||
- Created: `raw/articles/sample-llm-wiki-intro.md` (sample article)
|
||||
- Created: `scripts/wiki-search.sh` (grep-based search utility)
|
||||
- Key takeaways: Initial setup following Karpathy's LLM Wiki pattern. Three-layer architecture in place (raw → wiki → schema).
|
||||
|
||||
---
|
||||
*Log started: 2026-05-10*
|
||||
Reference in new issue
Block a user