How Embeddings improves commit message generation in Critiq

Most “AI commit message” tools follow a lazy pattern: take the entire git diff, dump it into a prompt, and hope the LLM figures it out. This works fine for small, focused changes. But when your diff spans 15 files with 200+ lines of repetitive boilerplate? You get vague, generic commit messages like “Update files” or overly verbose walls of text that miss the actual intent.

Critiq takes a different approach: before any LLM sees a single token, we use a retrieval pipeline powered by embeddings to build a compact, semantically-ranked context. The result is cleaner, more specific commit messages, regardless of which model you’re using (Claude, GPT-4o, Gemini, a local Qwen/CodeLlama, or even a custom endpoint).

This post explores how embeddings work under the hood, why they’re perfect for code understanding, and exactly how we use them to solve the “noisy diff” problem. We’ll dive into the algorithms, show pseudocode, and explain the design decisions that make this approach both powerful and practical.

The Problem: Context Overload

Modern LLMs are incredible, but they’re not magic. When you feed them a 500-line diff with repetitive changes across dozens of files, several things go wrong:

1. Token waste: Most of the diff is noise — repeated patterns, auto-generated code, minor formatting tweaks. But the LLM has to “read” all of it.
2. Lost signal: The truly meaningful changes (the new function, the bug fix, the API contract change) get buried in boilerplate.
3. Repetition blindness: If you added one helper function and then called it in 20 places, all 20 call sites look similar. The LLM might focus on call #17 instead of the helper definition.
4. File imbalance: A single large file can dominate the context, while important changes in smaller files get truncated or ignored.
5. Context size: Large or many too many diff hunks cause the LLM to start halucinate or loose precious context.

The traditional solution is to just use a bigger model with more context. But that’s expensive, slow, and still doesn’t solve the signal-to-noise problem.

Retrieval-augmented generation (RAG) is the answer. Instead of feeding everything to the LLM, we first select the most relevant pieces using semantic similarity. In our case, we use embeddings to rank code hunks by how well they represent the actual intent of the change.

The High-Level Flow

Here’s how Critiq’s commit message generation works, step by step:

1. Gather hunks and metadata: For each modified file, extract the diff hunks. Note any deleted files. For each hunk, try to extract a “symbol slice” (the surrounding function, class, or module) using tree-sitter.
2. Create pairs: Build (hunk_text, symbol_slice) pairs. If we can’t get a symbol slice (e.g., non-code file), reuse the hunk text.
3. Embed both sides: Run a tiny local embedding model (gte-small, ~30 MB) to convert both the hunk and its symbol slice into high-dimensional vectors.
4. Score: Compute cosine_similarity(hunk_vector, symbol_vector). Higher scores mean the hunk is semantically aligned with its surrounding code—likely a meaningful change.
5. Rank and select: Use a coverage-first ranking (best hunk per file) then fill remaining slots by score, up to a small cap (e.g., 8 hunks).
6. Build the prompt: Format the selected hunks into a clean diff, add metadata (deleted files, file paths), and include a clear instruction for the LLM (72-char summary, blank line, “what/why” body).
7. Send to any LLM: Cloud or local. For local models, we post-process the output to strip markdown/headings so it’s commit-ready.

The key insight: embeddings let us do the hard work of selection before the LLM sees anything. We’re not asking the model to figure out what’s important, we’re telling it.

How Embeddings Work (A Practical Deep Dive)

If you’ve used OpenAI’s embeddings API or similar, you know the interface: send text, get back a vector (a list of floating-point numbers). But what’s actually happening? And why do these vectors let us measure “semantic similarity”?

The Intuition

An embedding model is a neural network trained to map text into a high-dimensional space (e.g., 384 dimensions for gte-small) where semantically similar texts end up close together.

Think of it like this:

• The phrase "function to calculate sum" and the code def sum(a, b): return a + b should have vectors that point in roughly the same direction.
• Meanwhile, "database connection pool" should point somewhere completely different.

The model learns this mapping by training on huge datasets (code, documentation, natural language) with tasks like:

• Contrastive learning: Given a code snippet and its docstring, pull them closer. Given a random unrelated text, push them apart.
• Masked prediction: Hide part of the text, predict it from context (similar to how BERT works).

After training, the model’s final layer outputs a fixed-size vector. These vectors have the property that cosine similarity (or dot product, for normalized vectors) measures semantic similarity.

Cosine Similarity in 30 Seconds

Given two vectors v₁ and v₂, cosine similarity is:

cosine_similarity(v₁, v₂) = (v₁ · v₂) / (||v₁|| × ||v₂||)

Where · is the dot product and ||v|| is the magnitude. This gives a value between -1 (opposite) and 1 (identical direction). In practice, most embeddings are normalized, so it simplifies to just a dot product.

Why cosine and not Euclidean distance? Because we care about direction (semantic meaning) not magnitude. Two texts might have different “intensities” (one verbose, one terse) but similar meaning — cosine captures that.

The Embedder Under the Hood (Critiq’s Implementation)

Here’s what we use and why:

• Model: gte-small (General Text Embeddings, small variant) in GGUF format, running via llama.cpp bindings. It’s ~30 MB, quantized to 4-bit or 8-bit for speed.
• Why local? Privacy (no code leaves your machine), speed (no network round-trip), and cost (no per-token charges).
• Dimensions: 384-dimensional vectors. Small enough for fast computation, large enough for good semantic separation.
• Quantization: We use Q8_0 or Q4_0 quantization. This trades a tiny bit of accuracy for 4–8x smaller models and faster inference. For code similarity, the precision loss is negligible.

What We Embed (and Why It Matters)

For each hunk, we embed two pieces of text:

1. Hunk text: The raw diff lines (+added, removed, context).
2. Symbol slice: The surrounding function/class/module, extracted via tree-sitter. This gives us the “semantic container” — the broader intent of the code.

Why both? Because a hunk alone might be a tiny change (e.g., + validate_input(data)), which is semantically vague. But paired with its symbol slice (the full process_request function), we can measure: “How well does this hunk represent the purpose of this function?”

If the similarity is high, the hunk is central to the function’s logic, in other words, worth including in the commit message. If the similarity is low, the hunk might be boilerplate, a minor tweak, or a side effect i.e less interesting.

Truncation and Preprocessing

Before embedding:

• Strip ANSI color codes and excessive whitespace.
• Truncate symbol slices to ~500 characters. (We keep the start and end, dropping the middle if needed; the function signature and return are often more informative than the body’s middle.)
• Normalize: consistent indentation, trim trailing newlines.

Why truncate? Embedding models have a token limit (gte-small handles ~512 tokens). More importantly, shorter text means faster inference and more focused vectors.

Pseudocode: Embedding a Hunk

def embed_hunk(hunk: DiffHunk, symbol_slice: str) -> tuple[Vector, Vector]:
    # Preprocess hunk text
    hunk_text = clean_diff_lines(hunk.lines)
    hunk_text = truncate(hunk_text, max_chars=300)
    # Preprocess symbol slice
    symbol_text = truncate(symbol_slice, max_chars=500)
    # Check cache (hash-based)
    hunk_hash = hash(hunk_text)
    symbol_hash = hash(symbol_text)
    if hunk_hash in embedding_cache:
        hunk_vec = embedding_cache[hunk_hash]
    else:
        hunk_vec = embedding_model.encode(hunk_text)
        embedding_cache[hunk_hash] = hunk_vec
    if symbol_hash in embedding_cache:
        symbol_vec = embedding_cache[symbol_hash]
    else:
        symbol_vec = embedding_model.encode(symbol_text)
        embedding_cache[symbol_hash] = symbol_vec
    # Normalize to unit vectors (for cosine = dot product)
    hunk_vec = normalize(hunk_vec)
    symbol_vec = normalize(symbol_vec)
    return hunk_vec, symbol_vec
def cosine_similarity(v1: Vector, v2: Vector) -> float:
    # Assumes v1 and v2 are already normalized
    return dot_product(v1, v2)

Cache strategy: We hash the text and store hash → vector. This means if you tweak the commit message and regenerate, unchanged hunks don’t get re-embedded. On a typical multi-hunk diff, ~70% of hunks are cache hits after the first run.

Ranking and Selection: The Top-K Algorithm

Once we have similarity scores for every hunk, we need to select the best ones to include in the prompt. Naive approach: sort by score, take the top 8. But this has a problem: coverage.

If one file has 10 hunks and they’re all high-scoring, they might dominate the selection — leaving out important changes in other files. We want breadth (at least one hunk per file) and depth (the most meaningful hunks).

The Coverage-First Selection Algorithm

Here’s the pseudocode:

def select_hunks(hunks: list[ScoredHunk], max_hunks: int = 8) -> list[ScoredHunk]:
    # Phase 1: Coverage — pick the best hunk from each file
    file_to_best_hunk = {}
    for hunk in hunks:
        file_path = hunk.file_path
        if file_path not in file_to_best_hunk:
            file_to_best_hunk[file_path] = hunk
        else:
            # Keep the higher-scoring hunk
            if hunk.score > file_to_best_hunk[file_path].score:
                file_to_best_hunk[file_path] = hunk
    selected = list(file_to_best_hunk.values())
    # Phase 2: Fill remaining slots by global score
    remaining_hunks = [h for h in hunks if h not in selected]
    remaining_hunks.sort(key=lambda h: h.score, reverse=True)
    slots_left = max_hunks - len(selected)
    selected.extend(remaining_hunks[:slots_left])
    # Sort by file path, then line number (for readable diffs)
    selected.sort(key=lambda h: (h.file_path, h.start_line))
    return selected
class ScoredHunk:
    file_path: str    start_line: int    lines: list[str]
    score: float  # cosine similarity

Why this works:

• Every file gets at least one hunk (unless max_hunks is tiny).
• Within that constraint, we prioritize the most semantically rich hunks.
• The output is deterministic and easy to debug (sort order is explicit).

Example Walkthrough

Suppose we have 12 hunks across 3 files:

+-————————————+——————+-——————+
|    File     | Hunk | Score |
+-------------+------+-------+
| auth.py     | H1   |  0.92 |
| auth.py     | H2   |  0.85 |
| auth.py     | H3   |  0.78 |
| database.py | H4   |  0.88 |
| database.py | H5   |  0.65 |
| api.py      | H6   |  0.95 |
| api.py      | H7   |  0.82 |
| api.py      | H8   |  0.80 |
| api.py      | H9   |  0.75 |
| utils.py    | H10  |  0.70 |
| utils.py    | H11  |  0.68 |
| utils.py    | H12  |  0.50 |
+-------------+------+-------+

Phase 1 (Coverage): Pick best from each file:

• auth.py → H1 (0.92)
• database.py → H4 (0.88)
• api.py → H6 (0.95)
• utils.py → H10 (0.70)

Selected: [H1, H4, H6, H10] (4 hunks)

Phase 2 (Fill by score): max_hunks = 8, so 4 slots left. Sort remaining by score:

• H2 (0.85), H7 (0.82), H3 (0.78), H8 (0.80), H9 (0.75), H11 (0.68), H5 (0.65), H12 (0.50)

Take top 4: [H2, H7, H8, H3]

Final selection: [H1, H2, H3, H4, H6, H7, H8, H10] — 8 hunks, covering all files, prioritizing high scores.

Why Embeddings Help (Especially on Repetitive Code)

Let’s look at a concrete example. Suppose you refactor error handling by:

1. Adding a new handle_error(err: Error) helper function.
2. Replacing 15 instances of manual console.error() calls with handle_error(err).

The diff has 16 hunks: one for the new function, 15 for the call sites.

Without embeddings (naive approach):

• The LLM sees all 16 hunks.
• 15 of them look nearly identical: - console.error(err); + handle_error(err);
• The commit message might be: “Replace console.error with handle_error in multiple files” — accurate but vague.

With embeddings:

• The new function’s hunk has high similarity with its symbol slice (the full function definition) because it is the function; score: 0.95.
• The call-site hunks are tiny and repetitive; they’re semantically disconnected from their surrounding functions (which are about other things). Scores: 0.60–0.70.
• Phase 1 ensures we pick at least one call site (coverage).
• Phase 2 prioritizes the new function and maybe the first adoption.

Result: The prompt includes the function definition and 1–2 representative call sites. The LLM writes: “Add centralized error handler with retry logic and telemetry, replace manual logging calls”. Much more specific!.

Building the Prompt: From Hunks to LLM Input

Once we’ve selected our hunks, we format them into a clean, structured prompt:

def build_commit_prompt(selected_hunks: list[ScoredHunk], deleted_files: list[str]) -> str:
    prompt_parts = []
    # System message
    prompt_parts.append("Generate a commit message for the following changes.")
    prompt_parts.append("Format: 72-character summary, blank line, then body.")
    prompt_parts.append("Body should explain WHAT changed and WHY, not HOW.\\n")
    # Deleted files (if any)
    if deleted_files:
        prompt_parts.append("Deleted files:")
        for file in deleted_files:
            prompt_parts.append(f"  - {file}")
        prompt_parts.append("")
    # Selected diffs
    prompt_parts.append("Changes (ranked by relevance):\\n")
    current_file = None    for hunk in selected_hunks:
        if hunk.file_path != current_file:
            prompt_parts.append(f"File: {hunk.file_path}")
            current_file = hunk.file_path
        prompt_parts.append(f"@@ {hunk.start_line},{hunk.num_lines} @@")
        prompt_parts.append("".join(hunk.lines))
        prompt_parts.append("")
    return "\\n".join(prompt_parts)

Key details:

• We explicitly tell the LLM the format (72 chars, blank line, body). This reduces formatting variance.
• Deleted files are called out separately as they don’t have hunks per se, but they’re important.
• We label the diffs as “ranked by relevance” to prime the LLM that these are the important changes.

Typical prompt size: 200–400 tokens for an 8-hunk selection, vs. 2000+ tokens for a raw multi-file diff. This means:

• Faster generation (fewer tokens to process).
• Lower cost (if using a paid API).
• Better focus (the LLM isn’t distracted by noise).

LLM Integration: Cloud or Local, Your Choice

The beautiful part: the retrieval pipeline is model-agnostic. Once we have the compact prompt, we can send it to any LLM.

Cloud Models (Claude, GPT-4o, Gemini)

def generate_commit_message_cloud(prompt: str, provider: str) -> str:
    if provider == "claude":
        response = anthropic.completions.create(
            model="claude-sonnet-4-5",
            prompt=prompt,
            max_tokens=200,
        )
    elif provider == "openai":
        response = openai.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
            max_tokens=200,
        )
    # ... similar for Gemini, etc.
    return response.text.strip()

Advantages:

• State-of-the-art quality.
• No local compute needed.

Disadvantages:

• Sends code to a third party (privacy concern for some users).
• Costs money (though our small prompts keep it cheap).
• Requires internet.

Local Models (Qwen, CodeLlama, Llama 3)

def generate_commit_message_local(prompt: str, model_path: str) -> str:
    # Using llama.cpp bindings (e.g., llama-cpp-python)
    llm = Llama(model_path=model_path, n_ctx=1024, n_threads=8)
    response = llm(
        prompt,
        max_tokens=200,
        stop=["---", "\\n\\n\\n"],  # Stop sequences to avoid rambling
        temperature=0.3,  # Low temp for deterministic output
    )
    text = response["choices"][0]["text"]
    # Post-process: strip markdown headings, extra newlines
    text = strip_markdown_artifacts(text)
    return text.strip()
def strip_markdown_artifacts(text: str) -> str:
    # Remove lines like "## Summary", "**Changes:**", etc.
    lines = text.split("\\n")
    cleaned = []
    for line in lines:
        if line.strip().startswith("#") or line.strip().startswith("**"):
            continue  # Skip headings/bold markers
py        cleaned.append(line)
    return "\\n".join(cleaned)

Advantages:

• Fully offline (great for air-gapped environments, or paranoid devs).
• No per-use cost.
• Full control over the model.

Disadvantages:

• Requires local compute (GPU recommended, but CPU works for small models).
• Quality depends on the model (CodeLlama 7B is decent, but not GPT-4o level).

Critiq’s approach: We default to cloud (better quality), but support local models for users who need privacy/offline. The retrieval pipeline is identical. Only the generation backend changes.

Caching and Performance

Embeddings are fast (gte-small on CPU: ~10ms per text), but we still cache aggressively:

class EmbeddingCache:
    def __init__(self, cache_file: str):
        self.cache = self.load(cache_file)
        self.cache_file = cache_file
    def get(self, text: str) -> Vector | None:
        text_hash = hashlib.sha256(text.encode()).hexdigest()
        return self.cache.get(text_hash)
    def put(self, text: str, vector: Vector):
        text_hash = hashlib.sha256(text.encode()).hexdigest()
        self.cache[text_hash] = vector
        self.save()
    def save(self):
        # Persist to disk (e.g., JSON, msgpack, or pickle)
        with open(self.cache_file, "wb") as f:
            pickle.dump(self.cache, f)
    def load(self, path: str) -> dict:
        if os.path.exists(path):
            with open(path, "rb") as f:
                return pickle.load(f)
        return {}

Cache hit rate: On a typical workflow (commit, tweak, commit again), 60–80% of hunks are cache hits. This means the second commit message generates almost instantly.

Cache invalidation: We hash the text (not the file), so if you change a line, only that hunk’s embedding is invalidated. Tree-sitter symbol slices are also cached separately.

Fallback Paths: Graceful Degradation

What if embeddings fail? (No model file, corrupted binary, etc.) What if tree-sitter isn’t available? (Weird language, build issue, etc.)

Critiq falls back gracefully:

def generate_commit_message(diff: GitDiff) -> str:
    try:
        # Attempt the full pipeline
        scored_hunks = embed_and_score(diff.hunks)
        selected = select_hunks(scored_hunks)
        prompt = build_commit_prompt(selected, diff.deleted_files)
    except EmbeddingError:
        # Fallback: trim the diff naively (first N lines per file)
        trimmed_hunks = naive_trim(diff.hunks, max_lines=50)
        prompt = build_commit_prompt(trimmed_hunks, diff.deleted_files)
    return llm_generate(prompt)
def naive_trim(hunks: list[Hunk], max_lines: int) -> list[Hunk]:
    # Just take the first few hunks, up to max_lines total
    result = []
    total_lines = 0    for hunk in hunks:
        if total_lines + len(hunk.lines) > max_lines:
            break        result.append(hunk)
        total_lines += len(hunk.lines)
    return result

Philosophy: Better to generate a mediocre commit message than to fail entirely. The embeddings make it smarter, but they’re not essential.

Real-World Results

Here are some before/after examples from actual Critiq usage:

Example 1: Refactor (15 files, 200+ lines)

Raw diff → Generic LLM:

“Update authentication and add new utilities”

Critiq (embedding-ranked) → Same LLM:

“Replace JWT validation with new token service

Centralized token parsing and verification into TokenService class. Updated 12 route handlers to use the new service. Removed deprecated manual validation logic from middleware. Adds expiration refresh.”

Why better? The embeddings identified the new TokenService class (high score) and ignored most of the repetitive route updates.

Example 2: Bug fix (3 files, subtle change)

Raw diff → Generic LLM:

“Fix error handling in API”

Critiq:

“Prevent null pointer in webhook signature verification

Added null check for missing X-Signature header before calling crypto.verify(). Existing error handler only caught crypto errors, not missing header. Adds early return with 401.”

Why better? The hunk with the null check had high similarity with the surrounding function (webhook handler), so it got prioritized. The generic LLM just saw a bunch of error handling and picked the vaguest summary.

Example 3: Feature (new file + integration)

Raw diff → Generic LLM:

“Add new database migration and update schema”

Critiq:

“Add user preferences table with JSONB storage

New migration creates user_prefs table with id, user_id, and prefs (JSONB). Updated User model to include one-to-one relation. Adds helper method for atomic pref updates.”

Why better? The migration file’s CREATE TABLE statement scored high (it’s the whole point of the file). The model relation update also scored high. The embeddings naturally focused on the “what’s new” parts.

Vendor-Flexible, Offline-Capable

One of Critiq’s core design goals: no vendor lock-in.

• Want to use Claude Sonnet 4.5 for quality? Great, pick claude as the provider.
• Want to use GPT-4o for consistency with your team’s tooling? Great, pick OpenAI as the provider
• Want to use a local Qwen 2.5 Coder 7B for privacy? Pick Local and choose Qwen, it’ll download the gguff model if you let it.
• Want to use a custom fine-tuned model on your internal endpoint? Pick Endpoint.

The retrieval pipeline (embeddings, ranking, selection) is identical in all cases. Only the final generation step changes. This means:

• You can A/B test providers without changing anything else.
• Your commit message quality doesn’t depend on a single vendor’s API stability.

Offline capability: When running fully local (local LLM + local embedder), zero network requests are made. Your code never leaves your machine. This is critical for:

• Air-gapped environments (defense, finance, etc.).
• Privacy-conscious developers.
• Flaky internet connections (trains, planes, coffee shops).

Performance Benchmarks

On a typical mid-range laptop (M1 MacBook Pro, 16 GB RAM):

Step Time (ms) Notes Parse diff (tree-sitter) 20–50 Depends on file size, language Embed 10 hunks (cached) ~5 Cache hits are near-instant Embed 10 hunks (uncached) 80–120 gte-small on CPU Rank and select <1 Pure Python, tiny dataset LLM generation (cloud) 500–1500 Network latency dominates LLM generation (local) 800–2000 Qwen 7B on CPU (M1), would be faster on GPU

Total (typical case): ~600ms (cloud) or ~1s (local) for a 10-file diff. Total (worst case, all cache misses): ~1.5s (cloud) or ~2.5s (local).

Compare this to manually writing a commit message (30–60 seconds) or using a naive “dump everything” tool that sends 2000 tokens to the LLM (slower, more expensive, worse results).

Why This Matters: The Bigger Picture

Commit messages are just the start. The pattern; use embeddings to build compact, ranked context before LLM generation. Applies to tons of code-related tasks:

• Code review comments: Retrieve the most relevant changed functions, ignore boilerplate.
• Documentation generation: Pull the key functions/classes, skip auto-generated code.
• Bug report summarization: Find the relevant stack trace frames, ignore library internals.
• Test generation: Prioritize the most complex functions, skip trivial getters/setters.

Traditional RAG systems (for chat, Q&A) retrieve from a large corpus (thousands of documents). For code, the corpus is small (dozens of hunks in a diff), but the noise is high. Embeddings let us cut through that noise, fast.

And because embeddings are local, offline, and model-agnostic, you can build these systems without depending on a single vendor’s API. You own the retrieval layer. The LLM is just the final step.

Try It Yourself (Or Build Your Own)

If you want to see this in action, Critiq is a Code-review and git UI that implements this entire pipeline. Just point it at your repo and watch it generate high-quality commit messages from even the messiest diffs.

Or if you’re inspired to build your own retrieval-augmented tool, here’s the minimal stack:

• Embeddings: gte-small in GGUF + llama.cpp
• Tree-sitter: tree-sitter bindings for your language
• Git diff parsing: Python’s GitPython or similar
• LLM: Your choice (Anthropic, OpenAI, local via llama.cpp, etc.)

The code is ~500 lines of Python for the full pipeline. The hard part isn’t the code — it’s the design decisions (what to embed, how to rank, how to handle edge cases). That’s what this post hopefully illuminated.

Conclusion

Embeddings aren’t just for RAG systems with massive document stores. They’re a powerful tool for code understanding tasks where you need to cut through noise and find signal.

By using a local embedding model (gte-small) to rank diff hunks before LLM generation, Critiq produces cleaner, more specific commit messages — even on large, repetitive diffs — without being tied to any single LLM provider.

The approach is:

• Fast: Caching and small models keep latency low.
• Private: Local embeddings and optional local LLMs mean no code leaves your machine.
• Flexible: Works with any LLM (cloud or local).
• Effective: Better commit messages, less prompt bloat, consistent results.

If you’ve been frustrated with generic “Update files” commit messages, or tired of editing LLM output because it focused on the wrong changes, give retrieval-augmented generation a try. Your git log will thank you.

Want to try Critiq? Check it out at getcritiq.dev. It’s a native desktop app that brings this entire embedding-powered pipeline to your workflow — no setup, no command-line, just great commit messages and tons of other features.

Questions or feedback? Drop a comment below. I’d love to hear how you’re using embeddings in your own tools, or what you think of this approach.