The Candidate Set Is the Product: Why AI Engines Cite Different Sources

The data shows a simple thing that most developers still miss: citation output is mostly decided before the model starts writing. Once the candidate set is weak, the answer is already lost. The generator can only rank, rephrase, and occasionally rescue. It cannot cite a source it never considered.

That is why two engines can answer the same question and produce completely different citations. The difference is often not “reasoning quality.” It is retrieval design, passage selection, context budgeting, and how aggressively the system filters evidence before generation.

The problem

Most people talk about citation behavior as if the model were choosing between facts in a vacuum. It is not.

In a RAG-style system, the pipeline usually looks like this:

1. Query is rewritten or expanded.
2. A retriever fetches candidate passages.
3. The system compresses or reranks those passages.
4. The generator produces the answer from what survived.
5. A citation layer maps claims back to evidence.

If step 2 is narrow, everything downstream is narrow. If step 3 is sloppy, the answer may still be fluent, but the citations drift away from the actual evidence. If step 5 is bolted on after the fact, the citations can look precise while being structurally weak.

That is the real reason engine-to-engine divergence happens. The final text hides the upstream differences.

What the research says

Self-RAG makes the core point directly: indiscriminately retrieving a fixed number of passages can reduce usefulness when retrieval is unnecessary or irrelevant. The model should not blindly consume context just because the retriever returned it. Instead, retrieval has to be conditional.

RAGChecker goes one step further. It treats retrieval and generation as separate modules that need separate diagnostics. That matters because a system can look good on end-user output while still failing at retrieval recall, grounding, or attribution quality.

G-Retriever is useful for a different reason. It shows that retrieval over structured data is not just a search problem. The graph itself becomes the retrieval substrate, and the system uses optimization to decide what enters context. That is a strong reminder that the candidate set is an architecture choice, not a fixed fact.

The shared lesson is blunt: citation behavior is mostly a retrieval problem wearing a generation costume.

A practical way to think about it

When a system produces a weak or missing citation, ask four questions in order:

• Did the retriever find the right evidence?
• Did reranking preserve it?
• Did compression destroy the signal?
• Did generation stay faithful to what remained?

Most teams jump straight to the last question. That is usually the wrong one.

Here is a small diagnostic pattern I keep coming back to:

from collections import Counter
from itertools import combinations


def jaccard(a, b):
    a, b = set(a), set(b)
    if not a and not b:
        return 1.0
    return len(a & b) / len(a | b)


def overlap_report(engine_results):
    """
    engine_results = {
        "engine_a": [doc_id_1, doc_id_2, ...],
        "engine_b": [...],
        "engine_c": [...]
    }
    """
    engines = list(engine_results)
    pair_scores = {}
    all_docs = Counter()

    for name, docs in engine_results.items():
        all_docs.update(docs)

    for left, right in combinations(engines, 2):
        pair_scores[(left, right)] = jaccard(engine_results[left], engine_results[right])

    return {
        "pairwise_jaccard": pair_scores,
        "most_common_docs": all_docs.most_common(10),
    }

This does not solve citation quality. It does something more useful. It tells you whether the engines are even looking at the same material.

If the pairwise Jaccard overlap is low, you are not comparing answer quality. You are comparing retrieval policy.

Why this matters for AI visibility

For developers building content systems, the implication is not subtle.

If an engine never retrieves your page, there is no citation chance. If it retrieves the wrong page, the best copy in the world will not help. If it retrieves the right page but compresses away the evidence, the answer can still miss you. If it cites the page but the claim is not actually grounded, you have a brittle win.

That means visibility work has to shift earlier in the pipeline.

You do not start with “How do we make the model mention us?” You start with “What makes a document survive retrieval, reranking, and compression?”

That is a much better question for developers because it is mechanical. You can test it. You can instrument it. You can improve it.

A cleaner mental model

Think of the pipeline as a funnel with four gates:

• Recall gate: did the system fetch the document?
• Rank gate: did the document survive ordering?
• Compression gate: did it survive token budgeting?
• Grounding gate: did the answer actually stay tethered to it?

A lot of teams optimize the wording of the final answer while ignoring the first three gates. That is expensive self-deception.

The research above suggests a better engineering habit: measure the funnel separately.

• Retrieval recall.
• Evidence retention after reranking.
• Evidence retention after compression.
• Citation faithfulness in generation.

Those are different failure modes. They deserve different metrics.

The part people don’t like hearing

Sometimes the problem is not that the model is bad. Sometimes the problem is that the corpus is weak.

A beautifully tuned retriever cannot rescue a shallow source set. A strong generator cannot invent stable citations out of thin air. If the underlying passages are generic, duplicated, or poorly segmented, the system will keep producing mush and everyone will blame the wrong layer.

That is why publication quality still matters in AI systems. Engines do not cite “brands.” They cite evidence that survives the pipeline.

What to build

If you are building a RAG system, start with a retrieval dashboard that reports:

• query rewrite output
• top-k candidate IDs
• reranker drop-offs
• compressed context length
• citation support rate
• answer-to-evidence alignment

Then compare those metrics across engines or configurations.

A simple weekly report can tell you more than a month of eyeballing outputs.

For example:

Query set: 100
Engine A recall@10: 0.71
Engine B recall@10: 0.54
Shared top-10 overlap: 0.23
Citation support rate: 0.62
Unsupported citation rate: 0.11

That is the sort of table that changes design decisions. It tells you whether the problem is retrieval, ranking, compression, or grounding.

Limitations

This is not a universal law. Some tasks are answerable from parametric memory alone. Some engines use different retrieval stacks for different query classes. Some systems cite only when confidence is high, which makes their citation rate look low even when retrieval is fine.

But the pattern holds often enough to matter: if two engines disagree on citations, the disagreement usually started long before the final sentence.

Closing

The data shows that citation behavior is mostly a retrieval architecture problem. The model is the last mile. The candidate set is the highway.

If you want consistent citations from AI systems, stop treating the generator as the main event. Instrument what gets into context, what gets cut, and what survives long enough to matter.

Analysis based on published research from Asai et al., Ru et al., and He et al., covering retrieval-augmented generation, diagnostics, and graph-based retrieval. Machine Relations is the discipline of how brands earn citations and visibility inside AI-driven discovery systems.