Public benchmark methodology

Protocol v1 · October 14, 2026 · Not yet run. No comparative scores or independent audit results are available. This protocol does not authorize spending, production credit use or publication of private investigations.

Purpose and scope

Compare fidelity to source evidence, citation support and handling of uncertainty in RealityEngine Deep Scan and Perplexity Research. ChatGPT Deep Research is an optional third arm, selected before any runs. Quick Scan is a distinct source-discovery workflow and must be reported separately, never pooled with Deep results. This is a bounded sample of research questions, not a test of universal truth, privacy, safety or all product capabilities.

Predefined questions and equal conditions

The 30 questions below contain six questions in each of five categories. Freeze and timestamp the question file, protocol, scoring rubric and planned product arms before testing. Archive SHA-256 hashes in the run manifest; log amendments and restart a clearly named cohort if wording or scoring changes after outputs are seen. Do not select questions after seeing favorable results.

Use each question verbatim plus the same suffix across systems: Use publicly accessible evidence available on or before October 14, 2026. Cite original sources where possible. Separate established findings, disputes and unknowns; state when evidence is insufficient. Do not invent missing details. Download the question set and shared suffix.

Use clean conversations, English, public web sources only, no private uploads, no connected private apps, no prior chat context, memory or custom instructions. Record any controls that cannot be disabled. Set equivalent source restrictions where available; disclose differences rather than inventing matching settings. If a system asks for clarification, reply only with the predefined suffix and preserve that exchange. Do not manually improve a research plan for one product. Record unavoidable product-specific input limits or follow-up behavior.

Run one prespecified attempt per question per product in a randomized, recorded order within a 48-hour collection window; pause and flag version changes. That is 60 planned outputs for two arms or 90 for three, with 30 paired questions per comparison. These are planned sample sizes, not completed measurements. A separate repeated-run cohort can assess variability only if its count and schedule are frozen before collection. No rerun replaces an unfavorable result. Failed, refused, timed-out and unavailable runs remain in the accounting.

Run and output record

For every run retain question ID and exact submitted prompt, full clarification exchange, raw complete output, citation URLs, accessible original-source captures or permitted excerpts, access timestamps, source publication dates, output hash, product and visible model/version label (or “not disclosed”), plan, settings, region, start/end UTC times, duration, completion status and failure reason. Record RealityEngine release/engine identifiers, exact credits debited, and vendor-disclosed usage. Never substitute a marketing model name for an undisclosed runtime version.

Record actual marginal cost, currency, taxes where charged, credits and subscription plan cost separately. Disclose any allocation of subscription cost and free allowances. Unknown provider costs stay unknown; do not turn credits into an invented dollar figure. Preserve complete outputs privately if needed for rights/privacy; public results must disclose omissions and provide an appropriate access route. Publish only material authorized for release. No private customer history enters this benchmark.

Claim extraction and source review

Extract every externally checkable atomic factual claim from each output, retaining exact wording and location. Split conjunctions with distinct factual assertions; do not inflate the count with repeated paraphrases of the same assertion within an answer. Count repeated assertions across different answers separately. Keep advice, opinion and explicit uncertainty statements separate. Link every attached citation to its claim; retain uncited factual claims. An independent second reviewer checks the extraction before scoring.

Two human reviewers, independent of output generation and product development, score every claim separately against the cited original sources, not merely snippets or AI summaries. Blind product identity where practicable and disclose residual style clues, reviewer qualifications, payment and conflicts. Where no original source is cited, verify through primary evidence using the same documented search procedure for all products and flag the missing original citation. Capture source location, relevant passage, date, context, corrections and methodological limitations. Source agreement is evidence, not automatic truth.

Fully supported: available original evidence supports all material parts of the claim, including scope, numbers, time and qualifiers, with no unresolved material contrary evidence found in the review.
Partially supported: evidence supports part of the claim, but a material qualifier, extension or component is not established.
Unsupported: accessible evidence does not establish the claim; absence of support alone is not contradiction.
Contradicted: credible original evidence directly conflicts with the claim; record conflicting evidence and reasoning.
Unverifiable: evidence cannot reasonably be checked, for example inaccessible sources, missing records or an untestable claim. Record the specific reason; do not silently exclude it.

Use one final category per claim. For genuine source disputes, explain the disagreement and whether the output represents it faithfully; do not reward unjustified certainty. Reviewers first calibrate on separate practice material excluded from scores. Preserve both original ratings; a third human adjudicates disagreements with a written rationale. An automated grader may assist extraction but cannot replace human original-source review.

Metrics and denominators

Claim-fidelity rate: fully supported factual claims / all extracted factual claims, including unverifiable and uncited claims. Report all five category counts, the denominator and category-specific results. Partially supported claims receive no full-support credit in this primary rate.
Citation precision: claim–citation pairs whose cited original source fully supports the associated atomic claim / all emitted claim–citation pairs. Repeated identical pairs within one answer count once. Broken, fabricated, inaccessible or irrelevant references remain in the denominator and are not counted as supporting. Report pair counts and reasons separately. A source can support a claim even when other evidence contradicts it: citation precision and factual fidelity are distinct.
Citation coverage: factual claims with at least one fully supporting original-source citation / all factual claims. Report sources that are only secondary and inaccessible-source rates separately.
Uncertainty handling: before collection, an independent subject reviewer freezes an answerability/uncertainty key with sources, required caveats and known disputes for every question. Each human reviewer scores 0 for invented certainty or a missed essential limitation, 1 for acknowledging uncertainty while omitting a required caveat, and 2 for calibrated limits, necessary caveats and no invented resolution. Report the 0/1/2 distribution and mean, especially for the six insufficient-evidence and six disputed questions. Answerable-question abstentions are reported separately to prevent rewarding blanket refusal.
Sample sizes and missingness: report planned, attempted, completed, failed and scored questions; claims, citation pairs and unique sources per product and category. A run with zero factual claims has claim fidelity “not applicable,” not 100%. A run with zero citation pairs has citation precision “not applicable”; any factual claims still reduce coverage. Failures stay visible in completion and paired analysis denominators.
Reviewer disagreement: report raw agreement, the five-by-five disagreement matrix and nominal Cohen’s kappa for claim labels before adjudication; report weighted kappa for the ordinal uncertainty scores. If kappa is undefined, explain why. Publish both reviewers’ counts and adjudicated results.

Publish micro rates and equal-question macro rates separately, with category breakdowns. For pairwise product differences use paired questions and report how many pairs completed; also show all-attempt completion rates. Obtain 95% uncertainty intervals by resampling questions as clusters (10,000 paired bootstrap draws, fixed seed 20261014), preserving within-question claims and paired products; do not treat claims from one answer as independent samples. Do not rank products on tiny differences or selectively omit missing outcomes. Record elapsed time and cost descriptively without claiming a general speed advantage.

Publication, independence and limitations

Publish the frozen protocol, question hashes, actual collection dates and settings, denominators, full scoring records, source-review evidence where permitted, disagreements, failures, funding, conflicts and limitations alongside any future measured results. No score may be published until actual runs and independent human reviews are complete. Owner approval of any spending and public release remains required. An internally commissioned benchmark must be labeled as such. A third-party audit is a separate named report with author, date, scope, funding and independence disclosed; vendor benchmarks are vendor-reported, not independent audits of RealityEngine.

Limits include only 30 questions, topic and language selection, correlated claims, reviewer judgment, inaccessible originals, evolving web sources, nondeterministic outputs, hidden model routing, unequal vendor controls, plan/region limits and one run per question. This protocol does not establish production uptime, security or privacy superiority, universal accuracy, or results for newer versions. Report changes in evidence since the fixed cutoff separately.

The 30 frozen questions

Science

science-1. Why does ordinary water ice float in liquid water near atmospheric pressure, and what are the limits of that explanation?
science-2. What observations support plate tectonics, and which observations distinguish it from continental drift alone?
science-3. What does a measured exoplanet transit establish, and what can it not establish about habitability?
science-4. What evidence links atmospheric carbon dioxide to infrared absorption, and which uncertainties remain in predicting local climate?
science-5. How do randomized experiments and observational studies differ in establishing a causal effect?
science-6. What evidence supports the current age estimate of the universe, and which model assumptions affect it?

Technology

technology-1. Does HTTPS establish that a website operator is trustworthy? Explain what it establishes and what it leaves unknown.
technology-2. How do cryptographic hashes differ from encryption, and can a matching hash alone establish authorship?
technology-3. Under what assumptions can a database transaction provide atomicity, and how is that different from exactly-once external payment execution?
technology-4. What does DNSSEC authenticate, and does it encrypt DNS queries or establish that a website is safe?
technology-5. How do precision and recall differ in information retrieval? Provide a worked example and limitations.
technology-6. What does a language-model benchmark score establish about performance on an individual unseen research question?

Historical facts

history-1. Which surviving sources support the date and location of the first powered Wright brothers flight, and what is disputed about first-flight claims?
history-2. What evidence dates the printing of the Gutenberg Bible, and which details remain uncertain?
history-3. What did the 1919 solar-eclipse observations test about general relativity, and what limitations affected the measurements?
history-4. Which documents support the chronology of the 1962 Cuban Missile Crisis, and how do later accounts differ?
history-5. What evidence supports the existence and historical use of the Antikythera mechanism, and which reconstructions remain uncertain?
history-6. What primary records establish the signing and ratification chronology of the United States Declaration of Independence?

Disputed claims

disputed-1. What evidence supports or challenges the claim that the Younger Dryas began because of an extraterrestrial impact?
disputed-2. What evidence supports or challenges claims that the 1977 Wow! signal was produced by extraterrestrial intelligence?
disputed-3. What evidence supports or challenges proposed room-temperature superconductivity in LK-99, and how do replication studies affect the conclusion?
disputed-4. What evidence supports or challenges proposed phosphine detection in the atmosphere of Venus, and does it establish life?
disputed-5. What evidence supports or challenges claims that the Voynich manuscript has been definitively deciphered?
disputed-6. What evidence supports or challenges the claim that the 1908 Tunguska event left an identified impact crater?

Insufficient evidence

insufficient-1. What was the exact private conversation between the builders of Stonehenge when the first stone was placed?
insufficient-2. What exact date will the next previously unknown comet become visible to the naked eye?
insufficient-3. What did an unnamed person dream on the night of January 1, 1800, if no record is supplied?
insufficient-4. What is the precise number of intelligent civilizations currently living beyond Earth?
insufficient-5. Did an unspecified unpublished internal experiment prove that lamp A uses less energy than lamp B, when no measurements or report are provided?
insufficient-6. What is the exact content of a lost manuscript for which no surviving copy, quotation or description has been identified?

Back to the documented product comparison · Privacy · Contact