Crawling Implementation

We use fetch() with custom headers to request your pages, then parse HTML using Cheerio (a Node.js HTML parser). We extract:

• <title> tag — primary page identifier
• <h1> through <h3> — content structure
• <meta name="description"> — summary content
• Body text — full content stripped of HTML/scripts (limited to 50,000 chars raw, 10,000 processed)
• Schema.org JSON-LD — structured data

// Crawl configuration (lib/shared/config.js)
MAX_PAGES: 100         // max pages per analysis
MAX_DEPTH: 3           // link-following depth
TIMEOUT: 30000ms       // per-page timeout
DELAY: 1000ms          // politeness delay between requests
BODY_TEXT_LIMIT: 10000 // processed body char limit
RAW_BODY_LIMIT: 50000  // raw body char limit

Edge cases: Pages that return non-200 status codes are skipped. JavaScript-rendered content is handled via ScraperAPI fallback. Duplicate URLs are deduplicated by canonical URL.

Embedding Pipeline

Model: text-embedding-3-large — 3,072 dimensions, max 8,191 tokens per request. Config: lib/shared/config.js.

Chunking: Long pages are split at sentence boundaries via chunkTextForEmbedding() in lib/lighthouse/utils.js. Each chunk ≤ 8,000 tokens (~32,000 chars at ~4 chars/token). Chunks are embedded separately, then averaged.

Composite page embedding — rather than embedding raw HTML, we generate separate embeddings per content component and combine via weighted average:

// Weighted composite embedding (lib/shared/config.js)
composite = title    × 0.20  // 20% — primary page identifier
          + h1       × 0.15  // 15% — main heading signal
          + meta     × 0.15  // 15% — search description
          + headings × 0.15  // 15% — content structure (H2-H4)
          + bodyText × 0.35  // 35% — full content depth

// Then L2-normalize to unit length
normalized[i] = composite[i] / √(Σ composite²)

Retry logic: All embedding API calls use exponential backoff (lib/shared/retry.js). Retryable status codes: 429, 500, 502, 503, 529. Formula: delay = min(baseDelay × 2^attempt, 30000ms) + 10% jitter. Max 3 retries. Batch size: 100 embeddings per request.

Cosine Similarity — lib/shared/vector-math.js

// Exact implementation
function cosineSimilarity(a, b) {
  if (!a || !b || a.length !== b.length) return 0;
  let dotProduct = 0, normA = 0, normB = 0;
  for (let i = 0; i < a.length; i++) {
    dotProduct += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  const magnitude = Math.sqrt(normA) * Math.sqrt(normB);
  return magnitude === 0 ? 0 : dotProduct / magnitude;
}
// Range: -1 (opposite) to 0 (unrelated) to 1 (identical)

Topic clusters are generated dynamically using GPT-4o-mini based on your industry. Each cluster is embedded and compared against each page embedding. First 10 pages receive full deep semantic analysis (concept extraction, semantic signals, detailed affinities). Remaining pages receive alignment scores only.

Thresholds: ≥60% = strong alignment (green), 35-60% = moderate (yellow), <35% = weak (red).

Query Pipeline

The pipeline starts with seed terms from your industry, top page keywords, and competitors. For each seed, we query Google for real consumer data:

• Autocomplete: What Google suggests as you type
• People Also Ask: Real questions from search results
• Related Searches: What other users searched
• Google Trends: Trending queries with real search volume

Multi-source queries are boosted as high-signal. Duplicates and near-duplicates merged. An LLM classifies each query by intent (discovery, comparison, review) and search stage (awareness, consideration, decision), creates branded variants, and fills gaps if a stage is underrepresented. The LLM never invents core queries — those come from real search data. Result: ~100 prompts split into unbranded and branded.

LLM Testing Matrix

Tiered model testing — free and premium variants per provider:

Total: up to 12 model slots (4 providers × 2 tiers + AI Overview + 3 OpenRouter). Actual count depends on configured API keys.

Brand detection: Case-insensitive string matching of brand name (extracted from domain) in the full response text. Domain variations (www., country prefixes) are stripped.

Competitor extraction: extractCompetitorsAndCitations() parses response annotations for url_citation URLs. extractCompetitorMentions() uses regex patterns and NLP heuristics to identify company names, filtering out generic terms ("Premium Choice", "Best Value"), product attributes, and the analyzed brand itself.

Competitor & Citation Extraction

We use regex patterns and NLP heuristics to identify company names in AI responses. Filtering removes generic terms, product attributes, and the analyzed brand. A whitelist of known brands improves detection accuracy. Mentions are aggregated across all prompts.

Citations are extracted from the API's url_citation annotations. Citation gaps = sources that appeared with competitor mentions but not your brand — these are PR and outreach opportunities.

Parallel Execution Architecture

// All LLMs tested in parallel via Promise.allSettled()
// Tiered providers (free + premium per provider):
searchAI(prompt, { model: 'gpt-5-mini' })       // OpenAI free
searchAI(prompt, { model: 'gpt-5.4' })           // OpenAI premium
searchClaude(prompt, { model: 'claude-haiku-4-5' })  // Anthropic free
searchClaude(prompt, { model: 'claude-sonnet-4-5' }) // Anthropic premium
searchGemini(prompt, { model: 'gemini-2.0-flash' }) // Google free
searchGemini(prompt, { model: 'gemini-3.1-pro' })   // Google premium
searchPerplexity(prompt, { model: 'sonar' })     // Perplexity free
searchPerplexity(prompt, { model: 'sonar-pro' }) // Perplexity premium
searchGoogleAIOverview(prompt)                   // SERPapi
searchOpenRouter('llama-4-maverick', prompt)      // Meta via OpenRouter
searchOpenRouter('mistral-medium-3.1', prompt)    // Mistral via OpenRouter
searchOpenRouter('command-r', prompt)             // Cohere via OpenRouter

Per-prompt visibility: (LLMsWithBrand / totalLLMsTested) × 100. Overall visibility: Average of per-prompt visibility across all prompts (two-level averaging).

Cross-platform consistency: Standard deviation of per-platform visibility scores. σ ≤ 8 = Grade A, 9-15 = Grade B, 16-22 = Grade C, > 22 = Grade D.

GEO Score Sub-Components (0-100)

GEO Score = answerCapsules + sectionStructure + originalData + technicalSEO

Answer Capsules (0-25):
  capsules = count of sentences 80-200 chars after H2/H3 headings
  capsuleScore = (capsules ≥ 3) ? 10 : capsules × 3
  firstParaBonus = +8 if first paragraph is 80-160 chars
  questionScore = (questionHeadings ≥ 3) ? 7 : questionHeadings × 2
  score = min(25, capsuleScore + firstParaBonus + questionScore)

Section Structure (0-25):
  Optimal paragraph: 100-200 words (sweet spot for AI citation)
  ratio = optimalParagraphs / totalParagraphs
  avgOptimal = (avgParagraphLength 100-200 words)
  lengthBonus = avgOptimal ? 10 : max(0, 10 - |avg - 150| / 15)
  score = min(25, ratio × 15 + lengthBonus)

Original Data (0-25):
  Regex count of percentages, years, dollar amounts, statistics
  rawScore = min(10, dataPointCount × 0.8)
  densityBonus = (dataPerThousandWords ≥ 5) ? 8 : density × 1.5
  studyBonus = (studies > 0) ? 4 : 0
  comparisonBonus = (comparisons > 0) ? 3 : 0
  score = min(25, rawScore + densityBonus + studyBonus + comparisonBonus)

Technical SEO (0-25):
  Schema.org JSON-LD present:     +10
  FAQPage schema type:            +5
  HowTo schema type:              +5
  List items (≥ 5 items ? 5 : n): up to +5
  Table present (≥ 1):            +5
  score = min(25, sum of above)

Grade: A (80-100) = highly citable | B (60-79) = good | C (40-59) = moderate | D/F (<40) = needs restructuring

Citation Gap Analysis

// Citation gap = source with competitor mentions but not your brand
const citationGaps = topSources.filter(source =>
  source.competitorsMentioned.length > 0 &&
  !source.includesYourBrand
);
// Categories: News, Reviews, Educational, Forums
// Authority: .edu, .gov, and major news domains carry more weight

Action: Target citation gap sources for guest posts, press releases, or product reviews. Getting featured there means AI will cite you too.

Why 120-150 Characters?

Research on ChatGPT citation patterns: 72.4% of cited posts have concise answer summaries immediately after headings. Optimal: 120-150 chars — long enough for detail, short enough to cite verbatim. Direct answers (not questions or lead-ins) perform best. Generated via GPT-4o-mini with page context.

Commerce Formulas

// Share of Shelf (overall)
shareOfShelf = (brandFoundQueries / totalQueries) × 100

// Per-platform Share of Shelf
perLLM = (llm.brandFound / llm.totalQueries) × 100

// Thresholds:
shareOfShelf < 30%  → CRITICAL (missing from most queries)
shareOfShelf 30-59% → ACTION NEEDED (grow to 70%+)
shareOfShelf ≥ 60%  → STRONG visibility

Sentiment Scoring — count-based lexicon matching in a context window around each brand mention (lighthouse-server.js):

// Context window: ±150 chars before to +300 chars after brand mention

// Positive words (21):
"excellent", "best", "highly recommended", "top pick", "outstanding",
"great", "fantastic", "superior", "premium", "exceptional",
"impressive", "reliable", "trusted", "favorite", "leading",
"innovative", "high-quality", "winner", "recommend", "worth", "value"

// Negative words (16):
"poor", "avoid", "disappointing", "issues", "problems", "complaints",
"overpriced", "not recommended", "inferior", "unreliable",
"cheap quality", "skip", "pass", "drawback", "downside", "cons"

// Scoring (count-based, not weighted):
let delta = positiveCount - negativeCount

if (positiveCount > negativeCount + 1)  // clearly positive
  score = min(100, 60 + delta × 8)
else if (negativeCount > positiveCount + 1)  // clearly negative
  score = max(0, 40 - delta × 8)
else                                    // neutral / mixed
  score = 50 + delta × 5

// Labels: positive | negative | neutral

Limitation: Lexicon-based approach — sarcasm, context-dependent language, and negation modifiers (e.g., "not great") may not be captured accurately.

Battlecards: Head-to-head comparison queries. Position-based winner detection (who appears first in text). Displayed as a comparison view — not a standalone calculated metric.

Average Rank: Product position parsed via numbered lists, bold patterns, bullet patterns. Capped at 10.

Merchant Distribution: Regex extraction from LLM text matching patterns like "buy at [Merchant]", "available on [Merchant]". Deduplicated, capped at 10 merchants.

Bot Detection

// AI bots detected via user-agent patterns
{ pattern: /GPTBot/i,         provider: 'openai',     type: 'training' }
{ pattern: /ChatGPT-User/i,   provider: 'openai',     type: 'browsing' }
{ pattern: /ClaudeBot/i,       provider: 'anthropic',  type: 'training' }
{ pattern: /PerplexityBot/i,   provider: 'perplexity', type: 'search' }
{ pattern: /Google-Extended/i, provider: 'google',     type: 'training' }
// + QwenBot, DeepSeekBot, and others

Human tracking: 1x1 transparent pixel on page load records visits, referrers, and device info. Bot tracking: Server-side user-agent analysis. Key metrics: Human pageviews, bot visits by provider, bot traffic %, top crawled pages, top referrers.

Sentiment Scoring — Visibility Context (0-100)

See Commerce Intelligence (Step 11) for the full word lists. The same count-based lexicon algorithm is used: ±150/+300 char context window around brand mentions, positive/negative word counting, scored via the 60 + delta×8 / 40 - delta×8 / 50 + delta×5 formula.

// Default: 50 (neutral) when no brand mentions found

// Display thresholds (dashboard rendering):
score ≥ 75  → Very Positive (green)
score 60-74 → Positive      (green)
score 45-59 → Neutral       (yellow)
score 30-44 → Negative      (orange)
score < 30  → Very Negative (red)

Sentiment Scoring — Focus Group Context

The Consumer Focus Group uses a separate, more nuanced sentiment pipeline (lib/focus-group/ingestion/sentiment.js) with fractional word weights and square-root normalization:

// Fractional weights (0.3 to 0.9 per word):
Positive: "love" (0.8), "excellent" (0.8), "highly recommend" (0.9),
  "best" (0.7), "perfect" (0.8), "game changer" (0.7), "quality" (0.4)...
Negative: "worst" (-0.9), "hate" (-0.8), "terrible" (-0.8),
  "don't recommend" (-0.8), "waste" (-0.7), "useless" (-0.7)...

// Normalization:
score = totalScore / √matchCount     // dampen high match counts
score = clamp(score, -1, +1)

// Labels (threshold ±0.2):
score > 0.2         → positive
score < -0.2        → negative
|score| < 0.2 AND matchCount > 2 → mixed
otherwise           → neutral

// Also detects: emotions (8 types) and intents (8 types)

Limitation: Both sentiment systems are lexicon-based. Sarcasm, double negation, and context-dependent language may not be captured.

Two-Priority Classification

Priority 1 — URL pattern matching (most reliable):

Awareness:     Wikipedia, blogs, news, guides, tutorials,
               "what-is" pages, educational sites

Consideration: CNET, Wirecutter, PCMag, RTINGS, Tom's Guide,
               TechRadar, comparison pages, "vs" content,
               review sites, "best-of" roundups

Conversion:    Amazon, Best Buy, Walmart, Target, Newegg,
               /buy, /shop, /pricing, /checkout, /cart URLs,
               retailer domains, product pages

Community:     Reddit, Quora, Stack Overflow, GitHub discussions,
               forums, Discord, community boards

Priority 2 — Prompt text patterns (fallback): If URL doesn't match, classify based on prompt text ("best X vs Y" → Consideration, "buy X" → Conversion). Default: Awareness.

Deduplication: Same domain across prompts is merged — counts summed, unique LLMs and URLs collected. Sorting: Brand first → Competitor → Other, then by citation count.

Gap Severity Algorithm

// GPT-4o-mini identifies missing concepts per page
// Concepts are: deduplicated by name, frequency-ranked,
// importance-labeled (high/medium/low)

highGaps ≥ 3  → "Weak"     (significant content gaps)
highGaps  1-2 → "Moderate" (some gaps to address)
highGaps    0 → "Strong"   (comprehensive coverage)

// Top 10 gaps shown, sorted by frequency
// Topic gaps: clusters with zero aligned pages

Audit Checks & Thresholds

// Per-page technical SEO scoring (part of GEO Score)
score ≥ 18/25 → Good       (schema and structure present)
score   12-17 → Needs work (partial coverage)
score <  12   → Weak       (priority fix needed)

// Site-wide audit verdict
0 weak pages     → "Strong"   (all pages well-structured)
≤ 30% weak pages → "Moderate" (some pages need attention)
> 30% weak pages → "Weak"     (widespread technical gaps)

// Checks: JSON-LD schema, meta descriptions (flag <120 chars),
// lists & tables, heading hierarchy (H1→H2→H3)

Recommendation Pipeline

Top 20 content gaps fed into GPT-4o-mini with full context (gap severity, AI response snippet, topics, best matching page, similarity score). Output: structured JSON with issue, impact, action steps, implementation guide, effort estimate, priority, and suggested page. Rate-limited: 200ms between calls. Sorted by gap severity (highest-impact first).

Percentile-Based Tier Assignment

Data source: DataForSEO keywords/data endpoint for search volume signals per prompt.

// Percentile-based tiers (within analysis batch)
volumes = prompts.map(p => searchVolume(p))
p75, p50, p25 = percentiles(nonZeroVolumes)

volume ≥ p75  → HIGH DEMAND   (score: 3)
volume ≥ p50  → MED DEMAND    (score: 2)
volume >  0   → LOW DEMAND    (score: 1)
volume == 0   → NICHE         (score: 0)

// Usage: demand badges on prompt cards, smart sorting
// (high-demand gaps surface first), demand-weighted
// visibility summary metric

Why relative tiers? Raw volumes vary wildly by industry. Percentile tiers show which prompts matter most relative to your market.

Three-Tier Extraction Pipeline

// 1. Schema.org Extractor (lib/ground-truth/ingestion/schema-extractor.js)
//    Parses JSON-LD structured data
//    Confidence: 0.85 base, +0.10 for high-signal fields
//    (name, url, price, address), +0.05 for medium-signal
//    (description, logo, brand). Capped at 0.95.

// 2. Meta Tag Extractor (lib/ground-truth/ingestion/meta-extractor.js)
//    Reads <meta>, Open Graph (og:*), Twitter Card tags
//    Confidence: 0.70 base meta, 0.75 OG/Twitter, 0.80 canonical

// 3. LLM Extractor (lib/ground-truth/ingestion/llm-extractor.js)
//    Claude/GPT extracts facts from page body text
//    Confidence clamped: 0.40 (inferred) to 0.60 (very explicit)
//    Lowest tier — LLM-inferred facts are less reliable

Deduplication: Same (category, fact_key) keeps highest-confidence source. 11 valid categories: identity, product, pricing, claim, policy, location, leadership, competitive, contact, technical, visual.

Two-Pass Evaluation — lib/ground-truth/comparison/evaluator.js

Query generation (lib/ground-truth/comparison/query-generator.js): 5 framings per fact — direct, comparative, recommendation, negative, conversational. Temperature 0.7 for variety.

// Pass 1: LLM Judge (temperature 0)
// Classifies response as: correct, partially_correct, incorrect,
// hallucinated, not_mentioned, or refused

// Pass 2: Embedding Similarity
similarity = cosineSimilarity(embed(groundTruth), embed(extractedAnswer))

// Overall Accuracy
accuracy = ((correct + 0.5 × partial) / (total - refused)) × 100

// Confidence scoring
correct + similarity > 0.8 → 0.95
correct alone              → 0.85
similarity > 0.9           → 0.70
else                       → 0.50

// Severity classification
hallucinated                       → critical
incorrect + confused_with_competitor → high
incorrect                          → medium
partially_correct / not_mentioned  → low
refused / correct                  → info

// Issue types: outdated_info, wrong_number, wrong_name,
// confused_with_competitor, fabricated_detail, missing_context,
// oversimplified, contradictory

Models tested: GPT-4o (500ms rate limit), Claude Sonnet 4.5 (1000ms), Gemini 2.0 Flash (500ms), Perplexity Sonar (1000ms).

Two Distinct AI Readiness Scores

1. Brand-Level Readiness — appears in the Brand Safety Hub. Measures overall AI preparedness across your brand's fact base:

// lib/ground-truth/scoring/readiness-scorer.js
readiness = 0.20 × schema_markup       // JSON-LD score (0-100)
          + 0.20 × content_structure   // context(40%) + diversity(30%) + FAQ(30%)
          + 0.20 × fact_coverage       // coveredCategories / 11 × 100
          + 0.15 × freshness           // factFreshness(70%) + sourceFreshness(30%)
          + 0.10 × crawlability        // robots(20) + AI access(30) + sitemap(30) + URLs(20)
          + 0.15 × query_coverage      // coveredQueries / totalQueries × 100

// Content structure sub-scores:
//   Context: (factsWithContext / totalFacts) × 40
//   Diversity: min(30, uniqueSubcategories / totalFacts × 100)
//   FAQ: min(30, faqFactCount × 10)

// Freshness decay:
//   Fact freshness: linear decay over 90 days
//   Source freshness: linear decay over 30 days

// Crawlability breakdown:
//   robots.txt present:    +20 pts
//   AI crawlers allowed:   +30 pts
//   Sitemap found:         +30 pts
//   URL count (scale/50):  +20 pts max

2. Page-Level Readiness — appears per-page in the Pages tab. Measures individual page AI readiness:

// lighthouse-server.js — calculateAIReadinessScore()
readiness = 0.40 × geoScore            // GEO content score (0-100)
          + 0.30 × alignment           // avgAssociationScore × 100
          + 0.15 × technicalHealth     // 100 baseline, deductions for:
              // no schema (-30), weak meta (-20), no H1 (-20),
              // thin content <300 words (-15), slow >3s (-10),
              // redirect chains >2 (-5), broken links (-15)
          + 0.15 × linkAuthority       // 50 baseline, +10/inlink (max 30),
              // -40 if orphan, +20 if hub

// Grade: A (80+), B (60-79), C (40-59), D (20-39), F (<20)

Sensitivity & Pattern Analysis

// Prompt Sensitivity Score (0-100)
// Variance of accuracy across 5 query framings per fact
sensitivity = min(avgVariance / 0.25, 1) × 100
// 0 = consistent across all framings
// 100 = highly framing-dependent

// Hallucination Pattern Analysis
// Groups hallucinated responses by category and issue type
// Identifies which fact categories and error types are
// most common per model

// Causal Attribution
// Compares 7-day accuracy windows before/after fact changes
// Measures impact of content updates on LLM accuracy

// Benchmarks (percentile-based)
// Min sample: 5 brands for comparison
// Segmented by: industry, company size, model

Three-Stage Pipeline

1. Discovery (lib/reddit-intel/discovery.js):

// SerpAPI searches with 6 query variants:
site:reddit.com "{brand}"
site:reddit.com "{brand}" review
site:reddit.com "{brand}" vs
site:reddit.com "{brand}" alternative
site:reddit.com {domain}
site:reddit.com "best {category}" recommendation

// 20 results per query, 500ms between queries, max 2 retries

2. Scraping (lib/reddit-intel/scraper.js): Reddit JSON endpoint .json?limit=200&sort=top. Recursive comment flattening (max depth 10). Filters AutoModerator and deleted content.

3. Analysis (lib/reddit-intel/analyzer.js): Claude Haiku 4.5 (claude-haiku-4-5-20251001, max 2000 tokens) extracts:

• Brand mentions with sentiment (positive/negative/neutral/mixed)
• Domain mentions with context
• Topics with relevance scores (0-1)
• Recommendations with upvote counts

Scoring: Sentiment breakdown by count, competitor comparison by mention volume, recommendations grouped case-insensitive and sorted by count.

Ingestion Pipeline

// Quality score per response (lib/focus-group/ingestion/)
quality = length    × 0.25  // longer = more detail
        + engagement × 0.25  // upvotes, replies
        + specificity × 0.30 // concrete details vs generic
        + recency    × 0.20  // newer = more relevant

// Embeddings: text-embedding-3-large (3072 dims)
// Sentiment: Claude API (positive/negative/neutral/mixed)
// Rule-based fallback (lib/focus-group/ingestion/sentiment.js):
//   Fractional weights per word (0.3 to 0.9)
//   e.g., "highly recommend" +0.9, "worst" -0.9
//   Normalized: score / √matchCount, label threshold ±0.2
//   Also detects 8 emotion types + 8 intent types
//   See Step 13 for full details on both sentiment systems

K-means++ Clustering — lib/focus-group/personas/clustering.js

// K-means++ initialization:
//   First centroid: random selection
//   Remaining k-1: probability ∝ distance² (spreads centers)

// Iterative assignment/centroid update
//   Distance: Euclidean (√Σ(a[i]-b[i])²)
//   Convergence tolerance: 0.0001
//   Max iterations: 100

// Optimal K selection (k=3 to k=8):
combinedScore = silhouetteScore - 0.02 × (k - minK)
// Slight parsimony penalty for more clusters

// Silhouette score per point:
s(i) = (b - a) / max(a, b)
// a = avg distance to own cluster
// b = min avg distance to any other cluster
// Range: -1 (misclassified) to +1 (well-clustered)

// Cluster quality:
tightness  = exp(-avgDistance / 2)      // normalized 0-1
sizeScore  = min(1, clusterSize / 20)   // maxes at 20+
confidence = tightness × 0.6 + sizeScore × 0.4

Discussion Engine — lib/focus-group/discussion/engine.js

// Model: claude-haiku-4-5-20251001 (fast, cost-effective)
// Max tokens per response: 300

// 5 structured rounds per template:
1. First Impressions — gut reaction
2. Deep Dive — specific appeals vs concerns
3. Personal Relevance — would you realistically use this?
4. Competitive Context — how does it compare?
5. Final Verdict — decision scale + what would change mind

// Each persona's system prompt includes:
//   values, motivations, frustrations,
//   communication style, authentic voice samples
//   from their cluster

// Interaction: each persona sees previous responses
// for natural conversation flow

// Delays: 500ms between responses, 1000ms between rounds,
// 800ms between personas

Validation Thresholds — lib/focus-group/validation/question-generator.js

// Per-trait validation:
percentage = (validated / total) × 100
≥ 70% → confirmed    (trait matches real data)
40-69% → partial     (some evidence)
< 40% → weak        (insufficient support)

// Overall score: average of trait percentages

// Confidence levels:
50+ responses AND 70%+ score → high
30+ responses AND 50%+ score → medium
otherwise                    → low

Brier Score Calibration — lib/prediction-markets/calibration/brier.js

// Brier Score: (forecast - outcome)²
// 0 = perfect, 0.25 = coin-flip, 1 = maximally wrong

// Brier Decomposition:
Brier = Reliability - Resolution + Uncertainty

Reliability = Σ(nk/n) × (avgForecast - avgOutcome)²
  // Lower = better calibrated (forecasts match outcomes)

Resolution  = Σ(nk/n) × (avgOutcome - baseRate)²
  // Higher = better discrimination (separates events)

Uncertainty = baseRate × (1 - baseRate)
  // Fixed — inherent outcome variance

// Calibration: 10 probability bins (deciles)

// Platform comparison thresholds:
delta < 0.01  → similarly calibrated
delta 0.01-0.05 → slight advantage
delta > 0.05  → significant advantage

// Rate limits: Kalshi 50ms, Polymarket 34ms
// Relevance threshold: 0.3 cosine similarity to brand

Time horizons: days (≤7), weeks (≤30), months (≤90), quarters (≤270), years (>270).

Shadow Page Generation — scripts/lighthouse-server.js

// Content components generated per page:
1. Answer Capsule (≤150 chars)
   Priority: pre-extracted → potential capsule → first sentence

2. Key Facts Section
   Regex: sentences with %, large numbers, study refs
   Filter: 20-200 chars, limit 5 facts

3. Content Sections (H2/H3 hierarchy)
   Ceiling allocation across headings
   Paragraph truncation at sentence boundary (500 chars)

4. Expert Insights (300 char summary + quick wins)
5. Competitive Landscape (top 5 competitors)
6. Common Questions (top 5 queries)
7. Trusted Sources (top 5 citation sources)
8. FAQ Schema (schema.org/Question markup)
9. Related Pages Navigation (top 15 internal links)

// Schema types auto-detected:
Organization (always), Article, Product, HowTo,
FAQ, WebPage, BreadcrumbList

// Token reduction: reductionPct = (1 - shadow/original) × 100

// llms.txt: Markdown sitemap for AI models

// Deployment: Cloudflare Worker serves shadow HTML to
// AI bots (GPTBot, ClaudeBot, PerplexityBot, etc.)
// and original content to human visitors
// robots meta: "noindex, nofollow" (AI-only)