GoGuides Web Provenance Data Feed
Current Structured Provenance Layer
The current high-detail provenance layer began recording on August 8, 2026. It preserves structured states and repeat observations rather than simply recording that a URL was fetched.
Snapshot measured August 17, 2026. These figures describe the newer structured provenance layer and are intentionally separated from GoGuides' older historical measurement archive.
Web Crawl Corpus
GoGuides has built a substantially larger crawl corpus than the newer provenance layer alone represents. The primary pages dataset currently contains more than 5.65 million unique URLs, with recorded crawl timestamps extending back to December 4, 2025.
Snapshot measured August 18, 2026. The crawl-pipeline total includes URLs in completed, queued, error and active-processing states and should not be interpreted as 7.71 million successfully crawled pages.
A new system can begin observing a domain today. It cannot go backward and independently observe what that domain's DNS, TLS certificate, redirect behavior, robots state or content fingerprint looked like months earlier. Time makes accumulated observation history fundamentally different from software alone.
Historical Fingerprint Change Layer
GoGuides maintains current crawl state across its primary URL corpus and separately preserves historical fingerprint changes when a later measurement differs from the previously stored fingerprint. Unchanged re-observations are intentionally deduplicated rather than creating duplicate history rows.
The 376K+ records represent additional historical states created when fingerprints changed. They do not represent the total number of measurements performed, total crawl observations, or the number of URLs with current crawl state. The broader GoGuides corpus contains more than 5.65 million unique crawled URLs.
Historical change records include the URL, AI Rank, crawl timestamp, algorithm version and fingerprint associated with the changed state.
Current Crawl Infrastructure
Recent production measurements show GoGuides sustaining approximately 20,000–21,000 URL crawls per hour, while simultaneously processing, normalizing and preserving the resulting evidence.
The latest six completed production hours averaged approximately 20,696 URL crawls per hour. Throughput varies with destination response time, redirects, network behavior, safety checks, robots processing and other factors. GoGuides does not present peak throughput as a guaranteed service rate.
What a Provenance Record Can Contain
The value of the dataset is not simply domain coverage. GoGuides records machine-usable evidence describing what was independently observed.
status, content type, response size, response time
final URL, redirect count, redirect chain
canonical URL and title evidence
title SHA-256, content SHA-256, HTML SHA-256
robots permission, reason, robots.txt SHA-256
meta robots and X-Robots-Tag
certificate SHA-256, issuer, validity dates, HTTPS state
resolved IP, nameservers and DNS fingerprint
first observed, last observed, repeat count
AI Rank, algorithm version and evidence JSON
Illustrative Machine-Readable Record
Actual fields and delivery schemas can be tailored to the use case. A normalized record can expose the observation itself together with evidence required to interpret or validate it.
{
"domain": "example.com",
"requested_url": "https://example.com/",
"final_url": "https://example.com/",
"first_observed_at": "2026-08-08T15:32:01Z",
"last_observed_at": "2026-08-17T22:25:37Z",
"observation_count": 12,
"http_status": 200,
"redirect_count": 0,
"canonical_url": "https://example.com/",
"content_sha256": "...",
"html_sha256": "...",
"robots_allowed": true,
"https_enabled": true,
"tls_cert_sha256": "...",
"resolved_ip": "203.0.113.10",
"nameservers": ["ns1.example.net", "ns2.example.net"],
"dns_sha256": "...",
"ai_rank": 74,
"algo_version": 1
}
Public Change Intelligence
GoGuides now exposes /changes/{domain}
as Change Intelligence V1. It compares consecutive stored Web State V3 observations
of the same requested URL and classifies observable changes without triggering a new crawl.
Machine-readable results can distinguish content, structured-data, schema-type,
header, word-count, link, outbound-link, AI-policy, and HTTP-status changes and
provide refresh_recommended with a concise refresh_reason.
Why License the Feed Instead of Building It?
Organizations capable of building crawlers may still benefit from licensing independently maintained historical intelligence. Building collection software and acquiring historical observations are two different problems.
A GoGuides feed can complement an existing crawl stack by supplying prior observations, normalized provenance states and continuously maintained domain intelligence without requiring the consumer to reproduce the entire collection and history pipeline.
The harder-to-recreate asset is the accumulated observation record: what was seen, when it was seen, how the state changed, and the evidence fingerprints associated with those observations.
Public and Potential Delivery Models
Licensing can be scoped by domain set, fields, refresh cadence, historical depth and delivery method. GoGuides is interested in relationships with organizations building AI infrastructure, search, retrieval, web intelligence, cybersecurity, domain reputation, research and data products.
Designed to Complement Existing Web Infrastructure
GoGuides is not positioned as a replacement for large-scale crawling, search or retrieval platforms. The feed is intended to provide an additional independent evidence layer that can be combined with a buyer's existing web-access infrastructure.
For systems that already know what a website says now, GoGuides can help answer a different question: what has independently been observed about this domain over time?
Data Licensing & Integration
GoGuides is exploring direct data-feed and licensing relationships with organizations that can use longitudinal domain intelligence as part of their own products, models, retrieval systems, security infrastructure or data platforms.