DarkMatter AIDC Harvester is the intelligence gathering crawler for the AI Data Centers Project. It transforms open-web research into a controlled, evidence-aware workflow that discovers sources, retrieves documents, extracts structured claims, scores confidence, stages proposed changes, and requires human approval before information enters the primary AIDC knowledge base.
Harvester is designed to accelerate intelligence collection while preserving source traceability, evidence quality, and human authority. It does not treat discovered information as truth merely because it was found online. Instead, it moves research through a staged pipeline where every extracted claim can be inspected, scored, reviewed, accepted, or rejected.
Harvester is a crawler-assisted research subsystem embedded within the DarkMatter AIDC platform. It accepts research targets such as companies, brands, technologies, services, endpoints, or existing AIDC entities. Each target becomes a controlled research job with its own goal, priority, seed sources, status, attempts, evidence, and review history.
Harvester is not an automatic truth machine and does not directly overwrite the live AIDC database. It does not assume every web page is authoritative, and it does not publish extracted material without review. Its purpose is to reduce repetitive research work while keeping final decisions in human hands.
The primary mission is to map the organizations, services, technologies, infrastructure dependencies, ownership relationships, platforms, and public evidence surrounding the global AI data center ecosystem.
Discovery is automated. Acceptance is deliberate. Harvester can retrieve, parse, correlate, and propose, but the AIDC operator remains the final authority responsible for deciding what becomes part of the permanent intelligence record.
Each stage has a distinct responsibility. This separation prevents a successful crawl from being mistaken for verified knowledge and gives every proposed database change a visible path from source to decision.
Identify candidate sources from configured search providers, operator-supplied seed URLs, official websites, technical documentation, public reports, and other research targets.
Fetch source pages under controlled runtime limits and store retrieval details, source URLs, timestamps, response information, and document content for analysis.
Convert unstructured text into candidate claims such as ownership, location, service relationships, technology use, infrastructure dependencies, and organizational associations.
Assign confidence using source quality, directness, corroboration, extraction certainty, and the relationship between the claim and the original evidence.
Translate claims into proposed database operations without modifying the production intelligence graph. Each proposal records its target, value, source excerpt, source URL, confidence, and review status.
Present proposed changes to an operator who can inspect evidence, compare context, reject weak claims, and approve only information suitable for the AIDC knowledge base.
Apply only approved changes to AIDC. Rejected proposals remain excluded, preserving the distinction between collected material and accepted intelligence.
Harvester operates beside the main AIDC interface, sharing the same database environment while maintaining its own research queue, harvested documents, extracted claims, proposed changes, activity logs, configuration, and worker process.
Stores target type, optional AIDC entity ID, search term, research goal, seed URLs, priority, status, attempt count, error details, and creation time.
Processes pending jobs under configured limits for maximum jobs per run and runtime ceiling. Locking prevents overlapping workers from processing the same queue.
Maintains retrieved source material and source provenance so extracted intelligence remains traceable to the document from which it originated.
Represents extracted statements as subjects, predicates, values, excerpts, source relationships, and confidence data rather than leaving research trapped inside unstructured text.
Converts claims into staged insert, update, or relationship operations. Proposals remain isolated until accepted through the review interface.
Records worker events, review actions, errors, execution context, and operational history to support debugging, auditing, and future system analysis.
AIDC Harvester is built to investigate the layered network behind artificial intelligence services rather than treating a data center as an isolated building.
The most important feature of Harvester is not crawling speed. It is the separation between collection and publication. This boundary protects the integrity of AIDC.
Harvested content does not automatically become accepted intelligence. The crawler produces proposals, and proposals must pass through the review interface before they can alter the primary AIDC data model.
Proposed changes remain connected to the originating document, source URL, extracted excerpt, and confidence score. Reviewers can inspect the evidence instead of evaluating an isolated claim.
Worker runtime limits, queue size controls, priority values, attempt tracking, and process locking prevent the crawler from becoming an uncontrolled background process.
Queue history, worker events, proposal state, publication actions, and rejection decisions create an operational record that can be inspected during troubleshooting and project review.
Research Target
│
├── Entity Type
├── Existing AIDC Entity ID
├── Search Term
├── Research Goal
├── Seed URLs
└── Priority
│
▼
Harvest Job Queue
│
▼
Controlled Worker Process
│
├── Source Discovery
├── Document Retrieval
├── Claim Extraction
└── Evidence Scoring
│
▼
Proposed Changes
│
▼
Human Review
├── Reject
└── Publish
│
▼
AIDC Knowledge Base
worker.php and extractor.php form the processing spineHarvester 0.4.2 treats research as a controlled sequence of observable transformations. Each layer has a narrow job, an auditable output, and a clear boundary before publication.
worker.php claims one queued job, applies locking and attempt controls,
retrieves deterministic HTML sources, stores activity and errors, and passes bounded
content plus the operator-defined research goal into the extraction layer.
extractor.php calls the configured Ollama generate endpoint and requires
structured JSON. Canonical-name validation and root-entity guards reduce hallucinated
aliases, research-phrase entities, sentence fragments, and self-referential claims.
Harvested claims retain source document context, endpoint URL, endpoint category, extraction engine, excerpt, confidence, subject, predicate, and proposed value. The evidence remains attached to the proposal instead of dissolving into database fog.
Reviewers approve or reject staged proposals. Publication resolves the canonical
company first, remains duplicate-safe, and writes each accepted fact to its proper
table, including direct endpoint links through company_endpoints.
Queue Record
→ worker.php
→ source discovery / deterministic retrieval
→ extractor.php
→ schema-shaped JSON claims
→ provenance + confidence
→ staged proposals
→ human review
→ canonical entity resolution
→ typed publication
→ human interface + JSON APIs
Version 0.4.2 documents the production path from queued research through worker execution, Ollama structured extraction, provenance-aware claims, review-order-safe publication, canonical entity reuse, and separate human and machine-readable outputs.
Future development can extend source discovery, extraction quality, evidence correlation, review tooling, live worker status, duplicate detection, entity matching, relationship visualization, broader export contracts, and deeper source corroboration. Version 0.4.2 is the documented baseline for that work.
The 0.4.2 documentation set joins the original AIDC ownership-map canon with the current Harvester implementation, corrected endpoint publication model, provenance controls, and machine-readable data strategy.
AIDC exposes structured JSON API routes and database-backed entity records so companies, endpoints, technologies, services, brands, evidence, and relationships can be consumed without scraping presentation HTML. Human pages remain the visual intelligence console; machine outputs remain explicit data contracts.