Validate extracted triples before graph writes

Learning objective

Extract and validate triples from unstructured text with LLMs.

Core idea

An LLM turns a passage into candidate subject–predicate–object triples. It does not turn the passage into verified facts. A valid JSON array can contain a reversed relationship, an invented entity, or an assertion that ignores the word “not.” Validate the output’s structure and its support before publishing it into a graph that other agents traverse.

The graph model and provenance design belong to 1.7.1. This lesson covers the transformation that supplies that model: extracting candidates, checking them, resolving their endpoints, and deciding which candidates become graph writes. Two NVIDIA implementations expose why those stages need separate outcomes: txt2kg’s extraction route and Context-Aware RAG’s graph ingestion base.

How it works

Extract candidates with a constrained vocabulary

NVIDIA’s txt2kg extraction route accepts text. In its Ollama branch, it asks for subject–predicate–object triples as a JSON array, then parses the generated response. Its Triple interface names subject, predicate, and object, with optional confidence and usedFallback fields. Those identifiers describe the extraction result; they do not provide canonical business identifiers or a proof of support.

For a domain extractor, supply the permitted entity types, predicate definitions, and direction rules from your graph model. Include examples that preserve negation, uncertainty, and time. For example, “Aster may acquire Beacon” must not publish the same ownership assertion as “Aster acquired Beacon.” Your application can retain the former as a proposed relationship or reject it from a graph intended only for completed ownership.

Context-Aware RAG provides graph_ingestion with node_types and relationship_types, passed into LLMGraphTransformer as allowed_nodes and allowed_relationships on its structured path. The documented default is disable_entity_description: true, which clears both lists and sets ignore_tool_usage=True. Set disable_entity_description: false when relying on the structured path. Test the actual model and ingestion adapter together rather than treating a configured vocabulary as a guarantee.

Distinguish parse recovery from semantic validation

The txt2kg route first looks for an array and calls JSON.parse. If parsing fails, or no array is found, it calls parseTriplesFallback. That helper recognizes hyphen- or pipe-delimited lines such as Subject - Predicate - Object, as well as JSON-like objects. A hyphen inside an entity name can split it into the wrong fields. Recovery can salvage usable candidate text, but a parser recognizing three strings has established only that they fit a pattern.

The Ollama branch returns metadata.entityTypes: [], so its response supplies no entity types for the downstream type validator. It also attaches confidence: 0.8 to every returned triple. This is a fixed value in the route, not a calibrated probability that each relation is true. A downstream threshold of 0.75 would admit all those candidates regardless of their factual quality. Use source checks and a labeled validation set for acceptance decisions instead of interpreting the field as measured accuracy.

Context-Aware RAG has another recovery behavior. Its _extract_entities_with_fallback catches errors whose messages include Error code: 400, BadRequestError, or Invalid JSON: and retries with _configure_fallback_transformer. That transformer sets allowed_nodes=[], allowed_relationships=[], disables properties, and uses ignore_tool_usage=True. The method replaces self.transformer and sets self.disable_entity_description = True on the ingestion instance. Later batches processed by that instance therefore retain the unconstrained transformer; the downgrade extends beyond the failed batch. This is an availability trade-off: extraction can continue through a less constrained path. Keep the same downstream checks after either outcome.

After one failed batch, later batches processed by that instance retain the unconstrained transformer.

Validate the candidate before publication

Use a staged application validator with explicit rejection reasons. The stages are different tests, so a success in one stage cannot stand in for another:

These validation artifacts are application design choices. NVIDIA’s ingestion base implements a narrower sanitation step, handle_backticks_nodes_relationship_id_type: it cleans backticks and filters relationships with empty endpoint identifiers or types. That protects graph representation from malformed content; it does not establish that a source asserts the relationship.

Keep the original candidate and its source reference with a rejection reason for correction. For a source-support check, compare the exact passage with the predicate’s definition, including direction and qualifiers. If an LLM performs that comparison, evaluate its errors too; a second generation is another model judgment. Domain review remains appropriate for unresolved high-impact claims.

Measure the errors that change the graph

Build a labeled set of source passages and expected triples with canonical endpoints. Count accepted unsupported triples as false positives and missed supported triples as false negatives. Report relation precision and recall separately from JSON parse success. A pipeline can parse every response while admitting wrong relationships.

Include negatives such as denied relationships, conditional statements, tables with repeated names, and conflicting document revisions. Record errors by predicate and monitor the warning log beginning “Disabling entity description, LLM doesn’t support structured output” to identify when the instance switches modes. Compare validation errors before and after the switch, including later batches. If endpoint-type failures increase, repair model compatibility or the fallback handling rather than expanding the vocabulary to accept invalid content.

Extraction consumes inference tokens before the graph is queried. Larger chunks can preserve cross-sentence references but require more input and can mix unrelated entities. graph_ingestion.params.chunk_size and chunk_overlap expose that choice in Context-Aware RAG. Tune them on the labeled passages; overlap can produce duplicate candidates that still need canonical resolution.

Key distinctions and trade-offs

Prompt-only extraction is flexible but leaves more normalization and validation work downstream. A structured path narrows representational errors but still needs source verification. A permissive fallback improves batch completion while potentially weakening type restrictions. Choose between these paths by accepted-triple quality and processing cost, not by how many JSON arrays return successfully.

Separate an extraction failure from a disputed fact. Malformed JSON can justify a bounded retry; a passage that never asserts ownership needs a different interpretation or rejection. Repeatedly asking for the desired edge can manufacture apparent agreement without additional support.

Architecture / decision scenario

An insurance research assistant ingests 8,000 broker bulletins overnight. A labeled test produces 1,000 candidate ownership triples: 70 are unsupported proposals or negations. The business permits at most 10 unsupported ownership assertions per 1,000 accepted assertions and allows a six-hour ingestion window. These are the team’s acceptance criteria, not product limits.

The tempting shortcut accepts every parsed triple with confidence >= 0.75. Because txt2kg’s Ollama branch assigns 0.8 uniformly, that rule cannot separate the 70 bad candidates. The analytics owner rejects it and requires document-span checks, canonical company resolution, and measured precision before publishing the accepted set.

The platform team also tests the Context-Aware RAG fallback with the selected model and monitors the mode-switch warning. Candidates from the failed batch and every later batch on that instance pass through the same vocabulary validator, even after the extraction service resumes returning results. If the overnight window cannot accommodate unresolved records, they remain pending for domain review; the published graph stays smaller but avoids unsupported ownership paths.

Common exam traps

• Equating parse success with factual correctness.
• Treating a fixed confidence field as a calibrated probability.
• Assuming structured-output recovery preserves allowlists.
• Merging entity aliases before checking their canonical identity.
• Retrying a semantically unsupported relation until it appears.

What to remember

Extract candidates, validate structure and meaning separately, then write the accepted set. Track precision, recall, and fallback behavior against labeled source passages. Schema constraints reduce representational errors; source support determines whether a triple belongs in the graph.

Knowledge check

References

• NVIDIA, txt2kg extraction route
• NVIDIA, txt2kg triple interface
• NVIDIA, Context-Aware RAG configuration
• NVIDIA, Graph ingestion base implementation