NVIDIANVIDIA-Certified Professional: Agentic AI2026-106.4 Conduct data quality checks, augmentation, and preprocessing.

Repair, strip, and deduplicate corpus text

Completed by the knowledge check

Learning objective

Clean and normalize text: deduplication, boilerplate removal, and encoding.

Core idea

Text cleaning changes what a retrieval assistant can find and quote. A footer repeated across thousands of documents wastes retrieval capacity; a removed exception clause changes the answer. Choose transformations by the information they preserve, rather than by the percentage of bytes they delete.

Separate the readable document from the representation used to match duplicates. NeMo Curator’s normalize_text option in its exact and fuzzy deduplication workflows lowercases text, collapses whitespace, and trims surrounding whitespace for matching while preserving the original text. That is a different operation from permanently rewriting the corpus through Modify.

Extraction belongs to 6.3.1 and chunking to 6.1.1. Here you decide which extracted characters and repeated passages survive before either embeddings or retrieval can amplify their defects.

How it works

Repair encoding before comparing content

Apostrophes and accented characters can become garbled when text is decoded incorrectly. Curator’s UnicodeReformatter, under nemo_curator.stages.text.modifiers.unicode, uses ftfy to repair broken Unicode. Apply it through Modify; the documented default target is the dataset’s text field.

Repair is a transformation to inspect, not a reason to discard the source. Compare repaired samples containing product identifiers, accented names, mathematical symbols, and punctuation against the original documents. If a repair changes a meaningful identifier, fix the source extraction or configure a narrower repair rather than accepting more readable but incorrect text. A string that looks clean still needs the validity checks in 6.4.2.

NewlineNormalizer replaces three or more consecutive newline characters with two. It does not turn every newline into a space. That distinction matters for lists, headings, and procedures: preserving paragraph structure gives downstream chunking useful boundaries. Choose the sequence of repairs explicitly and keep it consistent between corpus runs so duplicate matching sees comparable inputs.

Remove boilerplate at the right granularity

Curator’s BoilerPlateStringModifier checks paragraphs for boilerplate strings such as “terms of use” and “privacy policy.” With its default remove_if_at_top_or_bottom=True, it removes matching paragraphs only when all matches lie at the document’s top or bottom. If a match occurs in the interior, it returns the document unchanged. With remove_if_at_top_or_bottom=False, any matching paragraph returns an empty string for the whole document. A paragraph containing “lorem ipsum” returns an empty string with either setting. Where the matches lie decides the outcome only under the default; with False, one match empties the document.

A privacy-policy document containing a matching phrase in its interior therefore remains whole under the default, but becomes empty with False. Inspect examples from each document family before choosing that behavior. For a recurring navigation line, LineRemover offers a narrower operation: it removes a line only when the line exactly equals one of the supplied patterns.

BoilerPlateStringFilter, used through ScoreFilter, judges documents using max_boilerplate_string_ratio. A filter makes a keep-or-reject decision; the modifier returns transformed text, which can itself be empty. A high boilerplate ratio can mean extraction captured the wrong page region; repairing extraction can recover useful content that rejection would lose.

UrlRemover removes all URLs in a document. It is useful only when those URLs are noise for the intended task. For maintenance instructions that direct technicians to a specific approved download, removing the URL destroys usable information. Choose the installed modifier by its behavior, rather than applying every module in the documentation’s example pipeline.

Detect duplicates, then remove them

ExactDeduplicationWorkflow computes MD5 hashes of document text and identifies identical hashes. Set text_field to the column being compared. With normalize_text=False, matching targets character-for-character copies; enabling normalization also groups differences in casing and whitespace. Test whether those differences matter in your corpus before changing that setting.

The workflow’s perform_removal parameter is reserved and must remain False. Identification writes duplicate IDs; TextDuplicatesRemovalWorkflow performs removal using ids_to_remove_path. A successful identification run therefore does not prove that the output corpus contains fewer documents.

Keep the ID assignment consistent across both steps. When the workflow assigns IDs, its id_generator_path connects removal to the same mapping. Losing that mapping turns a list of duplicate IDs into an unreliable deletion instruction. Existing stable document IDs are another documented option through assign_id=False and id_field.

FuzzyDeduplicationWorkflow handles small edits and reformatting using character n-gram MinHash signatures, locality-sensitive hashing, and connected components. num_bands and minhashes_per_band control matching strictness: fewer bands or more hashes per band makes matching stricter. These are candidate-detection controls, not a universal proof that two documents are interchangeable.

Both documented distributed workflows require a Ray cluster with GPU support. Fuzzy matching adds signature, shuffle, and graph work; reserve that cost for repeated material that exact matching misses. With auto-generated fuzzy IDs, use the same input_blocksize for identification and removal, as different partitioning can change the ID mapping and break removal.

Key distinctions and trade-offs

Choose each operation from the defect it addresses:

Operation	Deciding condition	Mechanism	Cost or failure risk
Encoding repair	Text contains garbled characters	`Modify(UnicodeReformatter())`	A readable repair still needs comparison with source meaning
Boilerplate removal	Repeated framing obscures the body	`BoilerPlateStringModifier` or `LineRemover`	Removes whole edge paragraphs; with `False`, one match empties the document
Exact deduplication	Copies differ only within approved normalization	`ExactDeduplicationWorkflow`, then `TextDuplicatesRemovalWorkflow`	Aggressive normalization can collapse meaningful distinctions
Fuzzy deduplication	Small edits leave repeated documents	`FuzzyDeduplicationWorkflow`	GPU processing and false grouping require stricter review

A duplicate decision also differs from a version decision. Two manuals with one changed torque value are near-identical text but distinct operating instructions. Treat their revision metadata as a reason to inspect or separate comparison groups, rather than interpreting high similarity as permission to delete a version. Source identity and retention ownership are covered in 6.3.3.

Architecture / decision scenario

An industrial manufacturer builds a technician assistant from 120,000 manuals and service notices. Its proposed overnight refresh must finish in two hours on a shared two-GPU curation allocation. In a reviewed sample of 1,000 documents, exact copies account for 180 records; near-copies account for another 70, including 12 notices with changed safety instructions. These are scenario measurements, not Curator benchmarks.

One engineer proposes URL removal and broad fuzzy deduplication across the whole collection to maximize compression. The content engineering owner rejects that recipe: approved firmware links must remain, and deleting any of the 12 changed notices is unacceptable.

The team repairs garbled text with UnicodeReformatter, normalizes excess blank lines, and uses a source-specific LineRemover for the repeated site navigation. It first runs exact identification and explicit removal, with reviewed normalization settings. It then tests stricter fuzzy settings within compatible manual families, inspects changed-value pairs, and measures whether the additional GPU work fits the two-hour window. If it does not, exact cleanup ships while fuzzy removal waits for a validated run. Compression rate cannot override retained safety content.

Common exam traps

Treating Unicode repair as proof that extracted facts are correct.
Assuming BoilerPlateStringModifier trims matches wherever they appear; its default removal depends on paragraph position.
Assuming UrlRemover distinguishes useful links from irrelevant ones.
Setting perform_removal=True instead of running the removal workflow.
Treating near-duplicate text as an obsolete document version.
Reassigning generated IDs between detection and removal.

What to remember

Repair characters, remove only verified framing, and choose exact or fuzzy matching according to meaningful differences. Curator separates duplicate identification from removal. Preserve the mapping between them, and measure retained information alongside processing cost.