Connect and index enterprise sources

Learning objective

Connect and index structured and unstructured enterprise sources for Agent Search. This lesson focuses on source preparation and indexing, not access-control propagation or retrieval tuning.

Core idea

Treat a source connection as a data pipeline, not as a one-time upload:

source -> connector or import -> document model -> schema and metadata -> index

Structured and unstructured sources both become documents in an Agent Search data store, but their preparation differs. The source remains authoritative; the index is a searchable representation that needs freshness, quality, and failure monitoring.

How it works

Start by inventorying the corpus. Identify its owner, system of record, data type, update pattern, stable identifier, permissions, and deletion requirements. Choose separate data stores when source types, schemas, regions, lifecycles, or access policies require independent management.

Agent Search supports several data-store types. In its data stores documentation, a structured document is a table row or JSON record, while an unstructured document is a supported file such as PDF, HTML, DOCX, PPTX, or text. Website and media sources have additional document models.

For structured data, define or verify the schema before relying on search. Keep a stable document ID and map fields by purpose. Mark fields as searchable, retrievable, indexable, or filterable only when the workload needs that behavior. Explicit schemas usually make types and field use more predictable than automatic detection. Imports can come from sources such as BigQuery, Cloud Storage JSON, or the API, depending on the supported data-store configuration.

For unstructured data, validate file type, extraction quality, language, layout, and metadata. Preserve titles, source URIs, timestamps, categories, and other fields needed for filtering and attribution. Scanned or layout-heavy files may need optical character recognition or layout-aware parsing. Agent Search supports unstructured imports from Cloud Storage and synchronization with supported connectors such as Google Drive; verify the current source-specific requirements before creating the store.

Then create the data store, configure its region and data type, connect or import the source, and attach it to a search app. The custom search checklist separates preparing data, importing data, creating the app, and configuring serving. Completion of an import job does not prove good retrieval.

Validate indexing with known documents and representative queries. Check document counts, rejected records, extracted text, schema fields, metadata, and source links. Define synchronization or re-import behavior for updates and deletes. Monitor connector failures and index lag so stale content does not silently remain available.

Key distinctions and trade-offs

Data store versus app: A data store contains indexed documents. An app uses one or more compatible data stores to serve search or recommendations. They are related resources, not synonyms.

Structured versus unstructured: Structured sources expose typed fields and a schema. Unstructured sources require extraction and often layout-aware processing. Adding metadata to an unstructured document does not turn its body into a fully structured record.

Import versus synchronization: A batch import captures a point in time. A connector can synchronize changes on a supported schedule. Either approach needs a defined freshness objective and deletion behavior.

Combining stores can give users one search surface, but blended retrieval adds schema, ranking, access-control, and evaluation complexity. Separate stores improve isolation while requiring explicit composition in the app.

Architecture / decision scenario

A manufacturer wants search across product records in BigQuery and maintenance manuals in Cloud Storage. The team creates a structured store with a declared schema for product ID, region, lifecycle status, and description. It creates a separate unstructured store for PDFs with product ID, revision, source URI, and effective date metadata.

Both stores connect to one custom search app. Evaluation queries confirm that an exact product ID finds the record and that a natural-language repair question finds the current manual. A scheduled import updates product rows, while a document pipeline processes revised manuals and deletes superseded index entries.

The separation preserves source-specific ingestion while metadata lets the serving layer relate products and manuals.

Common exam traps

• Treating successful upload or connector creation as proof of search quality.
• Loading structured records without stable IDs or a deliberate schema.
• Discarding source URI, revision, or effective-date metadata from documents.
• Assuming every scanned or layout-heavy file produces usable extracted text without validation.
• Confusing the data store that holds documents with the app that serves them.
• Planning updates but not deletions, leaving removed enterprise content in the index.

What to remember

• Model connection and indexing as a monitored, repeatable data pipeline.
• Prepare structured schemas and unstructured extraction separately.
• Preserve stable IDs, provenance, freshness, and useful metadata.
• Validate indexed content and representative queries after ingestion.
• Define update, deletion, and connector-failure behavior from the start.

Knowledge check

References

• Google Cloud, About apps and data stores
• Google Cloud, Custom search checklist
• Google Cloud, Prepare data for ingesting
• Google Cloud, Configure field settings
• Google Cloud, Introduction to custom search