Choose native multimodal processing or preprocessing

Learning objective

Use native multimodal processing for images, audio, and video, and recognize when preprocessing remains necessary. This lesson focuses on the media-processing decision, not connector setup, access-control propagation, or retrieval tuning.

Core idea

Use this rule: native first; preprocess for a named constraint.

Native multimodal processing means sending supported media and text instructions to a multimodal model in the same request. Your application does not have to turn every image into OCR text, every recording into a transcript, or every video into frame captions before the model can reason about it. This preserves relationships that a text-only derivative can lose, such as which spoken warning coincides with an action on screen.

Native does not mean that every file works unchanged. The selected interface and model still define supported formats, sizes, counts, durations, and locations. The service still converts media into model inputs, and video processing can sample frames. Preprocess when an input contract, quality target, retrieval design, precision requirement, or governance control demands it.

How it works

Gemini Enterprise Agent Platform supports adding images, audio, and video to Gemini requests. Give the model the original evidence and a concrete task, such as “Summarize the incident, identify visible hazards, and cite relevant timestamps.” Then evaluate the output on representative media.

Each modality has different failure modes:

• Images: Use native understanding for description, comparison, visual question answering, and semantic interpretation. Rotate images correctly, avoid blur, and use adequate resolution. Google documents that image models can be imprecise when locating or counting objects and can make mistakes on low-quality, rotated, or very low-resolution images. Use OCR, layout parsing, or object detection when exact text, structure, coordinates, or counts are the required artifact. See Image understanding best practices and limitations.
• Audio: Use native processing for speech-oriented tasks such as summarization, transcription, and translation. Configure the documented audio timestamp setting when accurate audio-only timestamps are required. Treat alarms, machine sounds, and other non-speech signals cautiously because the audio understanding limitations warn that non-speech sounds can be misrecognized. A dedicated audio classifier or signal-processing stage is appropriate when such sounds drive a decision.
• Video: Use native processing when the answer depends on both temporal visuals and audio. The documented static File API path samples one frame per second by default, so fast events can lose detail. The API supports clipping intervals and custom frame rates. Isolate the relevant interval or increase the sampling rate when rapid motion matters; use a specialist detector when you need frame-exact evidence. See Customize video processing.

Before adding a preprocessing stage, identify the contract it satisfies:

1. Check the exact entry point and model for supported media and limits. Transcode, resize, compress, or split only when the asset does not meet that contract.
2. Check whether the source quality exposes the needed signal. Rotate or rescan a poor image; trim silence or isolate a relevant clip.
3. Define the downstream artifact. Search may need timestamped text and metadata; an audit system may need stable fields and exact offsets rather than a fluent answer.
4. Apply required inspection, redaction, and retention controls before inference when policy requires them.
5. Evaluate the complete path. Preprocessing is useful only if it measurably fixes the named constraint.

Key distinctions and trade-offs

Native understanding versus preprocessing: Native understanding is flexible semantic reasoning over media. Preprocessing produces a normalized or structured derivative before that reasoning. They can be complementary.

One-request analysis versus corpus retrieval: Asking about an attached file is not the same as building a durable searchable corpus. Agent Search can parse and chunk supported unstructured documents, and its layout parser can annotate images for retrieval. For recordings, a searchable design may store a transcript, scene or chapter descriptions, timestamps, and filter metadata. See Parse and chunk documents.

Media file versus media document: An Agent Search media document typically contains metadata about a video, podcast, or other item. Creating that record does not prove that a model watched or listened to the underlying asset. The media document schema even provides an optional transcript field.

Native processing reduces pipeline stages, storage, synchronization work, and the risk of losing cross-modal context. Preprocessing adds latency, cost, and more artifacts to govern, but it can make content compatible, searchable, precise, and independently testable. Keep the original media as evidence and link derivatives back to it with stable identifiers and time ranges.

Architecture / decision scenario

A warehouse safety agent receives incident photos, voice reports, and security video. Investigators need an immediate incident summary and later search across months of cases.

For the immediate summary, the workflow sends a supported photo, voice report, and relevant video clip directly to Gemini. This lets the model relate spoken claims to visible events. It first checks image orientation and recording quality.

A conveyor fault occurs in less than one second. Because default video sampling can miss it, the workflow isolates the interval and uses a higher sampling rate. If investigators require exact object tracks, a specialist video-analysis stage produces them instead of relying on generative prose.

For later retrieval, the ingestion pipeline creates timestamped transcripts, scene descriptions, incident metadata, and links to the original Cloud Storage objects. Search retrieves likely segments; Gemini then analyzes the original segment natively. The team accepts the added pipeline cost because reusable search and precise evidence locations are explicit requirements.

In both paths Gemini analyzes original media; the derivatives serve search.

Common exam traps

• Assuming “multimodal” means every interface accepts the same formats and limits.
• Converting all media to text before testing native processing.
• Treating native image understanding as exact OCR, counting, or localization.
• Assuming a video model examines every frame at full detail.
• Confusing a searchable media metadata record with analysis of the media bytes.
• Adding preprocessing without a measurable compatibility, quality, retrieval, precision, or governance requirement.

What to remember

• Start with native media input when the interface supports it and quality is sufficient.
• Preprocess only to satisfy a named contract or correct a measured failure.
• Match preprocessing to the modality: image quality, audio signal, or video time sampling.
• Separate one-request media understanding from durable corpus indexing.
• Keep originals linked to searchable or structured derivatives.

Knowledge check

References

• Google Cloud, Gemini model inference
• Google Cloud, Image understanding
• Google Cloud, Audio understanding
• Google Cloud, Video understanding
• Google Cloud, Parse and chunk documents
• Google Cloud, About media documents and data stores