Sage Image Search

Try It NowView on GitHub

In the age of AI-powered tools, searching through thousands of images shouldn't just be about basic metadata. What if you could search by meaning? That’s exactly what this project aims to do: combine the best of semantic understanding and keyword precision into one powerful hybrid image search engine to help you find relevant image series for your project.

All images in this demo were captured by Sage nodes and retrieved from Beehive in near real-time.

Let’s walk through what it does, how it works, and where it’s going next.

What We're Building

This project is all about hybrid search — combining vector-based and keyword-based search — to enable smarter image retrieval. At its core, the production system:

• Generates captions for each image using a vision-language model (Gemma 4) — a detailed long caption, a short caption, and keywords
• Creates CLIP embeddings with DFN5B-CLIP-ViT-H-14-378 — a separate vector for the image and for the short caption + keywords
• Stores embeddings and metadata in NRP-managed Milvus
• Performs vector search against both image and caption embeddings
• Also does a traditional keyword search (BM25) on captions and SAGE metadata
• Combines the results with a three-way weighted hybrid ranker
• Finally, reranks the top hits with CLIP (query text vs. the image)

Why all this? Because combining these techniques helps you go beyond surface-level matching — it helps find the right image even if the words in the query aren’t an exact match to the caption.

Note: Individual components (captioning models, embedding models, rerankers, and so on) are interchangeable — the deployed system may use updated versions of those listed above, but the pipeline stays the same. For the latest models and services in use, see the Github Repository.

The Tech Stack & Pipeline

Here’s how the production stack fits together:

Step 1: Captioning with a vision-language model

We start with raw images from SAGE cameras. A VLM writes a long caption (rich description for keyword search), a short caption (visible content only), and keywords. Production uses Gemma 4 on the NRP AI Gateway. The teaching lab uses smaller Florence-2-base so it runs on a classroom GPU.

For example, an image of a storm in Chicago might be captioned as:

"A wet intersection of cars with a sky filled with dark clouds."

Step 2: Embedding with CLIP

We then use CLIP to generate vector representations for the image and for short caption + keywords (CLIP’s text window is 77 tokens). These live in the same vector space, so a natural-language query can be compared to both pixels and words. Production stores the two modalities as separate Milvus fields and blends them at query time.

Step 3: Vector + Keyword Search

At query time, we do two types of search:

• Vector Search: The user’s query is embedded with CLIP, and we search the most similar image_vector and caption_vector records.
• Keyword Search: The query is also run as a traditional text search against captions and metadata using BM25.

Each method has its strengths. Vector search understands meaning, while keyword search catches literal matches such as node IDs (vsn) and camera names.

Step 4: Hybrid Search Fusion

We merge image, caption, and BM25 results with Milvus hybrid_search and a WeightedRanker (roughly 46% image / 20% caption / 35% BM25). This hybrid strategy balances semantic relevance from vector search with the precision of keyword matching.

Step 5: Reranking with CLIP

Production then re-scores the top hits with CLIP: how well the query text matches each image, not just the caption. That is the finishing touch that boosts the most relevant images to the top. The Jupyter lab teaches the same idea with a classic CrossEncoder on query–caption pairs.

Hands-on Lab

This lab was built for Sage Grande: Summer of AI and is the best way to walk the pipeline yourself.

Launch the workspace on the National Data Platform:

Sage Image Search Lab Workspace

From there, open sage_image_search_lab.ipynb. Full NDP setup (GPU reservation, persistent storage, install steps) is in learning.md.

The notebook walks through:

1. SageBench subset
2. Captioning
3. CLIP embeddings
4. Indexing in Milvus Lite
5. Vector search, then keyword search
6. Hybrid search
7. Reranking
8. A Gradio search UI
9. Mini evaluation (MRR and Success@K)

Companion videos: Sage Image Search overview and Image Search Benchmarking.

Why This Matters

As our object database continues to grow, the challenge of finding the right image at the right moment becomes increasingly difficult. With thousands of visuals available, building rich datasets with Sage is possible — but navigating that volume can be frustrating and time-consuming. Traditional search methods in our cyberinfrastructure are starting to show their limits. Time-based filtering assumes users already know when the relevant images were captured, while metadata filtering often forces them to sift through large volumes manually to find what they need. That’s where this hybrid approach makes a real difference — by combining the semantic power of vector search with the precision of keyword matching, users can efficiently locate the exact image they’re looking for — and pinpoint the precise moment it was captured.

If you’re working on something similar or curious to dive deeper, feel free to reach out.

Try the live UI at portal.sagecontinuum.org/labs/image-search.

Publication

This work was presented at USRSE'25. For a high-level overview of the system architecture and motivation, see the official poster: Searching Sage: AI-Powered Image Retrieval on a Nationwide Edge Computing Cyberinfrastructure.

Resources

Everything you need to try the system, run the hands-on lab, and cite the work lives here:

Resource	Link
Try the live search	portal.sagecontinuum.org/labs/image-search
Source code	waggle-sensor/sage-nrp-image-search
National Research Platform	nrp.ai
Run the lab on NDP	Image Search Lab workspace
Jupyter notebook	sage_image_search_lab.ipynb
Lab guide (NDP setup, steps, production mapping)	docs/learning.md
Videos	System overview · Benchmarking
Documentation	Overview · Architecture · Using the Search UI
Evaluation	Image Search Benchmarking lab · docs/benchmarking.md · SageBench · FireBench · CloudBench · CommonObjectsBench · INQUIRE
Summer of AI	Sage Grande: Summer of AI
Publication	USRSE'25 poster

Try It NowView on GitHub