/xcode-26

Implementing Visual Intelligence in iOS

Overview

Visual Intelligence is a framework that enables iOS apps to integrate with the system's visual search capabilities. It allows users to find app content that matches their surroundings or objects onscreen by using the visual intelligence camera or screenshots. When a user performs a visual search, your app can provide relevant content that matches what they're looking at.

Key concepts:

• Visual Intelligence framework provides information about objects detected in the camera or screenshots
• App Intents framework facilitates the exchange of information between the system and your app
• Your app searches its content for matches and returns them as app entities
• Results appear directly in the visual search interface, allowing users to view and interact with your content

Setting Up Visual Intelligence

Required Frameworks

import VisualIntelligence
import AppIntents

Implementation Steps

1. Create an IntentValueQuery to receive visual search requests
2. Implement the values(for:) method to process the SemanticContentDescriptor
3. Search your app's content using the provided information
4. Return matching content as app entities

Working with SemanticContentDescriptor

The SemanticContentDescriptor is the core object that provides information about what the user is looking at.

Key Properties

// A list of labels that Visual Intelligence uses to classify items
let labels: [String]

// The pixel buffer containing the visual data
var pixelBuffer: CVReadOnlyPixelBuffer?

Accessing Visual Data

You can use either the labels or the pixel buffer (or both) to search for matching content:

// Using labels
func searchByLabels(_ labels: [String]) -> [AppEntity] {
    // Search your app's content using the provided labels
    return matchingEntities
}

// Using pixel buffer
func searchByImage(_ pixelBuffer: CVReadOnlyPixelBuffer) -> [AppEntity] {
    // Convert pixel buffer to an image and search your content
    return matchingEntities
}

Creating an IntentValueQuery

The IntentValueQuery protocol is the entry point for Visual Intelligence to communicate with your app.

Basic Implementation

struct LandmarkIntentValueQuery: IntentValueQuery {
    @Dependency var modelData: ModelData
    
    func values(for input: SemanticContentDescriptor) async throws -> [VisualSearchResult] {
        // Check if pixel buffer is available
        guard let pixelBuffer = input.pixelBuffer else {
            return []
        }
        
        // Search for matching landmarks using the pixel buffer
        let landmarks = try await modelData.search(matching: pixelBuffer)
        
        return landmarks
    }
}

Using Union Values for Different Result Types

If your app needs to return different types of results, use a union value:

@UnionValue
enum VisualSearchResult {
    case landmark(LandmarkEntity)
    case collection(CollectionEntity)
}

Providing Display Representations

Visual Intelligence uses the DisplayRepresentation of your AppEntity to present your content in the search results.

Creating a Display Representation

struct LandmarkEntity: AppEntity {
    static var typeDisplayRepresentation: TypeDisplayRepresentation {
        return TypeDisplayRepresentation(
            name: LocalizedStringResource("Landmark", table: "AppIntents"),
            numericFormat: "\(placeholder: .int) landmarks"
        )
    }
    
    var displayRepresentation: DisplayRepresentation {
        DisplayRepresentation(
            title: "\(name)",
            subtitle: "\(location)",
            image: .init(named: landmark.thumbnailImageName)
        )
    }
    
    // Other required AppEntity properties and methods
}

Opening Items in Your App

When a user taps on a search result, your app should open to display detailed information about that item.

Implementing AppEntity for Deep Linking

struct LandmarkEntity: AppEntity {
    var id: String
    var name: String
    var location: String
    var thumbnailImageName: String
    
    // Required for deep linking
    static var typeDisplayRepresentation: TypeDisplayRepresentation {
        // As shown above
    }
    
    var displayRepresentation: DisplayRepresentation {
        // As shown above
    }
    
    // Define how to open this entity in your app
    var appLinkURL: URL? {
        URL(string: "yourapp://landmark/\(id)")
    }
}

Linking to Additional Results

If your app finds many matches, you can provide a "More results" button that opens your app to show the full list.

Creating a Semantic Content Search Intent

struct ViewMoreLandmarksIntent: AppIntent, VisualIntelligenceSearchIntent {
    static var title: LocalizedStringResource = "View More Landmarks"
    
    @Parameter(title: "Semantic Content")
    var semanticContent: SemanticContentDescriptor
    
    func perform() async throws -> some IntentResult {
        // Open your app's search view with the semantic content
        return .result()
    }
}

Complete Example

Here's a complete example of implementing Visual Intelligence in a landmarks app:

import SwiftUI
import AppIntents
import VisualIntelligence

// Define the search result types
@UnionValue
enum VisualSearchResult {
    case landmark(LandmarkEntity)
    case collection(CollectionEntity)
}

// Define the landmark entity
struct LandmarkEntity: AppEntity {
    var id: String
    var name: String
    var location: String
    var thumbnailImageName: String
    
    static var typeDisplayRepresentation: TypeDisplayRepresentation {
        return TypeDisplayRepresentation(
            name: LocalizedStringResource("Landmark", table: "AppIntents"),
            numericFormat: "\(placeholder: .int) landmarks"
        )
    }
    
    var displayRepresentation: DisplayRepresentation {
        DisplayRepresentation(
            title: "\(name)",
            subtitle: "\(location)",
            image: .init(named: thumbnailImageName)
        )
    }
    
    var appLinkURL: URL? {
        URL(string: "yourapp://landmark/\(id)")
    }
}

// Define the collection entity
struct CollectionEntity: AppEntity {
    var id: String
    var name: String
    var landmarks: [LandmarkEntity]
    
    static var typeDisplayRepresentation: TypeDisplayRepresentation {
        return TypeDisplayRepresentation(
            name: LocalizedStringResource("Collection", table: "AppIntents"),
            numericFormat: "\(placeholder: .int) collections"
        )
    }
    
    var displayRepresentation: DisplayRepresentation {
        DisplayRepresentation(
            title: "\(name)",
            subtitle: "\(landmarks.count) landmarks",
            image: .init(systemName: "square.stack.fill")
        )
    }
    
    var appLinkURL: URL? {
        URL(string: "yourapp://collection/\(id)")
    }
}

// Define the intent value query
struct LandmarkIntentValueQuery: IntentValueQuery {
    @Dependency var modelData: ModelData
    
    func values(for input: SemanticContentDescriptor) async throws -> [VisualSearchResult] {
        // Try to use labels first
        if !input.labels.isEmpty {
            let landmarks = try await modelData.search(matching: input.labels)
            return landmarks
        }
        
        // Fall back to pixel buffer if available
        guard let pixelBuffer = input.pixelBuffer else {
            return []
        }
        
        let landmarks = try await modelData.search(matching: pixelBuffer)
        return landmarks
    }
}

// Define the "more results" intent
struct ViewMoreLandmarksIntent: AppIntent, VisualIntelligenceSearchIntent {
    static var title: LocalizedStringResource = "View More Landmarks"
    
    @Parameter(title: "Semantic Content")
    var semanticContent: SemanticContentDescriptor
    
    func perform() async throws -> some IntentResult {
        // Open your app's search view with the semantic content
        return .result()
    }
}

// Example model data service
class ModelData {
    func search(matching labels: [String]) async throws -> [VisualSearchResult] {
        // Search your database for landmarks matching the labels
        // Return matching landmarks as VisualSearchResult objects
        return []
    }
    
    func search(matching pixelBuffer: CVReadOnlyPixelBuffer) async throws -> [VisualSearchResult] {
        // Use image recognition to find landmarks in the pixel buffer
        // Return matching landmarks as VisualSearchResult objects
        return []
    }
}

Best Practices

1. Performance: Return results quickly for a good search experience

   • Limit the number of returned items (consider showing 10-20 most relevant results)
   • Use the "More results" button for additional items
   • Optimize your search algorithms for speed

2. Quality: Provide high-quality display representations

   • Use clear, concise titles and subtitles
   • Include relevant images that help identify the content
   • Ensure all text is properly localized

3. Relevance: Focus on returning the most relevant results

   • Prioritize exact matches over partial matches
   • Consider the context of the search (location, time, etc.)
   • Filter out irrelevant or low-confidence matches

4. User Experience: Make it easy to navigate from search results to your app

   • Implement deep linking to open specific content
   • Maintain context when transitioning to your app
   • Provide a consistent experience between search results and your app

Testing

To test your Visual Intelligence integration:

1. Build and run your app on a device
2. Use the visual intelligence camera or take a screenshot
3. Perform a visual search on content relevant to your app
4. Verify that your app's results appear in the search results
5. Test tapping on results to ensure they open correctly in your app

References

• Integrating your app with visual intelligence
• SemanticContentDescriptor
• IntentValueQuery
• DisplayRepresentation
• TypeDisplayRepresentation
• App Intents framework
• Making actions and content discoverable and widely available