microsoft

Phi-3.5 Vision Instruct

Released

2024-08-22

License

MIT License

Family

Phi 3

Type

Foundation Model

Huggingface Page Official Post

Model Report

Overview

Phi-3.5 Vision Instruct is a multimodal generative AI model developed by Microsoft, extending the Phi-3.5 family of Small Language Models (SLMs) with both textual and visual reasoning capabilities. Released in August 2024, Phi-3.5 Vision Instruct is engineered to deliver efficient, high-quality performance across a variety of commercial and research tasks, focusing primarily on English language use. The model emphasizes robust multi-frame image analysis, instruction following, and scalable deployment within constrained computational environments, supporting uses ranging from general visual understanding to advanced document summarization and reasoning. For further details, consult the model documentation on Hugging Face and technical disclosures from Microsoft's release announcement.

Scatter plot showing Phi-3.5 model efficiency and quality — Figure 1. Phi-3.5 model family efficiency and performance, highlighting Phi-3.5-mini and Phi-3.5-MoE in comparison to other small language models.
Full size Source

Model Architecture and Training

Phi-3.5 Vision Instruct is built on a 4.2-billion-parameter architecture that integrates an image encoder, connector, projector, and the Phi-3 Mini language model core. The system uses supervised fine-tuning (SFT) and direct preference optimization (DPO) to ensure instruction adherence, as well as Reinforcement Learning from Human Feedback (RLHF) to further align outputs with safety and usability standards, as outlined in the model card. This architecture allows the model to process and reason over both text and multiple images within the same context.

The model training set is a composite of rigorously filtered publicly available documents, educational datasets, synthetic structured data (including "textbook-like" materials for math, coding, and reasoning), and diverse image-text pairs. Specialized datasets for multi-image and short video understanding were newly created, ensuring that Phi-3.5 Vision Instruct could handle summarization, comparison, and storytelling tasks over sequences of images or video frames. Data collection procedures include filtering to remove undesirable content and scrubbing of potentially personal information to enhance privacy, in accordance with Microsoft’s Responsible AI guidelines.

The implementation leverages PyTorch, Hugging Face Transformers, and Flash-Attention, optimizing performance for modern GPU accelerators and supporting context windows of up to 128,000 tokens.

Technical Capabilities

Phi-3.5 Vision Instruct offers a broad set of multimodal capabilities. Its primary strength is the ability to process both single and multi-frame images in conjunction with textual prompts, enabling detailed visual question answering, comparison across multiple images, chart and table understanding, document OCR, and video or sequential image summarization. The model demonstrates enhanced performance in reasoning tasks that require understanding temporal or comparative relationships across multiple frames—a feature developed in response to user demand for richer context handling, as described in the official release.

A typical usage paradigm involves structured chat-style inputs, where image and text data are presented sequentially within the supported context window. The model is particularly suited to scenarios where computational or memory constraints demand high efficiency, and where latency requirements are paramount.

Phi-3.5-vision output for slide summarization task — Figure 2. Example output of Phi-3.5 Vision Instruct summarizing a presentation about dogs. Prompt: 'Summarize the deck of slides.'
Full size Source

Performance and Benchmark Evaluation

Phi-3.5 Vision Instruct demonstrates substantial improvements across established vision-language and multi-frame benchmarks, particularly in single-image and sequence-based reasoning. On single-image tasks, the model records increased scores in key benchmarks such as MMMU (43.0), MMBench (81.9), and TextVQA (72.0), as reported in the benchmark summary on Hugging Face. For multi-image and video benchmarks, Phi-3.5 Vision Instruct achieves an overall score of 57.0 on the BLINK suite—outperforming models like LlaVA-Interleave-Qwen-7B and InternVL-2-8B, and remaining competitive with significantly larger systems.

BLINK benchmark results across Phi-3.5-vision and peer models — Figure 3. BLINK multi-frame visual reasoning benchmark comparison. Phi-3.5 Vision Instruct demonstrates strong performance across diverse sub-tasks.
Full size Source

In video-oriented evaluation (Video-MME), the model secures a score of 50.8, surpassing several competing small and mid-sized LLMs. Across representative datasets—such as ScienceQA, MathVista, ChartQA, and POPE—Phi-3.5 Vision Instruct maintains robust performance, illustrating versatility not only in general visual analysis but also in structured document interpretation and educational domains. Full results and comparative data are accessible via the official model benchmark table on Hugging Face.

Relationship to Phi-3.5 Model Family

Phi-3.5 Vision Instruct is released alongside other notable Phi-3.5 models, including Phi-3.5-mini and Phi-3.5-MoE. Phi-3.5-mini features 3.8 billion parameters and emphasizes multi-lingual support across over twenty languages, with a 128K token context window and competitive performance in language and reasoning tasks, as detailed in the Tech Community blog post. Phi-3.5-MoE introduces a Mixture-of-Experts architecture, activating subsets of its total 42B parameter model to offer high performance at lower active compute, particularly in multi-lingual and long-context scenarios.

Comparative benchmark performance table for Phi-3.5-mini and peer models — Figure 4. Phi-3.5-mini performance on central benchmarks relative to other open and proprietary large language models.
Full size Source

Though Phi-3.5-mini and MoE variants excel in multi-lingual and conversational contexts, Phi-3.5 Vision Instruct is principally optimized for English and advanced multimodal tasks, focusing on vision-language integration and multi-frame analysis.

Limitations and Responsible Use

While Phi-3.5 Vision Instruct demonstrates high accuracy and efficiency, certain constraints and responsibilities should be considered in deployment. The model is primarily optimized for English, and its performance may degrade in other languages unless fine-tuned for multi-lingual use. As with other large AI models, outputs may contain inaccuracies, reflect biases or stereotypes, or generate content that is inappropriate for sensitive applications. It is not intended for high-risk decision-making without further validation, and developers are advised to implement user transparency, feedback mechanisms, and additional mitigations for safety and fairness, as outlined in the responsible AI statement and Microsoft Responsible AI Standard.

Special caution is warranted in scenarios that might involve identification of individuals in images, or any context where the reliability and provenance of generated information are critical. The model is distributed under the MIT License information.

Typical Applications

Phi-3.5 Vision Instruct is suitable for deployment in memory- or compute-constrained environments, latency-sensitive use cases, and applications involving complex visual reasoning. Example scenarios include detailed analysis of document layouts, chart and table extraction, multi-image comparison, summarization of slide decks or video sequences, and serving as an intelligent assistant for visually-rich workflows in domains such as education, office automation, and analytics.

Further Resources

About Phi 3: The Phi-3 family of AI models, developed by Microsoft, are compact yet powerful small language models (SLMs) that outperform larger counterparts in language, reasoning, coding, and math benchmarks, and are efficient enough to run on devices like smartphones.