News

Introduction: Understanding Generalization in Deep Generative Models Deep generative models, including diffusion and flow matching, have shown outstanding performance in synthesizing realistic multi-modal content across images, audio, video, and text. However, the generalization capabilities and underlying mechanisms of these models are challenging in deep generative modeling. The core challenge includes understanding whether generative models truly generalize or simply memorize training data. Current research reveals conflicting evidence: some studies show that large diffusion models memorize individual samples from training sets, while others show clear signs of generalization when trained on large datasets. This contradiction points to a sharp phase transition between memorization and generalization. Existing Literature on Flow Matching and Generalization Mechanisms Existing research includes the utilization of closed-form solutions, studying memorization versus generalization, and characterizing different phases of generating dynamics. Methods like closed-form velocity field regression and a smoothed version of optimal velocity generation have been proposed. Studies on memorization relate the transition to generalization with training dataset size through geometric interpretations, while others focus on stochasticity in target objectives. Temporal regime analysis identifies distinct phases in generative dynamics, which show reliance on dimension and sample numbers. But validation methods depend on backward process stochasticity, which doesn’t apply to flow matching models, leaving significant gaps in understanding. New Findings: Early Trajectory Failures Drive Generalization Researchers from Université Jean Monnet Saint-Etienne and Université Claude Bernard Lyon provide an answer to whether training on noisy or stochastic targets improves flow matching generalization and identify the main sources of generalization. The method reveals that generalization emerges when limited-capacity neural networks fail to approximate the exact velocity field during critical time intervals at early and late phases. The researchers identify that generalization arises mainly early along flow matching trajectories, corresponding to the transition from stochastic to deterministic behaviour. Moreover, they propose a learning algorithm that explicitly regresses against the exact velocity field, showing enhanced generalization capabilities on standard image datasets. Investigating the Sources of Generalization in Flow Matching Researchers investigate the key sources of generalization. First, they challenge target stochasticity assumptions by using closed-form optimal velocity field formulations, showing that after small time values, the weighted average of conditional flow matching targets equals single expectation values. Second, they analyze the approximate quality between learned velocity fields and optimal velocity fields through systematic experiments on subsampled CIFAR-10 datasets ranging from 10 to 10,000 samples. Third, they construct hybrid models using piecewise trajectories governed by optimal velocity fields for early time intervals and learned velocity fields for later intervals, with adjustable threshold parameters to determine critical periods. Empirical Flow Matching: A Learning Algorithm for Deterministic Targets Researchers implement a learning algorithm that regresses against more deterministic targets using closed-form formulas. It compares vanilla conditional flow matching, optimal transport flow matching, and empirical flow matching across CIFAR-10 and CelebA datasets using multiple samples to estimate empirical means. Moreover, evaluation metrics include Fréchet Inception Distance with Inception-V3 and DINOv2 embeddings for a less biased assessment. The computational architecture operates with complexity O(M × |B| × d). Training configurations demonstrate that increasing sample numbers M for empirical mean computation creates less stochastic targets, leading to more stable performance improvements with modest computational overhead when M equals the batch size. Conclusion: Velocity Field Approximation as the Core of Generalization In this paper, researchers challenge the assumption that stochasticity in loss functions drives generalization in flow matching models, clarifying the critical role of exact velocity field approximation instead. While research provides empirical insights into practical learned models, precise characterization of learned velocity fields outside optimal trajectories remains an open challenge, suggesting future work to use architectural inductive biases. The broader implications include concerns about potential misuse of improved generative models for creating deepfakes, privacy violations, and synthetic content generation. So, it is necessary to give careful consideration to ethical applications. Why This Research Matters? This research is significant because it challenges a prevailing assumption in generative modeling—that stochasticity in training objectives is a key driver of generalization in flow matching models. By demonstrating that generalization instead arises from the failure of neural networks to precisely approximate the closed-form velocity field, especially during early trajectory phases, the study reframes our understanding of what enables models to produce novel data. This insight has direct implications for designing more efficient and interpretable generative systems, reducing computational overhead while maintaining or even enhancing generalization. It also informs better training protocols that avoid unnecessary stochasticity, improving reliability and reproducibility in real-world applications. Check out the Paper. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. The post Why Generalization in Flow Matching Models Comes from Approximation, Not Stochasticity appeared first on MarkTechPost.

Multimodal LLMs: Expanding Capabilities Across Text and Vision Expanding large language models (LLMs) to handle multiple modalities, particularly images and text, has enabled the development of more interactive and intuitive AI systems. Multimodal LLMs (MLLMs) can interpret visuals, answer questions about images, and engage in dialogues that include both text and pictures. Their ability to reason across visual and linguistic domains makes them increasingly valuable for applications such as education, content generation, and interactive assistants. The Challenge of Text-Only Forgetting in MLLMs However, integrating vision into LLMs creates a problem. When trained on datasets that mix images with text, MLLMs often lose their ability to handle purely textual tasks. This phenomenon, known as text-only forgetting, occurs because visual tokens inserted into the language sequence divert the model’s attention away from the text. As a result, the MLLM starts prioritizing image-related content and performs poorly on tasks that require only language understanding, such as basic reasoning, comprehension, or textual question-and-answer (Q&A) tasks. Limitations of Existing Mitigation Strategies Several methods attempt to address this degradation. Some approaches reintroduce large amounts of text-only data during training, while others alternate between text-only and multimodal fine-tuning. These strategies aim to remind the model of its original language capabilities. Other designs include adapter layers or prompt-based tuning. However, these techniques often increase training costs, require complex switching logic during inference, or fail to restore text comprehension entirely. The problem largely stems from how the model’s attention shifts when image tokens are introduced into the sequence. Introducing WINGS: A Dual-Learner Approach by Alibaba and Nanjing University Researchers from Alibaba Group’s AI Business team and Nanjing University have introduced a new approach called WINGS. The design adds two new modules—visual and textual learners—into each layer of the MLLM. These learners work in parallel with the model’s core attention mechanism. The structure resembles “wings” attached to either side of the attention layers. A routing component controls how much attention each learner receives based on the current token mix, allowing the model to balance its focus between visual and textual information dynamically. Low-Rank Residual Attention (LoRRA): Balancing Efficiency and Modality Awareness The WINGS architecture uses a mechanism called Low-Rank Residual Attention (LoRRA), which keeps computations lightweight while enabling the learners to capture essential modality-specific information. In the first stage of training, only visual learners are activated to align image features. In the second stage, both visual and textual learners are co-trained with a router module that uses attention weights to allocate responsibility. Each learner uses efficient attention blocks to interact with either the image or the surrounding text, and their outputs are combined with those of the main model. This ensures that visual attention doesn’t overwhelm textual understanding. WINGS Performance Benchmarks Across Text and Multimodal Tasks In terms of performance, WINGS showed strong results. On the MMLU dataset, it achieved a text-only score of 60.53, representing an improvement of 9.70 points compared to a similar baseline model. For CMMLU, it scored 69.82, which is 9.36 points higher than the baseline. In reasoning tasks like Race-High, it gained 11.9 points, and in WSC, an improvement of 11.12 points was recorded. In multimodal benchmarks like MMMU-VAL, WINGS achieved an improvement of 4.78 points. It also demonstrated robust results on the IIT benchmark, handling mixed text-and-image multi-turn dialogues more effectively than other open-source MLLMs at the same scale. Conclusion: Toward More Balanced and Generalizable MLLMs In summary, the researchers tackled the issue of catastrophic text-only forgetting in MLLMs by introducing WINGS, an architecture that pairs dedicated visual and textual learners alongside attention routing. By analyzing attention shifts and designing targeted interventions, they maintained text performance while enhancing visual understanding, offering a more balanced and efficient multimodal model. Check out the Paper. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. The post This AI Paper Introduces WINGS: A Dual-Learner Architecture to Prevent Text-Only Forgetting in Multimodal Large Language Models appeared first on MarkTechPost.

The Importance of Symbolic Reasoning in World Modeling Understanding how the world works is key to creating AI agents that can adapt to complex situations. While neural network-based models, such as Dreamer, offer flexibility, they require massive amounts of data to learn effectively, far more than humans typically do. On the other hand, newer methods use program synthesis with large language models to generate code-based world models. These are more data-efficient and can generalize well from limited input. However, their use has been mostly limited to simple domains, such as text or grid worlds, as scaling to complex, dynamic environments remains a challenge due to the difficulty of generating large, comprehensive programs. Limitations of Existing Programmatic World Models Recent research has investigated the use of programs to represent world models, often leveraging large language models to synthesize Python transition functions. Approaches like WorldCoder and CodeWorldModels generate a single, large program, which limits their scalability in complex environments and their ability to handle uncertainty and partial observability. Some studies focus on high-level symbolic models for robotic planning by integrating visual input with abstract reasoning. Earlier efforts employed restricted domain-specific languages tailored to specific benchmarks or utilized conceptually related structures, such as factor graphs in Schema Networks. Theoretical models, such as AIXI, also explore world modeling using Turing machines and history-based representations. Introducing PoE-World: Modular and Probabilistic World Models Researchers from Cornell, Cambridge, The Alan Turing Institute, and Dalhousie University introduce PoE-World, an approach to learning symbolic world models by combining many small, LLM-synthesized programs, each capturing a specific rule of the environment. Instead of creating one large program, PoE-World builds a modular, probabilistic structure that can learn from brief demonstrations. This setup supports generalization to new situations, allowing agents to plan effectively, even in complex games like Pong and Montezuma’s Revenge. While it doesn’t model raw pixel data, it learns from symbolic object observations and emphasizes accurate modeling over exploration for efficient decision-making. Architecture and Learning Mechanism of PoE-World PoE-World models the environment as a combination of small, interpretable Python programs called programmatic experts, each responsible for a specific rule or behavior. These experts are weighted and combined to predict future states based on past observations and actions. By treating features as conditionally independent and learning from the full history, the model remains modular and scalable. Hard constraints refine predictions, and experts are updated or pruned as new data is collected. The model supports planning and reinforcement learning by simulating likely future outcomes, enabling efficient decision-making. Programs are synthesized using LLMs and interpreted probabilistically, with expert weights optimized via gradient descent. Empirical Evaluation on Atari Games The study evaluates their agent, PoE-World + Planner, on Atari’s Pong and Montezuma’s Revenge, including harder, modified versions of these games. Using minimal demonstration data, their method outperforms baselines such as PPO, ReAct, and WorldCoder, particularly in low-data settings. PoE-World demonstrates strong generalization by accurately modeling game dynamics, even in altered environments without new demonstrations. It’s also the only method to consistently score positively in Montezuma’s Revenge. Pre-training policies in PoE-World’s simulated environment accelerate real-world learning. Unlike WorldCoder’s limited and sometimes inaccurate models, PoE-World produces more detailed, constraint-aware representations, leading to better planning and more realistic in-game behavior. Conclusion: Symbolic, Modular Programs for Scalable AI Planning In conclusion, understanding how the world works is crucial to building adaptive AI agents; however, traditional deep learning models require large datasets and struggle to update flexibly with limited input. Inspired by how humans and symbolic systems recombine knowledge, the study proposes PoE-World. This method utilizes large language models to synthesize modular, programmatic “experts” that represent different parts of the world. These experts combine compositionally to form a symbolic, interpretable world model that supports strong generalization from minimal data. Tested on Atari games like Pong and Montezuma’s Revenge, this approach demonstrates efficient planning and performance, even in unfamiliar scenarios. Code and demos are publicly available. Check out the Paper, Project Page and GitHub Page. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. The post PoE-World + Planner Outperforms Reinforcement Learning RL Baselines in Montezuma’s Revenge with Minimal Demonstration Data appeared first on MarkTechPost.

Ever felt like trying to find a needle in a haystack? That’s part of the process of building and optimizing machine learning models, particularly complex ones like ensembles and neural networks, where several hyperparameters need to be manually set by us before training them.