In this episode, Julia Kasper discusses how AI models are rapidly evolving within the VS Code environment, emphasizing the importance of continuously evaluating and fine-tuning models like Raptor Mini to optimize developer workflows based on task complexity. She also highlights the challenges of AI model evaluation and encourages developers to actively experiment with different models and provide feedback to help improve the tools.

In this episode of the VS Code Insiders podcast, host James Monttoagno welcomes Julia Kasper to discuss the evolving landscape of AI models in software development, particularly within the VS Code environment. Julia shares her excitement about how AI is transforming developer workflows, blending elements of low-code development with traditional coding by accelerating tasks and enhancing productivity. She highlights the rapid pace of change in AI models, comparing it to the frequent emergence of new JavaScript frameworks, which keeps developers continuously adapting and experimenting with different models.

Julia explains that developers often start by choosing a single AI model and sticking with it, but her experience on the VS Code team has shown her the importance of continuously evaluating and switching models based on the task at hand. She notes that both the models themselves and the prompts used to interact with them are regularly updated, sometimes by the model providers and often by the VS Code team, to improve performance and better suit developer needs. This dynamic approach means that the same model can behave differently over time as improvements are made.

The conversation delves into the concept of fine-tuning AI models, where a base model is adapted using specific data and test cases to better align with particular workflows, such as coding within VS Code. Julia describes how Microsoft’s in-house team fine-tunes models like Raptor Mini to optimize them for speed and repetitive tasks, making them suitable for less complex coding activities. She emphasizes that choosing the right model depends more on the complexity and nature of the task rather than the size of the codebase, with larger, more creative tasks benefiting from bigger models and simpler, faster tasks suited to smaller, fine-tuned models.

Julia also sheds light on the evaluation process for AI models, distinguishing between online evaluations, which analyze live user data and feedback, and offline evaluations, which use predefined test cases to benchmark model performance before release. She points out the challenges in evaluating AI outputs due to their creative and non-binary nature, which differs from traditional unit testing. The VS Code team uses both internal benchmarks and community feedback, including thumbs up/down ratings within the editor, to continuously refine models and prompts to enhance the developer experience.

Finally, Julia encourages developers to actively engage with the AI tools in VS Code by experimenting with different models and providing feedback through built-in mechanisms like the thumbs up/down feature. She highlights the importance of this feedback loop in driving improvements and invites users to explore reusable prompts and create their own evaluation routines. The episode closes with James thanking Julia for her insights and reminding listeners to subscribe and engage with the VS Code community for ongoing updates and support.