Microsoft's new coding LLM is coming after GPT 4

Microsoft is really taking a big step into AI development. Last month, AI was at the forefront of the Microsoft Build conference, and the Redmond-based tech even announced that Copilot is coming to Windows 11. As a native built-in app, for everything you need,

AI is coming to Microsoft Teams as well, in its Recap Ai tool. And a lot of other tools from Microsoft will use AI, including Microsoft Fabric.

But it seems, Microsoft is also funding AI research. Just recently, the tech giant announced Orca 13B will be open source. And LongMem is another good point in AI research: it is the answer for unlimited context length.

And now there is time for another big breakthrough in AI research, coming from Microsoft, of course. The new 1.3B-parameter model coding LLM, called phi-1, is reportedly outperforming GPT 3.5, in only 4 days of training.

What is phi-1 and how does it outperform GPT already?

Phi-1 is a new 1.3B-parameter language model for code, with a significantly smaller size than competing models. The language model was trained for 4 days, over 7B tokens (slightly over 50B total tokens seen) followed by finetuning on less than 200M tokens.

Despite being a lot smaller than the competing models, phi-1 attained 50.6% pass@1 accuracy on HumanEval and 55.5% pass@1 accuracy on MBPP (Mostly Basic Python Programs), which are one of the best self-reported numbers using only one LLM generation.

Moreover, despite being trained on much fewer tokens compared to existing models, phi-1 still has a lot of potentials.

The improvements on HumanEval are the phi-1 greatest achievements, as a language model. After tuning, phi-1 managed to execute tasks that were not featured in the finetuning dataset. This means the model adapted and improved the fine-tuning process.

And the most remarkable thing is that phi-1 reorganized and consolidated the knowledge acquired during pretraining, even though the knowledge was not explicitly present there in the first place.

In shorter words, phi-1 not only learns during training but also expands the knowledge on its own. It managed to outperform GPT 3.5 on every level, and it’s just a matter of time until the small model takes on the big ones, like GPT 4.

What do you think about this new AI model? Where do you think AI research is heading? Be sure to let us know your opinions in the comments section below.