According to a study by UC Berkeley, OpenAI software is currently superior to all other models
AI chatbots can analyse sentences like a trained linguist, according to a study by UC Berkeley. It provides insights into the development of AI models and also challenges the notion that only humans have the ability to think about language. According to IEEE Transactions on Artificial Intelligence, OpenAI’s ‘o1’ has proven to be clearly superior.
AI conquers metalinguistics
According to research leader Gašper Beguš, talking about language and deliberately influencing it, i.e. metalinguistics, is now also becoming a domain of AI chatbots. ‘The most advanced large language models are closing this gap. They can not only use language, they can also reflect on how language is organised.’
The researchers fed 120 complex sentences into several versions of OpenAI’s ChatGPT and Meta’s Llama 3.1. With each sentence, the system was instructed to analyse it and assess whether it had a specific linguistic quality. In addition, a diagram was created, a so-called syntax tree, i.e. a visual representation of the structure of the sentence and its components.
Only o1 sees ambiguity
In the sentence ‘Eliza wanted her cast out,’ for example, the researchers wanted to know whether the AI could recognise an ambiguous structure. Did Eliza want someone to be excluded? Or did she want a cast to be removed? Versions 3.5 and 4, as well as Llama, were unable to recognise this ambiguity. The OpenAI model o1, on the other hand, recognised the ambiguity and created a corresponding diagram. This model is designed for more complex questions.
Beguš analysed the systems for their ability to recognise recursion, i.e. the infinite possibilities of language. The AI models were tasked with determining whether recursion was present in a sample sentence and, if so, which specific linguistic version it was. In addition, the models were instructed to add a similar recursive clause.
o1 was also successful with the sentence ‘Unidentified flying objects may have conflicting characteristics.’ And it took the sentence to a whole new level with ‘Unidentified recently sighted flying objects may have conflicting characteristics.’
