Apple has created AI that can outperform GPT-4 in understanding commands
Apple researchers have developed an artificial intelligence system called ReALM (Reference Resolution as Language Modeling) that aims to dramatically improve the way voice assistants understand and respond to commands.
In a research paper reported via VentureBeat, Apple introduces a new system for solving the problem of recognizing references (indications, references, or links) by large language models. This takes into account deciphering ambiguous references or descriptions of elements on the screen and understanding the context of the conversation and the general background. As a result, ReALM can lead to more intuitive and natural interactions with devices.
Reference recognition is an important part of understanding natural language, allowing users to use pronouns and other indirect references in conversation without confusion. For digital assistants, this ability has historically been a significant challenge, limited by the need to interpret a wide range of verbal cues and visual information. Apple’s ReALM system aims to solve this problem by turning the complex process of reference recognition into a purely linguistic modeling task. Thus, it can understand references to visual elements displayed on the screen and integrate this understanding into the flow of conversation.
ReALM reconstructs the visual layout of the screen using textual representations. This involves analyzing the elements on the screen and their locations to create a text format that reflects the content and structure of the screen. Apple researchers have found that this strategy, combined with specific modifications to language models for reference recognition tasks, significantly outperforms traditional methods, including OpenAI’s GPT-4 capabilities.
ReALM can allow users to interact more effectively with digital assistants by keeping in mind what is currently on their screen, without the need for precise, detailed instructions. This has the potential to make voice assistants much more useful in a variety of settings, such as helping drivers control infotainment systems while driving or assisting users with disabilities by providing an easier and more accurate way to interact indirectly.
Apple has already published several research papers in the field of artificial intelligence. Last month, the company introduced a new method for training large language models that continuously integrates text and visual information. Apple is expected to unveil a number of AI features at the World Wide Web Developers Congress in June.
