ChatGPT failed to answer half of the questions correctly at the programming exam

August 10, 2023SUNDRIES1 min read

Generative artificial intelligence often makes mistakes, so even developers do not recommend using it to create program code. To test ChatGPT’s overall abilities and knowledge in this area, the system was asked more than 500 questions about software development. More than half of them were wrong, TechSpot reports.

ChatGPT failed to answer half of the questions correctly at the programming exam

Researchers from Purdue University in Indiana, USA, asked ChatGPT 517 questions from Stack Overflow, a popular resource for professional programmers and enthusiasts. They evaluated the answers not only for correctness but also for consistency, exhaustiveness, and brevity. We also analyzed the language style and sentiment of the answers.

According to the results of the experiment, ChatGPT gave only 48% of correct answers. At the same time, 77% of the answers were described by the researchers as wordy. The exhaustiveness and textbook style of the bot’s writing contributed to the fact that the wrong answer seemed correct to the volunteers.

The study says that even when ChatGPT’s answer was clearly wrong, two out of 12 participants still preferred it because of the AI’s pleasant, confident, and positive tone.

Sources

techspot

Share:Telegram Viber Facebook X

Latest

Artificial Intelligence in Google Maps: New Features for Convenient Trip Planning

Google is actively integrating artificial intelligence into its services, making them more convenient and functional. Recently, searching for information in Gmail was simplified, and now the updates have reached Google Maps. The new version of Google Maps has introduced a feature that allows finding locations using screenshots. The app analyzes images, identifies mentions of places, […]

March 29, 2025

Faster, Cheaper, More Powerful: DeepSeek Presents New Version of Its Language Model

Chinese technology company DeepSeek has introduced an improved version of its large artificial intelligence language model DeepSeek-V3-0324, just three months after releasing the first V3 version in December 2024. The updated model features enhanced efficiency and expanded capabilities, including the ability to create visually appealing web pages and high-quality reports in Chinese. DeepSeek-V3-0324 stands out […]

March 29, 2025

Video

The Skype Era is ending: Microsoft offers alternative for millions of users

Microsoft is shutting down Skype after more than 20 years of operation. This decision will affect millions of users worldwide who have been using this program for video calls, messaging, and file sharing for decades. Instead, the company is offering a transition to another product. Reasons for Discontinuing Skype This step is dictated by Microsoft’s […]

March 28, 2025

The United States plans to implement a system to jam Russian and Chinese satellites

The US Space Force is preparing to introduce a new ground-based satellite jamming system aimed at countering Russian and Chinese satellites. According to Bloomberg, the first 11 of the 24 remote modular terminals of the system, known as Remote Modular Terminal (RMT), are scheduled to be deployed in the coming months. These terminals should deprive […]

July 22, 2024

Global IT outage affects Vodafone, Sense Bank, and Nova Poshta electronic system

Updated: As of 10:24 a.m., Nova Poshta resumed its work, the press service said. Oleg Gorokhovsky also wrote about the global IT failure in his Telegram channel. On the morning of July 19, Ukrainians began to complain about the work of the mobile operator Vodafone, and there were also malfunctions in the electronic systems of […]

July 19, 2024

ChatGPT developers started working on a new AI project called Strawberry

OpenAI, a startup that has gained worldwide fame for its ChatGPT AI product, has started working on the Strawberry project. According to Reuters sources, this initiative is aimed at forming a neural network capable of conducting research on the Internet on its own (and not just answering questions). If the agency’s information is true, ChatGPT’s […]

July 15, 2024