Rapid development of AI is breaking the ability to assess them, safety tests spouting false results

The rapid advancement of AI is creating some massive challenges in the traditional methods that are used to assess them. This presents a major dilemma for businesses and public bodies on how to effectively keep AI under check and navigate the evolving AI landscape.

Some of the most commonly used evaluation criteria for assessing AI performance like accuracy, and safety are falling short of the standard that they are usually subjected to as more sophisticated models enter the market as per a report by The Financial Times.

According to experts in AI development, testing, and investment, these traditional tools are easily manipulated and too narrow in scope to accommodate the complexity of the latest AI systems.

More from Tech

How ChatGPT is becoming everyone’s BFF and why that’s dangerous America ready for self-driving cars, but it has a legal problem

The intense competition in the AI space, fueled by investments from venture capitalists and tech giants like Microsoft, Google, and Amazon, has rendered many older benchmarks obsolete.

With new AI models and updates being launched every month, the evaluation standards that we have now are quickly becoming obsolete.

As generative AI becomes a top investment priority for a majority of tech businesses, it is becoming increasingly important to ensure that the AI products we have are actually trustworthy.

Governments are also grappling with how to deploy and manage the risks associated with the latest AI models, prompting initiatives like bilateral arrangements on AI safety between countries.

Impact Shorts

More Shorts

America ready for self-driving cars, but it has a legal problem

Alibaba, Baidu begin using own AI chips as China shifts away from US tech amid Nvidia row

Concerns are also arising regarding the integrity of public tests, as the AI models’ training data may inadvertently include the exact questions used in evaluations, which in turn poses a challenge to the reliability of benchmarks.

In response to this, we now have startups that are solely focused on coming up with innovative approaches to evaluate emerging AI models. Some platforms offer bespoke tests set by individual users, providing a more direct reflection of user preferences. However, while such approaches may benefit individual users, they may have limited utility for companies with specific AI model requirements.

Ultimately, businesses are advised to conduct internal testing and human evaluation to supplement traditional benchmarks, recognizing that the selection of AI models is as much an art as it is a science. As AI continues to evolve, adapting evaluation methods to ensure accuracy and reliability remains paramount for effectively harnessing the potential of this transformative technology.

(With inputs from agencies)