A new report from Stanford University – the second ranked university in the world according to Times higher education rankings – ranked ten major artificial intelligence (AI) firms in a new Index based on the transparency of their AI models. 

The goal of this project, according to the researchers, was to help guide policymakers in regulating the burgeoning sector. Rishi Bommasani, a researcher at Stanford’s Centre for Research on Foundation Models, said that companies should strive for a score between 80-100%.

The scoring is based on a system that evaluates 100 different aspects of transparency. About a third of the indicators are related to how the model is build such as training data and labour used. Another third is looking at how trustworthy the model itself is. The final third is looking at how the model is used downstream, such as policy disclosure and user data protection. 

That being said, the top ranked AI model – Llama 2 from Meta – secured the top spot with only 54%. This suggests that Meta is taking strides in providing information about its AI technology, but the industry as a whole is behind where it should be. 

The AI model BLOOMZ by BigScience took second place, due largely to its outstanding scores in the building of the model, the only of all top ten models with a score above 50% for labour and data.

The Microsoft backed firm OpenAI’s flagship model GPT-4 which is best known for creating the chatbot chatGPT ranked third in the transparency index with a score of 48%. Google’s PaLM 2 landed fifth place with a score of 40%, followed closely by Amazon-backed Anthropic at 36%. 

While all of the companies scored around 50%  for methods, capabilities and other basics, all of the listed models scored lower on downstream impact. Some companies such as OpenAI, Google, and Anthropic did score well in usage policy, but that was out shadowed by lower than acceptable labour and data scores.

Transparency in AI is not just a matter of corporate responsibility: it has wide-reaching implications for consumer protection, policymaking, and the advancement of AI initiatives. Without transparency, regulators and lawmakers cannot adequately address critical issues related to intellectual property, labour practices, energy consumption, and bias.

Recommended reading

• Cyber Risk Index Has Never Been Better but Companies are Still Worried
• Google to Introduce EU Ad Transparency Tools
• European Commission Figures Signal Fall in UK Digital Ranking

“If you don’t have transparency, regulators can’t even pose the right questions, let alone take action in these areas,” said Bommasani.

“For many policymakers in the EU as well as in the US, the UK, China, Canada, the G7, and a wide range of other governments, transparency is a major policy priority,” Bommasani added.

This index underscores the significance of AI transparency and regulation going forward. In that vein, the UK is set to host an international summit in November, while the G7 countries called for action on AI during a summit in Japan earlier this year. The EU on the other hand, has taken a leading role in regulating AI, as it is set to enact the world’s first law governing the technology by the end of the year.