Last month we published the methodology behind the GDI ratings of the disinformation risks of news sites. We see the GDI’s methodology and approach as setting a “gold standard” for how to best measure a news site’s risks of carrying disinformation. Such risk ratings must be trusted, neutral and independent – and not done for profit. This is the bar that must be met. It is what we need to disrupt the funding of disinformation sites through online ads.
In building the ratings framework, we have been focused on creating a robust and objective risk assessment of disinformation. We have worked at each step to mitigate bias and ensure a methodologically-sound assessment.
It has been a challenging task that we have soundly met. Disinformation is a very complex problem with numerous parameters contributing to it. The GDI risk ratings have created a multi-dimensional framework to simplify and capture the different facets of this phenomenon. Moreover, the risk ratings provide a way to understand current and latent challenges to preventing the spread of disinformation – including inadvertently – by sites.
At GDI, our ratings must be reliable, universally applicable and trusted by our partners in order for them to understand the disinformation risks of different news sites in different markets. Hence, we have made bias mitigation and information quality integral to the way we work.
We take this challenge seriously. It is not a process step. We have instituted systems to reduce bias and enhance the validity of our measurement across two critical areas: our operations and our research.
1. Organisation and governance
We are a UK based not-for-profit organisation that derives no commercial benefit from the ratings we produce. The index reflects our three core organisational principles of neutrality, independence, and transparency.
We are building a global independent board of directors who have no ties to any media or social media company. Our forthcoming board will be in place by the second quarter of this year.
We pride ourselves and are working towards attaining our stated funding principle of no one funder or investor contributing to more than a third of our total funding share.
We also receive no money from any media or social media company that could have any potential influencing effects on our risk rating.
2. Methodology and approach
The GDI risk rating is a combination of both an automated and a deeper, human assessment of a news domain. Our approach in developing the underlying methodology is to ensure neutrality and prevent any bias throughout.
The human elements of our index rely on a well-developed and externally-vetted methodology. We have built in multiple bias reduction mechanisms to enhance the accuracy, reliability and usefulness of the ratings produced through a deep review of a site’s disinformation risks. This review is done for sites that are harder to assess using AI alone. The deep assessment incorporates both a country reviewer and an independent expert survey.
The GDI has striven for neutrality by including some of the following measures:
- All articles assessed have been anonymised to remove reviewer’s personal biases from influencing their scoring.
- Scores have been reviewed by GDI staff to ensure scores are backed up by relevant information and domains and articles have been consistently scored. Additionally, a subset of articles have also been reviewed anonymously by a third reviewer to assess inter-analyst reliability of scoring.
- The independent expert survey results have been statistically tested to ensure experts’ personal political views have not influenced their responses.
- The indicators used across our pillars have been designed to establish cross validation of results.
These checks and balances are instituted across our disinformation risk rating process.
Of course there is always room for refinement. We are still in our early days and think we can make our model more reliable. With every assessment, we have the opportunity to stress test our framework and its underlying assumptions.
Our goal is to ensure that GDI’s assessment – both automated and human – captures the risk of disinformation in the real world.
We thank the advertisers, ad tech, platforms and news sites that have helped us in this process. And we look forward to working with others as we continue this journey to set the ‘gold standard’ in rating disinformation risk.