#11 How Community Notes Work and Is it Working?
Taking a break from the LLM posts, this week I want to discuss Community Notes as pioneered by X/Twitter a few years ago. Earlier in the year, Meta announced that they will end their fact checking program in the US and move to a crowd sourced Community Notes approach, first pioneered by X/Twitter. As Meta starts testing out this program on Facebook, Instagram and Threads, let’s look at how this feature has been working on X/Twitter.
What is Community Notes on X?
Community Notes is a crowd-sourced model where the community provides additional helpful context to posts, particularly that could be misleading to better inform people. For example:
How does it work?
Contributors sign up to write and rate notes. Any regular user like you or me can sign up - there are a few requirements but that are quite basic. Sign up here.
Contributors can write a new note on a post that they may find misleading or could benefit from additional context.
Once a contributor writes a note, it is first shared with the rest of the contributors community to see if the note is helpful or not as it would not be great to show a bad quality note next to a post.
Once there are enough votes on a note and it is rated helpful by people from diverse perspectives, the note becomes eligible to be seen publicly by all users on X. To find notes that are helpful to the broadest possible set of people, Community Notes takes into account not only how many contributors rated a note as helpful or unhelpful, but also whether people who rated it seem to come from different perspectives. Different perspectives are not based on demographic information or their political affiliation, rather the algorithm works like a recommendation engine where it tends to group people who have historically agreed in the past in one group and another group for people who have not agreed with this group. For a note to get elevated to public, both sides of the camp must find it helpful. The algorithm dynamically updates as new notes are created every day. This is the most fascinating part of the overall solution. Read more here.
What does success look like for Community Notes?
Success for Community Notes could be seen as:
If every post that deserved a community note got one. This is similar to how recall is defined in technical terms.
When a note shows up, it is indeed helpful to the broader community. This is similar to how precision is defined in technical terms.
A note should show up as fast as possible as it’s not too beneficial if a note is published after millions of views have already happened on the original post. This is how latency is defined in technology.
So, a good Community Notes system should have high recall, high precision and low latency, the trinity of an impossible triangle wanting good, cheap and fast.
X Community Note solution is open-sourced and provides all data to download. I downloaded the data and here is what I found.
Recall: In 2024, there were 1.18M notes written but only 8.3% of them were rated as helpful and eventually were shown to the broader user community. If we assume that almost all the notes were potentially written for posts that would benefit from additional perspective, then it means that the system has a quite low recall of <10%. Here is the code that I wrote to get these statistics.
Precision: I did not spend enough time to understand the precision of the system but given the algorithm is prioritizing alignment from a diverse community, this means the precision must be very high. I will look into this in later posts.
Latency: For the notes written in 2024, I found that the average latency for a note to become visible to the community was 38.5 hours while the median latency was 7 hrs. This tells us that it takes a long time for a note to receive enough helpful notes before it could become public. If I were to find how long does it take for a post to receive a note, it would be even longer.
Overall, I think that Community Notes is a very strong initiative but it suffers from severe limitations like an overwhelming majority of the posts that should be receiving a note do not get one, and even when they do, the notes take on average a very long time to show up.
Questions I would like to research going ahead:
For Notes that are helpful and visible to all users - Do they really need Community participation or are these easy enough cases where the platforms could be leveraging Large Language Models to tag them as misleading on their own?
For truly social controversial topics like politics, elections, religion and climate change, are there ways for community to converge on an aligned narrative? This is the true potential for Community Notes and should be explored further.
Please leave me a note with your thoughts on this article. I will continue to dive deep into this area over time.