- Tool uses machine learning to differentiate between ‘joke’ and verbal abuse
- Verbal abuse can be in the form of humiliation, sexual harassment or ‘hatred of identity’
- A pop-up box asks recipients if ‘all is well’ if it detects ‘vulgar language’
Dating app Badoo has launched a ‘vulgar message detector’ that will automatically flag any abusive, discriminatory or overly sexual messages.
The tool uses machine learning, a form of artificial intelligence (AI), to distinguish between ‘joke’ and genuine verbal abuse, such as ‘identity hate’, towards transgender people.
It is able to identify abusive or harmful messages sent between chat partners in real time, and then gives users the option to block and report them immediately.
With 521 million users in 190 countries, Badoo says the tool is one of the latest steps in a “wider commitment to security.”
It has been launched for all the users around the world whether they are chatting with a male or female or not.
Rude Message Detector identifies abusive or harmful messages sent in chat in real time and encourages users to block and report them directly to ‘create a safe and respectful place for daters’. A pop-up box (pictured) will appear asking the user if ‘all is well’ when machine learning detects some ‘vulgar language’.
How does this work?
Rude message detector comes into action when users are chatting in the app.
Machine Learning flags messages in three different categories:
– sexual: any message with explicit sexual connotations or overtly sexual remarks on one’s appearance
– Insult: any explicit insult directed at the recipient of the message
– identity hateAny discrimination or assault against race, sexual orientation, gender identity and expression, ethnicity, nationality, disability, religion and sexual orientation
Users get the option to block and report the offender.
According to Badu, machine learning identifies ‘subtle differences’ in potentially offensive words to determine whether messages are hurtful or prank.
For example, ‘That’s f**king amazing’ will not be flagged, but ‘Go f**k yourself’ will be flagged.
“The safety of our users is of utmost importance to us and we are constantly developing new technology to protect daters at Badoo UK,” said Natasha Briefel, Brand Marketing Director at Badoo UK.
‘We know that using witty and playful language is part of the fun of dating for many people, but we want to make sure there is a clear distinction between what is acceptable behavior and what is not.
‘We hope the launch of the Rude Message Detector feature will encourage minority people to think before typing rude messages, as well as help recipients of these messages feel empowered to block and report this behavior. Will get it.
Like Tinder, Badoo uses right or left swipe functionality to indicate romantic interest in another user.
When starting the app, Badoo shows you potential matches in your area and gives both men and women the chance to start a chat conversation – unlike rival ‘feminist’ dating app Bumble.
With Rude Message Detector, Badoo shows recipients a timely pop-up message asking if all is well and encourages them to report the sender if the message made them uncomfortable.
The pop-up says: ‘It looks like this person used some hate speech, so we’re checking if you still want to hear from them.’
It then gives users two options to click on – ‘Yes, keep chatting’ and ‘No, block and report’.
‘is everything OK?’ Rude message detector comes into action when users are chatting in the app
What is Machine Learning?
Machine learning (ML) is a branch of AI that is based on the idea that systems can learn from data, identify patterns, and make decisions.
ML systems can learn to improve their ability to perform a task without being explicitly programmed to do so.
Such systems can find patterns or trends in a set of data to come to conclusions or help humans make better decisions.
Machine learning systems become more effective over time as they learn.
The feature flags messages into three different categories – ‘sexual’, ‘insults’ and ‘hate identities’, which include discrimination against gender identities.
In severe cases, Badoo also proactively investigates the messages sent and takes direct action against the user.
A Badoo spokesperson told MailOnline that a ‘serious matter’ would be a violation. Community Guidelines, as outlined on its website, such as sending ‘explicitly sexual messages’ at the beginning of a conversation to multiple users, or body-shaming other users.
In its Community Guidelines, Badoo says its user base is a “very diverse community” and therefore takes “a strong stance” against hate speech and discrimination.
We consider content to be hate speech if it promotes racism, bigotry, violence, hatred, dehumanization or harm against individuals or groups based on race, ethnicity, disability, age, nationality, sexual orientation, gender, gender identity, religion gives or condemns it. appearance (including body shaming),’ it says.
Badoo uses automated systems and a team of human moderators to monitor and review accounts and messages for content that violates its guidelines.
Users who do not respect the guidelines may have access to the app limited or may be permanently removed from Badoo.
A Badoo spokesperson told MailOnline that human moderators are looking at machine learning as well as user messages, to continually improve the model and ensure that the firm is not unfairly banning any users.
In 2020, Badoo launched a security feature that uses AI to detect and blur unwanted nudes sent through the app. This warning message appears when Badoo’s AI thinks it has detected an unwanted nude from a chat partner
In its quest to make its apps secure first, Badoo last year launched a feature that…