Let's agree that disagreeing makes better models: creating perspectivist and soft label models for cross-domain hate speech detection

Everyday, hundreds of millions of messages are being posted on social media platforms. A relevant amount of these posts are abusive and contribute to make Internet an unsafe and negative place for a lot of users. This phenomenon is known as hate speech and even though it is relatively recent there is vast literature about the topic. The amount of messages that have to be checked is too high to have human moderators do it, therefore there is a need for automatic detection of hate speech. In research, there have been many proposals of systems that take on this challenge, with recent technology performing well under certain testing criteria. The problem is that most of these works focus only on particular facets of hate speech (e.g., racism or sexism) creating models that have good performance only in a single domain. Meanwhile, the goal of hate speech detection should be creating systems that work for all domains. All of these are motivations that contribute to the need of robust systems that recognize hate speech consistently. The best way to judge this quality is using cross-domain testing, which consists in testing models on datasets that focus on topics that are different compared to the datasets used for training. Another problem in traditional hate speech detection is the use of datasets that are gathered through majority voting, thus only one opinion is represented. To solve these problems, the approach we are going to take is to exploit disagreement between annotators in order to create better models. We do this because hate speech is a subjective phenomenon, thus making it necessary to consider different perspectives of the annotators. This has been a recent trend in Natural Language Processing as scholars are starting to become critical of classical approaches. Learning with disagreement approaches have the objective of training models in line with the different opinions of humans about a specific phenomenon. This means that corpora that are aggregated through majority voting techniques are too limited, and we need disaggregated datasets and, when possible, the information about who are the annotators(i.e., their cultural identity). In our work, we are going to introduce DisaggregHate It, a perspectivist dataset that satisfies these needs by keeping all annotations separated to allow different methods of aggregations or learning. We are going to utilize this corpus to test two approaches for taking into account the subjectivity of annotators: the opinions based on their social-demographic information and the distribution of their opinions for each instance. For the former, we overcome the idea of ground truth by creating models that take into account the different demographic categories of the annotators. Meanwhile, for the latter, we used soft labels by calculating functions that return a real number in the [0,1] range and not binary labels resulting from aggregation by majority voting. In our work, we created models following these principles and found they perform better under certain criteria.