LLMs are applied across domains and adopted by various organizations. With their rising popularity comes their increasing downstream impact on individuals and society. One significant aspect of their influence is the biases introduced throughout the AI pipeline, originating from both data and algorithms.
Therefore, understanding these biases is crucial before we engage in meaningful discussions about their implications and take steps to mitigate and account for them. This includes understanding the concept of bias, acknowledging the sources, processes, and histories that produce, propagate and perpetuate these biases, and recognizing the potential risks and harms they may pose to our society.
Bias is an overloaded term. For instance, the "social bias" refers to prejudices, inclinations, or discriminatory attitudes against an individual or a group. It typically includes biases based on inherent or acquired characteristics of an individual or a group, such as gender, age, sexuality, physical appearance, disability, ethnicity, race, nationality, socioeconomic status, profession, religion, and culture (the tendency to interpret a word or phrase according to the meaning derived from a given culture assigned to it; e.g. eating meat).
Bias may also refer to "statistical bias", which designates the difference between a model's prediction and the ground truth. In machine learning, minimizing statistical bias is a component of reducing error. In language models, "bias" generally concerns distributional skews that lead to unfavorable impacts for particular individuals or social groups.
In practice, the boundaries between the social and the statistical sense of the term are not always clear. Social and historical biases rooted in world knowledge enter the ML/AI systems; they are encoded in text corpora and perpetuated in a cascading fashion. For instance, word embeddings trained on large corpora of text containing human biases can lead to a large language model to produce undesirable outcomes, even if data is perfectly sampled or measured.
Bias per se is not necessarily bad. Biases can also be morally neutral such as those related to insects or flowers, or mirroring the existing distribution of gender in relation to professions or first names. Biases become problematic when they, as input to predictive systems, exacerbate existing inequalities between users. Besides, it is argued that many of the biases can act as a diagnostic tool about the state of society.
LLMs heavily depend on data that is either generated by humans or collected through systems designed by humans. Data is central to the performance, fairness, robustness, and safety of the AI systems. The quality of data carries significant scientific implications for the generalizability of the results. Moreover, it also has ethical implications that could potentially lead to real-world harm.
This section focuses on sources of bias introduced throughout the data-for-AI pipeline. Each stage of data and model development involves choices, practices and contexts that can lead to undesirable downstream consequences.
Besides, literature from the fields of Human-Computer Interaction (HCI) and Science and Technology Studies (STS) provide complementary perspectives to this discussion along the data-for-AI pipeline. For instance, HCI scholars examine the practices in working with data that shape the design and use of data-driven systems. STS researchers highlight sociopolitical contexts in which data, people, and things are situated. Readers may explore literature from these areas on their own.
Data Collection
Data labeling
Data processing
Model development
Model evaluation
Model deployment
* This part is under construction and coming soon.
Reference