Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP

Dimensions overall score 3.0

GitHub Code Pulse

No public code linked for this paper yet.

• The paper proposes seven evaluation desiderata for subjectivity-sensitive models in NLP.method 95%
• A subjectivity-sensitive model should recognize when an input needs to be treated subjectively versus objectively.technical 90%
• A subjectivity-sensitive model should represent a range of diverse viewpoints, minimizing bias toward dominant perspectives.technical 90%
• Current evaluation practices in NLP show that various aspects of subjectivity are understudied, including the distinction between ambiguous and polyphonic input.limitation 85%
• For subjective tasks, model calibration should shift to whether predicted probabilities reflect the human judgment distribution.technical 85%
• Subjectivity should be expressed in a way that does not hamper user experience.technical 90%
• A model should predict subjectivity accurately after recognizing its source.technical 90%
• There is a lack of interplay between different evaluation desiderata for subjectivity in current NLP research.limitation 80%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card