Skip to content

Latest commit

 

History

History
105 lines (91 loc) · 5.27 KB

README.md

File metadata and controls

105 lines (91 loc) · 5.27 KB
Raw

empathic-stories

Latest model checkpoint available here:

Model checkpoint available for download through Google Drive

Sample code setting up server to remote access model

Sample how to use:

from trainer import EmpathicSimilarityModel
from numpy import dot
from numpy.linalg import norm
import numpy as np

model_BART =EmpathicSimilarityModel.load_from_checkpoint("./lightning_logs/version_164/checkpoints/epoch=15-step=752.ckpt", model="BART", pooling="MEAN", bin=False, losses="MSE", use_pretrained=False)

# get embeddings
story1 = "this is my story"
e1 = model_BART(story1).detach().numpy().reshape(-1)

story2 = "this is my story 2"
e2 = model_BART(story2).detach().numpy().reshape(-1)

def get_cosine_similarity(a, b):
    cos_sim = dot(a, b)/(norm(a)*norm(b))
    return cos_sim

print(get_cosine_similarity(e1, e2))

File Structure

  • /annotation contains all MTurk annotation templates
  • /data contains all data folders for train, dev, test sets
  • /models contains all lightning modules and our pretrained BART model
    • EmpathicSimilarityModel takes in a story pair (2 stories) and fine tunes on empathic similarity score
    • EmpathicSummaryModel takes in a single story and fine tunes on empathy reasons (main event + emotion description + moral)
  • /config contains yaml config files for different model training settings
  • /user_study contains the frontend and server side code for our user study interface
  • dataset.py contains the dataloaders
  • special_tokens.py definitions of special tokens
  • trainer.py contains training code and input of config files for different model training settings
  • utils.py contains extra model utilities
  • evaluator.py contains an evaluation class to compute all evaluation metrics

Dataset Overview

Stories

  • Data Source: which data source the story came from
  • story: raw text of the story
  • story_formatted: the story formatted with breaks
  • story_summary: ChatGPT summarized story
  • comments: (if pulled from social media), top level comments to the story
  • url: (if pulled from social media), the original url of the story
  • post_id: (if pulled from social media), the original id of the story
  • post_time: (if pulled from social media), the time the story was posted
  • post_score: (if pulled from Reddit), the score of the post
  • toxicity_score: toxicity score rated by Detoxify
  • WorkerId: worker ID of annotator
  • LifetimeApprovalRate: annotator's lifetime approval rate
  • AcceptTime: when the annotator accepted the HIT
  • SubmitTime: when the annotator submitted the HIT
  • WorkTimeInSeconds: how long the annotator took for the HIT
  • Age: annotator age
  • Gender: annotator gender
  • Race: annotator race
  • Arousal: annotator's arousal before the task (1-10)
  • Valence: annotator's valence before the task (1-10)
  • Main Event: main event of the story as rated by human annotator
  • Emotion Description: emotion of the story as rated by human annotator
  • Moral: moral of the story as rated by human annotator
  • Empathy Reasons: reasons why people may empathize with the story as rated by human annotator
  • Main Event (gpt3.5): main event of the story as rated by ChatGPT
  • Emotion Description (gpt3.5): emotion of the story as rated by ChatGPT
  • Moral (gpt3.5): moral of the story as rated by ChatGPT
  • Empathy Reasons (gpt3.5): reasons why people may empathize with the story as rated by ChatGPT
  • Empathizable: how generally "empathizable" the story is
  • Well-Written: how well-written the story is
  • fake_score: how likely the post is written by AI tools, as predicted by the Writer AI Content Detector
  • num_sentences: number of sentences in the story
  • num_words: number of words in the story
  • num_sentences_event: number of sentences in the event
  • num_words_event: number of words in the event
  • num_sentences_emotion: number of sentences in the emotion
  • num_words_emotion: number of words in the emotion
  • num_sentences_moral: number of sentences in the moral
  • num_words_moral: number of words in the moral
  • num_sentences_empathy_reasons: number of sentences in the empathy reasons
  • num_words_empathy_reasons: number of words in the empathy reasons

Story Pairs

  • pairs: pair ID (matches with story file index)
  • binned: which sampled bin the pair belongs to (based on SBERT sampling)
  • story_A: first story in story pair
  • story_B: second story in story pair
  • story_A_summary: summary of first story in story pair
  • story_B_summary: summary of second story in story pair
  • Empathic Similarity (gpt3.5): empathic similarity score as rated by ChatGPT
  • Empathic Similarity Binned (gpt3.5): binned empathic similarity score as rated by ChatGPT
  • Empathic Similarity Reasons (gpt3.5): reasons why two stories are empathically similar as rated by ChatGPT
  • similarity_empathy_human_AGG: empathic similarity score as rated by human annotators
  • similarity_event_human_AGG: event similarity score as rated by human annotators
  • similarity_emotion_human_AGG: emotion similarity score as rated by human annotators
  • similarity_moral_human_AGG: moral similarity score as rated by human annotators