Add a section to show model average score for all eval data #4050

shifucun · 2024-03-19T21:14:57Z

Score for each model is passed from child component to the App. Use state in the parent component is very slow due to the large number of children components. Here uses a ref object to collect and a timer to check if all scores have been collected.

Also

Updated some golden stat vars which are not good match
Removed SDG queries since the current golden is for base medium index.

…to gcs

chejennifer

This is cool! Can you also include a screenshot of what this looks like?

chejennifer · 2024-03-20T00:34:45Z

static/js/apps/nl_eval/model_score_box.tsx

+  isExpanded: boolean;
+  goldenStatVars: string[];
+  overrideStatVars: EmbeddingObject[];
+  setScore: (modelName: string, sentence: string, score: number) => void;


nit: add a comment about what score we're setting here? maybe also rename to onScoreUpdated (here and everywhere else)

chejennifer · 2024-03-20T00:37:39Z

static/js/apps/nl_eval/app.tsx


 import { stringifyFn } from "../../utils/axios";
 import { QuerySection } from "./query_section";
 import { EmbeddingObject } from "./util";

+function OverallScoreTable(props: {


have a separate file for this component?

chejennifer · 2024-03-20T00:46:19Z

static/js/apps/nl_eval/app.tsx


 import { stringifyFn } from "../../utils/axios";
 import { QuerySection } from "./query_section";
 import { EmbeddingObject } from "./util";

+function OverallScoreTable(props: {
+  data: Record<string, Record<string, number>>;
+  count: number;


add a comment about what this is a count of? or have a better name

chejennifer · 2024-03-20T00:50:22Z

static/js/apps/nl_eval/app.tsx


 import { stringifyFn } from "../../utils/axios";
 import { QuerySection } from "./query_section";
 import { EmbeddingObject } from "./util";

+function OverallScoreTable(props: {
+  data: Record<string, Record<string, number>>;


comment what the keys and values are?

chejennifer · 2024-03-20T00:50:34Z

static/js/apps/nl_eval/app.tsx

+  useEffect(() => {
+    // Check if data is complete. If so, set mark to an empty string so that no
+    // new render is triggered.
+    if (Object.values(Object.values(props.data)[0]).length === props.count) {


is Object.values(props.data) always going to be non empty? If it can be empty, should check before accessing the first element

Not needed anymore as the logic is moved to App where each model should have the same length of values as the eval dataset.

chejennifer · 2024-03-20T01:10:30Z

static/js/apps/nl_eval/app.tsx

+      return;
+    }
+    // Update mark to be a new timestamp frequently if data is not complete
+    const timerId = setInterval(() => {


instead of the mark and timer, could this work:

add a new completedOverallScore state in App component

in handleUpdateOverallScore, check if scores are complete. If complete, update completedOverallScore

pass completedOverallScore to OverallScoreTable

This is a really good suggestion. Works perfect.

shifucun

Thanks for the review, it's very helpful!

shifucun · 2024-03-20T02:18:54Z

static/js/apps/nl_eval/app.tsx


 import { stringifyFn } from "../../utils/axios";
 import { QuerySection } from "./query_section";
 import { EmbeddingObject } from "./util";

+function OverallScoreTable(props: {


shifucun · 2024-03-20T02:19:29Z

static/js/apps/nl_eval/app.tsx


 import { stringifyFn } from "../../utils/axios";
 import { QuerySection } from "./query_section";
 import { EmbeddingObject } from "./util";

+function OverallScoreTable(props: {
+  data: Record<string, Record<string, number>>;


shifucun · 2024-03-20T02:21:51Z

static/js/apps/nl_eval/app.tsx


 import { stringifyFn } from "../../utils/axios";
 import { QuerySection } from "./query_section";
 import { EmbeddingObject } from "./util";

+function OverallScoreTable(props: {
+  data: Record<string, Record<string, number>>;
+  count: number;


shifucun · 2024-03-20T02:37:49Z

static/js/apps/nl_eval/model_score_box.tsx

+  isExpanded: boolean;
+  goldenStatVars: string[];
+  overrideStatVars: EmbeddingObject[];
+  setScore: (modelName: string, sentence: string, score: number) => void;


shifucun · 2024-03-20T02:40:31Z

static/js/apps/nl_eval/app.tsx

+      return;
+    }
+    // Update mark to be a new timestamp frequently if data is not complete
+    const timerId = setInterval(() => {


This is a really good suggestion. Works perfect.

shifucun · 2024-03-20T02:41:15Z

static/js/apps/nl_eval/app.tsx

+  useEffect(() => {
+    // Check if data is complete. If so, set mark to an empty string so that no
+    // new render is triggered.
+    if (Object.values(Object.values(props.data)[0]).length === props.count) {


Not needed anymore as the logic is moved to App where each model should have the same length of values as the eval dataset.

chejennifer

thanks for the changes!

Add a section to show model average score for all eval data

17dcbcd

shifucun requested a review from chejennifer March 19, 2024 21:14

shifucun added 2 commits March 19, 2024 17:03

update goldens

6d9a636

Merge branch 'master' of https://github.com/datacommonsorg/website in…

3705059

…to gcs

chejennifer reviewed Mar 20, 2024

View reviewed changes

resolve comments

c313cb4

shifucun commented Mar 20, 2024

View reviewed changes

shifucun requested a review from chejennifer March 20, 2024 02:44

chejennifer approved these changes Mar 20, 2024

View reviewed changes

Merge branch 'master' into gcs

c9d5ea0

shifucun enabled auto-merge (squash) March 20, 2024 16:32

shifucun merged commit 9063c6c into datacommonsorg:master Mar 20, 2024
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a section to show model average score for all eval data #4050

Add a section to show model average score for all eval data #4050

shifucun commented Mar 19, 2024 •

edited

Loading

chejennifer left a comment

chejennifer Mar 20, 2024

shifucun Mar 20, 2024

chejennifer Mar 20, 2024

shifucun Mar 20, 2024

chejennifer Mar 20, 2024

shifucun Mar 20, 2024

chejennifer Mar 20, 2024

shifucun Mar 20, 2024

chejennifer Mar 20, 2024

shifucun Mar 20, 2024

chejennifer Mar 20, 2024

shifucun Mar 20, 2024

shifucun left a comment

shifucun Mar 20, 2024

shifucun Mar 20, 2024

shifucun Mar 20, 2024

shifucun Mar 20, 2024

shifucun Mar 20, 2024

shifucun Mar 20, 2024

chejennifer left a comment

Add a section to show model average score for all eval data #4050

Add a section to show model average score for all eval data #4050

Conversation

shifucun commented Mar 19, 2024 • edited Loading

chejennifer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shifucun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chejennifer left a comment

Choose a reason for hiding this comment

shifucun commented Mar 19, 2024 •

edited

Loading