Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ModelingLLM: Add Structured Grading Instruction Generation and Restructure Module #340

Merged

Conversation

LeonWehrhahn
Copy link
Contributor

@LeonWehrhahn LeonWehrhahn commented Sep 9, 2024

Motivation and Context

Structured Grading Instructions (SGIs) lead to more accurate feedback generations. However, instructors do not always provide SGIs for every exercise. To address this gap and maintain consistency in the grading process, this PR implements automatic SGI generation when manual ones are not present.

Additionally, as the module has grown with more functions for processing and manipulating data, this PR includes a restructuring to enhance organization and maintainability.

Description

A follow-up PR will address caching the structured grading instructions

This PR introduces the following key changes:

  1. Automatic SGI Generation: Implemented a new function to generate Structured Grading Instructions when they are not manually provided for an exercise.
  2. Module Restructuring: Reorganized the module structure to better accommodate the growing number of processing and manipulation functions.
  3. Core Functionality Updates:
    • Restructured generate_suggestions.py to focus solely on the feedback generation process and moved other functionalities into separate functions and files for improved modularity.
    • Add get_structured_grading_instructions.py to include the new SGI auto-generation feature
  4. Model and Utility Improvements:
    • Added new models for Structured Grading Instructions.
    • Added new utility functions to support the core functionality changes
  5. Prompt Updates: Refined existing prompts and added new ones to support the automatic SGI generation

Steps for Testing

  1. Make direct API calls to the /feedback_suggestions endpoint of the module.
  2. Test with various diagram types (BPMN, UML Class, UML Activity, etc.) to ensure the functionality works for all Apollon-supported diagrams.
  3. Test both scenarios where Structured Grading Instructions (SGIs) are manually provided and where they are not.
  4. Verify that appropriate feedback is generated in both cases.

Testserver States

Note

These badges show the state of the test servers.
Green = Currently available, Red = Currently locked
Click on the badges to get to the test servers.


Screenshots

@LeonWehrhahn LeonWehrhahn added deploy:athena-test1 Athena Test Server 1 and removed deploy:athena-test1 Athena Test Server 1 labels Sep 11, 2024
Copy link

⚠️ Unable to deploy to test servers ⚠️

The docker build needs to run through before deploying.

@github-actions github-actions bot added the deployment-error Added by deployment workflows if an error occured label Sep 11, 2024
Copy link
Collaborator

@FelixTJDietrich FelixTJDietrich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve the conflicts

@LeonWehrhahn LeonWehrhahn added deploy:athena-test2 Athena Test Server 2 and removed deployment-error Added by deployment workflows if an error occured labels Sep 13, 2024
Copy link

⚠️ Unable to deploy to test servers ⚠️

The docker build needs to run through before deploying.

@github-actions github-actions bot added deployment-error Added by deployment workflows if an error occured and removed deploy:athena-test2 Athena Test Server 2 labels Sep 13, 2024
@LeonWehrhahn LeonWehrhahn added deploy:athena-test2 Athena Test Server 2 and removed deployment-error Added by deployment workflows if an error occured labels Sep 13, 2024
@LeonWehrhahn LeonWehrhahn temporarily deployed to athena-test2.ase.cit.tum.de September 13, 2024 03:56 — with GitHub Actions Inactive
@github-actions github-actions bot added the lock:athena-test1 Is currently deployed to Athena Test Server 1 label Sep 16, 2024
@LeonWehrhahn LeonWehrhahn removed the lock:athena-test1 Is currently deployed to Athena Test Server 1 label Sep 17, 2024
@dmytropolityka dmytropolityka added the deploy:athena-test1 Athena Test Server 1 label Sep 17, 2024
@dmytropolityka dmytropolityka temporarily deployed to athena-test1.ase.cit.tum.de September 17, 2024 15:56 — with GitHub Actions Inactive
@github-actions github-actions bot added lock:athena-test1 Is currently deployed to Athena Test Server 1 and removed deploy:athena-test1 Athena Test Server 1 labels Sep 17, 2024
@LeonWehrhahn LeonWehrhahn added deploy:athena-test1 Athena Test Server 1 and removed lock:athena-test1 Is currently deployed to Athena Test Server 1 labels Sep 20, 2024
@LeonWehrhahn LeonWehrhahn temporarily deployed to athena-test1.ase.cit.tum.de September 20, 2024 19:53 — with GitHub Actions Inactive
@github-actions github-actions bot added lock:athena-test1 Is currently deployed to Athena Test Server 1 and removed deploy:athena-test1 Athena Test Server 1 labels Sep 20, 2024
@LeonWehrhahn LeonWehrhahn added deploy:athena-test1 Athena Test Server 1 and removed lock:athena-test1 Is currently deployed to Athena Test Server 1 labels Sep 21, 2024
@LeonWehrhahn LeonWehrhahn temporarily deployed to athena-test1.ase.cit.tum.de September 21, 2024 06:53 — with GitHub Actions Inactive
@github-actions github-actions bot added lock:athena-test1 Is currently deployed to Athena Test Server 1 and removed deploy:athena-test1 Athena Test Server 1 labels Sep 21, 2024
@maximiliansoelch maximiliansoelch changed the title Add Structured Grading Instruction Generation and Restructure Modeling Module ModelingLLM: Add Structured Grading Instruction Generation and Restructure Module Sep 23, 2024
@maximiliansoelch maximiliansoelch merged commit d9ff3bd into develop Sep 23, 2024
13 checks passed
@maximiliansoelch maximiliansoelch deleted the features/modeling/structured-grading-instructions branch September 23, 2024 21:35
@github-actions github-actions bot removed the lock:athena-test1 Is currently deployed to Athena Test Server 1 label Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants