Athena: Add LLM token usage tracking
#9548
Conversation
…na/track-token-usage # Conflicts: # src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java
github-actions
bot
added
tests
server
FelixTJDietrich
changed the base branch from
develop
to
feature/track-usage-of-iris-requests
WalkthroughThis pull request introduces several new data transfer objects (DTOs) and modifications to existing services and DTOs within the Artemis application. Key additions include Changes
Possibly related PRs
Suggested labels
Suggested reviewers
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's limits when posting large numbers of comments.
Actionable comments posted: 13
🧹 Outside diff range and nitpick comments (27)
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMServiceType.java (1)
1-8: LGTM with a minor suggestion for JavaDoc improvement.The
LLMServiceTypeenum is well-structured and follows the coding guidelines. It has a clear purpose, uses proper naming conventions, and adheres to the single responsibility principle.Consider expanding the JavaDoc to include a brief description of each enum constant:
/** * Enum representing different types of LLM (Large Language Model) services used in the system. */ public enum LLMServiceType { /** Represents the IRIS LLM service. */ IRIS, /** Represents the ATHENA LLM service. */ ATHENA }This addition would provide more context about each service type, enhancing code documentation.
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMRequest.java (1)
1-4: LGTM! Consider adding documentation.The
LLMRequestrecord is well-structured and adheres to the coding guidelines. It uses appropriate types for its fields and follows the single responsibility principle by encapsulating data related to an LLM request.Consider adding Javadoc comments to describe the purpose of the record and its fields. This would enhance code readability and maintainability. Here's an example:
/** * Represents a request to a Large Language Model (LLM). * This record encapsulates all necessary information for tracking LLM usage and costs. */ public record LLMRequest( /** The name or identifier of the LLM model. */ String model, /** The number of input tokens used in the request. */ int numInputTokens, /** The cost per million input tokens. */ float costPerMillionInputToken, /** The number of output tokens generated. */ int numOutputTokens, /** The cost per million output tokens. */ float costPerMillionOutputToken, /** The identifier of the pipeline used for this request. */ String pipelineId ) {}src/main/java/de/tum/cit/aet/artemis/athena/dto/ResponseMetaTotalUsageDTO.java (1)
5-7: Consider enhancing the class documentation.While the current documentation provides a basic description, it could be more informative. Consider adding details about the purpose of each field and their significance in the context of token usage tracking.
Here's a suggested improvement:
/** * DTO representing the total usage metrics for LLM token consumption. * * @param inputTokens The number of tokens in the input text. * @param outputTokens The number of tokens generated in the output. * @param totalTokens The total number of tokens processed (input + output). */src/main/java/de/tum/cit/aet/artemis/athena/dto/ResponseMetaLLMCallDTO.java (2)
5-7: Consider enhancing the class documentation.While the current JavaDoc comment provides a clear description of the DTO's purpose, it could be improved by adding more details about the fields it contains.
Consider expanding the documentation as follows:
/** * DTO representing an individual LLM call. * * @param modelName The name of the LLM model used. * @param inputTokens The number of input tokens processed. * @param outputTokens The number of output tokens generated. * @param totalTokens The total number of tokens (input + output) for this call. */
9-10: LGTM: Well-structured record declaration with a minor suggestion.The use of a record for this DTO is excellent. It provides immutability and automatic generation of necessary methods. The field names and types are appropriate, and the use of Integer allows for null values, which is consistent with the @JsonInclude annotation.
Consider adding a validation check for the totalTokens field to ensure it equals the sum of inputTokens and outputTokens. This could be done in a compact form within the record:
public record ResponseMetaLLMCallDTO(String modelName, Integer inputTokens, Integer outputTokens, Integer totalTokens) { public ResponseMetaLLMCallDTO { if (inputTokens != null && outputTokens != null && totalTokens != null) { assert totalTokens.equals(inputTokens + outputTokens) : "Total tokens should equal the sum of input and output tokens"; } } }This addition would ensure data integrity within the DTO.
src/main/java/de/tum/cit/aet/artemis/athena/dto/ResponseMetaDTO.java (2)
7-9: Enhance class documentation for clarity.While the current JavaDoc provides a basic description, it could be more informative. Consider adding details about the fields
totalUsageandllmCallsto provide a comprehensive overview of the DTO's structure and purpose.Suggested improvement:
/** * DTO representing the meta information in the Athena response. * <p> * This record encapsulates: * <ul> * <li>totalUsage: Aggregated usage metrics for the response.</li> * <li>llmCalls: List of individual LLM call details made during the response generation.</li> * </ul> */
11-12: LGTM: Well-structured record declaration with a minor suggestion.The
ResponseMetaDTOrecord is well-defined and adheres to the coding guidelines for DTOs. It uses appropriate field names and types, following the single responsibility principle.For improved type safety and to prevent potential null pointer exceptions, consider using an immutable list type for
llmCalls.Suggested improvement:
import java.util.List; import java.util.Collections; public record ResponseMetaDTO(ResponseMetaTotalUsageDTO totalUsage, List<ResponseMetaLLMCallDTO> llmCalls) { public ResponseMetaDTO { llmCalls = llmCalls == null ? Collections.emptyList() : Collections.unmodifiableList(llmCalls); } }This change ensures that
llmCallsis never null and cannot be modified after creation.src/main/java/de/tum/cit/aet/artemis/core/repository/LLMTokenUsageRequestRepository.java (1)
11-14: LGTM: Interface declaration and annotations are correct.The interface is well-defined and follows coding guidelines:
- Proper use of @Profile and @repository annotations
- Follows CamelCase naming convention
- Extends ArtemisJpaRepository for code reuse
Consider adding a brief Javadoc comment to describe the purpose of this repository interface, enhancing code documentation:
/** * Repository interface for managing LLMTokenUsageRequest entities. */ @Profile(PROFILE_CORE) @Repository public interface LLMTokenUsageRequestRepository extends ArtemisJpaRepository<LLMTokenUsageRequest, Long> { }src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/dto/competency/PyrisCompetencyStatusUpdateDTO.java (1)
20-20: LGTM: Record definition updated correctly with a minor suggestion.The
PyrisCompetencyStatusUpdateDTOrecord has been appropriately updated to include the newtokensparameter for LLM token usage tracking. This change aligns with the PR objectives and follows the DTO guidelines.Consider renaming
tokenstollmRequestsfor improved clarity, as it more accurately describes the content of the list:-public record PyrisCompetencyStatusUpdateDTO(List<PyrisStageDTO> stages, List<PyrisCompetencyRecommendationDTO> result, List<LLMRequest> tokens) { +public record PyrisCompetencyStatusUpdateDTO(List<PyrisStageDTO> stages, List<PyrisCompetencyRecommendationDTO> result, List<LLMRequest> llmRequests) {Don't forget to update the corresponding documentation if you make this change.
src/main/java/de/tum/cit/aet/artemis/iris/service/session/AbstractIrisChatSessionService.java (1)
19-20: Consider using Map interface and adding a comment.The new field
tracesis well-declared as protected and final. However, consider the following improvements:
Use the
Mapinterface instead of the concreteHashMapclass for better flexibility:protected final Map<String, LLMTokenUsageTrace> traces = new HashMap<>();Add a brief comment explaining the purpose of this field to improve code readability.
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsageTrace.java (3)
22-26: LGTM: Class declaration and annotations are appropriate.The class declaration and annotations follow the coding guidelines, including proper naming, entity mapping, caching, and JSON serialization configuration.
Consider adding a brief Javadoc comment to describe the purpose of this class, which would enhance code documentation and maintainability.
28-51: Field declarations are well-structured, but consider using a primitive for userId.The field declarations follow the coding guidelines with appropriate JPA annotations, nullable markings, and relationship configurations. However, there's room for improvement:
Consider changing the
userIdfield to a primitivelonginstead ofLongto align with the guideline of preferring primitives. This change would prevent potential null values and improve type safety. IfuserIdcan be null, make sure to document this requirement clearly.
1-108: Overall, well-structured class with minor improvements suggested.The
LLMTokenUsageTraceclass is well-designed and adheres to most of the coding guidelines. It demonstrates good use of JPA annotations, follows the single responsibility principle, and provides proper encapsulation.To further improve the class:
- Add a class-level Javadoc comment explaining the purpose and usage of this class.
- Resolve the
userIdtype inconsistency as mentioned in previous comments.- Consider adding validation logic in setters, especially for fields like
courseId,exerciseId, anduserIdto ensure data integrity.- You might want to override the
toString()method to provide a meaningful string representation of the object, which can be helpful for logging and debugging.Example of a class-level Javadoc and
toString()method:/** * Represents a trace of LLM token usage in the system. * This entity is used to track and store information about token consumption * across different services, courses, exercises, and users. */ @Entity @Table(name = "llm_token_usage_trace") // ... other annotations ... public class LLMTokenUsageTrace extends DomainObject { // ... existing code ... @Override public String toString() { return "LLMTokenUsageTrace{" + "id=" + getId() + ", serviceType=" + serviceType + ", courseId=" + courseId + ", exerciseId=" + exerciseId + ", userId=" + userId + ", time=" + time + ", irisMessageId=" + irisMessageId + '}'; } }These improvements will enhance the overall quality, maintainability, and usability of the class.
src/main/java/de/tum/cit/aet/artemis/iris/dto/IrisChatWebsocketDTO.java (2)
35-37: Update constructor Javadoc to include the new parameter.The constructor has been correctly updated to include the new
tokensparameter. However, the Javadoc comment for the constructor should be updated to include a description of this new parameter.Please update the constructor's Javadoc to include:
* @param tokens the list of LLM requests with token usage information
Line range hint
63-66: Updateequalsmethod to include the newtokensfield.The
equalsmethod should be updated to include the comparison of the newtokensfield. This ensures that equality checks consider all fields of the record.Please update the
equalsmethod as follows:@Override public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } IrisChatWebsocketDTO that = (IrisChatWebsocketDTO) o; return type == that.type && Objects.equals(message, that.message) && Objects.equals(rateLimitInfo, that.rateLimitInfo) && Objects.equals(stages, that.stages) && Objects.equals(suggestions, that.suggestions) && Objects.equals(tokens, that.tokens); }src/test/java/de/tum/cit/aet/artemis/iris/IrisChatWebsocketTest.java (2)
56-56: Update test to reflect newIrisChatWebsocketDTOstructureThe test has been updated to include a new parameter (
List.of()) in theIrisChatWebsocketDTOconstructor. This change aligns with the modifications made to the DTO structure.However, there are a few points to consider:
- The test method name "sendMessage" could be more descriptive (e.g., "testSendMessageUpdatesWebsocket").
- The test is using
verifyinstead ofassertThat. Consider usingassertThatfor more specific assertions.- The new parameter is an empty list. Consider adding a test case with non-empty token data to ensure proper handling.
To improve the test, consider the following changes:
- @Test - @WithMockUser(username = TEST_PREFIX + "student1", roles = "USER") - void sendMessage() { + @Test + @WithMockUser(username = TEST_PREFIX + "student1", roles = "USER") + void testSendMessageUpdatesWebsocketWithCorrectDTO() { // ... existing code ... - verify(websocketMessagingService, times(1)).sendMessageToUser(eq(TEST_PREFIX + "student1"), eq("/topic/iris/" + irisSession.getId()), - eq(new IrisChatWebsocketDTO(message, new IrisRateLimitService.IrisRateLimitInformation(0, -1, 0), List.of(), List.of(), List.of()))); + IrisChatWebsocketDTO expectedDTO = new IrisChatWebsocketDTO(message, new IrisRateLimitService.IrisRateLimitInformation(0, -1, 0), List.of(), List.of(), List.of()); + verify(websocketMessagingService, times(1)).sendMessageToUser(eq(TEST_PREFIX + "student1"), eq("/topic/iris/" + irisSession.getId()), eq(expectedDTO)); + + // Additional assertion for more specific checking + assertThat(expectedDTO.tokens()).isEmpty(); }Also, consider adding a new test case to verify behavior with non-empty token data.
Line range hint
1-77: Improve adherence to coding guidelinesThe test class partially adheres to the provided coding guidelines. Here are some suggestions for improvement:
- Use
assertThatfor assertions instead ofverify(assert_use: assertThat).- Implement DB query count tracking for performance monitoring (db_query_count_tests: track_performance).
- Consider using ArchUnit to enforce package rules (archunit_use: enforce_package_rules).
- Implement a utility service factory pattern for better test organization (util_service_factory_pattern: true).
- Add explicit measures to minimize context restarts (context_restart_minimize: true).
Positive aspects:
- JUnit 5 features are used (
@BeforeEach,@Test).- DB access is avoided using mock users.
- Fixed data is used (mock content).
- The test size is appropriate and specific.
- Static mocks are used.
Consider implementing these improvements to fully align with the coding guidelines and enhance the overall quality of the test suite.
🧰 Tools
🪛 ast-grep
[warning] 55-55: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (new IrisChatWebsocketDTO(message, new IrisRateLimitService.IrisRateLimitInformation(0, -1, 0), List.of(), List.of(), List.of()))
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 55-55: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (message, new IrisRateLimitService.IrisRateLimitInformation(0, -1, 0), List.of(), List.of(), List.of())
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 55-55: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (new IrisChatWebsocketDTO(message, new IrisRateLimitService.IrisRateLimitInformation(0, -1, 0), List.of(), List.of(), List.of()))
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 55-55: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (message, new IrisRateLimitService.IrisRateLimitInformation(0, -1, 0), List.of(), List.of(), List.of())
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfigurationsrc/test/java/de/tum/cit/aet/artemis/iris/IrisCompetencyGenerationIntegrationTest.java (1)
70-73: LGTM: Test method updated correctly with CompetencyExtractionJob.The changes align with the new implementation using CompetencyExtractionJob. The test maintains its descriptive naming and specific focus, adhering to the coding guidelines.
Consider adding a brief comment explaining the purpose of the CompetencyExtractionJob for improved clarity:
// Create a CompetencyExtractionJob to simulate the competency generation process CompetencyExtractionJob job = new CompetencyExtractionJob(jobId, course.getId(), userUtilService.getUserByLogin(userLogin));🧰 Tools
🪛 ast-grep
[warning] 72-72: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (job, new PyrisCompetencyStatusUpdateDTO(stages, recommendations, null))
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 72-72: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (job, new PyrisCompetencyStatusUpdateDTO(stages, recommendations, null))
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfigurationsrc/test/java/de/tum/cit/aet/artemis/iris/IrisTextExerciseChatMessageIntegrationTest.java (2)
401-401: LGTM! Consider adding a TODO comment for token tracking.The update to
PyrisChatStatusUpdateDTOconstructor aligns with the new token tracking feature. However, passingnullfor the new parameter might not fully test the new functionality.Consider adding a TODO comment to remind about implementing a test case that includes non-null token data:
- request.postWithoutResponseBody("/api/public/pyris/pipelines/text-exercise-chat/runs/" + jobId + "/status", new PyrisChatStatusUpdateDTO(result, stages, suggestions, null), + // TODO: Add test case with non-null token data for LLM usage tracking + request.postWithoutResponseBody("/api/public/pyris/pipelines/text-exercise-chat/runs/" + jobId + "/status", new PyrisChatStatusUpdateDTO(result, stages, suggestions, null), HttpStatus.OK, headers);🧰 Tools
🪛 ast-grep
[warning] 400-401: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: ("/api/public/pyris/pipelines/text-exercise-chat/runs/" + jobId + "/status", new PyrisChatStatusUpdateDTO(result, stages, suggestions, null),
HttpStatus.OK, headers)
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 400-401: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: ("/api/public/pyris/pipelines/text-exercise-chat/runs/" + jobId + "/status", new PyrisChatStatusUpdateDTO(result, stages, suggestions, null),
HttpStatus.OK, headers)
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
Line range hint
1-401: Consider adding a new test case for token trackingWhile the existing tests cover various scenarios, there's an opportunity to enhance the test suite by adding a specific test case for the new token tracking feature.
Consider adding a new test method to verify the behavior of the system when token data is provided. This will ensure that the new feature is properly tested. Here's a suggested outline for the new test case:
@Test @WithMockUser(username = TEST_PREFIX + "student1", roles = "USER") void sendMessageWithTokenTracking() throws Exception { var irisSession = createSessionForUser("student1"); var messageToSend = createDefaultMockMessage(irisSession); List<LLMRequest> mockTokens = List.of(new LLMRequest(/* add necessary parameters */)); irisRequestMockProvider.mockTextExerciseChatResponse(dto -> { assertThat(dto.execution().settings().authenticationToken()).isNotNull(); assertThatNoException().isThrownBy(() -> sendStatus( dto.execution().settings().authenticationToken(), "Response with tokens", dto.execution().initialStages(), null, mockTokens )); pipelineDone.set(true); }); request.postWithoutResponseBody("/api/iris/sessions/" + irisSession.getId() + "/messages", messageToSend, HttpStatus.CREATED); await().until(pipelineDone::get); // Add assertions to verify token tracking behavior // For example, check if the tokens are correctly stored or processed }Would you like me to implement this test case or create a GitHub issue to track this task?
🧰 Tools
🪛 ast-grep
[warning] 399-399: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (Map.of("Authorization", List.of("Bearer " + jobId)))
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 400-401: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: ("/api/public/pyris/pipelines/text-exercise-chat/runs/" + jobId + "/status", new PyrisChatStatusUpdateDTO(result, stages, suggestions, null),
HttpStatus.OK, headers)
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 399-399: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (Map.of("Authorization", List.of("Bearer " + jobId)))
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 400-401: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: ("/api/public/pyris/pipelines/text-exercise-chat/runs/" + jobId + "/status", new PyrisChatStatusUpdateDTO(result, stages, suggestions, null),
HttpStatus.OK, headers)
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfigurationsrc/test/java/de/tum/cit/aet/artemis/iris/IrisChatMessageIntegrationTest.java (1)
448-449: LGTM! Consider adding a comment for the new parameter.The changes look good. The new parameter in the
PyrisChatStatusUpdateDTOconstructor is correctly added. However, it might be helpful to add a comment explaining the purpose of this new parameter, especially since it's currently set tonull.Consider adding a comment like this:
request.postWithoutResponseBody("/api/public/pyris/pipelines/tutor-chat/runs/" + jobId + "/status", - new PyrisChatStatusUpdateDTO(result, stages, suggestions, null), + new PyrisChatStatusUpdateDTO(result, stages, suggestions, null), // null for tokens (not implemented/tested yet) HttpStatus.OK, headers);This will help other developers understand the purpose of the new parameter and why it's currently set to
null.🧰 Tools
🪛 ast-grep
[warning] 448-449: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: ("/api/public/pyris/pipelines/tutor-chat/runs/" + jobId + "/status", new PyrisChatStatusUpdateDTO(result, stages, suggestions, null),
HttpStatus.OK, headers)
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 448-449: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: ("/api/public/pyris/pipelines/tutor-chat/runs/" + jobId + "/status", new PyrisChatStatusUpdateDTO(result, stages, suggestions, null),
HttpStatus.OK, headers)
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfigurationsrc/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsageRequest.java (1)
60-62: Ensure consistency in parameter names of setter methodsThe parameter names in the setter methods
setCostPerMillionInputTokensandsetCostPerMillionOutputTokensare singular, whereas the field names are plural. For consistency and clarity, consider renaming the parameters to match the field names.Apply this diff to rename the parameters:
public void setCostPerMillionInputTokens(float costPerMillionInputToken) { - this.costPerMillionInputTokens = costPerMillionInputToken; + this.costPerMillionInputTokens = costPerMillionInputTokens; } +public void setCostPerMillionInputTokens(float costPerMillionInputTokens) { + this.costPerMillionInputTokens = costPerMillionInputTokens; +} public void setCostPerMillionOutputTokens(float costPerMillionOutputToken) { - this.costPerMillionOutputTokens = costPerMillionOutputToken; + this.costPerMillionOutputTokens = costPerMillionOutputTokens; } +public void setCostPerMillionOutputTokens(float costPerMillionOutputTokens) { + this.costPerMillionOutputTokens = costPerMillionOutputTokens; +}Also applies to: 68-70
src/main/java/de/tum/cit/aet/artemis/iris/service/websocket/IrisChatWebsocketService.java (1)
65-65: Correct grammatical error in JavaDoc commentThere's a grammatical error in the JavaDoc comment. Change "token usage and cost send by Pyris" to "token usage and cost sent by Pyris" to improve readability.
Apply this diff to fix the error:
- * @param tokens token usage and cost send by Pyris + * @param tokens token usage and cost sent by Pyrissrc/test/java/de/tum/cit/aet/artemis/iris/IrisChatTokenTrackingIntegrationTest.java (2)
123-153: Suggestion: Narrow the exception types in test methodsThe test methods
testTokenTrackingHandledExerciseChatandtestTokenTrackingExerciseChatWithPipelineFailcurrently declarethrows Exception. It's better practice to specify the exact exceptions that might be thrown or handle them within the test to improve clarity and maintainability.Apply this change to specify the exceptions:
- void testTokenTrackingHandledExerciseChat() throws Exception { + void testTokenTrackingHandledExerciseChat() throws IOException, URISyntaxException, GitAPIException {And:
- void testTokenTrackingExerciseChatWithPipelineFail() throws Exception { + void testTokenTrackingExerciseChatWithPipelineFail() throws IOException, URISyntaxException, GitAPIException {Adjust the exception types based on the actual exceptions that can be thrown.
Also applies to: 173-204
🧰 Tools
🪛 ast-grep
[warning] 127-127: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (new PyrisStageDTO("DoneTest", 10, PyrisStageState.DONE, "Done"))
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 127-127: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (new PyrisStageDTO("DoneTest", 10, PyrisStageState.DONE, "Done"))
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
225-229: Suggestion: Narrow the exception type insendStatusmethodThe
sendStatusmethod currently declaresthrows Exception, which is too broad. Consider specifying the exact exceptions that might be thrown (e.g.,IOException) to improve error handling and make the code clearer.Apply this change:
- private void sendStatus(String jobId, String result, List<PyrisStageDTO> stages, List<LLMRequest> tokens) throws Exception { + private void sendStatus(String jobId, String result, List<PyrisStageDTO> stages, List<LLMRequest> tokens) throws IOException {Adjust the exception types based on the actual exceptions that can be thrown within the method.
🧰 Tools
🪛 ast-grep
[warning] 225-225: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (Map.of("Authorization", List.of("Bearer " + jobId)))
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 226-227: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: ("/api/public/pyris/pipelines/tutor-chat/runs/" + jobId + "/status", new PyrisChatStatusUpdateDTO(result, stages, null, tokens),
HttpStatus.OK, headers)
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 225-225: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (Map.of("Authorization", List.of("Bearer " + jobId)))
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 226-227: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: ("/api/public/pyris/pipelines/tutor-chat/runs/" + jobId + "/status", new PyrisChatStatusUpdateDTO(result, stages, null, tokens),
HttpStatus.OK, headers)
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfigurationsrc/main/java/de/tum/cit/aet/artemis/athena/service/AthenaFeedbackSuggestionsService.java (2)
64-64: Fix typo in JavaDoc parameter descriptionThere's a typo in the JavaDoc comment for
athenaDTOConverterService. The word "exrcises" should be "exercises".Apply this diff to correct the typo:
/** * Create a new AthenaFeedbackSuggestionsService to receive feedback suggestions from the Athena service. * * @param athenaRestTemplate REST template used for the communication with Athena * @param athenaModuleService Athena module serviced used to determine the urls for different modules - * @param athenaDTOConverterService Service to convert exrcises and submissions to DTOs + * @param athenaDTOConverterService Service to convert exercises and submissions to DTOs * @param llmTokenUsageService Service to store the usage of LLM tokens */
159-166: Use primitivebooleaninstead of wrapperBooleanforisPreliminaryFeedbackIn the
storeTokenUsagemethod, the parameterisPreliminaryFeedbackis declared as aBooleanobject. Since this value is always required and does not need to benull, it's better to use the primitive typebooleanto avoid unnecessary object overhead.Apply this diff to use the primitive type:
/** * Store the usage of LLM tokens for a given submission * * @param exercise the exercise the submission belongs to * @param submission the submission for which the tokens were used * @param meta the meta information of the response from Athena - * @param isPreliminaryFeedback whether the feedback is preliminary or not + * @param isPreliminaryFeedback whether the feedback is preliminary or not (true or false) */ -private void storeTokenUsage(Exercise exercise, Submission submission, ResponseMetaDTO meta, Boolean isPreliminaryFeedback) { +private void storeTokenUsage(Exercise exercise, Submission submission, ResponseMetaDTO meta, boolean isPreliminaryFeedback) {Ensure that you update any method calls to pass a
booleanvalue instead of aBooleanobject.
🛑 Comments failed to post (13)
src/main/java/de/tum/cit/aet/artemis/athena/dto/ResponseMetaTotalUsageDTO.java (1)
9-9: 🛠️ Refactor suggestion
Consider using primitive types for the fields.
The coding guidelines suggest preferring primitive types. Since these fields represent counts, they should never be null. Consider using
intinstead ofIntegerfor all three fields.Here's the suggested change:
-public record ResponseMetaTotalUsageDTO(Integer inputTokens, Integer outputTokens, Integer totalTokens) { +public record ResponseMetaTotalUsageDTO(int inputTokens, int outputTokens, int totalTokens) {This change would also eliminate the need for the
@JsonIncludeannotation, as primitive types are always included in JSON serialization.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.public record ResponseMetaTotalUsageDTO(int inputTokens, int outputTokens, int totalTokens) {src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/job/CompetencyExtractionJob.java (1)
16-16: 💡 Codebase verification
Issues Found: Update
userLoginReferences in Tests
IrisCompetencyGenerationIntegrationTest.java:72is still usinguserLoginstring. Update it to pass aUserobject.🔗 Analysis chain
LGTM: Record declaration updated correctly. Verify impact on codebase.
The change from
String userLogintoUser useraligns with the single responsibility principle and follows the DTO guideline of using Java records. This change enhances the job's capability to handle user-related data.However, this modification might impact other parts of the codebase that were previously using the
userLoginstring.Please run the following script to identify potential areas that might need updates:
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Find usages of CompetencyExtractionJob constructor and potential references to userLogin # Search for CompetencyExtractionJob constructor calls echo "Searching for CompetencyExtractionJob constructor calls:" rg -p "new CompetencyExtractionJob\(" --type java # Search for potential references to userLogin within CompetencyExtractionJob context echo "\nSearching for potential references to userLogin:" rg -p "CompetencyExtractionJob.*userLogin" --type javaLength of output: 997
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsageTrace.java (1)
53-107:
⚠️ Potential issueInconsistency in userId getter and setter types.
The getter and setter methods are generally well-implemented, providing proper encapsulation. However, there's an inconsistency in the
userIdmethods:The
userIdfield is of typeLong, but thegetUserId()method returnslong. This mismatch could lead to potentialNullPointerExceptions. To resolve this, either:
- Change the field type to
long(preferred, as suggested earlier):private long userId;
- Or, change the getter return type to match the field:
public Long getUserId() { return userId; }Make sure to adjust the setter parameter type accordingly:
public void setUserId(Long userId) { this.userId = userId; }Choose the option that best fits the requirements of whether
userIdcan be null or not.src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsageRequest.java (2)
28-30:
⚠️ Potential issueUse
BigDecimalfor monetary values instead offloatUsing
floatfor monetary values can lead to precision and rounding errors. It's recommended to useBigDecimalfor precise monetary calculations.Apply this diff to update the data types:
+import java.math.BigDecimal; @Column(name = "cost_per_million_input_tokens") -private float costPerMillionInputTokens; +private BigDecimal costPerMillionInputTokens; @Column(name = "cost_per_million_output_tokens") -private float costPerMillionOutputTokens; +private BigDecimal costPerMillionOutputTokens;Remember to update the corresponding getters and setters to use
BigDecimal.Also applies to: 34-36
37-38: 🛠️ Refactor suggestion
Specify
FetchType.LAZYin the@ManyToOnerelationshipThe default fetch type for
@ManyToOneisFetchType.EAGER, which can lead to performance issues by loading related entities unnecessarily. SpecifyFetchType.LAZYto optimize performance.Apply this diff to modify the fetch type:
+import jakarta.persistence.FetchType; -@ManyToOne +@ManyToOne(fetch = FetchType.LAZY) private LLMTokenUsageTrace trace;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.import jakarta.persistence.FetchType; @ManyToOne(fetch = FetchType.LAZY) private LLMTokenUsageTrace trace;src/main/java/de/tum/cit/aet/artemis/iris/service/websocket/IrisChatWebsocketService.java (2)
56-56: 🛠️ Refactor suggestion
Refactor method to improve clarity by reducing null parameters
In
sendStatusUpdate, passingnullvalues for unused parameters can be confusing. Overloading thesendStatusUpdatemethod to handle different parameter sets or using default values can enhance code clarity and adhere to clean code practices.
45-45: 🛠️ Refactor suggestion
Consider refactoring to avoid passing multiple null arguments
Passing multiple
nullarguments to constructors reduces code readability and maintainability. Consider using the builder pattern or providing overloaded constructors forIrisChatWebsocketDTOto handle different parameter combinations more gracefully.src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisExerciseChatSessionService.java (1)
174-206: 🛠️ Refactor suggestion
Refactor
handleStatusUpdatemethod to improve readability and maintainabilityThe nested
if-elseblocks within thehandleStatusUpdatemethod from lines 186 to 206 increase complexity and reduce readability. Consider refactoring the token usage handling logic into separate helper methods or simplifying the conditional structures to adhere to the Single Responsibility Principle and improve code clarity.🧰 Tools
🪛 ast-grep
[warning] 176-176: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (new IrisTextMessageContent(statusUpdate.result()))
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 176-176: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (new IrisTextMessageContent(statusUpdate.result()))
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfigurationsrc/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisCourseChatSessionService.java (1)
64-67: 🛠️ Refactor suggestion
Consider refactoring the constructor to reduce parameter count
The constructor currently has ten parameters, which can make the class harder to maintain and test. To improve readability and adhere to the Single Responsibility Principle, consider grouping related parameters into parameter objects or using a builder pattern to manage object creation.
src/test/java/de/tum/cit/aet/artemis/iris/IrisChatTokenTrackingIntegrationTest.java (1)
139-141:
⚠️ Potential issueFix: Replace
getFirst()withget(0)when accessing list elementsThe method
getFirst()is not defined forListin Java. To access the first element of a list, useget(0)instead. This will prevent a compilation error.Apply this diff to fix the issue:
- assertThat(savedTokenUsageTraces.getFirst().getServiceType()).isEqualTo(LLMServiceType.IRIS); - assertThat(savedTokenUsageTraces.getFirst().getExerciseId()).isEqualTo(exercise.getId()); - assertThat(savedTokenUsageTraces.getFirst().getCourseId()).isEqualTo(course.getId()); + assertThat(savedTokenUsageTraces.get(0).getServiceType()).isEqualTo(LLMServiceType.IRIS); + assertThat(savedTokenUsageTraces.get(0).getExerciseId()).isEqualTo(exercise.getId()); + assertThat(savedTokenUsageTraces.get(0).getCourseId()).isEqualTo(course.getId());And similarly for the second test:
- assertThat(savedTokenUsageTraces.getFirst().getServiceType()).isEqualTo(LLMServiceType.IRIS); - assertThat(savedTokenUsageTraces.getFirst().getExerciseId()).isEqualTo(exercise.getId()); - assertThat(savedTokenUsageTraces.getFirst().getIrisMessageId()).isEqualTo(messageToSend.getId()); - assertThat(savedTokenUsageTraces.getFirst().getCourseId()).isEqualTo(course.getId()); + assertThat(savedTokenUsageTraces.get(0).getServiceType()).isEqualTo(LLMServiceType.IRIS); + assertThat(savedTokenUsageTraces.get(0).getExerciseId()).isEqualTo(exercise.getId()); + assertThat(savedTokenUsageTraces.get(0).getIrisMessageId()).isEqualTo(messageToSend.getId()); + assertThat(savedTokenUsageTraces.get(0).getCourseId()).isEqualTo(course.getId());Also applies to: 189-191
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java (2)
48-48: 🛠️ Refactor suggestion
Consider making database operations asynchronous
The
// TODOcomments at lines 48 and 78 indicate thatsaveLLMTokenUsageandappendRequestsToTraceshould ideally be executed asynchronously to improve performance. Implementing asynchronous operations can enhance responsiveness, especially if these methods are called frequently or involve significant processing time.Would you like assistance in refactoring these methods to be asynchronous or opening a GitHub issue to track this task?
Also applies to: 78-78
45-45:
⚠️ Potential issueCorrect the Javadoc parameter description
The Javadoc comment references
IrisTokenUsageBuilder, but the correct class name isLLMTokenUsageBuilder. This inconsistency can cause confusion and should be updated.Apply this diff to fix the Javadoc:
- * @param builderFunction of type Function<IrisTokenUsageBuilder, IrisTokenUsageBuilder> using IrisTokenUsageBuilder + * @param builderFunction of type Function<LLMTokenUsageBuilder, LLMTokenUsageBuilder> using LLMTokenUsageBuilder📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.* @param builderFunction of type Function<LLMTokenUsageBuilder, LLMTokenUsageBuilder> using LLMTokenUsageBuildersrc/main/java/de/tum/cit/aet/artemis/athena/service/AthenaFeedbackSuggestionsService.java (1)
166-175:
⚠️ Potential issueHandle potential
ClassCastExceptionwhen casting toStudentParticipationIn the
storeTokenUsagemethod, the participation is cast toStudentParticipationwithout checking its actual type. If the participation is not an instance ofStudentParticipation, this will result in aClassCastException.Consider checking the type of the participation before casting:
private void storeTokenUsage(Exercise exercise, Submission submission, ResponseMetaDTO meta, Boolean isPreliminaryFeedback) { if (meta == null) { return; } Long courseId = exercise.getCourseViaExerciseGroupOrCourseMember().getId(); Long exerciseId = exercise.getId(); - Long userId = ((StudentParticipation) submission.getParticipation()).getStudent().map(User::getId).orElse(null); + Long userId = null; + if (submission.getParticipation() instanceof StudentParticipation studentParticipation) { + userId = studentParticipation.getStudent().map(User::getId).orElse(null); + } llmTokenUsageService.saveAthenaTokenUsage(courseId, exerciseId, userId, meta, isPreliminaryFeedback); }This ensures that you safely handle cases where the participation might not be a
StudentParticipation.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.private void storeTokenUsage(Exercise exercise, Submission submission, ResponseMetaDTO meta, Boolean isPreliminaryFeedback) { if (meta == null) { return; } Long courseId = exercise.getCourseViaExerciseGroupOrCourseMember().getId(); Long exerciseId = exercise.getId(); Long userId = null; if (submission.getParticipation() instanceof StudentParticipation studentParticipation) { userId = studentParticipation.getStudent().map(User::getId).orElse(null); } llmTokenUsageService.saveAthenaTokenUsage(courseId, exerciseId, userId, meta, isPreliminaryFeedback); }
github-actions
bot
removed
tests
database
| * DTO representing an individual LLM call. | ||
| */ | ||
| @JsonInclude(JsonInclude.Include.NON_NULL) | ||
| public record ResponseMetaLLMCallDTO(String modelName, Integer inputTokens, Integer outputTokens, Integer totalTokens) { |
There was a problem hiding this comment.
Do these numbers have to be nullable?
| * DTO representing the total usage metrics. | ||
| */ | ||
| @JsonInclude(JsonInclude.Include.NON_NULL) | ||
| public record ResponseMetaTotalUsageDTO(Integer inputTokens, Integer outputTokens, Integer totalTokens) { |
There was a problem hiding this comment.
Do these numbers have to be nullable?
…' into feature/athena/track-token-usage # Conflicts: # src/main/java/de/tum/cit/aet/artemis/athena/dto/ResponseMetaDTO.java # src/main/java/de/tum/cit/aet/artemis/athena/service/AthenaFeedbackSuggestionsService.java
github-actions
bot
added
core
…' into feature/athena/track-token-usage
|
I messed up with a rebase. Will reopen another PR |
Checklist
General
Server
Client
authoritiesto all new routes and checked the course groups for displaying navigation elements (links, buttons).Changes affecting Programming Exercises
Motivation and Context
We want to track the token usage for Athena.
Description
Stores token usage of Athena from the response meta field for feedback generation requests.
Follow-up Issues
costPerMillionInputToken,costPerMillionOutputTokenand apipelineIdSteps for Testing
Prerequisites:
Exam Mode Testing
Prerequisites:
Testserver States
Note
These badges show the state of the test servers.
Green = Currently available, Red = Currently locked
Click on the badges to get to the test servers.
Review Progress
Performance Review
Code Review
Manual Tests
Exam Mode Test
Performance Tests
Test Coverage
Screenshots
Summary by CodeRabbit
New Features
Bug Fixes
Tests