feat: Vision models support in Anthropic #32103

vivonk · 2024-03-27T10:33:11Z

Description

Adding new vision models support in Anthropic

Fixes https://github.com/appsmithorg/appsmith-ee/issues/3681

Automation

/ok-to-test tags="@tag.Datasource"

🔍 Cypress test results

Tip

🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉
Workflow run: https://github.com/appsmithorg/appsmith/actions/runs/8706176169
Commit: 2c393e7
Cypress dashboard url: Click here!

Summary by CodeRabbit

New Features
- Enhanced chat functionality with version upgrade and refined message creation process.
- Added vision-related commands for handling image and text data in the Anthropic plugin.
- Expanded constants to support new messaging and vision features.
- Introduced new fields in request models to support system prompts and messages, while deprecating older fields.
- Implemented a VisionCommand in method strategy for better handling of vision tasks.
- Improved utility functions for message handling and configuration value extraction.
Refactor
- Streamlined request URI construction to support new vision functionality alongside chat commands.
- Removed unused constants and methods to clean up the codebase.

coderabbitai · 2024-03-27T10:33:21Z

Walkthrough

The update enhances the Anthropic plugin by improving request serialization, refining command functionalities, and introducing vision-related capabilities. It streamlines JSON request creation, refactors chat command operations, enhances message handling, and lays the groundwork for vision command functionalities. Constants and utility functions are updated to support these changes, ensuring a more robust interaction with the Anthropic API.

Changes

File(s)	Summary
`.../AnthropicPlugin.java`	Introduced `objectMapper` for serializing `anthropicRequestDTO` to JSON and improved error handling.
`.../commands/ChatCommand.java`	Updated chat command operations, refactored methods, and improved message handling.
`.../commands/VisionCommand.java`	Added vision-related command functionality for HTTP request creation and message processing.
`.../constants/AnthropicConstants.java`, `.../utils/AnthropicMethodStrategy.java`, `.../utils/RequestUtils.java`	Added and updated constants, enhanced URI construction, and method strategy for vision commands.
`.../models/AnthropicRequestDTO.java`, `.../models/Message.java`	Introduced new fields in `AnthropicRequestDTO` and added `Message` class for message content handling.
`.../utils/CommandUtils.java`	Provided utility functions for message handling and configuration value extraction.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

vivonk · 2024-03-27T10:33:42Z

/build-deploy-preview --skip-tests=true

github-actions · 2024-03-27T10:34:05Z

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/8450353422.
Workflow: On demand build Docker image and deploy preview.
skip-tests: . env: .
PR: 32103.
recreate: .

...ver/appsmith-plugins/anthropicPlugin/src/main/java/com/external/plugins/AnthropicPlugin.java

github-actions · 2024-03-27T10:46:40Z

Failed server tests

com.external.plugins.AnthropicPluginTest#verifyDatasourceTriggerResultsForChatModels
com.external.plugins.ChatCommandTest#testCreateExecutionUri
com.external.plugins.ChatCommandTest#testMakeRequestBody_withValidData

github-actions · 2024-03-27T11:05:09Z

Failed server tests

com.external.plugins.AnthropicPluginTest#verifyDatasourceTriggerResultsForChatModels
com.external.plugins.ChatCommandTest#testCreateExecutionUri
com.external.plugins.ChatCommandTest#testMakeRequestBody_withValidData

...plugins/anthropicPlugin/src/main/java/com/external/plugins/constants/AnthropicConstants.java

vivonk · 2024-03-28T11:34:52Z

/build-deploy-preview --skip-tests=true

github-actions · 2024-03-28T11:35:13Z

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/8466655993.
Workflow: On demand build Docker image and deploy preview.
skip-tests: . env: .
PR: 32103.
recreate: .

...ver/appsmith-plugins/anthropicPlugin/src/test/java/com/external/plugins/ChatCommandTest.java

github-actions · 2024-03-28T11:58:51Z

Failed server tests

com.appsmith.server.solutions.ApplicationForkingServiceTests#forkApplication_WhenContainsInternalFields_InternalFieldsNotForked

github-actions · 2024-03-28T12:05:26Z

Deploy-Preview-URL: https://ce-32103.dp.appsmith.com

vivonk · 2024-03-28T13:19:48Z

/build-deploy-preview --skip-tests=true

github-actions · 2024-03-28T13:20:21Z

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/8467958122.
Workflow: On demand build Docker image and deploy preview.
skip-tests: . env: .
PR: 32103.
recreate: .

github-actions · 2024-03-28T13:32:39Z

Failed server tests

com.external.plugins.AnthropicPluginTest#verifyDatasourceTriggerResultsForChatModels

github-actions · 2024-03-28T14:48:07Z

Failed server tests

com.external.plugins.AnthropicPluginTest#verifyDatasourceTriggerResultsForChatModels

vivonk · 2024-03-29T05:31:02Z

/build-deploy-preview --skip-tests=true

github-actions · 2024-03-29T05:31:26Z

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/8477630495.
Workflow: On demand build Docker image and deploy preview.
skip-tests: . env: .
PR: 32103.
recreate: .

…nthropic-claude-vision-support

github-actions · 2024-03-29T05:54:59Z

Failed server tests

com.appsmith.server.services.PageServiceTest#createPageWithNullName

github-actions · 2024-03-29T06:01:36Z

Deploy-Preview-URL: https://ce-32103.dp.appsmith.com

…nthropic-claude-vision-support

github-actions · 2024-04-03T10:37:23Z

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/8537297651.
Workflow: On demand build Docker image and deploy preview.
skip-tests: . env: .
PR: 32103.
recreate: .

vivonk · 2024-04-03T10:37:25Z

/build-deploy-preview --skip-tests=true --recreate=true

vivonk · 2024-04-11T09:12:08Z

/build-deploy-preview --skip-tests=true

github-actions · 2024-04-11T09:12:33Z

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/8644329228.
Workflow: On demand build Docker image and deploy preview.
skip-tests: . env: .
PR: 32103.
recreate: .

github-actions · 2024-04-11T09:42:38Z

Deploy-Preview-URL: https://ce-32103.dp.appsmith.com

...ppsmith-plugins/anthropicPlugin/src/main/java/com/external/plugins/commands/ChatCommand.java

app/server/appsmith-plugins/anthropicPlugin/src/main/resources/editor/vision.json

...r/appsmith-plugins/anthropicPlugin/src/main/java/com/external/plugins/models/MessageDTO.java

...ava/com/appsmith/server/migrations/db/ce/Migration050MoveAnthropicLegacyModelsInQueries.java

sondermanish

Minor change required in the migration, rest looks fine

vivonk · 2024-04-15T10:30:06Z

/build-deploy-preview --skip-tests=true

github-actions · 2024-04-15T10:30:31Z

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/8687593093.
Workflow: On demand build Docker image and deploy preview.
skip-tests: . env: .
PR: 32103.
recreate: .

github-actions · 2024-04-15T10:55:09Z

Failed server tests

com.appsmith.server.solutions.ApplicationFetcherTest#getAllApplications_WhenUnpublishedPageExists_ReturnsApplications

github-actions · 2024-04-15T11:00:25Z

Deploy-Preview-URL: https://ce-32103.dp.appsmith.com

feat: Vision models support in Claude 3

74dc726

github-actions bot added the Enhancement New feature or request label Mar 27, 2024

vivonk added the ok-to-test Required label for CI label Mar 27, 2024

coderabbitai bot reviewed Mar 27, 2024

View reviewed changes

...ver/appsmith-plugins/anthropicPlugin/src/main/java/com/external/plugins/AnthropicPlugin.java Show resolved Hide resolved

vivonk commented Mar 28, 2024

View reviewed changes

...plugins/anthropicPlugin/src/main/java/com/external/plugins/constants/AnthropicConstants.java Outdated Show resolved Hide resolved

test fiz

1d53e5e

coderabbitai bot reviewed Mar 28, 2024

View reviewed changes

...ver/appsmith-plugins/anthropicPlugin/src/test/java/com/external/plugins/ChatCommandTest.java Show resolved Hide resolved

test case fix

ec887df

Merge branch 'release' of github.com:appsmithorg/appsmith into feat/a…

46e0bff

…nthropic-claude-vision-support

vivonk added 2 commits April 3, 2024 15:15

Merge branch 'release' of github.com:appsmithorg/appsmith into feat/a…

a38f7c1

…nthropic-claude-vision-support

migration and backward compatible response

4738e92

sondermanish reviewed Apr 11, 2024

View reviewed changes

...ppsmith-plugins/anthropicPlugin/src/main/java/com/external/plugins/commands/ChatCommand.java Show resolved Hide resolved

sondermanish reviewed Apr 11, 2024

View reviewed changes

app/server/appsmith-plugins/anthropicPlugin/src/main/resources/editor/vision.json Show resolved Hide resolved

sondermanish reviewed Apr 11, 2024

View reviewed changes

...r/appsmith-plugins/anthropicPlugin/src/main/java/com/external/plugins/models/MessageDTO.java Show resolved Hide resolved

dto usage comments

cf24bf9

vivonk requested a review from sondermanish April 12, 2024 10:33

sondermanish reviewed Apr 15, 2024

View reviewed changes

...ava/com/appsmith/server/migrations/db/ce/Migration050MoveAnthropicLegacyModelsInQueries.java Outdated Show resolved Hide resolved

null check

21e53e4

vivonk requested a review from sondermanish April 15, 2024 08:55

sondermanish reviewed Apr 15, 2024

View reviewed changes

...ava/com/appsmith/server/migrations/db/ce/Migration050MoveAnthropicLegacyModelsInQueries.java Outdated Show resolved Hide resolved

sondermanish reviewed Apr 15, 2024

View reviewed changes

null check and dynamic query input

7687bbf

ui change

2c393e7

vivonk requested a review from sondermanish April 15, 2024 12:08

sondermanish approved these changes Apr 15, 2024

View reviewed changes

vivonk added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Apr 16, 2024

vivonk enabled auto-merge (squash) April 16, 2024 12:29

trishaanand added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Apr 16, 2024

vivonk merged commit bc3d46d into release Apr 16, 2024
98 of 102 checks passed

vivonk deleted the feat/anthropic-claude-vision-support branch April 16, 2024 13:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Vision models support in Anthropic #32103

feat: Vision models support in Anthropic #32103

vivonk commented Mar 27, 2024 •

edited by github-actions bot

Loading

coderabbitai bot commented Mar 27, 2024 •

edited

Loading

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

vivonk commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

vivonk commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

vivonk commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

vivonk commented Mar 29, 2024

github-actions bot commented Mar 29, 2024

github-actions bot commented Mar 29, 2024

github-actions bot commented Mar 29, 2024

github-actions bot commented Apr 3, 2024

vivonk commented Apr 3, 2024

vivonk commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

sondermanish left a comment

vivonk commented Apr 15, 2024

github-actions bot commented Apr 15, 2024

github-actions bot commented Apr 15, 2024

github-actions bot commented Apr 15, 2024

feat: Vision models support in Anthropic #32103

feat: Vision models support in Anthropic #32103

Conversation

vivonk commented Mar 27, 2024 • edited by github-actions bot Loading

Description

Automation

🔍 Cypress test results

Summary by CodeRabbit

coderabbitai bot commented Mar 27, 2024 • edited Loading

Walkthrough

Changes

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

vivonk commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

Failed server tests

github-actions bot commented Mar 27, 2024

Failed server tests

vivonk commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

Failed server tests

github-actions bot commented Mar 28, 2024

vivonk commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

Failed server tests

github-actions bot commented Mar 28, 2024

Failed server tests

vivonk commented Mar 29, 2024

github-actions bot commented Mar 29, 2024

github-actions bot commented Mar 29, 2024

Failed server tests

github-actions bot commented Mar 29, 2024

github-actions bot commented Apr 3, 2024

vivonk commented Apr 3, 2024

vivonk commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

sondermanish left a comment

Choose a reason for hiding this comment

vivonk commented Apr 15, 2024

github-actions bot commented Apr 15, 2024

github-actions bot commented Apr 15, 2024

Failed server tests

github-actions bot commented Apr 15, 2024

vivonk commented Mar 27, 2024 •

edited by github-actions bot

Loading

coderabbitai bot commented Mar 27, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)