Segment anything 2 pipeline image #165
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces a prototype implementation of the [Segment Anything v2](https://github.com/facebookresearch/segment-anything-2) (SAM2) pipeline within the AI worker. The prototype demonstrates the basic functionality needed to perform segmentation on an image. Note that video segmentation is not yet implemented. Additionally, the dependencies were updated quickly, which may temporarily break other pipelines.
This commit allows nested arrays to be supplied as JSON strings for SAM2 input. It also implements robust error handling to return a 400 error with a descriptive message when incorrect parameters are provided.
This commit ensures that we return the masks, iou_predictions and low_res_masks in json format.
This commit replaces the JSON.dump method with a simple str method since it is highly unlikely that the string contains invalid data. Co-authored-by: Peter Schroedl <[email protected]>
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces an implementation of the Segment Anything v2 (SAM2) pipeline within the AI worker.
This implements the basic functionality needed to perform segmentation on an image, including specifying points and labels. Returns the scores, logits, and low-res-masks.
In addition, this PR introduces a new method of adding pipelines dependencies without breaking existing implementations present in ai-runner. Example usage for local development:
The existing Dockerfile is specified as a 'base' image ( e.g. :
FROM livepeer/ai-runner:base) in cmd/segment-anything-2/Dockerfile.segment_anything_2.The go-livepeer PR here implements the logic necessary to launch this pipeline-specific container when appropriate : livepeer/go-livepeer#3131