new: Added jina clip v1 #408
Conversation
hh-space-invader
force-pushed
the
support-jina-clip
branch
from
86287ef to
82e2d4b
Compare
hh-space-invader
changed the title
hh-space-invader
force-pushed
the
support-jina-clip
branch
from
aa6be34 to
2de91d5
Compare
| height, width = image.height, image.width | ||
|
|
||
| # if the size is larger than the new canvas | ||
| if width > size or height > size: |
There was a problem hiding this comment.
should not it be if width >= size and height >= size ?
There was a problem hiding this comment.
It should be or. Assuming that size=500, width=600, height=400. With or, it will trigger cuz width>size, with and it will not trigger cuz hight <= size (and in theory we want either dimension if higher to be cropped)
There was a problem hiding this comment.
what will happen to the second dimension?
if height and width are 600 and 400, required size is 500, the result shape will be (500, 500)
So, what are those 100 pixels which occur in width? What are their values? Which color was used to fill them?
There was a problem hiding this comment.
It turns out that internally, the crop function pads the smaller side with zeros by default and the fill_color is not used if one of the sides > size. I modified the implementation.
| image = image.crop((left, top, right, bottom)) | ||
| return image | ||
|
|
||
| new_image = Image.new(mode="RGB", size=(size, size), color=fill_color) |
There was a problem hiding this comment.
What if we pass a grayscale image?
There was a problem hiding this comment.
In our post processor, the first operation is to change the image to RGB, so it shouldn't happen
| @staticmethod | ||
| def _interpolation_resolver(resample: Optional[str] = None) -> Image.Resampling: | ||
| interpolation_map = { | ||
| "nearest": Image.Resampling.NEAREST, | ||
| "lanczos": Image.Resampling.LANCZOS, | ||
| "bilinear": Image.Resampling.BILINEAR, | ||
| "bicubic": Image.Resampling.BICUBIC, | ||
| "box": Image.Resampling.BOX, | ||
| "hamming": Image.Resampling.HAMMING, | ||
| } | ||
|
|
||
| if resample and (method := interpolation_map.get(resample.lower())): | ||
| return method | ||
|
|
||
| raise ValueError(f"Unknown interpolation method: {resample}") |
There was a problem hiding this comment.
feels like it should not be a part of Compose class
There was a problem hiding this comment.
I felt the same. Got any suggestions ? fastembed/common/utils ?
There was a problem hiding this comment.
idk for sure, we can just move it out of the class
at least, this Compose._interpolation_resolver is super ugly, if we keep this method here, we need to make get_resize a class method, not a static, it is only used inside of Compose class anyway
There was a problem hiding this comment.
I've changed get_resize to cls method as _interpolation_resolver would only be used in here
new: added resize2square
Co-authored-by: George <[email protected]>
hh-space-invader
force-pushed
the
support-jina-clip
branch
from
74a381e to
7627d78
Compare
Adding jinaai/jina-clip-v1
They provided two examples, the first one works and the second one complains about missing
jinaai/jina-clip-v1/sentence_xlnet_config.json. The output of the first one seems to have small numbers, like they are normallized but its not mentioned so not sure tbh.Update:
The text model needs pooling and normalizing
The image model needs the image to be square
All Submissions:
New Feature Submissions:
pre-commitwithpip3 install pre-commitand set up hooks withpre-commit install?New models submission: