🔥 A web utility case of Segmenting images using Meta AI's SAM.

Under construction, more features are coming...

Problem

This project depends on the result of python script to achieve segmenting images. However, netlify doesn't support languages beyond TS/JS/GO, and Vercel serverless function has a timeout limitation of 10s at free plans (so does netlify).

It won't be free due to AI SAM computations at other belike platforms. So at current stage, this project is fully working only under development env.

If you want a deployed version, you have to be aware of all the potential costs.

what have done

• generate embeddings with python scripts when consuming a next api endpoint (currently locally)
• host onnx model file in cloud with Aliyun OSS
• host generated embedding npy files in cloud with Aliyun OSS
• tackle the cors issue of uploaded npy files
• tackle the host issue of sam_vit_h_4b8939.pth file
• put all pieces together and make the app run
• support uploading image
• refact: support local generate and store embedding files with uploaded image eliminating any process with cloud.

what to do

• support extracting specific piece of image
• cooler interaction with each piece
• refact: support local host onnx model
• deploy with docker on other platform
  • build new CI & CD pipeline
• use webassembly approach

warning: Downloading fee generated by aliyun oss is significant, therefore the refactor is a must to do.

Tips

20230507

• There is python environment issue which is it will throw an ENOENT error when executing python script
• Theoretically, this can be resolved by using netlify function or docker to keep dev and prod the same running env
• But netlify do not support deploy with docker, so the netlify function approach seems to be a good one.

20230508

• There is a timeout limit of 10s, and it is not available for custom setting if you are in a free plan;
• So netlify function approach is a deadend.

20240509

• Any code deployed at netlify will be run in Lambda function. which only support js/ts/go, see forums. So deployment using netlify is a big deadend.
• Compiling python to webassembly and invoking wasm file directly seems to be a workable approach. Give it a shot!