Segmentation Person Media

Person segmentation from a media element and manipulate background in the browser.

Try it out by live demo heer!

To segment person, it's using BodyPix from tensorflow project.

Installing

npm install segmentation-person-media

Usage

import { load } from 'segment-person-media'
const segmentationPersonMedia = await load()

// 1. Create a MediaStream from HTMLVidelElement
const localVideo = document.getElementById('local-video') as HTMLVideoElement
// 1.1 Mask
const maskedStream = segmentationMedia.createMaskedStream({ src: localVideo } })
// 1.2 Change background
const imageData = getSelectedBackgroundImage()
const changedBackGroundStream = segmentationMedia.createChangedBackgroundStream({ src: localVideo, backgroundImage: imageData })
// 1.3 Blur
const bluredStream = segmentationMedia.createBluredStream({ src: localVideo })

// 2. Create a ImageData from HTMLImageElement
const personImage = document.getElementById('person-image') as HTMLImageElement
// 2.1 Mask
const maskedImageData = await segmentationMedia.createMaskedImageData({ src: personImage })
// 2.2 Change background
const imageData = getSelectedBackgroundImage()
const changedBackGroundImageData = await segmentationMedia.createChangedBackgroundImageData({
  src: personImage, backgroundImage: imageData
})
// 2.3 Blur
const bluredImageData = await segmentationMedia.createBluredImageData({
  src: personImage
})

You can check demo code lines to get to know details of how to use.

https://github.com/joe-re/segmentation-person-media/blob/main/demo/main.ts

Loading the model

You need to load the model data to segment person at first.

const segmentationPersonMedia = await load()

SegmentPersonMedia accepts the same arguments as BodyPix and you can adjust it for your project.

ref: https://github.com/tensorflow/tfjs-models/tree/master/body-pix#loading-the-model

APIs

Requirements

For creating a stream APIs, it requires canvas.captureStream for the browser. https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream

Common params

• SegmentationConfig is the same as segmentPerson's config from BodyPix. https://github.com/tensorflow/tfjs-models/tree/master/body-pix#person-segmentation

createMaskedStream

Create a masked background stream from HTMLVideoElement.

const stream = segmentationMedia.createMaskedStream({ src: localVideo } }

input

{
  src: HTMLVideoElement,
  frameRate?: number,        // default: 30
  options?: {
    color?: {                // default: { r: 255, g: 255, b: 255, a: 255 }
      r: number,
      g: number,
      b: number,
      a: number
    },
    maskOpacity?: number,    // default: 1
    maskBlurAmount?: number, // default: 0
    flipHorizontal?: boolean // default: false
  },
  config?: SegmentationConfig
}

Options includes some parameters to give drawPix.drawMask. https://github.com/tensorflow/tfjs-models/tree/master/body-pix#bodypixdrawmask

returns

MediaStream

createChangedBackgroundStream

Create a changed background stream from HTMLVideoElement.

const stream = segmentationMedia.createChangedBackgroundStream({ src: localVideo, backgroundImage: imageData })

input

{
  src: HTMLVideoElement,
  frameRate?: number,         // default: 30
  backgroundImage: HTMLImageElement | HTMLCanvasElement | ImageData,
  options?: {
    maskOpacity?: number,     // default: 1
    maskBlurAmount?: number,  // default: 0
    flipHorizontal?: boolean  // default: false
  },
  config?: SegmentationConfig
}

Options includes some parameters to give drawPix.drawMask. https://github.com/tensorflow/tfjs-models/tree/master/body-pix#bodypixdrawmask

returns

MediaStream

createBluredStream

Create a blured background stream from HTMLVideoElement.

const stream = segmentationMedia.createBluredStream({ src: localVideo })

input

{
  src: HTMLVideoElement,
  frameRate?: number,              // default: 30
  options?: {
    backgroundBlurAmount?: number, // default: 9
    edgeBlurAmount?: number,       // default: 9
    flipHorizontal?: boolean       // default: false
  },
  config?: SegmentationConfig
}

Options includes some parameters to give drawPix.drawBokehEffect. https://github.com/tensorflow/tfjs-models/tree/master/body-pix#inputs-4

returns

MediaStream

createMaskedImageData

Create a masked background image data from MediaElement.

const imageData = segmentationMedia.createMaskedImageData({ src: personImage })

input

{
  src: HTMLImageElement | HTMLVideoElement | HTMLCanvasElement,
  options?: {
    color?: {                // default: { r: 255, g: 255, b: 255, a: 255 }
      r: number,
      g: number,
      b: number,
      a: number
    },
    maskOpacity?: number,    // default: 1
    maskBlurAmount?: number, // default: 0
    flipHorizontal?: boolean // default: false
  },
  config?: SegmentationConfig
}

Options includes some parameters to give drawPix.drawMask. https://github.com/tensorflow/tfjs-models/tree/master/body-pix#bodypixdrawmask

returns

Promise

createChangedBackgroundImageData

Create a masked background image data from MediaElement.

const imageData = await segmentationMedia.createChangedBackgroundImageData({
  src: personImage, backgroundImage: imageData
})

input

{
  src: HTMLImageElement | HTMLVideoElement | HTMLCanvasElement,
  backgroundImage: HTMLImageElement | HTMLCanvasElement | ImageData,
  options?: {
    maskOpacity?: number,     // default: 1
    maskBlurAmount?: number,  // default: 0
    flipHorizontal?: boolean  // default: false
  },
  config?: SegmentationConfig
}

Options includes some parameters to give drawPix.drawMask. https://github.com/tensorflow/tfjs-models/tree/master/body-pix#bodypixdrawmask

returns

Promise

createBluredImageData

Create a blured background image data from MediaElement.

const imageData = await segmentationMedia.createBluredImageData({
  src: personImage
})

input

{
  src: HTMLVideoElement,
  options?: {
    backgroundBlurAmount?: number, // default: 9
    edgeBlurAmount?: number,       // default: 9
    flipHorizontal?: boolean       // default: false
  },
  config?: SegmentationConfig
}

Options includes some parameters to give drawPix.drawBokehEffect. https://github.com/tensorflow/tfjs-models/tree/master/body-pix#inputs-4