Idea

Since it’s Halloween on Thursday, I want to have a ghost haunt you in the background of your camera using Depth Estimation and Mask Generation from Transformers.js.

https://editor.p5js.org/XXHYZ/sketches/jsNcYKPNF

ghostjs.mp4

Loading the Transformers.js

First I need to determine the depth and location in which the ghost will generate at. I followed the depth estimation example and made a basic setup for my depth estimation, I was wondering what the 2nd parameter of this code means, then I looked up the basic JS setup documentation and figured it was the model being used for depth estimation. The third parameter basically sets our device as GPU render for better performance.

depthEstimation = await pipeline(
  "depth-estimation",
  "onnx-community/depth-anything-v2-small",
  { device: "webgpu" }
);

Then I need to load the pixels from our depthImg, write the RGB of every pixel as our corresponding depthValue from our depthImg and finally update the pixels back to our canvas and display it as a p5 image.

if (results) {
  const { depth } = results;
  let depthImg = createImage(depth.width, depth.height);
  depthImg.loadPixels();

  for (let y = 0; y < depth.height; y++) {
    for (let x = 0; x < depth.width; x++) {
      let index = x + y * depth.width;
      let depthValue = depth.data[index];
      let pixelIndex = index * 4;

      depthImg.pixels[pixelIndex] = depthValue;
      depthImg.pixels[pixelIndex + 1] = depthValue;
      depthImg.pixels[pixelIndex + 2] = depthValue;
      depthImg.pixels[pixelIndex + 3] = 255;
    }
  }
  depthImg.updatePixels();
  image(depthImg, 0, 0, width, height);
}

When I finished setting up and ran it, there were two error messages from transformers but it doesn’t seem to impact my performance in any ways.

Mask Generation

I tried adding another mask generation task for this little project but in the Transformers.js official documents it’s unsupported, so I’ll use the body segmentation model from ml5.js to mask out our body instead. I don’t think it’s as accurate as mask generation, but it would suffice.

I was struggling to get my segmentation result to correctly mask out my ghost image, I don’t know why, it just keeps erasing parts of the ghost without replacing pixels with the original image.

// Masking out the ash baby with our person
if (segmentation) {
  image(segmentation.mask, 0, 0, width, height);
  segMaskImg = segmentation.mask;
  segMaskImg.resize(320, 240);
  ashBabyImg.mask(segMaskImg);
}

I ended up just scrapping this piece of code, and added a fade in fade out effect for the ghost.

if (babyPosition) {
  if (fadeIn) {
    opacity += 5;
    if (opacity >= 255) {
      opacity = 255;
      fadeIn = false;
      holdCounter = holdTime;
    }
  } else if (holdCounter > 0) {
    holdCounter--;
    if (holdCounter === 0) fadeOut = true;
  } else if (fadeOut) {
    opacity -= 5;
    if (opacity <= 0) {
      opacity = 0;
      fadeOut = false;
      fadeIn = true;
    }
  }
  // Set alpha tint for the baby image
  tint(255, opacity);
  image(ashBabyImg, babyPosition.x, babyPosition.y, 200, 192);
  // Reset tint for the rest of the layers
  noTint();
}

Ghost Position from Depth Map

Finally I placed my ghost on the furthest depth coordinate on the canvas with a simple algorithm for the minimum depth, and setting my babyPosition to the x y value of my current loop