MobileNet & Image Classification Ideas

If you feed garbage into algorithms, garbage comes out. Training sets are central to how AI interpret the world, so of course how we train these algorithms: what data to use, how we categorize them, label them matters. When I started using MobileNet, the pretrained image classification model, for various types of objects, I started realizing its greatest issue: lack of representation. It failed to identify my liger pictures as its not considered an animal, probably due to its rarity in nature, but I still wish it could. The model can’t identify images and label sets that it hasn’t seen before, so its up to the “trainers” to include all sorts of animals into the model, instead of the user, who have much less control over the outputs.

In the case of my video classifier, it just straight up doesn’t recognize half of the stuff on my desktop, and I had to search for stuff that matches MobileNet’s existing labels to get any consistent confidence level p > 0.8. Although it is much more responsive than what I initially expected.

Single Image Classifier

[ALL ASSIGNMENTS ARE UPLOADED ON MY GITHUB PAGE]

https://github.com/xxhyz233/LunaChen-MLArts-FA24/

Basically started by creating a “mobilenet” image classifier, imported my image, and have the classification done in a callback function, with Label and Confidence displayed. I started with some usual animals like dogs, penguins, cats and it recognizes them just fine, but then when I had an image of a liger to be classified, it identifies it as a tiger with 0.61 confidence, which is just totally wrong. The MobileNet model probably doesn’t have liger as a label.

let mobilenet;
let liger

// MobileNet is loaded from the cloud, this function runs when the model is loaded
function modelReady() {
  console.log('Model is ready!!!');
  mobilenet.classify(liger, gotResults);
}

// Error first callback, must include error msg as 1st arg
function gotResults(error, results) {
  if (error) {
    console.error(error);
    let label = "Label: " + error[0].label;
    let confidence = 'Confidence: ' + nf(error[0].confidence, 0, 4)
    fill(0);
    textSize(32);
    fill('white');
    text(label, 10, height - 100);
    text(confidence, 10, height - 50);
  }
  else {
    console.log(results);
    results[0].label
    let label = "Label: " + results[0].label;
    fill(0);
    textSize(64);
    text(label, 10, height - 100);
  }
}

function imageReady() {
  image(liger, 0, 0, width, height);
}

function setup() {
  createCanvas(640, 480);
  background(220);
  mobilenet = ml5.imageClassifier('MobileNet', modelReady);
  liger = createImg('media/liger.jpg', imageReady);
  liger.hide();
}

Video Classifier

https://youtu.be/L2pxJN6Yce4

Follow the references and tutorials, replaced the image source as video source and voila.

let classifier;
let capture;
let label;
let confidence;

function preload() {
  // 2nd arg = continous image source being classified
  capture = createCapture(VIDEO);
  classifier = ml5.imageClassifier('MobileNet', capture, modelReady);
}

// Declare a callback function when classifier is loaded
function modelReady() {
  console.log('Model ready!!!');
  classifier.classifyStart(capture, 3, gotResults);
}

// Callback function when classify starts
function gotResults(results, error) {
  if (error) {
    console.error('ERROR');
  }
  else {
    label = results[0].label;
    confidence = nf(results[0].confidence, 0, 4);
  }
}

function setup() {
  createCanvas(960, 540);
  capture.hide();
  capture.size(960, 540);
  background(0);
  fill('white');
  textSize(32);
}

function draw() {
  background(0);
  image(capture, 0, 0);
  text(label, 10, height - 100);
  text(confidence, 10, height - 50);
}