03_ml_models

Applications of Machine Learning models

Session A: Other pre-trained models

Objectives:

• Understanding how to use pre-trained models other than image classification.
• Ability to work with PoseNet and ml5.js
• Ability to work with Object Detection Models (CoCoSSD).
• Ability to work with Image Segmentation models (UNet and BodyPix) and ml5.js.
• Week 3 Slides

Models

• PoseNet
  • Read Real-Time Human Pose Estimation in the Browser with TensorFlow.js by Dan Oved, with editing and illustrations by Irene Alvarado and Alexis Gallo.
• HandPose
• Object Detection
• UNet
  • Trained by Zaid Alyafeai using mut1ny - Face/Head segmentation dataset
• BodyPix
  • Introducing BodyPix: Real-time Person Segmentation in the Browser with TensorFlow.js

Data and Model Biographies

• Slides from Ellen Nickles
• The Future Is Here!, a commission for The Photographers’ Gallery which examines the process of dataset creation by Mimi Ọnụọha
• Feminist Data Set tool kit to interrogate every step of the AI process by Caroline Sinders
• Ted Talk: The era of blind faith in big data must end and Cathy O'Neil's Risk Consulting & Algorithmic Auditing firm

Session B

ml5 video tutorials

• ml5.js Pose Estimation with PoseNet, code
• ml5.js Object Detection, code

p5 web editor examples

• PoseNet Webcam Part Selection
• PoseNet Webcam Full Skeleton
• PoseNet Single Image
• Object Detection - CoCoSSD
• UNet Image Segmentation
• BodyPix Image Segmentation

Related Projects

• Maya Man's PoseNet sketchbook, Body, Movement, Language: A.I. Sketches with Bill T. Jones
• PoseNet Whack a Mole by Vibert Thio
• Sidewalk Orchestra by Cristóbal Valenzuela
• Pose Music by Tero Parviainen
• The Treachery of Sanctuary by Chris Milk
• Gait Analysis from runnersneed

Assignment 3

2. Read Mixing movement and machine by Maya Man
3. Read Humans of AI by Philipp Schmitt
4. Pick one of the models above (PoseNet, HandPose, UNET, BodyPix, CoCoSSD) and following the examples and ml5.js documentation experiment with controlling elements of a p5.js sketch (color, geometry, sound, text) with the output of the model. (You may also choose a ml5.js model not covered here if you like!)
5. Considering the Model and Data Biography reflect on the the following questions:
   • What questions do you still have about the model and the associated data? Are there elements you would propose including in the biography?
   • How does understanding the provenance of the model and its data inform your creative process?
6. Document your response to the Data and Model biography as well as your p5.js sketch in a blog post and add a link to the post / your p5 sketch on the Assignment 3 Wiki. In your blog post, include visual documentation such as a recorded screen capture / video / GIFs of your sketch.