This is a repository for the "Machine Learning for the Web" class at ITP, NYU
Libraries like TensorFlow.js and ml5.js unlocked new opportunities for interactive machine learning projects in the browser. The goal of this class is to learn and understand common machine learning techniques and apply them to generate creative outputs in the browser.
This class will start with running pre-trained models and re-training models in the browser using high-level APIs from ml5.js and p5.js, as well as explore the Layer APIs from TensorFlow.js to create models from scratch using custom data. This class will also cover preparing the dataset for training models.
At the completion of this course, students will have a better understanding of common and popular machine learning models, how do they work, how to train these models, and their use case to creative projects. The output of the class will be interactive ML web applications.
The topics that will be covered are Image/Sound/Doodle Classification, Face/Pose Recognition, Image Style Transfer, pix2pix Image Transformation, and Image Synthesis. The techniques and neural networks we will use and build are Transfer Learning, Convolutional Neural Network, Generative Adversarial Network and more.
Prospective students are expected to have taken an ICM (Introduction to Computational Media) course, or have equivalent programming experience with JavaScript(p5.js), HTML, CSS.
- Yining Shi, Tuesday, 6:30PM - 9:00PM, Zoom meeeting, 9/8/2020 - 12/8/2020
- Office Hours
- All class dates
- Miro board
- How to push code to a Github Repo and host sketch on Github
- How to update your code and push it to Github
- Video: How to host p5 sketch on github pages
To run each examples, open your terminal, type in the following commands:
$ git clone https://github.com/yining1023/machine-learning-for-the-web.git
$ cd machine-learning-for-the-web
$ python -m SimpleHTTPServer # $ python3 -m http.server (if you are using python 3)
Go to localhost:8000 in your browser, you will see a directory list like this:
- week1-intro/
- week2-...
- week3-...
- week4-...
- week5-...
- week6-...
Click into each week, you will see the example for each week.
-
Notes: Introduction to Machine Learning
-
Coding session:
- Installing ml5.js
- Running Image Classification example with ml5.js
- Hosting p5 sketch on github or use p5 web editor
- How to updating homework wiki
-
Coding session:
- Sound Classifer with Teachable Machine
- Pose Classifier with Teachable Machine
- Image Classifier with Teachable Machine
- Classifiers with Arduino
-
Review: Showing the homework from last week, discuss any difficulties and other creative outputs
-
Coding session:
- Setup Spell.run training environment
- Training a new Style Transfer model
- Running Style Transfer model in ml5.js
-
Review: Showing the homework from last week, discuss any difficulties when you are training your own style trasnfer model
-
Coding session:
- Running pix2pix with ml5.js
- Setup Spell.run training environment
- Prepare dataset for pix2pix
- Training a new pix2pix model
-
Coding session:
- Using pre-trained styleGAN models in Runway
- Create latent space animation
- Train your own styleGAN
-
Homework: Continue final projects development
- ml5js
- TensorFlow.js
- Machine Learning Crash Course
- Google AI Adventures
- 3 Blue 1 Brown - Neural Networks
- fast.ai
- Spell.run
- RunwayML
- Machine Learning Workshop by @AndreasRef
- Coding Train Video Series:
- Two Minute Papers
- Udemy - Machine learning with Javascript
- Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems
You will need a modern laptop (4 years old or younger is a good rule of thumb). Most required software is freely available.
You are required to attend all class meetings and submit all weekly assignments and a final project.
Grading (pass/fail) will be based on a combination of factors:
- Attendance, participation in class discussion, and engagement in other students' projects (40%)
- Assignments (40%)
- Final Project (20%)
Please see ITP's statement on Pass/Fail which states that a "Pass" is equivalent to an "A" or a "B" while anything less would be considered a "Fail".
Attendance is mandatory. Please inform your teacher via email if you are going to miss a class. Two unexcused absences is cause for failing the class. (An unexcused lateness of 10 minutes or more is equivalent to 1/2 an absence.)
This class will be participatory, you are expected to participate in discussions and give feedback to other students both in class and participate with their projects. This (along with attendance) is 40% of your grade.
Class will culminate with final projects. You are expected to push your abilities to produce something that utilizes what you have learned in the class that is useful in some manner to yourself or the world. This will comprise 20% of your grade.
Plagiarism is presenting someone else’s work as though it were your own. More specifically, plagiarism is to present as your own: A sequence of words quoted without quotation marks from another writer or a paraphrased passage from another writer’s work or facts, ideas or images composed by someone else.
(The following is adapted from Golan Levin’s Interactivity and Computation Course (Fall 2018) at Carnegie Mellon University.)
with the exception of examples specifically provided by the professor that are demonstrated in the videos for this course. Please note the following additional expectations and guidelines:
When using others' code, pay attention to the license under which it has been released, and be certain to fulfill the terms and requirements of those licenses. Descriptions of common licenses, and their requirements, can be found at choosealicense.com. Some licenses may require permission. If you are confused or aren’t sure how to credit code, ask one of the course instructors and make your best good faith effort. Not properly citing code sources is grounds for a 0 on an assignment.
The use of general, repurposable libraries is strongly encouraged. The people who developed and contributed these components to the community worked hard, often for no pay; acknowledge them by citing their name and linking to their repository.
It sometimes happens that an artist places the entire source code for their sketch or artwork online, as a resource from which others can learn. Assignments professors give in new-media arts courses are often similar (e.g. "Clock"); you may also discover the work of a student in some other class or school, who has posted code for a project which responds to a similar assignment. You should probably avoid this code. At the very least, you should be careful about approaching such code for possible re-use. If it is necessary to do so, it is best to extract components that solve a specific technical problem, rather than those parts which operate to create a poetic experience. Your challenge, if and/or when you work with others' code, is to make it your own. It should be clear that downloading an artwork from someone's GitHub and simply changing the colors would be disgracefully lazy. And doing so without proper citation would be outright plagiarism.
The core of the educational experience at the Tisch School of the Arts is the creation of original academic and artistic work by students for the critical review of faculty members. It is therefore of the utmost importance that students at all times provide their instructors with an accurate sense of their current abilities and knowledge in order to receive appropriate constructive criticism and advice. Any attempt to evade that essential, transparent transaction between instructor and student through plagiarism or cheating is educationally self-defeating and a grave violation of Tisch School of the Arts community standards. For all the details on plagiarism, please refer to page 10 of the Tisch School of the Arts, Policies and Procedures Handbook, which can be found online at: http://students.tisch.nyu.edu/page/home.html
Please feel free to make suggestions to your instructor about ways in which this class could become more accessible to you. Academic accommodations are available for students with documented disabilities. Please contact the Moses Center for Students with Disabilities at 212 998-4980 for further information.
Your health and safety are a priority at NYU. If you experience any health or mental health issues during this course, we encourage you to utilize the support services of the 24/7 NYU Wellness Exchange 212-443-9999. Also, all students who may require an academic accommodation due to a qualified disability, physical or mental, please register with the Moses Center 212-998-4980. Please let your instructor know if you need help connecting to these resources.
Laptops will be an essential part of the course and may be used in class during workshops and for taking notes in lecture. Laptops must be closed during class discussions and student presentations. Phone use in class is strictly prohibited unless directly related to a presentation of your own work or if you are asked to do so as part of the curriculum.