CogWorks

The Autonomous Cognitive Assistant course, CogWorks, uses project-based learning to immerse students in exciting applications of modern machine learning and data science. CogWorks was created in 2017, and has been offered for four years as a course at the MIT Beaver Works Summer Institute (BWSI). BWSI is a rigorous four-week (over 160 hours) residential STEM summer program, held at MIT, for talented rising high school seniors from across the country.

The class consists of three distinct modules, which focus on applications of machine learning in the domains of audio, vision, and language, respectively. These modules bring students in contact with critical foundational concepts in applied mathematics, science, and machine learning via compelling projects that have important, real-world applications.

  1. Song-identification: students apply spectrogram-analysis, 2D peak-finding, and fingerprinting techniques to create an app that answers the question: “What song is this?” Playing several seconds of audio from one’s phone into a laptop’s microphone, this app can name that tune!

  2. Face-recognition & identity clustering: students will use a neural network and clustering techniques to detect and identify faces across pictures. Given a stack of pictures, this app will recognize the people in the photos, and will sort those photos accordingly.

  3. Image query by caption: students will create an app that takes in a written caption as a query, and returns images that resemble the caption. E.g. the caption “dog on a motorcycle” will prompt the neural network to find images in a database that suite this caption. This requires foundational techniques of natural language processing along with deep learning methods.

The central ethos to this course is that all of these projects can be completed without our depending on any “mystery boxes” to do so. This is, in part, a reaction to the growing popularity of online tutorials, which guide students to download and run industrial-grade algorithms and permit very little insight into how things are actually working. Students often express that this course illuminates many things that have remained hidden to them, despite their prior work through quick online tutorials.