Instructor: Yitzchak Elchanan Solomon (I go by “Elchanan”, “El” is fine too)
Required Background: Students should be familiar with the basics of linear algebra, multivariable calculus, and probability theory, as would be covered in an undergraduate course. All the case studies and homeworks will be written in Python, using Jupyter notebooks, so students should get themselves up to speed on the elements of Python.
Office Hours: TBD
Synchronous Meeting Times: MoWe 7:00PM – 8:00PM (DKU Time)
Semester: Oct 26, 2020 – Dec 10, 2020
Important Dates: TBD
Grading System: TBD
Thanks to Matthias Schröter for his invaluable help in the design of this syllabus!
The course will be organized into seven modules, one for each week. Each module will consist of:
- A unifying theme or topic.
- Assigned readings covering the basics of the topic. The readings will come from free, online textbooks.
- A collection of advanced topics I will lecture on (time permitting).
- Video lectures exploring the material in greater depth and developing the mathematical content.
- A case study consisting of a data set analyzed using module-specific techniques. The case study will happen during a synchronous lab session (2 hrs total per week).
- Homework, both written and in code, on the mathematical and practical principles of the module.
- An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani)
- Pattern Recognition and Machine Learnin (Bishop)
- Neural Nets and Deep Learning (Nielsen)
- Deep Learning (Goodfellow, Bengio, Courville)
- Hands-On Machine Learning with Scikit-Learn and TensorFlow (Géron) (neither mandatory nor free)
- Python Data Science Handbook (Jake VanderPlas) (neither mandatory nor free)
Module 1: The Basics of Machine Learning
Topics: Statistical Learning, Accuracy vs. Interpretability, Supervised vs. Unsupervised, Regression vs. Classification, Bias-Variance Tradeoff, Curse of Dimensionality, Validation, Resampling.
Advanced Topics: Decision Theory, Information Theory.
Assigned Readings: ISLR 1, 2.1, 2.2,5.1,5.2,6.4 Bishop 1.1, 1.3, 1.4
Module 2: Linear Models
Topics: Decision boundaries, Linear Regression, Logistic Regression, Discriminant Functions, Generative Models, Geometry of Least Squares, Subset Selection.
Advanced Topics: Shrinkage, Lasso, Bayesian approach to regression and model selection.
Assigned Readings: ISLR 3.1-3.4,4.1-4.3,6.1,6.2, Bishop 3.1,3.2, 4.1.1-4.1.3
Module 3: Tree-Based Models
Topics: Decision Trees, Classification Trees, CART, Pruning, Bagging, Boosting, Random Forest, Stacking, Bayesian Model Averaging.
Advanced Topics: Adaboost, Exponential Convergence, Chi-Squared Statistics.
Assigned Readings: ISLR 8.1,8.2, Bishop 14.1-14.4
Module 4: Unsupervised Learning & Dimensionality Reduction
Topics: PCA, SVD, Clustering, K-Means, Gaussian Mixture Models .
Advanced Topics: Random Projections and the Johnson-Lindenstrauss Lemma, Locally Linear Embeddings, IsoMap, MDS,t-SNE.
Assigned Readings: ISLR 6.3, 10.1-10.3, Bishop 9.1,9.2,12.1
Module 5: Introduction to Neural Networks
Topics: Neural Network, Sigmoids, Activations, Hidden Layers, Backpropagation, Cross-Entropy, Softmax, Regularization.
Advanced Topics: Universal Approximation Theorem, Optimization (e.g. momentum), Vanishing Gradients, Unstable Gradients.
Assigned Reading: Nielsen 1,2
Assigned Viewing: 3Blue1Brown playlist on Neural Networks.
Module 6: Convolutional & Recurrent Neural Networks
Topics: Convolutional Layers, Padding, Pooling, Segmentation, Detection, Localizaton, Recurrent Neurons, Backprop Through Time, NLP .
Advanced Topics: Long Short Term Memory(LSTM).
Assigned Reading: Nielsen 6, Deep Learning 10.
Module 7: Representation & Generative Learning
Topics: Word2Vec, CBOW, Skip-Gram, Encoders, Sparse/Convolutional/Variational Autoencoders, Generative Adverserial Networks .
Advanced Topics: Hierarchical Softmax, GANs and JS Divergence.
Assigned Reading: How Exactly does Word2Vec Work? (Meyer), Generative Adverserial Nets (Goodfellow et al.), Deep Learning 14
- Communication with myself and the TAs will happen by email. Content-related questions should be directed towards the Piazza page, administrative questions to myself and the TAs.
- If you need to request accommodation for exams, either to miss/reschedule an exam (due to an emergency) or to have more time, I need you to send me a note from a Dean, and get in touch with me about alternative plans, at least one week before an exam.
- If you are experiencing technical difficulties in submitting homework or exams, you need to reach out to me before the deadline is over, rather than after. I will add a “late submission” deadline for homeworks on Gradescope, to give a buffer zone for technical problems (no points will be taken off). Once the “late submission” deadline is passed, I can no longer help with technical problems, so if you didn’t get in contact with me before the “late submission” deadline, I will not be able to make any accommodations.
- The lowest homework grade will be dropped.
Civility is an essential ingredient for academic discourse. All communications for this course should be conducted constructively, civilly, and respectfully. Differences in beliefs, opinions, and approaches are to be expected. Please bring any communications you believe to be in violation of this class policy to the attention of your instructor. Active interaction with peers and your instructor is essential to success in this course, paying particular attention to the following:
- Be respectful of others and their opinions, valuing diversity in backgrounds, abilities, and experiences.
- Challenging the ideas held by others is an integral aspect of critical thinking and the academic process. Please word your responses carefully, and recognize that others are expected to challenge your ideas. A positive atmosphere of healthy debate is encouraged.
- Read your online discussion posts carefully before submitting them.
As a student, you should abide by the academic honesty standard of the Duke Kunshan University. Its Community Standard states: “Duke Kunshan University is a community comprised of individuals from diverse cultures and backgrounds. We are dedicated to scholarship, leadership, and service and to the principles of honesty, fairness, respect, and accountability. Members of this community commit to reflecting upon and upholding these principles in all academic and non-academic endeavors, and to protecting and promoting a culture of integrity and trust.” For all graded work, students should pledge that they have neither given nor received any unacknowledged aid.
Academic Policy & Procedures:
You are responsible for knowing and adhering to academic policy and procedures as published in University Bulletin and Student Handbook. Please note, an incident of behavioral infraction or academic dishonesty (cheating on a test, plagiarizing, etc.) will result in immediate action from me, in consultation with university administration (e.g., Dean of Undergraduate Studies, Student Conduct, Academic Advising). Please visit the Undergraduate Studies website for additional guidance related to academic policy and procedures. Academic integrity is everyone’s responsibility.
Academic Disruptive Behavior and Community Standard:
Please avoid all forms of disruptive behavior, including but not limited to: verbal or physical threats, repeated obscenities, unreasonable interference with class discussion, making/receiving personal phone calls, text messages or pages during class, excessive tardiness, leaving and entering class frequently without notice of illness or other extenuating circumstances, and persisting in disruptive personal conversations with other class members.
What resources can help me during this course?
Please consult with me about appropriate course preparation and readiness strategies, as needed. Consult your academic advisors on course performance (i.e., poor grades) and academic decisions (e.g., course changes, incompletes, withdrawals) to ensure you stay on track with degree and graduation requirements. In addition to advisors, staff in the Academic Resource Center can provide recommendations on academic success strategies (e.g., tutoring, coaching, student learning preferences). All ARC services will continue to be provided online. Note, there is an ARC Sakai site for students and tutors. Please visit the Office of Undergraduate Advising website for additional information related to academic advising and student support services.
For additional help with academic writing—and more generally with language learning—you are welcome to make an appointment with the Writing and Language Studio (WLS). To accommodate students who are learning remotely as well as those who are on campus, writing and language coaching appointments are available in person and online. You can register for an account, make an appointment, and learn more about WLS services, policies, and events on the WLS website. You can also find writing and language learning resources on the Writing & Language Studio Sakai site.
If you are experiencing technical difficulties, please contact IT:
- China-based faculty/staff/students 400-816-7100, (+86) 0512- 3665-7100
- US-based faculty/staff/students (+1) 919-660-1810
- International-based faculty/staff/students can use either telephone option (recommend using tools like Skype calling)
- Live Chat: https://oit.duke.edu/help
- Email: firstname.lastname@example.org