I am currently taking the course now. Spring 2025 semester and I thought I did well for project 1 only to receive 30 out of 100. Can I still salvage my grade for rest of the semester and at least get a B.

This was my first class and I had a lot of fun during the semester. I took this as first class because I wanted to learn RL for a long time and this was my golden opportunity.

Pros:

• The class will set you up for RL research, by the end of it you will be confident to do come up with some ideas yourself.
• It teaches a lot of RL algorithms from the basics to latest algorithms. (by teaching, I do not mean, there is a lecture covering it, rather, you'll have to learn it to solve the projects, otherwise, you'll fail the project). Be ready to learn cool algorithms like value iteration, Q-learning, SARSA, deep-q-learning, DDPG, TD3, REINFORCE, Vanilla Policy Gradient, PPO, Multi-Agent PPO.
• Lenient grading. Even if you make some mistakes on your paper, there is not a huge penalty. Even if you don't solve the problem asked in the project, you can explain what all you tried and you should not face huge issues. Analysis is the most important part in the paper. Also explain the problem first in the introduction section.
• Active discord and communities. Since this is my first class I do not have anything to compare, but I found the class discord to be pretty active. TAs are also active on ed discussions. They mod discord too.
• Project 1, Project 2 and Project 3 will give you a lot of learning. You will tonnes to talk about if you are interviewing for a RL/ML based roles.
• You receive detailed feedback on your papers, which if you implement will set you up for easy A.
• Grades are revealed in 3 weeks after submission.
• Regrade is fairly quick, 1 week.
• The instructor and TAs are quite knowledgeable and very nice people in general. Joe and Markian top tier TAs.
• OH are helpful, they also have private 1/1 OH, which really helped me in the initial projects. They also provide recording for public OH.

Cons:

• Totally disaster ed lectures, but they are needed for final exams. My strategy was to clear my concepts from David Silver Lectures (gold-standard) and learn game theory stuff from ed lectures. TBH, game theory ed-lectures are decent. There are two professors in the lectures who are top of the top in the field, but the lectures are setup in a way that they communicate with one another to teach stuff, which doesn't work. Not to mention, a lot of jokes in the lectures, which is just distracting and borderline irritating. Imagine preparing for finals and listening to jokes while trying to focus.
• Towards the deadline for submissions, many students ask question on ed, and then TAs get overwhelmed and hence some questions take longer than others to get answered. My strategy if my question was not answered was to create another private post for TAs, as a reminder, then the answers came.
• Project 4, I did not learn much from this project, it was quite basic. There was also a issue with AWS credits. This is the first semester they were running this project, so problems were expected. But the fact that this did not add anything new to the learning from what was an amazing class until project 3 was sad. I was expecting something more challenging in Project 4. It disappointed a lot of the students.
• Final exam: Its the most weird exam I have ever taken and it is given a weightage of 30%. It multiple choice and multiple answer (credit for partially correct answer is given). Its important to do at least average to get an A.

Recommendation: if you are okay by learning by yourself. Its a fun class.

Suggestion: Use OH and discord community to the fullest. Overall I got high 90s in all my projects, and above average in finals, and got an easy A.

The course consists of 4 projects and 1 final. I like the projects, and think they are well related to reality. However, I didn't like the lectures, because I think it is for those who have some or very good foundations of ML. Since it is my first ML course, it is difficult for me to understand some concepts mentioned in the lecture. Not recommended for those who have no ML knowledge.

TLDR; This class is very cool. Project 3 is very hard. If you're interested in the material, you should for sure take it.

This is a really cool course. The lectures are very similar quality to ML, so if you liked ML and the corny way Prof. Littman and Isbell lecture you will like this. I found the lectures to be fine. They cover the material, but there is a lack of depth that you are expected to extrapolate from the assigned readings. The other thing to note is that the lectures do not really cover material that is necessary for Project 2 and 3. Project 1 was really easy after the horror that was the 10-page papers from ML. If you want to do well read the paper literally everything you need is in that paper. Project 2 was also pretty easy. I don't recall if they gave the best advice about papers to read and reference for that, but look up DQN, DDPG, PPO, or REINFORCE and you should be able to figure it out. Project 3 was very difficult. This project covered MARL in a simplified Overcooked environment. And a big part of why this assignment is hard is because the environment is implemented in python and slow. Start this one as soon as it drops. Clear your calendar for the three weeks the assignment is available. I did not solve all of the layouts, but the penalty for not doing so is "small" (they don't specify what it is exactly). The HW is all very easy as well. I would recommend getting them out of the way quickly so that you can spend all your energy on the papers. The Final exam for this class is pretty annoying. There was no provided study guide and no clear communication of what would be on it. But don't worry because as long as you're ahead of the average you should be in good shape. Honestly, feels kinda disingenuous to have a final exam that is multi-choice/ multi-answer for a class that almost entirely focuses on analysis rather than definitions and rote memorization, but I also understand that grading a free response exam would take a lot of time. I got right around 50% and still got an A, so unless you really screwed up your projects or missed a homework I wouldn't sweat it too much.

I'm giving this course a 5 on quality because it was awesome, but recognize this is a graduate level course where most of the learning is done outside of lecture. TAs as always are going to give really vague and unhelpful answers to questions that get to the heart of assignments. I'm giving a 4 on difficulty because I'd say that is the average difficulty level, but Project 3 is seriously no joke and has been the hardest assignment I've worked on at OMS so far.

The lectures for this course were some of the better ones I've seen in OMSCS, of course there is always room for improvement and to teach more state of the art material, but overall was still extremely solid. The assignments were also pretty useful for the most part, especially the projects. I like classes like these where it requires a good deal of time and work BUT the assignments are designed in such a way where you actually learn a TON. Granted, a lot of the learning was done on the projects from my own research, but I also feel like they graded much fairer than other courses (CV and ML4T) and the assignments were not nearly as tedious as other classes (CV). The information learned in doing the assignments and projects was great in my opinion, especially if you have an interest in RL. I wish that continuous action spaces were covered more, along with a more formalized learning of multi-agent RL and how to use RL libraries (since everything was implemented from scratch except NN neurons and backprop). The projects, especially projects 2 and 3, require an insane amount of time so plan accordingly to start VERY early. I got a 100% on the third project but turned it in late and got a late penalty since I didn't allocate my time correctly. I highly recommend this class as it is probably the best one of the 8 courses I have taken.

I had taken AI and DL prior to RL, and I would rank the difficulty of the classes such that RL > AI > DL. I'm very glad that I took RL LAST because this was the class where I underestimated how much effort it would take to absorb the material, let alone just be able to do the assignments! RL is very conceptual and math-heavy, so if you do not understand how and why the math works then you're in for a world of hurt.

Student performance is judged based on 30% Homework (Gradescope submission work), 45% Projects (code to GitHub and research-style paper submissions), and 25% Final Exam (one closed-book exam proctored by HonorLock for 120 min).

Homework - Other people in OMSCentral have already remarked on the Homework assignments, so I have nothing new to contribute here. Seems unchanged from previous years for Spring 2024.

Project #1 - I highly recommend to put most of your effort on writing a high-quality research-like report for each project and not being fixated on mimicking the exact results of a research paper and/or getting the "best possible" results for each experiment. I learned this from Project 1 where even though I was able to get results very similar to Sutton's paper of TD Lambda, I still got a 10 point deduction for "not having sufficiently in-depth analysis" in my 5 page paper. To be fair, I agreed with this comment from the grader - I did not know how to best analyze the "whys" of what I was seeing and was not explaining it in a sufficient depth. I am pretty sure the deduction would have been far less if I had been unable to mimic the results of Sutton's paper but was able to explain the theoretical "whys" of why failures happened in my experiments.

Project #2 - This time around, I quickly coded up a solution that I thought worked sufficiently to address the Lunar Lander AI problem and spent over 1 week writing a report that elaborated much more on the "why" analysis of my experiments. There were pros and cons to this approach, but I wouldn't realize it until Project 3. The pros was that I got zero deductions on Project 2 (full score), which meant my report was solid), and I knew then exactly how yo format any future reports that I would ever need to turn in for RL. The con was that my agent for Project 2 was not the most efficient, and this would later affect me in Project 3.

Project #3 - This was only the 2nd time that the course had students work on a multi-agent problem that was the Overcooked AI problem, and I could tell they were still observing where they should fix any kinks if they were to have Project 3 remain the same for future semesters. This project made me sick...literally! I spent over 3 weeks spending all my free time reading more than 30 research papers and attempting so many different experiments in order to get some results for a writeup. I lost sleep, had constant headaches, etc. I was not looking for perfection of being able to solve all 5 layouts of Overcooked, but I did panic when I realized that I was getting stuck just solving 2 out of the 5 layouts and not making any progress despite my efforts. The teaching staff recommended that as a baseline people try using their single-agent approach from Project 2 to solve Project 3 and see how far they get before attempting any other new multi-agent approaches. This is where my con from Project 2 affected me - since I didn't spend countless hours making the most efficient agent, the agent from Project 2 was only able to solve 2 layouts in Project 3. I spent the next 3 weeks trying new algorithms (and failing), and then resorting back to trying to salvage my agent from Project 2 to solve at least one more layout so that it wouldn't have such a pathetic performance. I ended up being able to only solve 3 out of 5 layouts with my final implementation, but I wrote what I thought was a high quality paper explaining the whys of my successes and failures. Ultimately, I got very small deductions for not being able to solve the remaining 2 layouts and got a higher score than I did in Project 1. The class everage and median for Project 3 was low (in the 70s), and I think this is due to 2 potential reasons. Either people spent all their time doing experiments but didn't put the same amount of focus into their reports, OR people procrastinated and thought they could spend just a few days before the deadline to get everything done (code and report) and quickly realized that Project 3 is 10x harder than both Project 1 and 2 combined. I personally think that if I had spent more time in Project 2 making the most optimal agent possible, then I would have better experimental success in Project 3 because enhancing a very efficient single-agent approach to work for the multi-agent environment would have shaved off a lot of experimentation time for me.

Final Exam: I needed the high scores from Projects and Homework to save me. I failed this exam so terribly that if you were to look at my performance in relation to the rest of the class for just the exam, it looked like I did not learn a darn thing all semester! I studied and prepared for 7 days prior to the exam, and still "failed". My advice is to not over-prepare by studying the lecture videos - instead, read the relevant chapters from the class textbook and then just take the test. I spent too much time going over lecture videos and not enough time reading the high-level perspectives from the textbook. Also, blind guessing results in points deduction because the final exam is multiple choice with multiple answers for some questions. If you choose 1 wrong choice and 1 right choice for a multiple-answer question, then you get 0 points on that question!

Lecture videos - So awful...this is the first class where I felt like vomiting everytime I started a video to hear the non-stop bantering and joking of the 2 professors who star in the lecture videos. The joking got in the way of me learning anything at all, and they would purposely spend hours deriving something trivial and then wiping the math away to say they deliberately made errors to prove a point. It got to the point where I stopped watching the weekly allotment of videos for 1 month because I couldn't sit through all the unhelpful chatter of the videos. My feedback to the teaching staff was to please just do away with the current lecture videos because they were the worst part of the course (even worse than the experience of taking the final exam).

Teaching staff: This class is so darn RESPONSIVE! This is the first class where the TAs explained their background and hobbies on a Ed post at the beginning of the semester, and it was so nice to know these people were "real people" and not just names on a virtual board that answered questions or graded assignments. Some of the TAs had IMPRESSIVE backgrounds, and I found myself wondering "wow, how high is the caliber of expectations to be able yo qualify as a TA for RL?!". Also, the TAs are hovering around Ed Discussion around the clock, and responses to student questions are posted within a few hours. That is such a crazy difference compared to AI where questions from students could go unanswered for up to 7 days! The TA in charge of Project 3 also spent a lot of time asking for feedback about how to improve Project 3 for the future. The way the TAs helped to run the course is how all classes in OMSCS should run.

Extra Credit: We had an extra credit opportunity this semester, and I did not get it. But if I did, it would have added an extra 10 points to my grade on Project 3. I guess in the end I didn't really need any extra credit because I lost very few points on Project 3 despite dismal experimentation results (solving 3 out of 5 layouts is still just solving 60% of a problem, which is really not something to rave about if I were on a podium).

Conclusion: I learned SO much, and also worked SO much for Project 3 that I need to find a passion project to showcase everything that I can do now with reinforcement learning! Even though I failed the Final Exam, my performance on Homework and Projects was high enough to still result in an A (with the help of the class curve). I had estimated that this would be a B grade class for me since it took so much effort for me to understand the material, and therefore I was pleasantly surprised to see my final grade in the class!

Very difficult, but in the right ways. This class treats you like an adult and puts you in charge of managing your time and research to get to the finish line. Probably the first class in OMSA that fostered a collegial relationship with TAs (10 classes taken).

Challenging but achievable, and very engaging because of this. Just finishing the class won't get you an A, unlike many other OMSA courses. The projects are hard and you'll have to do some research of your own to succeed. But it's a process that will help you learn a lot in very practical ways. If RL is at all interesting to you, don't miss this class - just take it alone or pair it with something easy.

One thing that holds this course back is that there is an amount of disconnection between the lectures and everything else. The lectures follow the mathematical theory behind how the various RL topics work. Then the projects and weekly homework focus on practical applications. In between those two ends, you have the readings -- they should bridge the gap, but often fall short.
The readings consist of the Sutton and Barto book, which I did enjoy, and then research papers. I recommend buying the hard copy of the book because it is foundational and I found it handy to write in the margins and highlight. I'll be returning to it during self-study a lot. The research publications are just okay. Some are far more recent than others, and while the older papers provide some much needed insight into how the RL paradigm has shifted over the past 10+ years, the last third of the class is definitely in need of renovation. For example, the third project focuses on multi-agent reinforcement learning (MARL). It's been said before, but it is an absolute giant of a project that tested me in so many different ways. You would think that multi-agent learning would make up some portion of lectures, right? Negative. The last third of the class lectures focus on a smattering of topics, mostly game theory. While game theory was interesting, the MARL domain is so relative today that it was boggling that it wasn't covered in much detail. That was pushed to a lot of self-study centered around Project 3: we were given a laundry list of papers to read, pick an algorithm(s) to implement, and build a MARL solution to a cooking environment. It didn't help that there was a bug in the environment that I got conflicting responses about what it affected. Overall, this is a difficult, math-heavy course. It is in need of modernization. The lectures are dense, and sometimes confusing, and the interplay between the two lecturers is grating at times. When you just want things explaining simply, it can get frustrating. There needs to be a better bridge between theory and practice. Get ready to spend ungodly amounts of time looking at other resources, like the fantastic David Silver lectures or alternative textbooks. The TAs are great; they're just working within the framework that they've been given. I enjoyed the first two projects, and yes the final exam is very hard. I scored in the 50s and was above the course average so it is universally difficult.

If you were inspired to take reinforcement learning after hearing that it was used to train ChatGPT, avoid this course, because you’ll learn none of it unless you self-study, in which case you might as well not take the module at all. IMO, the short reinforcement learning section in ML4T is more applicable to real-life than the entire 7642 syllabus.

Lectures are absolutely useless except for Final Exam, and maybe HW1: • Charles is fine but he only teaches a negligible proportion of the syllabus • Michael is incomprehensible?? More than once he switches train of thought mid-sentence, and he jumps from concept to concept without any context or explanation. The bantering style of lecturing exacerbates this • Lecturing style aside, the syllabus is so theoretical, it teaches you absolutely nothing about how to implement anything, much less best practices in modern RL. This is a massive wasted opportunity because RL is super interesting (as I found out during self-study)

My recommendation is to read the relevant textbook chapters before watching the lectures, then the lectures might make a bit more sense. The rest of the weekly readings are mostly useless except for the final exam: • Believe it or not, the paper introducing deep RL is not even in the reading list. In fact, I don’t think any deep RL papers are on the reading list • Most readings were authored in the 1990s, i.e. were focused on solving simplistic computer games. It’s hard to see any application to real life • Project 3 has 12 suggested readings in it, all of which are 2016 or later, covering more recent techniques in RL. Why aren’t these in the weekly readings instead?

TAs are insanely helpful & trying their absolute best. I cannot stress this enough. I feel like they themselves know that the course content is useless, but there’s only so much they can do.

Project 3 will have you implementing a multi-agent model (which they don’t teach). As others have already mentioned, this project is an absolute beast, and combining multiple neural nets in 1 model can be very very tricky in PyTorch. Clear all other deliverables ASAP, and get started the moment it’s released.

Finals-wise, know that for multiple semesters, the average grade has been <50%. You’ll need a super deep understanding & memorization of the attributes and properties of the whole syllabus (which is massive). Most of the questions don’t have answers explicitly stated in the lectures or readings. And cheatsheets aren’t allowed.

Super hard course, haha check out my candid review here! https://youtu.be/LEwjFYt4XIQ

I took this course over the summer (doing my first week of homework on a cruise ship in the Mediterranean) with full knowledge that it is a tough class, but observing that the grade distribution for the class in the summer (https://lite.gatech.edu/lite_script/dashboards/grade_distribution.html) is generally about the same as for the fall and spring. Sure enough, this semester the distribution came out to be 41% A, 28% B, 1% C, 2% D, 2% F, 25% W.

I rate this class only a 3 because the lectures are poor, dated, and lacking relevance. The lectures contain many mistakes which aren't all annotated. Many results are presented only at a high level so that it is difficult to understand the meaning or how to practically apply the results. The continual banter, including debates about the meanings of things, adds to confusion. The lectures don't contain any material on policy gradient methods even though these are arguably the most important and commonly applied techniques in modern RL. It is also annoying that project 3 is all about policy gradient multi-agent methods for which the lectures provide no support at all. As other people have, I recommend the David Silver lectures on Youtube.

One unexpected downside of taking this class over the summer was that only 160 students were enrolled so that there was no critical mass for good Slack discussions. Consequently, everyone gave up on Slack and questions posted there went unanswered. You get better support from other students by taking the class in the Fall or Spring. Fortunately, I had manually copied the Slack discussions from the spring into a Word doc to preserve these before I'd lose access. Having these available and searchable was useful.

As other people have said, project 3 is a beast and really represents a capstone project even though the grade allocation is the same as for projects 1 and 2. I finished project 2 and the homeworks early so I'd have 3 weekends on project 3. I also took some days off work, ending up spending about 90 hours on the project.

I was able to significantly beat the baselines on project 3 using CCPPO (following the ray lib example), increasing the size of the central critic NN (because coordinated value estimation is hard), and with lots of reward shaping to incentivize better and more aggressive defense. Reward-shaping is tricky and easily becomes pathological, so this tip may not be much of a tip since it takes a ton of training experimentation to get this working.

Watch the video visualizations generated by the provided Jupyter notebook. The best baseline (3) wins nearly all games in which the agent team starts with the ball (the odd episodes), but ties most games in which they do not (the even episodes). So the best strategy for improving on this may be to find a way to add an aggressive defense and try to steal the ball and also score on the even episodes. I did this mostly by adding reward shaping to incentivize more aggressive defense. I found this wasn't enough with PPO alone: I needed an algorithm with more coordination.

For my setup, I used docker on my Windows 11 host system using an RTX3090 GPU and a CPU with 20 cores and 64gb ram. Lots of people had trouble with cloud setups and some people were hardly able to complete any training at all, spending most of their time troubleshooting problems and complaining about this on Ed. Following a suggestion someone made in Slack the previous semester. I was able to speed up training significantly by increasing the SGD minibatch size and related parameters, and also the learning rate--so that I could complete a 20 million step training run in around 10 hours (instead of 24).

The final exam was an exercise in remembering details from the lectures and two papers (KWIK and Sutton 88). I wish I had made a table of methods/techniques/algorithms and their properties and memorizing this as someone suggested in another review. I studied by rewatching all of the lectures and making notes, and studying the posted student answers to study questions, finding other sources for answers when those were weak. This study was enough to get me a score above the upper quartile, but I think I could have done better with added table memorization. The mean score was 46, median 48, and upper quartile 56.

I ended up with an A, scoring (94,95,97) on the projects, and a 61 on the final.

Reinforcement Learning Review

Reinforcement Learning (RL) is a fascinating class, but I have mixed feelings now that the course has concluded. On the plus side, RL feels like it just might be the next "big thing." The field is a fascinating fusion of classical computer science, statistics, cognitive and neuroscience, and mathematics. Before taking this class and having taken the AI, ML4T, and ML courses, this class felt like what AI should be, or at least, what I expected AI and machine learning to be like; that is, an agent that explores an environment, acquiring feedback and evaluating its progress towards some goal at each step along the way. The agent-centric aspect of RL distinguishes it from other areas of AI and ML, and it is a novel way to think about problems in this space.

On the downside, RL is still very theoretical and has yet to gain traction in industry, as it is not applicable to most commercial use cases. The domains where RL has been successful have been confined mainly to robotics and gaming - although I have heard that large language models like chatGPT used specific RL techniques. RL is an active area of research, and I hope it will find broader appeal as success in new domains and use cases accrue. RL is definitely something to keep your eye on if you are in the Machine Learning or Interactive Intelligence specializations.

There are three projects (45%), six homeworks (30%), and a final exam (25%) that make up the final grade. There are course lectures, office hours, and a lot of readings. This class is demanding, and the projects require a lot of time. In terms of effort and workload, it is up there with ML and AI, and I easily spent at least 40-60 hours on each project. Preparing for the final was chaotic, and I was "burned out" after P3. The instructors included a series of game theory lectures at the end. Including this material was a little puzzling, at least to me, since none of the homework or assignments involved game theory. However, conceptually there is an overlap between game theory, a la John Nash's "a beautiful mind" equations, and multi-agent RL.

I finished with a final grade of 74%, which was a B. I did below average on the first project and the exam and was at or above the median for the rest. With a little effort, you can get 100s on all HWs - the first one is the hardest, and they get progressively easier and less time-consuming. Understand how to write a good report for the projects - results are less important than showcasing your understanding. For the final, don't overthink it - know the big picture. The average/mean was around 46-48% (I scored a 42%), but it is easy to go down the wrong path in studying since there is so much material. I certainly did.

Lectures and Class Material The lectures for this class could be better. I did not enjoy them and stopped watching them halfway thru. Some are good, while other modules were just confusing. The focus was mainly on mathematical proofs and lacked a coherent narrative. I don't think the course modules were helpful for the final - they went into too much detail.

I watched most of the David Silver lectures, which were terrific. The David Silver lectures were coherent and well-paced, each one building on the previous one in a coordinated fashion. The course lectures were less connected and felt arbitrarily stitched together in comparison. Often it was unclear why a proof or equation was being introduced or how it fit into a larger RL context. I often felt "lost in the details." Combing both the Silver lectures and the course lectures is likely the best approach since the Silver lectures provide a layout while using the course lectures to fill in the mathematical details whenever needed - but who has the time to watch so many videos?

Many of the readings were excellent. After you bang your head for a while on an academic paper, it does eventually begin to make sense. I learned quite a bit from the papers and the book. I didn't read everything, and some readings were harder to grasp or seemed to fringe, but the course designers did an excellent job selecting the essential readings in the field. The DQN journal articles and the Sutton paper on TD learning are foundational readings. The Barto and Sutton text was also quite good, at least the early chapters (i.e., 1 thru 7) I had read.

The TAs for the class were talented and generally impressive, particularly the lead instructors. They were very positive and supportive, but your YMMV for the office hours. The TAs preferred facilitating class discussions, letting things flow organically, instead of a more structured guided office hour. Often, I felt they were reluctant to provide solid answers or advice, instead encouraging students to give it their best shot and not be afraid to take some liberties or explore the topic independently.

Ed discussions were good - lots of class participation but not too much direct involvement from the TAs. Again, they preferred to stay in the background and referee more than actively participate. But they did help direct and clarify topics when necessary.

Overall a great class and an important and emerging area for those interested in AI. It gets a bit theoretical at times, but the projects were fantastic and provided the necessary hands-on RL experience I was looking for. If you want to prepare for the class, study a bit of dynamic programming and read up on and understand the grid-world MDP problem using value and policy iteration. Learning the basics of PyTorch will come in handy. Understand gradient functions (or read Sutton's Temporal Difference Learning paper). Also, I would learn how to spin up and configure a virtual machine using one of the major cloud providers - that may come in handy for P3 if you don't own a souped-up home machine. Finally, if you have a lot of spare time, watch David Silver's lectures, read his DQN paper, and the first 5 or 6 chapters of Sutton.

Good luck.

I won't spend a lot of time re-iterating stuff that other people already accurately described. TAs good (and very nice, can't praise them enough), assignments very easy, lots of reading, projects generally hard. I'll just provide some more insight into project 3. You could probably submit this project as the capstone project for DL, the one that you have multiple months to do and must complete with a group, and I'm very sure the TAs would OK it; it's that complex. If you aren't intimately familiar with RLLib, I would strongly, strongly recommend spending time earlier in the semester looking through the source code and the examples it contains. Learn how to modify the structure of a neural network or the overall behavior of one of the pre-existing algorithms. That was probably what most of us spent most of our time doing, and it's a real shame, because that cuts into valuable experimentation time. Also, have a fairly modern 6 (really 8) core CPU and 32 GB of RAM MINIMUM if you don't want to be renting a VM from someone. I know everyone always says "start early," which is often not even an option for a lot of people, but really do clear some time for this one, if possible.

That said, I think grading was more lenient on P3 than the other projects. The median ended up higher, while the mean was lower. I did quite well with, and I can't stress this enough, garbage results. They give a lot of points for demonstrating one's understanding of the principles behind multi-agent RL, and for thoroughly explaining why one's agents performed in the manner they did.

The final was extremely dumb. Assuming they keep the multiple-choice format, my recommendation would would be to simply re-watch the lectures, making note of any time they define something or describe the performance characteristics of some algorithm. (how well it does/what kind of environments it can operate in/what information it needs in order to work). You should be able to beat the curve with that alone. They provided a lot of study questions in Ed, but I wish I spent less time on them, as rote memorization would have almost certainly earned me a better score.

I hope they are able to improve/streamline P3, because it seems like a project with a lot of potential that was simply too difficult and complicated for most people to get working and then have time to conduct detailed experimentation in a timeframe of 3-4 weeks (Again, start early if you can. Neglect your children if you have to!). I think the idea was to give people this powerful RL framework with which would allow them to spend less time implementing and more time experimenting, but I think the opposite ended up happening.

This class is pretty great and feels like a grad class. The homeworks are easy expect HW1 which I found to be pretty stressful and difficult. I will say the TAs were pretty amazing throughout the entire class and were very supportive. A lot of students too were very involved which made for a fun time as a fellow student.

The projects are all good:

Project 1- Pretty easy to follow pseudo code from the book and other students discuss little tricks they had to do to get the exact results on here as well as in your slack.

Project 2- Fun project, I used a DQN. Feels sort of out of left field to be learning some DL, but once you get past that hump its not too hard. Still a lot of work and time to train and tune, but its pretty fun to do and overall enjoyable.

Project 3- New project as of the last year or two. Really really really hard. I was not able to beat the baselines we had to train against despite trying all sorts of strategies. I trained locally, and you need at least 18gbs of memory to train your model. Spent lots of time and effort on this and felt bad to not get good results. Most of my time was spent scouring github libraries for MARL to try and understand how to implement a decent agent and make it work in RLLIB.

All of these projects require you to start the minute they open otherwise your chances of finishing them and getting a good grade is very slim.

Exam- The exam was total BS. Some of the questions were about an algorithm that we never talked about. The review questions did not really prepare you. The TAs were suppose to post answers but they never did. While I liked the idea of us learning off each other, please also post the answers!

I also tried to take it at my parents house while on vacation so I bombed it as I rushed through it. Oh well, my exam grade doesn't reflect how much I learned. I had pretty much aced every single homework and project and still did so incredibly poor on the final.

Lectures- I didn't mind the jokey tone that smoov and curly took in this one, but a few of the topics in the middle where just like straight math proofs with no real explanation of how they connect to the topics. It was fine, and I would never say the lectures are bad, but I wouldn't say they blew me away either. As others have mentioned David Silver's lectures are much better and I would recommend watching those. Like a good grad class, you will need to do a lot of learning outside of the provided materials. In this class it's cool to read papers that are cutting edge and interesting, and be able to understand them.

Overall- TAs were really good. TJ and Jake in particular (as well as others!) who ran our office hours were always very supportive. It's a fun, challenging class that despite the final I'd take again.

This is probably the most difficult course I've taken in my entire academic career. Thankfully, the difficulty isn't artificial and doesn't come from needless busywork, but rather from the intensity of the material itself.

The course is broken down into six homeworks, three projects, and one exam. The six homeworks are auto-graded in Gradescope and cover the material from the lectures/readings and help give a good foundation for thinking about the projects. The projects are interesting in that you are not graded on your implementation at all, but rather on your interpretation of your results, and the exam is a multiple choice exam that covers all of the material in the course.

The homeworks don't have anything of note in them that makes them stand out - they're fun and interesting, but a bit foundational. Once you crack the main idea of the problem, the implementation is generally straightforward.

The projects are real work. You'll end up writing a ton of code and submitting it, but it's basically only checked to make sure it runs and isn't a copy of another implementation. The effort comes from writing the reports and coming up with justifications for how you approached the problem, and (more importantly) showing that you understood the problem. My undergraduate degree was a writing degree, and these projects felt like a real synthesis of all of my skills. I enjoyed the process, though I generally spent a single day writing the report, which I wouldn't recommend unless you have experience writing a 15 page paper in 10 hours. It melted me and affected basically everything else in my life.

One thing to mention about the projects is that the work starts hard and gets mind-blowingly difficult. The third project leans heavily on compute power and understanding the RLlib abstraction, and if you wander off in the woods for any amount of time, you will likely not be able to solve the problem. I was not able to beat the benchmarks, mainly because I had a significant amount of difficulty understanding the RLlib abstraction and figuring out how to write and debug my own code. I spent about 100 hours on the project, but beat the median only by a little bit, mainly because I wasn't able to come up with excellent points because I didn't fully understand what was going on. It was humbling, but (most importantly), it made me thirsty for more.

The final, though, was awful. They give you a study guide with open-ended problems, but don't give you the answers they think are right. Given that the students argue over definitional true/false questions, this makes it really difficult to study and know if you are on the right track. Couple that with the fact that the exam doesn't reflect what was in those study guides, and the average for the final was about a 45 and the highest grade was an 80.

I don't think the exam was difficult enough to warrant this, and the main problem with it was the lack of getting the students prepared. I think they can solve this by offering gigantic question banks they draw from for the exam, and offering guidance on how to approach the problems.

Outside of the exam, this course was excellent and felt like a real graduate course (instead of undergraduate+, like GIOS and ML4T) as the focus was on interpretation and analysis rather than code.

I would recommend this course, but understand there are points where you're likely going to want to give up and/or cry and/or hallucinate from lack of sleep.

Heavy workload if taking another course, TAs are engaged and not snarky, 2 office hours sessions a week, covers the classics, but also gets into the newer DRL stuff. Lots of papers to read and process, which is fun and feels more like a graduate class than some other courses.

Assignment grading is slow, which seems by design (given there is no shortage of TAs). You just have to have confidence that you're doing the right things.

It is the best class that I took in OMSCS so far. You definitely learn a lot. The topics are eye opening. It is harder than AI, ML and DL imo.

Overall - It is the most insightful and difficult course I've taken so far (KBIA, ML4T, AI4R), which is to be expected from the reviews. Loved project 1 and project 2, reading papers and trying to implement TD(lambda) and DQN was super fun!

Project 3, I have no idea what I was doing. I feel like it was such a huge step up from P1 and P2, and I was not making any progress in any meaningful way. rllib is a monster of abstraction which is pretty much the crux of the project. I would recommend anyone taking this course in the future, try to implement P2 in rllib after doing it in just PyTorch. Also, you'll need a machine that is good enough the train for P3. I know the TAs did not explicitly say this but if you can't train 1 million timesteps within 1 hour, you need a better/faster machine. This is the ONLY way to know that what you are doing is making an impact on the agent.

Homework was trivial, given that this is not your python course, you'll be fine.

The Final was brutal. As stated in various posts, the average was in the 40s. I rewatched the entire lecture videos, read various sections of the book, and did the practice exam twice. After all of that, I still got the average.

Time spent per week: 20+

Conclusion: ML specialization, this is a must. Even if ML is not your specialization, you might as well take it. Everything else will be a walk in the park.

Very interesting and engaging material, but extremely time consuming course. Would not recommend taking another course at the same time if you're a part time student. I found the lectures in the course material hard to follow and keep up with, the David Silver lectures on YouTube cover a good chunk of the same material and I found those much easier to understand. The final project is very computationally demanding, you should have a decent GPU, a high end CPU, and at least 64GB of RAM available you if want to train on your own system. After dedicating over 90% of my free time after my job to this course, I was able to pull off a final grade of B.

It was my 10th course in OMSCS. Some feedback may not contain too many details due to the large class size. Nonetheless, this course is well-managed by TAs.

1. Homework. HW1-6s are simple and easy to implement. HW5-6 may require additional readings regarding KWIK and game theory.

2. Projects. They are most time-consuming. I didn't work well in Project 1 but achieved 100 in Project 2 and Project 3. The key is to explain your methods, experiments, and analysis thoroughly. That's the reason why TAs recommend IEEE templates. The main structure of the analysis is describing phenomena -- providing hypotheses -- validating your conclusion theoretically and experimentally. IEEE papers usually separate experiments and results. But I recommend combining them because of the 5-pages limitation. It is also convenient to read and prune. Besides, I strongly recommend following Kaiming He's writing style in the Masked Autoencoders paper. You can highlight or bold key concepts at the beginning of paragraphs.

Project 1 focuses on TD-lambda. You need to introduce TD methods, replicate experiments and provide your own analysis. Showing your understanding of each experiment is the crux. But I couldn't realize it until submission.

Project 2 is about deep reinforcement learning, specifically, the DQN series. Two experiments include the training phase and hyperparameter-tuning. It is easy to talk a lot about hyperparameters, like model architecture, learning rate, discount factor, and epsilon-greedy. However, don't forget to describe your training result and analysis in detail.

Project 3 introduces multi-agent reinforcement learning and plays a Google research football game. The provided baselines depend on PPO. PPO is such a robust algorithm that you may fail to achieve some improvements. So I suggest using QMIX, MAPPO, or other SOTA methods directly because each trial may take several days (So sorry that I was not able to reply to some questions in Edstem). I trained MAPPO on Google cloud and paid 50 dollars plus 300 free credits. Although it is manageable to use GCP with only the free credits, the high computing requirements (especially CPUs) cause much trouble in coding and debugging. For RLlib-based implementation, I recommend seeing some GitHub extensions, such as epymarl and MARLlib. As for the report, except for win-rate improvements, you need to select or customize parameters to display team coordination.

3. Exam. Important Relevance: Sutton & Barto's book > David Silver's slides > lecture videos. Although lecture videos don't cover all concepts, you can still achieve an above-average grade by only watching them.

Overall, I think this course is a real graduate-level course in OMSCS. If you have some questions, don't be afraid to ask TAs on Edstem and watch the office hours.

Background: Non-CS, process engineering Prior Courses: KBAI (A), ML4T (A)

I wanted to get into ML but couldn't get past the waiting list. So I took this first.

My current standing is: Project 1: .5 STD above median Project 2: (probably) .5-1.0 STD below median Project 3: not due for 5 weeks HWs: All 100% Final: ???

ML isn't necessary for taking RL.

ML4T helped in some respects because I already had a general idea of what 'learning' was, what is a Markov Decision Process, how to deal with NumPy, and how to do Q-learning. Q-learning is one of the HW assignments and is a general idea behind DQN which has many use for project II. ML4T was an advantage, but I don't think it's necessary.

This course requires a LOT of reading. This course for reading is as KBAI is for writing. The lectures are really thick with theory and math as well. You will spend more time reading research papers and textbooks than writing code. If you are a theoretically inclined person, this course will suit your strengths. If not, you will struggle. I haven't had to 'use' or derive any math yet, but you need to follow to understand.

I enjoy the projects and the homework assignments are fun little challenges. I have yet to start on the final project or take the final exam, but the final project looks like a real challenge.

My advice is to re-read everything after you get about halfway into the course. So much of the readings were illegible to me until after completing project I. Then, what was previously alien, became plain English. In addition to that, I will watch the David Silver lectures. I would expect to do well on the final with this strategy.

Just watch David Sliver's lectures. Ambiguous instructions in projects. No rubrics. Grading depends on TA's mood.

Avoid this course if you have better options. UCL or Edinburgh RL course is much better.

The video quality is really low. You have to learn by yourself. The project sucks. The project prescrption is much less to be detailed. You don't know the points for each part. The grade you got would be unbelievably low. If you argue, you would be deducted 10 points. Seemed decided on your luck to meet a good TA. Totally ruined my semester and good mood. QUit.

Not much to say here that hasn't been said in previous reviews: Taking this course in the summer is a behemoth and I do not recommend anyone take this course with another course in the summer (or even in the fall and spring). Not because of most of the course, but because of the final project.

(6) HW's that accout for 30% of the grade and start with a first really difficult HW and get progressively easier as the course goes on. HW 5 and 6 are an absolute joke compared to HW 1. Aim to get 100's on all HW's, which is very doable.

(3) Projects, 45% of the grade (15 each). The best and most rewarding part of the class IMO. 1st project on re-proving TD learning, which is quite easy to do but doubly easy to get a bad grade on since TA's are hit-and-miss on what they take points off for. Highly recommend focusing on theory and analysis of data for the reports and little on the actual implementation. The TA's want to know what you learned from the results and implementation, and why you implemented it; not how you implemented it. Embody that in your report and a better grade will follow. The second project is the OpenAI Lunar Lander which is really fun and allows you to use whichever agent and algorithm of your choosing. The perfect project, harder than the first but very rewarding in terms of knowledge and application if done properly. The third project is an absolute behemoth. Prepare yourselves for hours to just get acquainted with the project. Start as soon as possible. Read as much literature as possible. Fully understand the provided code and objective before throwing stuff against the wall and seeing what sticks. I highly recommend approaching the problem like a scientific experiment, and write the report as such. Have a hypothesis on why a particular technique will work, explain why you think it will work, then implement it and gather good results. Write the report detailing this hypothesis - theory -(small bit on) implementation - (large chunk on) analysis of results. My report followed this format and, even though the agent did not improve upon baseline, since my report showed clear understanding of the literature and results I still received an A. The average is very low for this project since most people just use an inbuilt algorithm and see if it works without really understanding what it is they are doing or why they are doing it. Highly recommend going the reward-shaping route (not based on my project but on high-performing others). (1) 25% Exam which is honestly a gotchya. They should really change the exam as this has been a real black eye for the class for as long as reviews have been here. Nothing new, 50% avg grade. Know the material and you should do well enough to beat the average, but nothing will properly prepare you for the exam since it is all gotchya's.

ML really put me through the wringer. I expected RL to be less stressful because the assignments are less open-ended. It was, in much the same way that being punched in the stomach is less stressful than being kicked in the crotch.

The projects are great; I learned a ton and I'm excited to go do more experimentation on my own. In each project, it felt like there was some mysterious thing that I never figured out standing between me and the expected results, hence the stress. Fortunately, this doesn't prevent you from doing well in the class. Scoring is less harsh than ML but perhaps more harsh than most courses; correspondingly, the grading curve is less extreme than ML but still unusually generous (the cutoff for an A during this term seemed to be somewhere a little below 80%).

The TA team was fantastic. I think I got a lot out of both the textbook and the (large list of) assigned papers. The lectures were hit-and-miss - not as great as the ML lectures (but lectures are my least favorite way to learn anyway).

I used about $430 worth of Google Cloud resources for project 3. Currently Google gives you $300 for free if it's your first time using it; I paid the rest out of pocket. I'm not sure this was necessary, but I think it made things easier for me. GPUs aren't important for the project, but having lots of processor cores is.

Advice:

• Don't take it in summer. The class has a ton of reading, and the projects involve experiments that take significant time to run, so spreading things out more would have been really nice. In particular, having only 2 weeks for project 2 sucked.
• On project 3, if you aren't seeing good results quickly, try several different approaches. I think I wasted too much time trying to figure out what was going wrong with one particular approach, when other options would have yielded good results more easily.

Conclusion: This is a course with both significant advantages and disadvantages. Be very careful of taking it during a summer session.

Background: Quant, with a B.S. in Physics and M.S. in Financial Engineering.

Final grade: A

Pros

• Extensive coverage, including some new research in RL field.
• Course is taught in a relatively clear way.
• Final grade is curved generously.
• You actually do not need that much maths.

Cons

• The last project is way too time-consuming and requires A LOT OF computational resources. Better have a nice GPU before start.
• The final exam does not have a lot to do with the lectures. It covers some readings, and details in the homework/projects as well. You can easily spend plenty of time preparing for it and get a low grade (as I did :D)
• The professors talk about too many topics, but do not express some fundamental concepts in a clear way.

Somehow I found this class more straightforward than Machine Learning, despite it being a similar format. I scored 100% on all projects/homeworks, and 61% on the final. I really enjoyed it, especially the projects.

The homeworks are guaranteed points, which you will need for your grade to survive the harder assignments. The first couple of them are the hardest, and the last 2-3 are trivial.

There are people (not me) who got 100% on projects with only 3.5/5 pages. The most important thing is touching on every part of the assignment prompt in some depth. The grading is both harsher than you expect if you don’t do this, and more lenient than you expect if you do.

The final exam is total BS. It’s all gotchas. The highest score was 78%, and the mean (excluding 0’s) was slightly over 50%. You must do well on the homeworks and projects to survive the hit from the final exam. They release a study guide ahead of time, but it does not cover everything. Official statements were that the exam would only cover lectures. This was not true. There were some concepts on the Final that were only covered in the readings.

For Project 2, DO NOT TUNE BY HAND. Use a tuning library like optuna. This will save you so much personal time, even if not compute time.

The best way to prepare is to get familiar with Ray and Rllib libraries at least several weeks before the release of Project 3. My weekly workload was 10 hours or even less some weeks, 30 hours in the last week of Project 2, and 50+ hours per week throughout the whole Project 3 window.

This is a great course. The TAs are particularly great; I have never seen such an engaging TA team. (Usually the TAs are not responding to questions in EdStem or Piazza, and I hardly seen a TA in Office Hours. But in this course, the TAs usually respond to your EdStem question on the same day, and you can even see many of them actively explaining things in Office Hours, which really surprised me a lot.)

The Assignments are usually easy, except Assignment 1, which is super hard, in particular if you want to get 100% correct; the key to Assignment 1 is that you have to rethink and simplify your model until it trims down to just a table or list with only a few columns that can represent the infinite games for that particular problem. It is not easy so please start early. The other assignments, particularly the last two, are easy in order to give your more project time (I think).

Project 3 (i.e. final) is super hard. Projects 1 and 2 are OK. Project 1 is about Sutton famous TD-learning paper and represents a fair amount of work. Project 2 is about a game in Google AI Gym and is also fair; I used DQN and got good experiment results, though the project report did not yield my expected score (which I think TA should give more textual review on my report content, but anyways). Unfortunately, Project 3 is a nightmare! (I suggested TAs to open this final Project 3 a few weeks earlier in the next semester.) Project 3 is so diffcult because it asks you to use RLlib (which is an open source tool for deep reinforcement learning) and you have to work on Dec-POMDP, which is perhaps the most difficult topic in reinforcement learning. TAs already simplified the problem, but still it is hard and takes very long to train. The problems that I encountered in Project 3 includes:

• No good environment for problem solving: TA suggested either Docker, native setup, or Colab, but in fact all of them have problems.

  • Docker: It is easy to deploy/setup for running, but I cannot scale it up since Docker network sucks on Mac and Windows. I wanted to setup RLlib cluster so that RLlib can train with my 3 old machines, but since the RLlib node is running inside a Docker container, the RLlib head node cannot talk with the other RLlib nodes via the Docker bridge/host network (which only works on Linux but not on Mac or Windows; a problem haunting in Docker for years). Docker is great but has so many network problems on desktops (perhaps Docker network options are designed for servers), therefore Docker is a limitation to upscaling of training.
  • Native setup: It has so many compat problems with Python / make / C and everything. My recommendation is: don't try it, for the sake of limited project time.
  • Colab: It is good for GPU but not good for CPU, since Google only offers 2 vCPUs. This project needs a GPU and many CPUs (to rollout the samples to environment for "bootstraping", I guess), so even 24 or 48 CPUs will not be too many.
  • Google Cloud Platform GCP or Amazon Cloud AWS: Personally I think this is the best option, since you can create a VM with lots of CPUs and one high-end GPU to run your training efficiently. Nevertheless, this comes with a cost (USD 300 for me), which is equal to 1/3 of the course fee! Note that the TAs already recommend us NOT to spend money on cloud platform to get a good grade, and they also promised that they will only look at my analysis writing on project report instead of my source code performance; nevertheless, I admit I am spiritually weak so I cannot tolerate 50 hours of turn-around time for experiment or hyperparameter tuning. Having said that, if you are a new subscriber to GCP, Google may grant you a USD 300 credit (which perhaps expires in months) which you may use for this project.

• RLlib: It is a defacto standard for deep reinforcement learning, but it is still in active development, so many features are experimental. Limitations are not well documented. You have to keep the rollout size low or the multi-CPU rollout process will consume your expensive GCP 300GB memory very quickly. The framework is flexible but is also complex, so even if experiments are achievable, they need much time to work with. QMIX is achievable but you have to pay attention with state action grouping and python/string/dict issues. In order to make QMIX works, I modified RLlib source codes and patched them into the Docker container image, which requires a long investigation. So please count this task into your project schedule as well.

• 60 hours for each training: I am using a Macbook 2018 Pro with 6-core CPU and it took 60 hours to finish training just one default PPO agent. Please note if you design your own RL agent then it may converge poorly due to incorrect design or bugs, so waiting 60 hours for deciding another retry is just too luxurious for a 4-week project. That was why I switched to GCP, which accelerates the training to 10 hours (at the cost of USD 300), yet I still have to permaturely terminate many trainings or hyperparameter tunings in order to speed things up.

Understanding Dec-POMDP and relevant algorithms are relatively easy, when compared to all these hardware/platform issues. I did QMIX and even modified the algorithm a bit, and I think QMIX is achievable. Many others did PPO with center-critic, since PPO is kind of like RLlib preferred algorithm, though I found it hard to hack around the RLlib framework to sneak in the center-critic. On the other hand, since PyTorch and deep learning is a must for using RLlib in project 3, I would also recommend you to take the Deep Learning course before you take this course.

Exam is hard since it contributes to 25% of final grade (while project 3 only contributes to 15%) and it requires you to remember lots of material. Frankly I like Prof. Isbell and Littman, and their worked exmaples in the video lectures are really good for understanding, but the videos contain lots of irrelevant inside jokes, which is not good for memorization. I also could not find any presentation slide pack. So I recommend you to re-watch all videos (incl. all Office Hours, which explains Sutton 88 and KWIK) and write down all equations in clear text (in Microsoft Words) with standardized symbols, so that you can remember all equations or even cross reference them. I also recommend you to download all videos (either via UdaCity or via a browswer plugin) so that you can rewatch them at 2x speed on any device without network latency. Also the TAs are so kind to post a list of review questions on EdStem; answering all of them should help you get a pass (though not necessarily a high score). That being said, I did not score well in exam; I score 60, while the top score was ~80 and the average was ~45.... Now you see how difficult it is. (The exam choices are ambiguous and your score can easily be deducted by a few missing ticks).

Prof Littman is good; he said he is not officially one of the course instructor, yet he allocated chat time for us on Office Hours. He is also very responsive to our review questions when we are preparing the exam. Prof Isbell is diffcult to be commented, since I never saw him anywhere on EdStem or Office Hours, but he is busy working as the Dean of Computing College, and he did arrange Prof Littman to chat with us. On the other hand, I have to appreciate the TAs again, since they are really the best TAs that I have ever seen in my OMSCS journey, where I already have taken 8 courses.

Final verdict: It is a contemporary course, despite that Bellman equation was invented 60 years ago. Before I took this course I thought I will be taught only the classic theories, like dynamic programming or so. It is until the last project that I know deep reinforcement learning can be the hottest topic, incl. AlphaGo and others. The amount of things that you can play with Deep RL is tremendous. (Though Prof Littman mentioned that many students turned out to use algorithms other than RL to solve specific problems.) Overall, this is a good course. Thank you TAs!!

WOW! This is an amazing class - both challenging and exciting at the same time.

This was my 9th course in OMSA. I have a non-cs background and I took this class because I really enjoyed CDA.

I think the OMSA C-track experience would have been incomplete without taking this course. This course has amazing content and really awesome projects. The only pre-requisite is having enough time.

Assignments: They are really easy if you understand the concepts. No assignment is more than 20lines of code. Try to finish them as soon as you can so that you can spend more time on the project. Scoring full marks is really easy so make sure you do it.

Project: Project 1 and 2 are way easier than project 3. Hence, it is important to do well in the first two of them. The trick is to 'think and communicate'. I would implement the required algorithm a week before it is due and spend the week thinking about the results I got, conduct additional experiments and write all the work in an orderly manner. For Project 3, my only advice is to start early. The detailed reviews of the past students were a huge help. I got a near perfect score 299/300 on the projects.

Exam: I found Exams to be the most difficult part because there is so much content to cover. Also, the multiple answers add to the difficulty. Only way to prepare is to consistently spend 1-2hrs a week (which I did not do) and then spend the last three days revising everything. I re-watched Prof David Silvers first five videos before the exams.

I really enjoyed this course. TAs are awesome and Prof Littman is very active on Ed. We also had a special OH by Prof Littman (Yeah we were lucky ducks :))

Overall: This was a very fun but demanding class. Fortunately, grades are heavily curved so ~80 probably gets you an A. I had an 88 final grade and got an A. Lectures were OK. I'm not a fan of Professor Isbell and Littman going back and forth making bad jokes, surprisingly, because I was fine with CV lectures' bad dad jokes. Make sure to watch David Silver's RL lectures, they really helped me understand the concepts.

• HWs: The HWs start off being "hard" and then get "easier". What I mean by that is that RL is a whole new way of thinking and takes some time to adjust. The first HW isn't actually hard but takes everyone the longest because they're just not used to it. The HWs are the best way to boost your overall grade. It is entirely on gradescope and the TAs are really helpful for answering questions both on Ed and during office hours. So make sure to get 100s on these.

• Projects: The projects are the hardest part of this class. During the week before the projects are due, I would often spend ~30 hours working on the projects. This is because they are both hard and writing the paper is a lot of work.

• Project 1 is about fully understanding temporal difference learning paper by Sutton and replicating the graphs. This project isn't particularly fun since you have to replicate a paper, but I appreciated this assignment later in the course, because TD learning is fundamental in a lot of RL algorithms.

• Project 2 is a really fun project, I enjoyed it a lot. You are basically trying to beat a game using RL algorithms. Most people use the DQN algorithm as it is really popular. There is a good amount of hyperparameter tuning though, so try to make each run as fast as you can to do additional tuning. Heres a hint, gamma will be your most important hyperparameter.

• Project 3 is a new project. The old project, I believe is replicating another paper on multi-agent RL. The new project involves trying lots of different multi-agent RL algos and other aspects specific to multi-agent. Since this was the first semester they did this project, there were a lot of bumps that hopefully they improve on in the future. If I had to redo the project, I would probably get started on the project earlier and get as much computing resources I can (without spending too much money). I wish they gave us another week or two to do this project, because it felt a bit rushed with only 3 weeks. Overall, this was a fun project but I was extremely stressed doing this project. Luckily, the grading on this project was fairly lenient.

• Some advice for writing the papers: a. Order of importance: Analysis/Results -> Methods/Solution Setup -> everything else. The ultimate goal of the paper is to demonstrate to the TAs you understand what you were doing and why you were doing it, and that primarily shines through in the analysis/results and methods section. b. Use the IEEE latex format they recommend. It is more space efficient and make your life easier to fit the 5 page constraint. c. Spend some time talking about pitfalls and obstacles but not too much as it can bog things down.

• Exam: The exam was my least favorite part of the class. It makes up 25% of the grade, but a lot of the questions are vague/ambiguous and based on small facts the professors might have mentioned in passing. I rewatched the lectures at 2x speed right before the exams and got a 62/100. I wish they would either lower the grade percentage or make the exam test more on general principles in RL instead of obscure facts.

My Background: This was my first semester in OMSCS and I took RL and RAIT. My prior ML/DL experience was useful for RL, especially the projects.

Here's my review of the course content:

1. The video lectures are great overall in terms of breadth but cover some outdated or seemingly not very useful topics (like averagers in module 8, Bayesian RL in module 9). Nevertheless, the breadth of video lectures is simply amazing - covering a wide range of topics (like RL algorithmic analysis, reward shaping, exploration-exploitation tradeoff in bandits and MDPs, POMDPs, Options, Game Theory, multi-agent coordination-communication-coaching) with great clarity. The instructors also do not shy away from presenting the necessary math and proofs, which is great for a Masters class. But, policy gradient methods (which are driving current RL research) are not covered in the videos. Also, basic RL concepts (like Policy Iteration, Value Iteration, Prediction vs Control, On vs Off Policy Learning, Online vs Offline algorithms, Planning vs Learning, etc.) are either not emphasised much or not explained properly in the videos. For this, I would advise what I did before the semester started: watch at least the first seven RL lectures by David Silver on Youtube and take notes since these will prove very useful for the projects and final exam. Most of the material in David Silver's lectures is directly from Sutton & Barto textbook, and thus, it also helps cover some of the assigned readings from the textbook.

2. Homeworks: Fairly easy to get 100% except for HW1. OH discussions are useful in solving HWs.

3. Project 1 (Sutton 1988 replication): Fairly straightforward. To replicate paper's graphs closely, use 5 sequences in a set instead of the 10 mentioned in the paper - this trick worked for most students. My report consisted of introduction, env description, TD-lambda desc, experiments (with pseudo-code), and analysis. For writing analysis, David silver lecture is very useful. For all reports, I would suggest to use Latex and have 2+ pages for analysis since the bulk of the grade lies in showing your understanding. Remember to mention the pitfalls encountered in the project (make up some pitfall if you did not encounter any for such a simple project) or else you will be penalised some points irrespective of how good your analysis is - this is what happened to me - I ended up with 94/100.

4. Project 2 (Lunar Lander): Most students use DQN. DQN implementation is not very difficult if you know PyTorch. My report consisted of introduction, env description, desc of Q-Learning, Deep Q-learning (NN architecture, experience replay, target network, exploration-exploitation tradeoff), DQN pseudo-code, experiments, analysis, pitfalls and future work. Hyperparameters experimented with were hidden layer size, num of hidden layers, gamma, alpha, epsilon_start, epsilon_final, epsilon_decay. I also performed ablations by removing experience replay and target network. I got 100/100 since I had done extensive experimentation and also analysed all the hyperparams' results. Choose gamma wisely (very close to 1) since it can make or break training.

5. Project 3 (Football): This was the first semester for this new project. It is a great research project since the students need to figure out everything from the algorithm to the evaluation metrics. It requires reading quite a few of papers. Students tried methods like QMIX, MADDPG, COMA, centralized critic PPO, reward shaping, curriculum learning, feature engg, etc. I tried QMIX and spent a lot of time getting it to train successfully but it didn't. Most successful methods seemed to be centralized critic policy-gradient methods (like centralised critic PPO) and I would suggest trying these straightaway instead of wasting time on QMIX. My report consisted of introduction, football env desc, Dec-POMDP desc, QMIX and pseudo-code, policy-gradients' brief desc (since baselines were trained using PPO), experiments, analysis, pitfalls and future work. I was able to show using metrics that QMIX improves coordination although not the win rates. Eventually, because of clear explanation, well articulated experiments, nicely explained motivation and challenges, and choice of effective metrics for analysis, I managed to score 93/100 - This is consistent with what the TAs stated about the project - You can write a solid paper even if you cannot beat the bselines' win rates. Two major downsides of this project are machine setup (definitely use docker if possible since it simplifies a lot of things - I setup the native environment (to have access to more RAM compared to docker setup) by manually installing all required packages only to find python version incompatible with the provided checkpoints and ultimately had to also setup docker) and long training times (takes at least a day on the very best of machines to perform a single run).

6. Final Exam: Make great notes while watching the videos (note each piece of detail carefully) and study it thoroughly before the exam to score high on the exam. I did not find any paper / textbook readings useful for the exam though I would suggest glancing over two of them (Sutton 1988 - Project 1 and KWIK - HW5). Also, David Silver slides / lecture notes are a great refresher for the early part of the course and prove helpful for scoring well in the exam. By following this process, I ended up with 88/100.

Final Grade: 95/100.

Finally, TAs are amazing and conduct the course smoothly although a few appearances by the instructor in the OHs would be definitely appreciated.

It's quite difficult. Especially the theoretical part of the RL's (convergence, etc) are well covered. Project 3 requires a decent amount of RAM and patience because the training takes a while.

Okay class. I am not a fan of the lectures but the TA's do a great job of helping people out when they have questions. The homework assignments are fairly straightforward and if you follow the classes pace you're pretty much able to do them when they come out.

There are three big projects in the class and each one is a monolith requiring a significant amount of time to pick away at. This is the area where I feel the class is both the most challenging but also immensely satisfying once concluded.

The only big complaint I have is that for some reason they don't release all the assignments and projects at the beginning of the semester. Also the grading feels really inconsistent sometimes, for two of the projects I felt that I answered all of the prompts, but I was still graded poorly for things unrelated to those.

For anyone planning on taking this, here's what I would advise:

• Watch the class lectures and supplement them with the popular YouTube series by DeepMind. The professors do a good job covering the topics but I always felt like there were always gaps when I was doing the assignments. The reading also patches this so don't skip.

• Start the projects as early as you possibly can, projects 2 & 3 in particular are incredibly challenging if you haven't done any sort of reinforcement learning or deep learning before. A lot of your project time is more setting up and writing the paper than experimenting, so you want to get through things early.

• If you're stuck on HW or a project, don't be afraid to ask the TA's for help and watch the office hours. The office hours are incredibly useful.

This class was really tough in Summer semester. I would recommend taking it during Fall or Spring so you have more time to digest the many interesting papers and have time to properly tackle the assignments.

Projects were quite difficult and would have preferred more focus on developing agents like the Lunar Lander project rather than replicating papers twice.

The homeworks can be easy or hard depending on the week.

The amount of reading + lectures is intense, but it's possible to get by primarily with just video lectures. There were tons of topics covered from basic methods to game theory, would have liked to see simple non-academic applications covered.

This is the most philosophically interesting class in the program so far for me (4 classes in) and has a strong academic and research focus. This is a great class if you are interested in theoretical CS, theoretical AI, and social science (i.e. economics, cognitive science). To succeed, learn how to write a good research paper, read the assigned readings, and don't skip the lectures when things get hectic as the final exam will cover that material.

Reinforcement Learning is a bit of a weird course in that it has a strong academic flair to the class. Personally, I agree with the way the class is taught. Rather than focusing on the applications of RL, the class requires you to truly understand the thoughts of the researchers that published cutting edge work in the field. This is much more valuable for understanding the fundamental difficulties of working in RL than hacking together some code for something that works.

The core of the class is in the 3 major projects, each covering an important algorithm in RL. You're required to write everything in 5 pages, which can be a challenge for people who are not used to writing and research. One of the biggest difficulties with the class is figuring out how to write the papers to get an A, but if you focus on theory over experiment results, you'll get a high score.

Homework assignments, except for the first one, are fairly easy. Only the first one will be a major conceptual hurdle, but it's quite important for the rest of the class. Since the projects are so massive, I recommend doing the HWs ahead of time to leave more than 1 week for the project.

The most disappointing part of the class was the final exam. It felt poorly instrumented and had lots of confusing questions. Compared to how intellectually fulfilling the projects were, the exam felt like a major let down in terms of challenge. Most of the questions were on specific terms taught in the lectures, with some being more conceptual and interesting.

I hadn't taken intro to ML ahead of this, but I studied RL for my senior project during my Bachelors, including some practice with OpenAI Gym. I was pretty disappointed with the class being previously interested with RL, mostly because the material felt super theory heavy to me.

I made the mistake of only watching the lectures and sometimes doing the weekly readings. The lectures especially sometimes strayed too far into explaining proofs and writing down equation after equation (many of which were not super important to remember in the context of the class). The first third of the class I felt lost because of this. Watching David Silver's RL lectures was way more productive in my opinion, and those lectures are similarly based on the textbook for the class.

Homework assignments were really easy and helped teach more granular topics. Projects were a beast, mainly because of the condensed summer schedule. Definitely need to start a week or more early if you're not absolutely familiar with each topic. You have 5 page papers on top of writing code, and the projects are pretty open ended so sometimes it is difficult to decide if you're going in the right direction. I pulled last minute all nighters for all 3 projects, granted I did not take my own advice about working ahead. Despite all of that, I think it was a cool format that requires actual knowledge of the topic, you can't get away with just writing correct code. Two of the 3 topics weren't really that interesting, but the second project was awesome.

Overall this class definitely soured my feelings about RL. The topic seems like it's still very limited to the academic community compared to other ML subsets. Even though there's some cool problems, didn't really explore real-world applications. That was also emphasized by the class curriculum being pretty heavy on research papers (replicating them, learning course content from them, etc.).

TLDR

I have mixed feelings about this class. On one hand, the material is very interesting. While I have a lot of experience in supervised and unsupervised ML, I had none in RL and learnt a great deal. The TAs were super knowledgeable and helpful. The homeworks are fairly easy to get 100% in.

On the other hand, this is an incredibly demanding class. Most weeks I'd put 20-25 hours a minimum, some up to 40. The lectures are not great so I always had to supplement them with David Silver's lectures online, as well as attending office hours every week. There's also a ton of readings. The projects are incredibly time consuming. I would not recommend taking 2 classes if RL is one of them.

Lectures

I didn't love the lectures. Some people really enjoyed the informal format of student-professor like lecture, but it didn't work for me. The examples were not clear and I found it much easier to watch David Silver's lectures on youtube. Definitely watch the lectures though, the final exam will be heavily based on what you learn in lectures.

Homework

There are 6 homework that account for 15% of the grade total. They actually get easier and easier. I'd say the first two homework are definitely the trickiest ones, and we had a lot of people post on Piazza saying they were dropping the class because they couldn't figure them out. If you end up in a similar situation, don't despair. They do get easier (towards the end, it would take at most an hour to figure them out), and the TAs always hold super helpful office hours

Projects

There are 3 projects in this course, each accounting for 15% of your final grade. The first project asks you to read a paper and replicate some experiments in it. The second project is about developing some form of Q learning to solve a game (Lunar Lander) and the third project is another replication project.

The first project is the most straightforward. You should try your best to get 100% in that project since it only gets harder from there. The replications are fairly easy - most of the points will come from your report. Use the format the TA suggest, and write as much as you can (5 pages max). Talk about all your pitfalls and obstacles throughout the process, and talk to all the differences in replication results. Try to make sense of them as opposed to just stating them. Really detail your process as much as you can.

It's important to do well on the first project since the other two are so much harder, especially the second. In that project, you'll have to make use of some basic NN in a Q learning implementation to solve a game. Start early as humanly possible - seriously. Report wise, I have the same comments as above, be as detailed as possible.

The last project is like the first one in that you need to replicate some experiments but more complicated. This time you are asked to replicate a grid soccer game and implement some forms of multi agent q learning that require linear programming. Again, start early. It is easier than the second project, but still very time consuming, and a lot of students struggle with it.

Exam

There only one exam that accounts for 25% of your grade. The average score on the exam is ~50%. It's a multiple choice format, with no prep exam or sample exam. It was a little daunting since we covered a ton of material in this course. Overall, it wasn't too bad. Knowing what I know after having taken the exam, my best advice would be to write down all the algorithms you learnt about, and summarize them; control type, on/off policy, run time, convergence and so on. Of course, there are also other questions in regards to other topics covered throughout the class, which you can cover by mostly watching the lectures. I'd re watch them at 2x speed and take some notes. I did not find that the readings were relevant for the exam.

This is a very difficult course due to reinforcement learning concept is rather new to me.

The course covers full aspect from the math behind to the programming.

Replicating the paper is a difficult project but super rewarding once you get through the pain in reading paper, understand the notation, what does the author try to convey and then finally replicate it just by reading the paper...

TAs are super helpful and expert in this topic. We also covered Alpha Go and Alpha Star in TA session which definitely help expand my knowledge on this and some latest development in Deep Reinforcement Learning.

You will come out of it understand on Reinforcement Learning as a topic and also some commonly seen RL algorithm now like DDPG, DQN, DDQN , Q Learning, SARSA...etc

The lecture only covers the concept. You will need to dive-in deeper using external resources such as AIMA 4th edition.

David Silver's is a good lecture video as companion

I put 20 hours per week, but at peak, I probably clock like 30~40 hours.

The lectures suck ass. Ignore the lectures and readings.

Just watch David Silverman's lectures and you can get 100% in homeworks and exams.

Good luck in the stupid projects.

Summary

• This course does a great job of teaching the core concepts of Reinforcement Learning. While it won’t teach you how to code up state-of-the-art RL algorithms, there’s enough freedom particularly in Project 2 to really expand into more current algorithms if you so choose. The class gives great experience in reading research papers, replicating research results from scratch, and writing research reports – all very important for a master’s student. By the end of the course I feel knowledgeable enough to be able to pick up a novel research paper in RL, navigate the basics and know where to go to learn more.
• Unless you have PTSD of Smoov & Curly from ML, I’d recommend this class for anyone interested in RL and ML in general. It has solid lectures, fantastic projects, and is well-run.

Course Format

• The projects are a great balance of programming and writing reports (5 pages), and make up the majority of your grade, along with Homeworks and a final exam.
• Pre-Reqs:
  • You should feel comfortable coding and debugging in Python.
  • Knowledge in machine learning and deep learning is helpful.

Instructors

• The TA’s were really just superstars. Extremely knowledgeable and very active on Piazza answering questions. They held 2x hour-long office hours every week, and really facilitated learning by asking provocative questions and getting students on the right path for the HW and Projects.
• The professors were 100% absent. I found that was fine; I suppose one is the Dean and the other is at Brown, but it would have been nice to have a professor attend a handful of the office hours, to at least connect the voices on the lectures to the course we’re actively working on.
• Instructors are consistently working to improve the course, updating the Projects, and seem receptive to feedback.

Course Materials

• Well-done lecture videos. With both professors throughout, the “Lecturer” and “Student” roles make for good back-and-forth and make the lectures very engaging and enjoyable. However, they are fairly theory-heavy when compared to the overall course – lots of time spent on proofs and math that don’t appear anywhere else except their own research. Speaking of which…
• The material seems pretty biased toward the professors’ own work. Many of the required research articles are the professors’ work. It does make me wonder if the course is truly comprehensive, or biased RL through the lenses of Isbell and Littman, Smoov & Curly.
• There is a good deal of reading in this course. I found the Sutton & Barto textbook very well written and really helped supplement my learning. There are also many required research papers, some more important than others, that you should come away with at least a high-level understanding of. The papers the projects are based on, however, you should read multiple times and ensure you know them like the back of your hand.
• There were six Homework assignments to help guide you into the larger projects and solidify understanding of the content. There are no hidden tests so you can submit freely on Gradescope until you are sure you have 100%, which everyone did. I agree with others that some of these could be slightly more difficult. HW 1 and 3 were great, while HW 4 and 6 I finished in about an hour with very little thought.

Projects

• Project 1 covered TD Learning, Project 2 one could use any RL algorithm to solver OpenAI’s Lunar Lander environment, and Project 3 covered Game Theory and Correlated Equilibria.
• Project 2 is a beast – start as early as humanly possible. I had a bit of experience with deep learning and like many in the class implemented a Deep Q Network just by reading DeepMind’s research paper on it and a helpful two lectures from the TA’s on deep learning and DQNs. It still was extremely challenging as I had not used much PyTorch before. I spent almost 70 hours on this project, including coding and writing the report. Even though I used deep learning, you do not need a GPU for this – a CPU is just as fast if not faster in most implementations. Even so, you’ll want to leave a good chunk of time just to iterate different models’ hyperparameters.
• I did very well on the projects, so I’ll go into some detail on what I did that I think worked well:
  • Don’t spend all your time on coding an immaculate program. Your code quality, speed, or anything like that is not graded whatsoever. As long as it works, and graphs what you need it’s good. Start early just jotting down random thoughts as you code and then jump into writing the report ASAP.
  • Your report needs to be at 5 pages – no more, no less. Do not include a separate cover page or reference page. As suggested by the TA’s, keep writing until you’re over 5 pages and then trim the fat. All my reports were over 4000 words.
  • Use a conference paper format. I used IEEE. I used a very simple section format for all the reports: Introduction (1-2 paragraphs), Methods (1-2 pages), Results (1 page), Discussion (2+ pages). That’s it. Methods should just demonstrate your understanding of the algorithm and explaining the environment or problem. Results should as shortly as possible state the results and show the graphs. The bulk of your paper, and your grade, is your analysis of your results in the Discussion. Explain them and connect them to the source material.
• Two out of the three projects (1 & 3) were reimplementing a research paper’s experiments and graph results. Personally, I think just one is enough to practice doing so and learning the benefits and pitfalls. I think for Project 3, where you already have a good amount of RL knowledge and have worked through two projects, a project with more freedom and discretion around what you implement would be more enjoyable. Project 2 already has great freedom in you choosing any RL algorithm to solve a set environment in Lunar Lander – only to go back to replicating an experiment and some graphs in Project 3 with not much direction to do tinkering or exploring. If you agree with this sentiment, the good news is it sounds like the TA’s are considering moving in that direction with P3, so this note may be outdated soon.

Exams

• The Final was redone this term. It was a two-hour closed book multiple choice/multiple answer exam. The questions are fairly straightforward, but the multiple answer bit can really trip you up if you’re not 100% sure.
• Mean was around 50% with a high score of 77%. I didn’t do great, but even so, I wouldn’t say the questions were particularly deceiving or even difficult. I think the students didn’t exactly know what the exam would be like (no prior exams given), and it was comprehensive of the entire semester, so it was hard to prepare for.
• Now knowing what the exam is like, I would recommend the best way to study being re-watching all of the lecture videos (maybe on 2x speed, or skip some familiar parts) and reviewing your notes. Then review the main research papers and if you have time, supplement your weakest knowledge with re-reading the relevant textbook material. You want to make sure you know the key concepts, key terms, and key algorithms well, even those from the very beginning of the class. Think high-level.

Review Context

• This was my first semester in the program, and I did not have a strong academic background in CS or relevant work experience. Going in I was comfortable in Python, and had taken some MOOCs in Machine Learning and Deep Learning. I say this to preface my ratings and average time spent per week; in short: your mileage may vary!

I thought this course was a very nice, quick intro to RL topics, that would serve well as a starting point for people who want to enter the field. The book by Sutton & Barto contains the fundamentals, and the lectures include summaries of key takeaways from various articles. Some of the articles are math-heavy, but understanding of that math was not required nor tested.

I have no ML background and this is my first ML-related course, so I found project 2 extremely challenging. The other assignments were straightforward. The final was a bit silly; it mostly tested stuff from the very first part of the semester. The grading seems inconsistent and it's hard to know what to expect, however the curve at the end made up for that.

Before, I get too far into my review and scare people, note that this was my first class in OMSCS because it was the only one I could get into at the time.

This class was generally enjoyable though a bit tricky at times. I found the lectures to be both entertaining and helpful. I'm more of a visual/audio combo learner so seeing and hearing equations and workflows explained in the course lectures was helpful for me. They go into enough depth for you to generally understand the premise of an algorithm and maybe a quick example but nothing to deep or complex. They leave that for the practice on homework or assignments to implement.

With regards to the homework and projects, I found them to be both challenging and interesting. There was a good mix of algorithm implementation and working towards specific knowledge that would help you on the projects. I have noted 25 hours here as my workload, which is an accurate representation but I feel I probably could have cut that by 5-10 hours a week had this not been my first OMSCS class. I would have really benefited from the Algorithms and ML courses prior to taking this course. None the less, those classes were not specifically needed, as I was able to teach myself some extra bits that were helpful for implementing algorithms on the projects and HW.

There is a LOT of reading to be done in this class. Lots of papers on different algorithms and RL techniques that are interesting but quite dense. It's a lot to distill but they make reference to the papers quite a bit in lecture and test you on principles from them on the exam. I would not say that they are explicitly needed to do well in the class but there are certainly some key ones that you'll want to read all the way through (specifically ones that you are re-creating or that discuss a specific algorithm). Personally, I didn't have time to read all the papers and noticed a few gaps in understanding at exam time, but it only cost me a question or two.

I was able to take a lot of the knowledge I gained from this class and put it to work directly at my job which is a great feeling. I think this class achieves what it sets out to do - enabling the student to feel comfortable implementing RL solutions.

This was my second course in the program. Overall, this course was really meh. Here is my analysis, and suggestions on how they could've improved.

The projects were pretty interesting, but the lack of a grading rubric make the grading process opaque and confusing. There isn't much guidance to what they really want beside general guideline. I think they are trying to give flexibility, but it seems odd because they have requirements for grading. They should just settle on a format and give a grading rubric to help guide our learning instead of just saying "do whatever you think is best." I found it to be a lazy practice, it can be greatly improved. It also keeps grades consistent across graders, which may be a problem, though I thought my graders were very fair so far (still haven't got the final project grade).

The homework did not teach me anything. They were a huge portion of the final grade and it was really easy to get 100% on them, but they offered by also zero learning value. They should revamp the projects to be more useful problems to understand the subject.

The final exam was really scary leading up to it, but it wasn't all that bad. It was all concepts. It was one of those exams where you either knew the material or didn't, there was no figuring it out on the exam. Watch the lectures and take great notes and it will get you most of the way there. TAs recommended using lectures as the primary studying tool.

The average on the final was a 50%, which always sucks because what did we really learn then? I think flipping to a quiz format where there are 5-6 big quizzes each worth 5% of the final grade with gradually more material ending with the exact same final exam would have resulted in really strong scores, better learning, and better retention. A single exam with so much weight isn't really useful pedagogically in comparison.

I felt like I learned a lot from the first two projects, watching the David Silver lectures, and reading Sutton's book. So the first 2/3 of the class was pretty good! I felt like I learned a lot and retained it.

The last third of the class was Game Theory, which I find to be a complete drag. A lot of toy problems and kinda useless theories that don't generally apply to real life imo. Researchers seem to love the subject because this is my forth time learning about game theory in a college course haha. Overall, it wasn't too bad though.

I wouldn't really recommend this class. None of it is inherently difficult, but it isn't a really well designed course.

This is the first class I've taken in OMSCS.

Overall, the class is fairly easy to understand, and the only real difficulty was fixing bugs in code and some linear algebra I needed to brush up on.

The lectures could ramble on and on, and I didn't appreciate the humor mixed in, but overall the class was useful for getting a high level overview of what reinforcement learning is.

The final exam is actually much easier than what the practice questions would suggest, and many of the practice questions appear on the exam. Be sure to study these questions thoroughly for free points, and the other questions should be fairly easy to answer as long as you've watched the lectures.

The projects do take quite a bit of time, and the final project does use linear algebra to implement constraint optimization.

The course was challenging. The fun part were the projects, it was awesome and it seems the faculty targeted to teach via projects and home-works. At one point of time I stopped watching the lecture videos and started preparing for home-works and projects. Read S&B book (holy grail for this course). For the projects I often had to visit the lectures which were mentioned as referral material.

The TAs were awesome and very helpful, we also had a healthy discussion of AlphaGo paper over one of the office hours. The interactions are insightful. Piazza will be your best friend though.

I personally found this course very difficult and wouldn't had finished many of the HWs or projects without the helps and hints in piazza.

I had picked this subject in my first semester, I wonder how difficult would other subjects be which are rated more difficult in OMSCentral.

This is my first course in OMSCS and has been a great learning experience. I have a non-CS undergrad degree but have spent couple of years in machine learning domain. The course content was easy to grasp but for someone with no prior ML experience, I would suggest taking CS 7641 first as there were quite a few references to CS 7641 course (I did the udacity version of CS 7641 couple of years ago) . The lectures are great. Prof Charles and Littman made the lectures quite enjoyable. I found David Silver's lectures a good supplementary material for the course. The course had 6 homeworks. They consisted of jupyter notebook having some skeleton code from a concept discussed in lectures or from some research paper. We were asked to complete the code and submit it on gradescope. They were autograded and results were given instantly. They were simple enough and I was to able to finish each of them within 2-3 hours. The projects were the best part of this course. There were 3 in total, each tougher than the previous. They required implementing parts of a research paper and then writing a report on the methods and results obtained. The grading rubric was a bit vague, but what I observed was the TA's were mainly looking if the learner could express the ideas of the main paper in their own words or not. To get A, you definitely need to ace the projects. Start them as soon as they are released and keep atleast one week for writing the report. I felt the lectures and the deliverables were not fully in sync and this resulted in considerable additional reading, especially the part on game theory. The final exam was easy. It covered the concepts discussed throughout the lectures as well as some questions were from assigned readings. The TA's had been fantastic. The office hours were great and they were responsive on Piazza. Overall the course was a great introduction to reinforcement learning and I definitely recommend this course to anyone who is remotely interested in this area.

Overall, I liked this class a lot and learned a lot. This was my first class in the OMSCS program. The subject is very interesting, very current and this seemed to provide a good basic foundation over a lot of material. Personally, I liked the video lectures, but could see where the not so funny, constant jokes could be a turn off. There is pretty good linkage from the lectures to the assigned papers for reading, which is a lot. There is some balance for full understanding on the papers and effort/time. Homework was useful, edge cases probably the biggest challenge for a 100%.
Projects are the big lift and were very time consuming. The first and last were tedious reproductions of papers - the first one was probably justified, the reproduction (3rd project), more tedious to me. You really need to get ahead on the lectures and read the papers, they are helpful for the projects, which wasn't clear at the time and would have been helpful for me on the last project. Exam was a bit questionable - 60-70% seemed like it was reasonable core knowledge, the other questions were kind of tangential \ ambiguous trick questions - hard to tell as you don't get your graded exam back (?). TA's were good overall, with a few outstanding ones. Workload was on average 20+ hours, a bit up and down, with project crunch times definitely higher. Good class.

Course was challenging, even with being fluent in python and having a DL background. Projects and homework both take time to figure out the trick (or find the bug). Don't procrastinate or you won't have time to solve the problems that crop up.

To my background: I am a mechanical engineer by training with an academic history including a PhD. This was my 2nd semester and I was (and still am) relatively new to coding as well as computer science and algorithm design in general.

To the course: I was and still am fascinated by the idea to let an agent figure out the world on its own. The course teaches just that in a way that really allows the student to understand the fundamentals. There is criticism that the problems are toy problems and that RL in general is not industrially used. For the first, I think after the course one has a really good foundation to build off of for solving real world problems step by step. For the second: That is not entirely true, I have seen usage of RL for industrial systems and also RL is still a research area - you're not only supposed to be made "fit for industry" (this mindset is a bit short sighted, I find).

The course is heavy on mathematical concepts. Granted, I have some background that has dealt with math always, but I am not the most gifted person in grokking math. However, lectures and the book do a really good job in bringing this closer to you. So do NOT be affraid of the math, seeing similar things over and over again (nod to Bellman), you eventually know whats going on.

The projects are super fun (even replicating papers) but the last one turns to bid of a nightmare after a while.

I found the course very, very rewarding giving me the confidence to tackle hard courses in the program, allowing me to get good foundation in RL and also beeing more cool about math.

This class is tough. I had to drop it.

I'll start with the positives. This class is run extremely well. Office hours were informative; students and TAs are helpful on Piazza. I enjoy the delivery style of the lectures on Udacity and the material itself is rigorous and interesting. The first project has you replicate a research paper from Richard Sutton, for example.

Why I dropped. I was not prepared for this course. Before RL I've taken both ML4T and ML. The main difference I see between RL and ML is that in ML you're allowed to use any package or library you want and the focus is on discussing the outcomes of different ML algorithms applied to real-world problems. In RL, you need to first implement the algorithm, and then you get to discuss the differences based on the code you wrote. There are no packages allowed here (at least not for the first 3-4 assignments). I think that raises the difficulty for someone like me, who is not a software engineer by employment (though I do have a CS background).

I would suggest you have strong Python programming skills and get very comfortable with mathematical notation before taking this course. It's very rewarding but I need about 15 more hours a week that I currently don't have.

Good luck!

Very hard class. They give you a final exam with no feed back just to judge how poorly they instructed the class. Class average on final was 50%, true/false, with explanations. This class made me hate machine learning, unlike my undergrad work on the subject. Lectures are worthless. Instructors do not cite their sources or explain the topics in the lectures. The material itself is garbage. Reinforcement learning is useless in the real world, and the instructor's explanation is "Just do Q learning cause it works." Then they give you a final exam with no feed back just to judge how poorly they instructed the class, rather than to correct the students. Clearly they don't care about teaching reinforcement learning so why should we care about learning it. Instructors act as if their questions have answers, but don't provide feedback to prove it. In a a science experiment like machine learning, instructors have a burden of proof to bare. I don't believe deep learning will ever work if we hide information from students. Make sure you have a good graphics card, their software hogs resources like no tomorrow. Takes a whole day of processing to test your code. Avoid this class.

This is a good course with good TAs. TAs run the show and may not be clear at times when you ask a question but digging deeper will definitely get you through. I found the HWs more valuable than the first and the last project. The second project was the best because you can implement any algorithm of your choice like DQN, or policy gradient methods. You will need to write high quality detailed papers to get a good grade. I didn't know what to expect in the class so I only received 59 in my first project. So I worked my ass off for the next projects and got 100s in both. The third project and the final HW weren't so good because of lack of resources for linear solvers online. The best part of the class was that you will read a TON of papers! I mean a TON! This is what I liked about the class. The standard Richard & Sutton book is only a part of the reading you'll do, I'm happy that this class helped me improve my technical reading skills. The final was fair enough with 25 questions, each question is a True/False and a written explanation and each question was worth 1% of the final grade. Also, it is closed book and closed notes so it was difficult but the professor brings the grade bars down so you'll likely get at least a B or an A. My average was 87 and made an A. This is good class regardless of the cons I mentioned because it will give you the required skills necessary skills to get to the next level.

This is my 4th OMSCS course and the best so far (good enough for me to write my first review..).

The content is very interesting and the Piazza community is always active and helpful. If you are like me who wants to dig deep in the concepts and learn not only how, but why, this is the course for you.

This is also a course that the more you put in, the more you get out of. I am kind of glad that I am taking it during the summer because it is the only course on my plate and I can fully commit to it.

Personal tips - start projects early. good project scores will make your final grade a lot nicer.

My fourth course and arguably my favorite. It was the most challenging but the most rewarding. As others have mentioned, put the priority on the three projects, particularly the paper itself. Naturally, you will spend time on the code trying to replicate accurately, but the TAs are looking more for your understanding in the paper than they are your ability to code. I got two 70s and one 90 on the paper, and still ended up with an A in the course.

The positives:

1. Engaged TAs and helpful office hours
2. You will learn a lot, both through the homework and projects
3. Curving really helps

The negatives:

1. Grading/feedback is fairly vague on the papers
2. Lectures are a bit hard to watch, you get a lot more out of the textbook and conversations on Piazza/Slack
3. Any final where the average is less than 65 is annoying, did we really learn anything if that is the case? (Yes, but make it better than T/F). Practice questions are exactly that - don't expect a lot of reuse on the exam itself.

Another thing to note - I didn't actually read a lot of the papers and managed just fine, again make it through the lectures and put the most time into understanding Sutton/Barto.

This is a solid class that covers a wide range of RL fundamentals. The class structure is very similar to the OMSCS Machine Learning class and I would highly recommend taking ML before taking this class - RL takes about 30% of the lecture material from ML and a lot of important things like report grading expectations and exam format carry over from ML.

The class deliverables consists of:

• 3 projects each of which involve a good amount of coding and a 5 page written report. You should start these as soon as they're released.
• 1 homework assignments. These assignments are quite straight forward and shouldn't take more than an hour or so to complete.
• 1 final exam.

Pros

• The lecture content is interesting and thorough. You get enough foundation to pursue specific areas more in depth if you so choose.
• The homework assignments and projects are really useful for learning. I personally learned a lot about the minutia of RL algorithms by working through the projects.
• The exam is fair. You'll need to study for the exam since it pulls from the entire course, but it's a very straightforward exam.
• Grades are curved - the cutoffs vary by semester but I got an A with a 80-85% raw score.

Cons

• I found the Littman / Isbell tag team lectures to be a bit grating - especially after having gone through the ML lectures a semester prior. Also IMO Isbell is a better lecturer in terms of structuring content, but this class leaned more towards Littman which made the lectures a bit harder to follow.
• The "reproduce this paper" projects could be annoying - sometimes it was more challenging to figure out the minutia of how the authors of a paper set up their experiment than to actually implement the experiment itself. I would've much rather work out more complicated algorithms or come up with my own experiments.
• The game theory aspects of the course were more challenging if you didn't already have some background in Linear Programming - it's not necessary, but having some kind of optimization background is useful for this class.

I just finished RL this summer and I also took it before any ML class (granted I took a "Fundamentals of Data Science" course in undergrad which admittedly helped a bit when we covered function approximators).

I don't think it's as bad as everyone makes it out to be so long as you...

• Watch (and take meaningful notes) on David Silver's YouTube lectures.
  • I cannot stress this enough; the notes I took were indispensable and I frequently referred to them throughout the course.
• Knock out the homeworks ASAP (preferably the weekend they are released) and spend the rest of the time on the project & paper.
  • You are graded mostly on the paper, so I would start writing that as early as possible, or at least jot down some high-level points of discussion while you are doing the project.

I watched a few of the course lecture videos during the semester to better understand some key concepts related to the homeworks/projects, but for the most part, I delayed them until the week before the final and did well enough to net an A after the curve

The course is solid. The videos provide a more theoretical view of RL which I find to complement RL bible book by Sutton well in the sense that they cover slightly different things from different angle. The jokes by two professors are lame but when you watch second time in final revision with 2.0x play speed they become not so bad... There is actually a reading schedule for you to read the RL book while the lecture progresses which I followed and found rewarding. While assignment 1 and 3 are replicating outdated paper, I found the two to be rewarding. I enjoyed assignment 2 the most as I was able to experiment both policy gradient and Deep Q-learning techniques. My only compliant is the game theory stuff is probably a bit less practical in real life because the assumptions like opponent can see your Q-values and the two will select optimal action by reasoning sounds unrealistic. However, I enjoyed solving those things with linear programming, which I have no experience before this class. I did not read most of the papers. The prof. sniffed in a lot of his own work. I found Sutton's book to be more accessible and is way better conceptually and logically organized than the ordering that lecture and assigned papers provided.

You learn a lot, also it has the best project of the program, but the lectures are the worst with distracting jokes everytime.

Quick background on me: I'm using OMSCS for a career transition and to get into more interesting or high level work. NOT PhD or academia.

This class gives the the real grad school experience. Read a research paper, try to implement, then realize the big wigs made some unpractical assumptions. Fantastic.

The TAs are passionate and my classmates were phenomenal.

The course itself is interesting homeworks, projects, and one final. Projects are graded on write-up only not code, so if you understand the research papers well and got something working, you can get to an A with a great write-up.

The final is insane. No advice to prepare because you can't... but there's an awesome curve.

Disappointing, although to be fair, a lot of people seemed to like it a lot more than I did. The Abbott and Costello routine of the 2 profs gets tiresome and makes the videos hard to watch. 2/3 projects are theoretical--replicating a moldy academic paper. Have fun describing that at a job interview ("Yes I know you are trying to solve real world problems, but I replicated these graphs from a paper from 1988!") The other project is more interesting (lunar lander). At least you could explain that at an interview and describe how you solved a real problem with RL. The projects require a lot of work, but I learned less on these projects than, say, AI4R, where they maximize your learning and minimize the dog work. The course is run by the TAs but the head TA was nowhere to be seen after like 3-4 weeks. You won't hear from the prof the whole term. The TAs are passionate about the subject, but the grading is arbitrary and demotivating. I worked hard to get my code working, and turned in well written reports and still got grades that were lower than I expected, with vague comments. The final was OK--focus on watching the the key videos again and make sure you know the various theorems, etc., you'll be OK. I haven't taken ML yet but I might just move over to a different specialization, and learn ML through a MOOC, rather than go through this again. There are better resources out there on the web. You could easily create a better course than this by: 1) Using David Silver's videos and dropping these videos 2) Assigning 3 interesting projects and using an autograder. Skip the replications of old research papers and subjective grading by people who took the course a year or two ago. 3) Hire a real instructor who is going to commit time to this. The TAs are good but we're paying real money here and should get a real instructor. Understandably, Isbell has other things to worry about now that he's dean, so fine, hire someone else.

I loved this course. This is the best course I have taken so far. Some advice for those thinking about taking this course.

Advice 1: Don’t just watch the lectures. Take notes when you watch them, ask questions in piazza if you did not understand something and try to learn the concepts of the lectures in each week. The amount of material is just huge and you don’t need to understand everything in the lectures to do the project and homeworks. So if you don’t take the time to understand some parts of the lectures because you don’t need it for the homeworks, you will struggle (as I did) when the final (and only) exam approaches and then you HAVE TO study all the material. It’s just not possible to study all of the and keep a full-time job. You will have to study throughout the semester to do well in the final exam. Trying to study everything in the last week of the semester is not a good strategy.

Advice 2: Other have already said this, but I will repeat it because it’s worth it. Start working on the project as soon as they are released. They always endup taking more time than you expected. Par of the reasons is that training the models can take hours and you have to replicate experiments from papers where there are missing details about the implementation, so you have to try several ways of solving the problems and that takes time.

Advice 3: The TAs are great. This is a hard course, so make sure you use the office hours and piazza if you are struggeling. They do help a lot, but you have to ask for help and do it as soon as possible. Remember that the projects are long and even if they helped you with a hint or a useful comment, you won’t have time to implement if if you asked for help the day before the deadline.

Advice 4: Keep in mind that historically there has been a very generous curve on this course, so if you follow the advice above and do your best, you should be able to get a B in this course, even if you are rounding the 75’s in your final grade. Historically the average grade in the final test has been around 50, so as you can imagine, they curve the course a lot. I did well in all the homeworks and project except for the second project, where I got around 59 due to lack of time (a relative got sick and my life got hectic around that time). Even though, I ended up with an A in the course (I got above 95 in all other projects and homeworks). To sum up, if you don’t do well in one of the projects, don’t drop the course right away.

Advice 5 (For OMSA). If you are an OMSA student (as I am) make sure to take CSE 6040 (Introduction to Computing for Data Analytics) before taking this course. I think CSE6050 is a good way of measuring your programming skills. If you struggled with that course and don’t feel comfortable programming on your wont without any guide, then you might not be ready to take this course. This course requires you to implement the algorithms from scratch. In the last project we had to create the environment and the rules associated to the environment where the agents would be placed to learn and then implement the learning algorithm on top of that. If the environment was not properly created, the learning algorithm would not work. So, there is no time in this course to struggle with the details of the programming language you are trying to use.

Workload: Depends. The amount of material that was crammed into this course makes it impossible for a student without reinforcement learning background to digest them all in a single semester. If someone wants to accomplish that, workload is 50 hours per week. If not, 20 hours per week to get a decent performance in the course.

I think the course creators did this on purpose. The point is to provide you with a map of reinforcement learning instead of showing you around of some locations on the map. Given the time these are the two options. Showing you all the locations is not realistic. After the course is over you will be able to embark the journey of reinforcement learning by yourself. What papers to read? What ideas are floating around in the community? You should be able to answer those questions after taking this course.

This course has a veteran TA team that are knowledgeable. They host extra study sessions and reading sessions that discuss most recently published papers.

Forgive me for not talking about homework assignments, projects, exams, grading, etc. They are not that interesting after they are behind you. What's interesting is now you have the MAP, use it to navigate through the reinforcement learning jungle!

This is a tough course but my interest in the subject helped me to succeed. I was already comfortable with the concept of RL prior to the course. I've used python daily at work for past 4-5 years. This was my 4th course after KBAI, ML4T, and AI.

The course mostly hits on the earlier research in the field, prior to Deep RL. So once you're done, you have some work to do to catch up with the field still. This, however, is ideal for me since I want to focus in RL in my career and for personal projects.

Homeworks:

• 6 homeworks
• Each took me 5-10 hours

Projects:

• 3 projects
• Each took me 30-40 hours
• Write the full 5 page length, no more, no less. And you'll likely get >80%.
• Write down your thoughts, notes, and anything you can while you work on the coding portion
• Summarize and compile all of your notes and you'll easily fill in 5 pages after you include the specified content they request.
• DO NOT HAND IN <5 PAGE REPORT (or >5 page)

Test:

• 1 test
• I waited until the end to go through all lecture material and white-papers and I think that was the right call since I had multiple full days to study. If you don't, then I would be sure to keep up on the readings and lectures.

I loved this course. And while I thought at one point that I wanted to quit because of all the stuff happening in the world, I was very glad that I pushed through and finished strong. I understand the topic much more deeply and have momentum now to go and research more about Deep RL.

This is a course to take if you want to learn a lot, but you'll have to work for it. Each week you'll typically have 1-2 hours of lecture, a chapter of textbook, 2-3 research papers, and a homework or project due. I would not recommend pairing this with anything else.

When it comes to the actual assignments there are:

• Six Homeworks: Each of these builds on the lectures. I would not say they were incredibly easy, but to each their own. I did not find the lectures tied amazingly well to the homeworks and had to Google a bit for some. They're worth 30% of your grade, and you will want every point of that.

• Three Projects: The first and third project, you are given a research paper and have to reimplement what's described in the paper to reproduce some figures. The rubric for these is... not great. Basically, explain every little choice you made, every assumption, every reason why your figures might be different. These will easily take 40+ hours to finish. The second project is actually solving the Lunar Lander using an RL algorithim. This one is far easier to do well on, due to the fact that it's solving something instead of just reimplementing something, HOWEVER it is very computationally intensive, so give yourself plenty of time to solve it. In your paper describe how you solved it in detail, and all the assumptions, etc... These are worth 15% each

• Final: The final exam is worth 25% of your grade and can cover material from any lecture, any paper, any project. They're not going to try to trick you on it, it's actually pretty fair, but it's still a ton of material to remember.

My recommendation for taking this course is first and foremost just stay on top of all the readings, they add up but are critical. Read everything on Piazza and Slack and get started early, you'll need the time.

This class was by far my favorite course in the program so far (and it is my 6th, so I have seen things). Please don't let the negative reviews deter you from taking this class. I personally did not find it nearly as difficult as some people said it was, but maybe that's because I went in feeling extremely worried by all the harsh reviews I had read on here and I was pleasantly surprised that it was much more manageable than I was expecting. To put things into perspective, I work full-time and have a 1-year old at home, so my free time is limited. I still got an A in this course.

The course consists of:

-Udacity lectures. Don't make the mistake I did of getting behind on these of you will have a rough couple of weeks before the final. Watch them and take notes. Make flash cards of the important points. You will thank yourself when it's study time.

-6 homeworks. I found the homeworks to be mostly incredibly easy. The first couple I spent maybe a few hours on, but most of that was reading and watching lectures and trying to figure out what the homework assignment was actually asking us to do. Later homeworks were finished in as little as 20 minutes. (The other students on Slack were a great help.) You write the code and then when you get it running you answer a 10-question "quiz" that gives you inputs and you provide the output of your code. You can take this quiz up to 10 times, so if you get it wrong, go back to your code and debug. There is basically no reason you should get less than 100% on the homework.

-3 projects. These are time-consuming but doable. They feel intimidating, but just jump in. Remember you have to write a 5-page paper on each project, so consider starting the paper at the same time you start writing your code so you aren't rushing to finish it when you get the code running. Even if you don't get correct results from your code, you can address your issues in your paper and discuss why you think your results were wrong, and still recoup most of the points, if the rest of the paper is good. Don't skimp on the background and experimental setup sections.

-comprehensive final exam. Watch all the lectures and do the reading and you will be fine.

It's hard to judge the workload because it fluctuated. When I was working on projects it was higher, like 15-20 hours a week. When I wasn't, it was basically just watching the lectures (or not, since I put that off a bit too much). I have a job and a kid at home so most of this course I did on weekends only. A little in the evenings but not much. I did take a day off work to study for the final but I probably didn't really need to.

The TAs for this course were great. They held office hours and a weekly reading session. I did not participate in most of these due to time constraints. But it was nice to know they were available. They were very responsive on Piazza as well.

The lectures were excellent. At first I was a bit irritated by "Smoov and Curly's" banter, but once I got past that I found the explanations and proofs to be a great help in understanding the concepts. Yes, there is lots of math in the course. This is a Computer Science masters degree. You can't expect to go through the entire program and not deal with math. Don't be afraid of it, though. The lectures and readings will help you to understand the math you need to get the projects and homeworks done. You don't need math for the final so don't be intimidated by all the proofs in the lectures--you don't have to know how to recreate them.

I really can't say enough great things about this class. I wasn't going to take it because I don't need it for my specialization and it was "hard", but then I jumped in at the last minute and I'm so glad I did! This class is challenging, but I did not find it difficult. I'm honestly surprised by so many people who rated it "very hard" (maybe they aren't very good at writing papers so they didn't like that aspect?). If you put in the effort you can get an A. And you will learn a lot and have a lot of fun!

I have a mixed feeling about this course because the content is very very good and you actually learn some awesome things but the way in which the content is delivered is not very good. The lectures are pretty much useless. There is a lot of material to cover - videos, papers. For the first half of the course, I was clueless as to what I am doing. But eventually, I started understanding. If I had to take this course again, then I would do some things differently. Suggested to do:

1. Watch David Silver's lectures on RL before the course starts. They are exceptional!
2. Sutton and Barto's - Reinforcement Learning An Introduction is as simple as it will get. Read and reread the book. The concepts are clear and written in very easy to understand fashion.
3. This course is based entirely on probability. If you are feeling rusty in it, then revise it before the course starts.

This class is no joke. There is much more material covered between the lectures, readings, and projects than you can reasonably expect to absorb in a semester.

Most of the other reviews are accurate in the specifics of the workload and other technical details. I won't repeat them here. But I do want to address the complaints that you learn outdated material in this class...

Every discipline has a set of foundational ideas that don't change. In the case of Reinforcement Learning, this idea is the Bellman Equation. Every project and homework in this class gives insight into some aspect of the Bellman Equation, under different settings and assumptions.

It is true that this class doesn't go into the latest deep learning approaches. But gaining the intuition from these supposedly "outdated" ideas, I feel equipped to read/implement the state of the art papers.

Take this class if you want to be challenged. It really is fulfilling to finish.

This class covers very broad topics crammed together in one course, from MDP to game theory. Most of the topics covered was from old research papers which has little real world application.

You will be heavily graded on your project report, (45% of grade), which has no clear grading rubric. There is no way to be really prepared for exam (25% of grade) as the questions are pretty tricky and vague (you need to answer with T/F and concise explanation). So 70% of your grade is subjective, only 30% of your grade from HW is objective.

If you take this class, you better start way ahead by watching David Silver's lectures and other lectures like Pieter Abbeel or Emma Brunskill, and reading the RL text book as much as you can before class even start.

I took this class SPR2020 as my first class in the program.

Before I go into my actual review, I need to address all the negative reviews for this course, as I am totally irked:

As a main point, if you want to work on and create AI systems using techniques like RL and ML, you should understand mathematics. PERIOD. Researchers absolutely need to know how the math works to make breakthroughs, and appliers of the technology need to understand it, otherwise, its just a black box - and when your black box stops working on your use case, you better know why it works. On top of that, all the mathematical concepts for AI are built on one another, and compared to the latest buzzword deep neural net that has zero interpretability and nobody actually knows WHY it works, this class takes you from A to Z on the entire field of RL, and tells you why it works at each step. The 'cool, new stuff' everybody wants to use is built upon that so-called 'outdated' mathematics (which, by the way, will never become outdated in comparison to your ML library function) that you really need to know.

/rant over

The lectures in this class were quite mathematical - you didn't need to 100% understand everything, just the general idea, but its great that they went the length to tell you why it works.

On the other hand, the homeworks (aside from homework 2) were quite applied programming could be completed within a few hours and tested your ability to implement the RL technique at hand - i felt there needed more depth.

Project 1 and 3 replicate papers, and project 2 is fully application, and the writeup is where most of the points are gained, and you can really demonstrate your knowledge of what is going on. Paper replication (as they say in the class) is a very important technique for a researcher, as actually many papers published in conferences today are just taking someone else's paper, and then improving it by some amount! The projects are very open ended in the analysis, which is what you expect from a graduate class - nobody is here to hold your hand. Like everyone else says, start early - they take a lot of time.

I actually didn't read any of the required papers in the class, and I think I did fine - they mostly expand on the topics covered in the lectures from the ORIGINAL source material of the lecture topic; this is GREAT. Additionally, there are optional textbook readings from literally the BIBLE OF RL - Sutton & barto; if you ask anybody how to learn RL, they will refer you to the textbook. The textbook has very good explanations, but can be even more detailed and mathematical than the lecture videos and what is expected of you.

I was actually disappointed at how much math there seemed to be vs how much there really is, but the class enables you to really learn this subject in depth on your own time WHILE you take the class. However, given that this is a CS class, it has the perfect amount of math, application, theory, and comprehension involved to take you through the subject.

There's one final exam, and its just true/false and covers basically everything in the class (which is a LOT). I think its a good rating of your understanding of the material - if you can't hold a conversation with someone about the material, do you really know it?

I absolutely recommend this class. I wouldn't take it with any other classes though (although, I don't think I'm ever going to take any of my classes with other classes). Supervised learning is nice to know for project 2, and being comfortable with mathematical notation is a given.

This class is extremely demanding. I didn't dislike this class, but it made me realize that I am not interested in the theory behind RL/ML/AI work and instead am more interested in the application. Most of the class is taught from a theoretical and mathematical point of view. It can be hard to really grasp the "whys" of what Prof. Isbell and Littman are explaining but they are great in their delivery none the less.

I had been on the fence about my specialization between ML and Computing Systems and this pushed and confirmed me into the camp of CS. There were parts that I thought were interesting, but in the syllabus Prof. Isbell addresses the students as "future researchers" and that is what he will expect out of you. I really enjoyed the LunarLander project, but I liked it because it was very applied and I got to learn about Neural Nets much more than I thought.

Technically, I really disliked this course. But that is a very subjective view. I think if you know what you are getting into and have a passion for understanding the research and literature behind many RL problems, then you will love this class. I think this class highlights one of the frustrating things about OMSCS which is a lack of pre-reqs or concrete curriculum through the program. It feels like you should go into this class having a better foundation. It seems crazy that this could be your first class into the program.

One piece of advice I would give future students of this class is to make sure to give yourself enough time to write the paper that goes along with each project. There is a significant coding portion to the projects, but the TAs are looking for a good analysis of what you did and largely don't really care about the code. Don't assume that once you're done with the code the paper will be a breeze. It won't.

Overall I was disappointed in this course.

Projects were reproductions of very old papers, you were given very little direction from TAs, No grading rubric for reports.

Homeworks were fairly straight forward but most often unrelated to the projects.

The final was full of tricky , not 'trick', questions but were written by someone who seem to be showing off how smart they think they are. Very little direction on where to focus study efforts.

I would not recommend this course to anyone.

Mixed feelings about the course. There are 3 projects and 6 homeworks. The homeworks are spread out throughout the semester where the cadence is 2 homeworks to 1 project.

The first and third project sets out to have you replicate the graphs from a research paper. The second project has you implement an RL algorithm to solve the lunar lander problem.

These projects require a lot of time so it's important to start early. You need to account for not only writing the program but to also then account for the time it takes to write the paper.

Ontop of all of this, there are lectures and an overwhelming number of papers to read every week. The papers aren't strictly necessary per se, but they can be helpful in giving a better understanding of RL as a whole.

The final exam is 25 questions true/false with explanation. Due to the breadth of the course material, it's hard to know where to focus your studying other than to study everything. The exam will still be a hit or miss unless you're able to absorb all of the material to a T.

Overall, I liked the class - I felt that projects were fun and allowed me to learn a lot more about RL than I knew previously. It can be overwhelming if you aren't a super mathy person, but you'll realize that RL really comes down to different implementations of the Bellman equation.

This course is poorly constructed. The ambiguity makes the course extra hard. The material itself should not cause this much painful experience. Also, as someone who already has some experience in reinforcement learning, I found certain topics are weird. I don't believe the instructor makes effective or clear discussions.

This was my 3rd class in this program and I had enjoyed this program up until I made the HUGE mistake of taking this class. DON’T take this class. You WILL NOT learn anything, the professors and TA’s are the absolute worst, the office hours, study sections, etc don’t clarify any questions, they will just confuse you even more, because the TA’s don’t answer the questions clearly, they will just go off and explain absurd philosophical concepts, rather than tell you what you need to do to solve your problems. You will spend hours reading outdated papers and books from 1988 or so. Also, the workload is pretty heavy, you will have 6 homeworks, 3 projects and a final. If you happen to get a bad grade and decide that you want a re-grade, don’t even think about it. The re-grading policy is that they take 10 points off if you don’t get 5 points when they re-grade you. The TA’s won’t share the rubric, they will basically enter a bunch of comments on the things you did wrong and assign you an arbitrary grade. So, you won’t really know which amount of points you got taken off as a result of each mistake. The grading process is arbitrary. This class is a waste of your time and money. Think very carefully before taking it. I hope this helps someone.

This was a phenomenal class. The material is very interesting and the curriculum is structured so that you receive a healthy cross section of the different parts of reinforcement learning. As this field is very new, many of the concepts learned involve cutting edge research, including implementation of algorithms from research papers that are less than a decade old. Finally, the assignments and projects push you to learn about as much as is possible in a first course on the subject.

But then why aren't all the reviews for this course 5 stars?

This is the question I'll try to answer in this review.

• This class is much harder than it appears; don't underestimate it.

Part of the confusion here comes from the fact that there are no barriers to taking this course. There are no hard prerequisites and no checks for mathematical maturity, even though this is a highly theoretical course and the potential student body has a very wide range of backgrounds.

Couple this with grade inflation, where well over half the class that sticks with the course gets an A or a B, and you have a situation where many people finish the course but don't grasp the fundamentals very well. This phenomenon is, unfortunately, true for many courses across OMSCS, but is especially acute here. Therefore:

• Be prepared for this course

Most important by far is to have a strong background in basic probability theory: random variables, conditional probability, expectation. If you want a deep understanding of the topics and not a superficial one, you need fluency with probability.

Some exposure to supervised learning, e.g. CS 7641 or Andrew Ng's Coursera course, is helpful, especially for project 2, but not strictly necessary.

In general, the stronger you are at math the easier this course will be for you; any deficit there translates to many more hours of effort during the course. This leads to the following point:

• Put in the work

I averaged over 40 hours a week in this course, and didn't feel like I was doing anything more than conscientiously learning the required material. I think that the stated average of ~22 hours a week grossly underestimates the time investment required of the average enrolled student to do this course properly.

Underestimating this course will lead to a shaky understanding of topics, unexpected grades on assessments, and for the less self-aware, a rush to blame something other than themselves once the course is over.

As said at the start, this course is great. The curriculum is masterfully structured. The textbook (Sutton and Barto) provides an excellent theoretical foundation to reinforcement learning. I thought the class lectures were a good complement to the textbook -- there is a wide variety of topics covered in them, many of them without the rigor of the textbook, but with intuitive explanations.

The projects involve a lot of learning that is on top of what the textbook/lectures provide (e.g. linear programming for project 3), and take a ton of time, but are really rewarding. Finally, the selection of assigned research papers is a snapshot into the most important developments in the field. Getting through them is not easy, and a strong preparation in math and probability theory will pay dividends here as well.

Giving these myriad sources of learning their due, not to mention giving the projects the effort they deserve, requires a lot of time. Be prepared, respect the course, and I think you will love it.

I took this class during summer 2019 without taking ML.

This is one of the more difficult classes in OMSCS. Class material is a bit on the mathy side, and you could get lost easily.

I really enjoyed and learned a lot from the projects, and I learned TONS from the projects. There are two projects where you get to replicate the result of a paper and discuss the results. One project was training an agent via open ai gym, where we got to learn tons about value function approximation and other more recent SOA RL implementations.

One of the challenges of the project was that it takes a while to get decent enough results to write the paper. In retrospect, I should have stubbed out sections and wrote as much as possible despite not having the perfect results. I eventually was able to replicate everything perfectly, or got decent performance results, but the last minute writing was stressful.

There are homeworks which really helps you materialize what you have learned in the lectures.

The TAs are fantastic - make sure to go to all of the office hours.

Make sure to read all of the assigned paper. I only realized in the end that many of the lectures are actually based off of the papers. This is a seminar style course, so it makes sense that they have several lectures based on key papers in the RL field.

I enjoyed this class and definitely learned a lot. The homework assignments were easy and sometimes helpful for understanding the material. They only took a few hours each and were graded as you submitted. The projects required way more time and were more interesting. The grading for the projects seemed a bit all over the place. I received my lowest score on what I thought was my best work and my highest score on what I felt I had rushed. The final exam covers a lot of material that you never really had to think much about throughout the entire semester. I didn't get to study much for the final and was genuinely concerned I failed the course, but somehow ended up with an A so there must be a very generous curve.

Way too much theory for 1 course. Final exam is closed book and it is impossible to remember all the concepts taught in the course. Homeworks are manageable, though not easy. Projects are brutal.

Videos were refreshing and format of a two way discussions between professors was engaging. Homeworks were not super easy, and there was some learning from them also.The projects are hard, more than just getting the code to work, forces you to learn more by asking to write reports and explain the results.The videos were a very good summary of the reading material books/ publications. Be prepared to put in a lot of effort. I think the overall final grading curve was liberal (I got an A with >80% overall ) even though the individual project and final scores were strictly graded. I did the course before ML but after AI. Overall, highly recommend this class

This class is an amazing dive into Reinforcement Learning. Here are my pointers for anyone considering this class.

• The TAs are amazing and engaged so take advantage of the office hours and study sessions.
• Be prepared to write a a lot.
• Be prepared to read a lot.
• Start the projects as soon as possible.
• Brush up on probability, statistics, and notation if it has been a while.
• The pace is quick so try to avoid falling behind.

This is my first course in OMSCS, loved it.

The workload is heavy but you learnt a lot. I read the previous OMSCentral reviews carefully so I came prepared. I think it's really helpful to read the tips given in a previous post especially these couple points, so I will emphasize again here:

1. Start Projects early, all of them. I started all my projects 1 full month ahead and i can barely finish up it couple days before the deadline. Especially project 2, the run time for each experiment is very long, so you have to run and wait for couple hours and make change then run again.

2. Videos are more important than readings, except for Projects-related.

3. Google the projects, will save you huge amount of time. Don't use the code though, only principles.

I don't recommend take this course at the same time with another course. I only took this one course in my first semester and I almost used up all my after-work time for the project and lectures. Assignment is easy enough you can usually finish within a day, but project is way longer time consuming. But i found the concepts learnt very interesting. TA/instructor's reply time is fast enough on Piazza, usually within half a day. I would definitely recommend this course.

The only thing I don't quite like the course is the structure. I feel the topic are like jumping around. I would learn better if we were introduced at the beginning about the big concept like model-base vs model-free learning, deterministic MDP vs stochastic, on-policy vs off-policy, online vs off-line etc. instead of just mentioning certain topic in different lecture videos. It's a bit difficult for me to connecting the dots before final exam. So I end up did really well in the assignment and projects, almost get 100% on all of them except one. But I did not so well in the final.

I've lived in the US for almost 10 years now, and watching the video of this class just makes me feel like I don't know English. I really hope they can show the contents in a clearer way. Watching the videos is just so painful :'(

My main grip with the course was the "lel epic bacon so randum xdeee" humor that was way overdone. We get it, this class is so nerdy and dorky and warm and fuzzy.

Although it is a theory heavy topic assessments were all on the applied front for the first month of the class before I dropped e.g. experiment reproduction from the Sutton paper or Heroku coding labs. Still the lectures were focused on the theory but then they were rather surface level showcases, alternating between either basic intuition, rote theory or partial discussions on the rather more obvious concepts like contraction and convergence, and at the end they were not really helpful to actually passing the class or understanding the material at all -- too redundant, trivial and esoteric at the same time. Assigned readings were foundational papers like the aforementioned Sutton paper which is nice but then not exactly an opportunity for a deep dive or anything and frankly not that useful compared to alternatives.

At the end it all seemed like added up to an introductory level MOOC rather than an in-depth class on RL supposed to be taken after ML. A rather lukewarm experience, although it should take minimal time to get an A with some ahead-of-time studying with the Sutton textbook.

The Homeworks all required the student to get a non-obvious trick on their own, but beyond that, the actual solutions were very easy to get.

All three projects were challenging to a degree, but only in the sense of the amount of time, they required a lot of parameter tuning and experimenting, not that the subject was challenging.

The final exam was a test on paper reading and watching the video. I considered it largely a pointless exercise in trying rememeber random trivia about what you studied in the class.

Dang! This class was the most difficult and the most rewarding class I have ever taken. But it is graded with a generous curve, so I somehow pulled an A. The lectures are quite enjoyable for the most part, although they get pretty involved in the math at times. I found that I was drifting off as they discussed proofs over conctractability or something like that (not the most memorable as you can see). But for the most part the interactions of the professors is funny and instructive, a nice combination to be sure!

The assignments were either interesting and engaging or a complete waste of time, nothing really in the middle. I liked the assignments dealing with Q-learning and Deep Neural Nets, but the multiple agent linear programming assignment was a nightmare, as I had to learn what linear programming was and how to get it to work in my environment in a couple of weeks. And let me warn you that there is no such thing as an intuitive linear programming library. They are all awful. Why do we even need that project??? Same kind of goes for the KWIK learner. I didn't use any RL techniques, I just hardcoded the bot because that was the only way I could figure out how to get it to pass. Perhaps they could have explained KWIK better instead of just mentioning there is a paper on it and hoping you can understand it enough to solve the problem they present. I guess I didn't?

The second project about broke me but it was so cool! And even though my agent was under performing (due to a misunderstanding of the Keras API), I still got a decent grade because I was able to explain with great detail and understanding how it should work.

The final got an average close to 50% so I got above average at 60%. One of the most useful things for me was watching all of the lectures again at 1.5x speed and skipping the really math heavy sections.

My big frustrations were that they expected you to have a linux setup and it almost felt like "any self-respecting computer scientist should be using a linux to begin with". Well 90% of desktop computers are Windows, so maybe add a section on how to setup on Windows. Most helpful thing I found was setting up Anaconda and letting it do the work for me. That solved most of my import issues. But I spent way too much time before that just trying to get environments up and running. Otherwise, the TAs were very helpful, especially in the office hours where they drop all sorts of hints to help make your life easier.

Overall a great class, just be ready to dump a ton of time into it because those projects take about 20 hrs/week easy for 2-3 weeks.

Great course overall !!

Projects and homeworks were top notch, but lectures weren’t that good. I suggest to see few of David silvers lectures.

Keep noted that this courses introduces you to several RL topics, it doesn’t go in depth except for some where you will be doing projects, homeworks etc.

There are lot of papers to read for the exam, but don’t worry you just need to understand the gist of the paper.

I would recommend this as a first course as well. But don’t pair this up with another hard course.

For projects and homeworks, just start asap. Try to attend office hours as well. Projects require coding and writing report to present the results.

Overall, you will definitely enjoy this course. Go for it!!

Took RL as a summer course and it was a lot of material to absorb during the shortened summer term. They do not remove any assignment or exams during the summer so be prepared to hit the ground running. There are 6 fairly easy assignments and 3 intensive projects. The first project was nice because you got implement TD lambda algorithm but it was somewhat annoying because you're asked to recreate the experiments from a seminal paper that was extremely lacking on the details of how the experiments were set up. The second project involved implementing a Deep-Q network to have an agent solve the OpenAI lunar lander environment and this was the highlight of the course. The third project involved recreating a paper on stochastic games and correlated-q learning. It was very lackluster because of the way it is set up and needs to be replaced in my opinion (the material in the paper was interesting though). The grading is not as severe as in ML but the curve is ultimately the same. Exams were very fair but comprehensive. It was nearly impossible to keep up with all of the papers during the summer term. There were some weeks where we were asked to read 5 papers in a week. I was unable to read all of the papers over summer but did skim the conclusions of most and pulled an A on the Final. There were also a lot of lectures to watch and I watched at 1.75x speed to get through them. The lectures for this class need to be refreshed and I recommend watching David Silver's lectures in conjunction (or better yet before) to bolster the concepts. Exams will be based on this courses lectures however so you must watch them.

This was my fifth class in the OMS CS program, and I would consider it "Very Hard" if taken over the summer because of the condensed schedule. There was a lot of material covered in the class between each of the 3 projects, and there was only around 1 week to study for the final exam (there was no midterm or other quizzes / tests / exams). There were also 6 homework assignments, but these were much easier by comparison because there was immediate feedback through the online console for whether your solutions were correct or incorrect.

I would strongly recommend taking CS 7641 Machine Learning beforehand, since this class is essentially a continuation of that with more focus on reinforcement learning topics. CS 7646 Machine Learning for Trading would also be a good prep class to take, since it uses Q-learning as well as Pandas / Numpy if you are not super comfortable with those. For the most part, the entire class used Python for all assignments.

In general, this class felt very much like a true graduate-level course where students replicated published results from papers for Projects 1 and 3. However, Project 2 "Lunar Lander" was by far my favorite and the most "real" / tangible application of reinforcement learning. Regardless though, the projects are very time consuming and cannot be completed over a single weekend.

Each project also involved a 5-page report, so there were many late nights trying to fit all the analyses and graphs into the page limit. Unfortunately, similar to CS 7641 Machine Learning, the project grading was a bit arbitrary and subjective between TAs with some grading much harder than others. There was a general outline and rubric to follow, but the grading criteria could have still been made a bit more explicit.

The final exam was 90 minutes and all multiple choice similar to the format from CS 7641 Machine Learning. It was difficult to study for since any of the material from the semester could be on the exam, but I felt the questions asked were fair for the most part. It's another one of those classes where you do not worry so much about your grade since there is a pretty decent curve at the end, so it is instead better to focus on learning and practice with the topics reviewed.

Overall, I definitely recommend taking this class if you are pursuing the Machine Learning specialization!

Took RL as my second course in the programme, after AI. Found RL to be significantly harder than AI due to the report writing required for the projects (which means you're never really "done" with a project, there are always more experiments to run, more things to write about) and the fact that the summer semester is shorter but we still covered all the content that would be covered in a regular semester.

Plenty has been said about the homeworks and projects, so I won't add to that here, except to say that I feel Project 3 should be changed to one that doesn't involve as much hyperparameter tuning, since that was something we did a lot of in Project 2. I would have liked to work on a project that used options as well.

Because I hadn't taken ML prior to taking RL, I prepared by watching the David Silver lectures and reading the first 6 chapters of the Sutton & Barto book. Those provided a better big picture view of RL than the official lectures, which jump into the math quite quickly.

Ultimately, I walked away feeling as though I'd learnt a lot. The course would have been even better if we'd delved deeper into some of the topics, such as POMDPs and options / hierarchical RL. I would also replace most of the readings that are not required for the homeworks / projects with chapters from the Sutton & Barto book because frankly, most of the papers had blurred together by the end. Reading the book would also have led to a more cohesive learning experience and perhaps allowed us to cover additional material not in the lectures, like policy gradient and actor-critic methods.

The course is good and the subject is really interesting to me, but taking ML first has increased too much my expectation for RL. I was hoping for a more practical coursework, but the lectures/readings are too theoretical. I've enjoyed way more ML than RL.

Projects 1 and 3 are a bit boring, you have to replicate 2 papers but I've spent more time guessing what hyperparameters the authors used instead of actually implementing and analyzing results.

Project 2, on the other hand, is super cool. You get to implement a real agent that lands on the moon in a simulated environment. I've learned a lot from it.

Besides the projects, there are 6 homeworks that you have to implement some RL algorithms. They fun to work with and helped me to learn.

To wrap up the course, there is a final proctored exam. It's not super hard, but it's hard, be prepared.

Overall, the course is not easy. Make sure to invest enough time, especially in the projects.

I don't know if that's because I took the course over the summer, but I felt that TAs didn't help much on Piazza, they simply ignored many of the questions made by students. I don't know also if that's because Isbell became the Dean but we haven't had any interaction with him other than by watching the recorded lectures. I remember having him very active on Piazza in ML.

I took this class in the Summer of 2019. I had taken ML before hence this class was not very hard to me. However in the summer, I had my share of anxiety and sleepless nights. RL is an amazing concept with many real life applications. However, this course was made atrocious by a disastrous sequence of lectures. It was total garbage and was the first course (this was my 5th) where I just couldn't finish watching the lectures because of its uselessness. Why on earth did they think of adding jokes every now and then in the hope of making it interesting. They ruined it actually. Lectures were insanely long. And I just couldn't watch them without falling asleep. They need to revamp the lectures to bring it to an acceptable level.

You need outside help to survive this class. David Silver or Stanford lecture series will help a lot. The Sutton book is extremely useful in understanding key concepts. There are 6 HW assignments and the first 2 are the hardest. The last 4 were easy and were done in a week total. P1 was easy, P2 was medium while P3 was hard because of lots of missing information. TAs were very good and the Prof never showed up on Piazza.

Taking this class in the summer is brutal as you have deadlines every week and you cannot take your foot off the gas. Be prepared to spend some sleepless nights especially for P3. There were students who were coding up until the last minute of the submission deadline. And if you haven't taken ML or have no background whatsoever, your only option to survive this class is to drop it.

Overall I enjoyed this class not because of the way it was run but because of the cool concept and my interest in learning it. Avoid taking in the summer if you can unless you can put in some significant effort.

If you're thinking of taking this in the summer....having NOT taken ML....just don't. Don't do it. The pace is absolutely brutal with a homework or project due every week, 5 papers to read, and 1-2 lectures to watch. It's even more brutal when you don't know what you're doing and this is your first ML course. Just don't do it.
The material is very interesting and it's a very important part of ML, but the lecture delivery is the worst I've ever seen. You will end up watching a second set of lectures from Silver or one of the Stanford classes. They throw you in the deep end in shark infested waters with nothing but a due date and a link to a paper as guidance. The material is too deep to absorb in such a short period of time. I'm going to have to properly re-read everything in order to absorb it. You may be able to earn an A, but was the PTSD really worth it? Take it, but take it in a regular semester. You'll suffer, but less.

Overall, this course was the most useful for me in OMSCS.

Pros: {1} You will learn a lot. {2} Very practical, hands on. No hand-holding. {3} Lectures are good (contrary to some opinions here) and enough for almost everything (except P2, read Mnih 2015 paper for that and google). {4} Homeworks are not hard. {5} Generous curve. {6} Final is general concepts only, no tricky details.

Cons: {1} Unclear hidden rubrics for projects. TAs get wild. Some people get 105 without good replications, some get 85 with ideal agents and extra explorations. {2} Regrading policy is strict. Either you gain 5+ points, or you lose 10. {3} Spiky workload. Almost none in some weeks and 50+ hours in others. {4} Unlikely to pair with other courses. {5} Instructors/TAs non-existent on Piazza. {6} A lot of readings, that are almost not used anywhere. {7} Curve for everyone does not let predict your grade. {8} Many concepts are barely touched, but present on the final. {9} Project 2 requires you to implement untaught concepts, like DQN. {10} Endless hyper-parameters tuning. You will see your life burn away.

Tips: {1} Start Projects early, all of them. At least 1 full week in advance. {2} Ignore readings (including Sutton) till the exam, except for Projects-related. Even before the final only skim, read Intro and Conclusion. {3} Re-watch videos as often as possible. {4} Max-out pages in reports, use 2 columns IEEE format. Write about everything you can think of. Go through pseudo-code, algorithms and formulas, implementation details, everything. Or risk losing marks. {5} Google the projects, will save you huge amount of time. Especially on P2. Don't use the code though, only principles. {6} Make sure your implementation is correct before hp-tuning. Unit tests and debugging, as usual. {7} Go to slack for advice, ignore Piazza. {8} Ask for clarifications in private post to Instructors and your grader on Piazza to get a potential regrade without risking -10. Don't forget to fully justify your opinion. {9} Watch David Silver's lectures in advance, will help a lot with developing a proper intuition. And doing this course after ML will help a lot as well.

This is a very cool course. Super recommended! Some of the lectures are a bit archaic, but overall, it is a very solid and fascinating theory class. Sometimes I felt the homework and projects were so hard that I almost wanted to give up several times, but eventually, I persisted in working and survived. The lecture has some proof and theory that really help students understand why some algorithms work. In fact, many of the convergence proof ideas are borrowed from the results from real analysis and metric spaces. If people have taken undergraduate real analysis class, all the proofs in the class will make more sense and they should be able to re-derive the proofs to strengthen the understanding of the material.

The best knock against reinforcement learning, I think, is that it’s not shown itself to be super-useful. The go-to examples of why it is important include Google’s DeepMind playing Go, Stanford getting a toy helicopter to do stunt tricks, and DeepMind getting a computer to play some Atari games really, really well. While all those are cool, none of them sound super-profitable. I guess there’s driverless cars too if you want to go for that. If you don’t, be warned that this is a rewarding class that is a lot of work, but you are unlikely to go do a reinforcement learning job afterwards simply because companies aren’t looking to fly stung helicopters.

I took this after ML, which is the recommended sequence. I think that’s fine, but it’d be fine reversing the order as well. ML is the more practical of the two.

If you are doing the ML spec, you should take this as one of the electives. Sure, CV is probably better. RL, however, is a better class than DVA for the reasons mentioned in the DVA reviews. ML4T is just numpy for babies.

Regarding the advice in other reviews, it’s pretty good. Watch the David SIlver lectures first. Isbell and Littman’s thing where they pretend to teach other like they aren’t already experts works better in ML than here where I think the one playing student is a bit too quick for my tastes. The projects are basically OK. Be aware that you are primarily graded on your report and really not at all on how your code performed, so write a good report and read the assignment about five times to make sure you hit everything.

The final exam is hard, but basically fair. Everything is stuff that was covered in lectures. I didn’t do any of the reading but will end up with an A in the course. You do have to make sure you actually understand the material though. Good luck!

Super hard course, a lot of reading, lectures are decently long, projects are tedious and hard to replicate the results from the papers (they involve replicating research papers). If you take this class during the summer, you will not have time for anything else. There will be homeworks every weekend (can take a decent amount of time) and 3 projects, with no free weekend. There are also tons of papers to read. Lectures are not really useful for projects, they just present some high level concepts from the readings. Also, a lot of the parts are kind of hand-wavey.

This course is complete garbage. The lectures are terrible. The projects are terrible. I really regret taking this course.

One of my favourite classes in the program! I loved the RL portion of ML and this class takes it further. RLDM surveys the important aspects of both RL and DM and will arm you with foundational knowledge so you can understand SOTA. Taking this over the summer was grueling with a constant stream of work every week, my advise to anyone looking to take this over the summer is to get a head start in Silver's lectures as well as course lectures.

I would say this course is not bad. Here are my suggestions Projects: write the report carefully and pay some attention to the project requirements. Don't need to spend a lot of time on the code. Homeworks: these are the easy part of this course, make sure you get 100 of all it. These are the only assignments that you actually can control how many scores you can get. exam: this is a joke. Just watch the video many times and take notes as much as possible.

I don't have much contribute beyond the reviews below. You will get 3 hard projects, 6 easy homeworks, and 1 one poorly structured final (50 average in our class, basically 1 or 2 questions per course section). The projects and the homeworks cover about 10% of the videos and 5% of the papers. The rest is covered in the final. If you think this is a strange distribution, then I would agree with you. I believe that for our semester anything above an 80% got an A; so you should be able to make it.

My biggest concern for the course was the distribution of learning; once again, it often felt that it was not worth investing the 2 hours a week watching the lectures. All those papers? They won't help you with the majority of assignments, I only know this because I did not really read any of the assigned readings, got 100's on all the homeworks and had a high average on the projects. That being said, you can write 3 papers of equal quality on the projects and odds are you will get 100 on one of them, a 90 on the other, and a 70 on the third since each will be graded by a different TA, and the grading rubric is hidden. At the same time its probably not worth asking for a regrade, since you will get penalized more from it, and there's a curve anyways that largely accounts for these distributions.

Still, I was mostly disappointed in this course because I loved ML. I felt that there were useful nuggets in almost every video section; it was probably the first class where I actually enjoyed watching them. I think, for me at least, the biggest issue with lectures was that the instructors quickly jumped to fairly complex mathematical definitions which do not truly work with the video lecture format (you can't ask them to clarify things as you could in a normal class, and pausing to take notes is fairly sh**tty).

I also just think that I"m less interested in say, a rigorous proof for why policy iteration converges, as I was in all of information theory.

All that being said, I do think there is a kernel of a great course here. The last lecture summarizes it -- you are learning more about how anyone learns, more about the philosophy of effective exploration -- then about any specific algorithm. I do find that fascinating, but I do not believe that this course quite lands that plane. The homeworks are too easy, the projects are good, but I do not think you learn that much from the part of the project that involves playing endlessly with hyperparameters (all 3 projects have some of these components, but especially project 2).

FInally, most reviews mentioned something about using Deep reinforcement learning. This is pure shenanigans. You can choose to use a neural network for the second project, but if you do odds are it will have 2-3 fully connected layers. You definitely won't use CNN's or anything similar. You will read a few papers related to Deep reinforcement learning, but you won't be learning from the pixel data, you'll be learning from an already extracted vector of features. Just wanted to point this out because I think a few people may take this course just for that project, and you should know what you're getting into.

There are 6 homeworks, 3 projects, and 1 final exam for this course.

Homeworks are mostly short coding tasks and the grades are given in an online grader immediately. They are not hard.

Project 1 and 3 are to reproduce some results in papers and project 2 is some computational experiments. The projects are interesting but you need to summarize all you've done in 5-pages reports. The rubrics are very ambiguous and I got some unreasonable points. Requesting regrading can cause an extra penalty.

The final exam is hard. It contains 25 T/F questions which require reasoning. The average score is around 50%.

Personally, I do not like the lectures AT ALL!! The teachers spend a lot of time joking and they keep distracting me; while I do not think they explain well in many important concepts (being too fast).

The final letter score is very generous.

I really enjoyed Reinforcement Learning - it's a lot work. I did very well (A) and felt like I understood everything but man, some projects were brutal (20-40 hours) and the final was tricky. RL is a broad survey course with the first third spending a lot of time on temporal difference learning, before moving to a variety of theoretical ideas (e.g. convergence, exploration, reward/policy shaping, generalization, POMDPs, and options), and finishing with game theory.

Project 1 was pretty easy and looking back I'm glad it forced me to master TD learning. Project 2 was the most enjoyable - you could reimplement DQN from DeepMind's Minh et al. 2015 paper on Lunar Lander. Project 3 was very hard, but classical RL + game theory will help you understand modern evolutionary game theoretic ideas (e.g. see AlphaStar). Project 2 really made me appreciate the need for RL hyperparameter tuning - DeepRL really won't work converge unless you find the goldilocks esque combination of hparams. Training instability is real. Even though HWs were relatively straightforward (some aha moments), the frequent volume of due dates and large number of survey topics make this class very time consuming.

Read all assigned 20+ papers! If you do, you'll be in a really strong position to read any current RL paper from arxiv.org and know what they're doing. The class lectures basically walk through a subset of the papers and explain some the derivations. Watch David Silver's lectures on YouTube beforehand or in tandem with the Georgia Tech lectures.

I thought RL and ML were really well done - I think criticism of it being 'outdated' (circa 2015 hah) miss the point of the class. Reading foundational older papers give you ideas to holistically 'stake out' the RL field. You soon realize there's < 50 core concepts, of which modern DeepRL are often reapplications of older concepts. You also have the chance to apply DeepRL (IMO just read Rainbow paper then implement DDQN) in Project 2 (but I already knew DL very well). You DO learn practical knowledge because you need to use theory to understand why your RL model isn't learning (as evidenced by a recent work project). Sorry, but sklearn plug-and-chug simplicity doesn't exist (yet) for applied RL problems. You'll need to debug your rewards, agent selection, and environment design and know wtf you're doing.

I do wish RL spent time on policy gradient methods though instead of just value-based methods. They also breezed through MCTS and model-based methods, and my takeaways was POMDPs aren't useful. I thought options were stupid, until I started reading recent Hierarchical DeepRL papers and now I'm glad I forced myself to read Sutton 1999's paper on it during class. But you can find more info from YouTube's DeepMind courses and some UC Berkeley DeepRL lectures. If you lack a strong math background you might struggle, but I didn't think the math was hard (I'm also an experienced data scientist, so take that for what you will).

'Deep RL (Need Not || Doesn't) Work Yet': https://www.alexirpan.com/2018/02/14/rl-hard.html

Took this one as a natural extension from ML last semester. Compared to ML, I liked this course a little bit more because RL, unlike ML, does have some coding homework and projects in addition to writing reports.

There are 3 projects:

1. replicate paper result on TD. (easy/medium, disliked)
2. train agent using DQN for "Lunar Lander". (medium/hard, liked)
3. replicate paper (again!) on multi agent learning/game theory etc (hard, disliked.)

The only project I enjoy is #2 where you gain practical knowledge on using deep q learning network to train an agent to play a game, in which you will spend a lot of time doing hyperparameter tuning, but the end result is rewarding as it opens the door of reinforcement learning application in game playing and some research done by DeepMind, e.g. AlphaGo, etc.

But I'm not a big fan of academic research so reading and replicating a paper is not for me. Project #3 costed me more than 40 hours to finish, while got 100/100 on grade, I don't feel I learned anything useful except the skill on translating math formula into python code, (and may be writing report in a pretty format helped too.)

The worse part of this course that I want to complain is the final exam. I got a B due to bad performance on final exam even though I received more than 90% of the points for everything else. This is because exam take 25% of the total grade while there're 25 T/F questions total in 1.5 hours. Do that math, one question takes 1 point in your final grade. and 40 hours spent on project 3 only worth 15 points total. Plus, there's no chance to request for regrade as they don't even release any information on where you did wrong and what should be the right answer.

I guess getting a B is not that bad, the worst part is that I have no chance to know why and no chance to learning from it at all.

I'd be neutral on recommending this course, yes you can learn something from it but you'll need to manage your expectation well to avoid getting disappointed by the quality of this course.

The projects were the only interesting part, but the last two are very time-intensive.

Lectures were terrible. I stopped watching them early into the course. It seems like many others did the same.

Zero engagement from the Prof, TAs were somewhat unhelpful and gave alot of vague answers (mostly because I don't think they knew the answers themselves).

The average for the final exam was ~50%... it's a difficult class due to both challenging material and poor structure.

I would say avoid if possible, maybe watch the Silver lectures on Youtube instead or just read Barto & Sutton.

Fairly high workload, but overall a very fun topic and taught fairly well.

The three projects in the course were demanding but rewarding. Lectures at times felt a bit disconnected from the projects/homework, but still added value.

If you haven't done anything with machine learning already, I recommend taking some other courses (coursera or CS7641). This course covers different material from general ML, but there are many shared themes.

TLDR; Great projects and bad lectures make for an OK survey of RL. Definitely not one of my favorite courses. Updating lectures would go a long way in improving this course.

Lectures - As many other reviewers have mentioned, the lectures are not very engaging and were generally very difficult for me to watch. There are several hours of watching Michael Littman quickly discuss intricacies of different equations without doing a great job of tying it back to a larger picture or concept. As with ML, lots of bad jokes between the instructors, which is fine but makes the lectures longer than they need to be.

Homework - Generally do a good job of testing material. I was able to finish most homework within one or two hours. Most homeworks required going back and doing some reading

Projects - There are three projects and the difficulty increases nonlinearly from 1st to 3rd. Overall, the projects were my favorite part of the course and where I ended up learning the most. 1st project can be finished in a weekend, but make sure to allocate more time for 2nd and 3rd projects.

Exam - One final exam. I thought the exam wasn't a great test of knowledge. Anything in the lectures is open game here and there are a lot of topics that are very quickly touched on in lectures and then never revisited in projects or homeworks. Take good notes during the lectures if you expect to do well on the exam.

Instructors / TAs - Instructors are more or less absent from piazza. However, I thought the TAs were great for this course. Very knowledgable and responsive.

-Pros

You will learn to manage your time, read research papers thoroughly and replicate their experiments.

You'll get the basics of RL.

-Cons

lecture videos are horrendous; too many jokes such that it hurts the learner from grasping concepts.

You don't get much help from TA's

Outdated material

I don't recommend this course, hopefully it gets a rework in the future. In the meantime, if you are curious about RL, just watch David Silver's youtube videos, you'll get more out of that then this course in it's entirety.

Pros:

1. This course will force you adapt to grad level study, which means you need to be able to read a paper and replicate the models in paper
2. TAs reply on piazza

Cons:

1. Poor lecture, to many jokes and off-topic discussions in lecture videos
2. Zero coding sample in lecture, this courses is basically teach basic markov chain, DP, and game theories. You won't learn practical knowledge

The material is engaging and fascinating. The lectures are competent, but cover almost no relevant theory and hand-wave all the interesting parts. I watched all of Andrew Silver's RL lectures beforehand, and I think that's necessary to avoid spinning your wheels on trying to understand RL through the lectures. The papers are well-curated and fascinating, while still accessible to the student audience. The TAs are decently responsive on Piazza, as is the knowledgeable portion of the students. Isbell is absent entirely, but as those who have taken ML will attest, this is entirely to RL's benefit.

Two of the projects are well-designed. The middle one, which involves trying to put together a video-game-player using tools and infrastructure not covered in this class (or, frankly, in ML) was one of the most frustrating experiences of my life, and I am no stranger to over-ambitious projects. As compared to nearly every other class I've taken in OMSCS, the busy-work:workload ratio was the lowest of any class except CCA. This applies pretty evenly for every assignment except Project 2, which was a huge waste of potential.

The concerns about grades are overhyped. The project averages hovered around 80%, and for some reason the exam average was 57%, but this was my first encounter with ML of any kind, and I never got below a 92% on any project or the exam. You just have to be willing to put in the research, work through the proofs, and understand the theory. From the piazza, it was pretty clear that there were a lot of people looking for shortcuts, which may account for the low average scores. With the extra credit, I wound up with 97%. Again, that's without any background in ML or CS. You just have to read the papers and follow the proofs.

I'd recommend this class. I took it before ML, and thank goodness I did, because the difference could not be more stark (I likely wouldn't have taken RL if I'd had to endure ML first).

Comments were pretty spot on for this class though opinions will very. Not a killer like ML, AI, or GA but definitely on the more difficult end like a 4 out of 5 on the scale. Based around reading papers and replicating results so expect a lot of learning a lot of the more difficult mathematical/algorithmic concepts on your own.

Background: This was my 4th class, and I have taken ML4T (and 6340 and 6250) but not regular ML before taking this course. I also have done a fair bit of reinforcement learning at my day job in the past and had watched the David Silver RL lectures a couple years ago.

Lectures: The lectures are starting to show their age. Deep learning for RL plays a much bigger part in the wild and is largely untouched in the lectures. Overall I thought the lectures were not too hard to follow but also didn't really help that much with the projects. They were more useful for the assignments and final review.

Paper readings: There are kind of a lot of papers in this class. I think most of them your could probably skip. Just read the ones that are needed directly for the assignments/projects and skim the rest.

Assignments: Much easier than the projects. I was able to 100% them all, finishing a few of them in only a few hours after really starting to code them up. Love it when classes have an autograder. Each one had 10 example inputs and you could attempt each one 10 times.

Projects: In general it is totally unclear to me how these are graded and it was sometimes difficult to cram everything into the maximum 5 pages. I thought the first one was the hardest because I couldn't reproduce the charts at all even though my calculations seemed to match those that we crowd-sourced as a class. On the second one I took the opportunity to use some deep learning and probably ended up doing way more work than was necessary but because it was more open ended was more enjoyable. The last one I took less time on thanks again to helpful hints by other classmates and the short linear programming tutorial by one of the TAs. I also found the feedback totally lacking on these projects. We got P1 feedback right as P2 was due and even then the only thing I was told for my 85% was "Good job, even though you couldn't reproduce the charts you seem to understand the experiments." P2 had slightly more feedback but also didn't come until after P3 was due and I have a hard time believing that P3 was really even properly read through.

Final: 30 T/F with explanation in 90 minutes on ProctorTrack. Several obscure questions and a few things that were easy to get flipped in your brain. When taking it I didn't think I had missed that much but ended up getting like a 70% on it, so my explanations must have been inadequate.

Overall: Thanks to the curve got an A. TAs were fine. BlueJeans for office hours was sometimes annoying. Poor audio a few times, screen messed up/resolution too low to read. I didn't listen to every one Piazza got quieter as the semester went on. I think Isbell made one comment the whole semester. In general I think a useful course but not sure how good it would have been if I didn't already know a lot about MDPs and Q learning beforehand.

Overview

This was my first ML related course and it was a doozy! Very challenging topic and be prepared to do a lot of self studying. I hadn’t taken ML or ML4T or really any linear algebra before this so I had to spend a lot of time wrapping my head around the concepts and the math. That being said, the TAs are phenomenal and never underestimate the value of Slack. Everyone is in the same boat as you and there are plenty of people wondering about the same questions you are so jump on slack and interact! I learned a lot through discussions in slack and was also able to contribute back as well.

Structure

Homework/projects were released in batches. I think it was something like 2 or 3 homework’s and a project would all be released at the same time. Then 1 hw was due each week and if you kept on schedule you would have 2 weeks to do the project. Do not wait to start the projects. Start them as early as you can. There was also generally 1 lecture and 1-3 papers to read each week. I did find the papers a bit overwhelming at times. They are deep in many cases and I definitely didn’t understand them all - though by the end of the course many of them did make a lot more sense, especially after applying and seeing some of the base material so many times. There was only 1 exam - 25 questions, T\F and you must justify your answer to get full credit. The exam definitely tests your comprehension of the subject. You can’t just read someone else’s notes or study guide and expect to do well.

Lectures

The lectures were great to watch. Isbell and Littman work very well together and make the lectures generally enjoyable and informative. That being said they can be very abstract so it really made you work to understand how you would actually apply the material.

Hw and Projects

I didn’t find the homework’s to be especially difficult and they mostly helped reinforce what you were learning. The projects were very challenging, especially the second one. It really gives you an appreciation for what’s going on and makes you learn how to approach and solve a problem, especially when the solution domain is incredibly open ended. The projects often require a lot of re-reading papers and lectures and that can be time consuming. Your solution may also take a long time to run depending on how you designed it and what algorithm(s) you decided to use. So start them early and ask questions. Each project consisted of you writing your own code and then a max 5 page report.

Conclusion

Overall a really challenging, but fun course. I found it very taxing and trying at times, but am glad I took it. If you can, probably better to take ML before this course. I ended up with an A, but did spend a lot of time to get there. None project weeks were reasonably light but project weeks were really heavy.

Took this course Fall 2018, not listed in the drop-down currently.

The six homework assignments were relatively straightforward for someone who has taken ML4T, DVA and ML first.

As with other students, I found project 2 (of 3) involving solving the Lunar Lander OpenAI gym to be extremely difficult. Despite a poor showing there, I still managed an A in the class due partially to the generous curve. My recommendation for project 2 is to read the relevant papers early and thoroughly.

Projects 1 and 3 involved reproducing paper results and required significant time but were not unreasonable in scope. The three projects included coding the reinforcement learning algorithms from scratch, analyzing behavior with graphs and text discussion in a 5 page paper (which seems not enough space at times).

There was no midterm, so ample rope to hang oneself on the final. The final used ProctorTrack, but that seems to be standard operating procedure for most classes in the program. Watch all the lectures, take good notes, rewatch the lectures at 1.5x and summarize notes for the final. Not easy, but doable in a semester.

I used python for all coding. Unlike ML, there did not seem to be many people using other languages in this class. Although one could certainly code projects 1 and 3 in a different language.

The TAs held weekly office hours that were recorded, which proved extremely useful given my schedule. TAs were fairly responsive in Piazza. Overall I am happy with the assistance provided and grading feedback.

Almost 30 papers were assigned readings. The relevant papers were mostly important for doing well on the projects. Most final exam questions seemed to come from the video lectures.

The mean score on the final exam was 57.2. The high score was 92. So, do not freak out if some of the questions seem impossible.

NOTE: I took the course in FALL 2018, but this wasn't available to select in the dropdown.

This course was a disappointment overall. The workload was challenging, but not for the right reasons. Most of the TAs were condescending and unhelpful. The regrading policy was draconian: If you asked for a regrade, but didn't get at least five points in the regrade, you would lose ten points (for failing to internalize criticism). This would be fine, except that you have no access to the grading rubric, because... reasons. Additionally, grading was super-slow, with often a month elapsing between submitting a project and getting the grade back. This was particularly annoying when they missed getting an important grade back prior to the drop deadline. To make matters worse, approximately 2/3 of the way through the course, they "clarified" that you could ask questions about your grade without requesting a regrade. Presumably, they'd received complaints, but it struck me that this system was put in place to save the TAs from having to work, rather than for any benefit of the students.

The material is dated, and the lectures are very math heavy. The problem with that is that the math is presented as background material, and is poorly explained and has key details glossed over. Moreover, you don't actually need to know it for the course, and it's there at the expense of other explanations and more up-to-date material.

On top of that, two of the three projects require techniques that aren't mentioned at all (or are only barely touched upon) in the lectures. (Deep Learning and Linear Programming, respectively).

The projects are interesting, but more often than not, you are fumbling in the dark for solutions/hints that can only be found by searching the web, or watching David Silver's lectures.

This course needs a serious revamp. For one, the field has advanced a lot since the lectures were recorded. Secondly, the teaching style, was fairly adversarial. I get that it's supposed to challenge the students, but the challenge should come from the material, not from the attitudes of the professor and TAs.

Overall, I would not recommend this course. It was more difficult than it should have been, and for the wrong reasons. I want to feel like I'm being challenged to learn on a course, not condescended to. I was not the only one to feel this way.

Note that I did well in this course, getting an A, so this is not just sour grapes on my part. This course has genuine problems. You should only take it if you really want to learn RL (outdated), and really don't mind being in the dark for most of the course.

An additional tip if you take this course: Watch David Silver's lectures in addition to the course lectures. They cover the material much more thoroughly and clearly.

In short, this course is a lot more work than it should be for the depth of material, and most of that time will be figuring out where to find missing information.

This RL class is a foundational class towards your understanding and further study of Reinforcement Learning. It will be a compilation of the basics of RL. If one realizes, the videos are a distillation of the required reading. It is a very well-run class with excellent TAs whose enthusiasm will just spillover to the class. Office hours are something I look forward to, and definitely regret missing.

Assignments are tough but doable - these actually test the intimate understanding of a topic - a great way to learn. Not only do you "know" something - you actually get to do something.

Projects are tough but fun! It takes A LOT of time - but with proper time management - it should be doable - they give a month to finish a project! With the difficulty of the project comes learning also.

Finally the final exam is super tough - but that will be a true test of your understanding of the course. Read and internalize the lessons. No such thing as winging it.

The course will lay the foundations for RL - but will leave you yearning for more time to cover more material and to explore more things in depth. By going through the class though, you're likely be in a state capable to go about exploring and getting more information, or exploiting what you already know. Either way you should be able to reap some rewards.

This course isn't so much hard as time consuming & frustrating.

• The material is complex but not impossible, especially if you have taken ML and/or ML4T in advance.

• The projects will make you nuts -- not implementing the concepts, but trying to reproduce with accuracy the original author's results. The time invested in testing and tuning doesn't match the knowledge accumulated. And many find the lack of an explicit rubric leads to surprising grades.

• The final exam format is, well, not my favorite. True/False Explain, and each question is crafted more as an inference trap then a test of your understanding.

• Professor Isbell is completely absent. This is a TA-run class, with mixed results. Some are quite good and supportive, while others only contribute to the confusion.

Others are right, the Lunar Lander project is a gas. In all, I was glad to take this after ML, as I was prepared for the teaching style and test format.

This is a tough course. The content is difficult to understand, the syllabus is huge, the tests are difficult, the projects too.

Having said that there is a lot to learn. I will probably get a B, but I am happy to have learnt so many things.

To succeed in the course:

1. You need to be ahead of time. Watch David Silver's lectures online
2. Watch all the Udacity video content (And more importantly make notes, so that you can refer them later. Not possible to remember everything)
3. Explore OpenAI Gym
4. Attend or atleast watch all recorded OH
5. Be active on Slack

And you'll be good to go!

The best part of this course is the TA support and the infrasttucture they have. Office Hours are priceless, even if you cant attend - the videos are recorded.

The TAs are passionate about the subject: Miguel, Chris, Alec, Tim are very active on Slack as well on Piazza. I wish Prof. Isbell showed some love to CS7642 as much as CS7641

Disclaimer: I have taken DVA, ML4T, AI4R before. This was my fourth course

I was only mildly interested in RL before taking this class and have preferred it to Computer Vision only because I liked Prof. Isbell and his teaching style after taking ML. But the class has certainly peaked my interest in RL and I now I want to learn more. There is no question about it - it's a demanding class and I spent more than 30 hours per week, overall, and easily more than 40 hours on projects. But I really enjoyed all of them. Projects and homeworks have been fun to work on, but writing papers was not as fun.

Lectures were interesting, often long and required rewatching some of them to get some concepts.

Professor was mostly absent from class, but the TA have done a great job of managing the class and helping students along the way.

Very challenging class. Challenging projects include trying to replicate results of RL papers.

Good video lectures. Some overlap with ML (RL videos from ML are reused for this class).

I don't know why Isbell is still shown as the instructor for this class, he has 0 involvement beyond the lecture videos, which IMO is a very weak excuse to have him as lead instructor when he doesn't even once communicate with students during the semester. Miguel Morales, the head TA should be the head instructor for this class and he should have the power to modify the course, he's writing a book on RL, so he's pretty qualified.

Regarding the workload it varies, I was very busy this semester and was still able to deliver all homeworks with some help from GitHub. The projects require replicating paper results, which is an absolute hell given that authors give very lackluster descriptions and not enough information to replicate their experiments, but even if you aren't able to replicate them 100% you can still get a decent grade, TAs are very fair as long as you show effort. The last project requires linear programming with CVXOPT, the second project requires knowledge of POMDPs, linear function approximation or Deep RL, the first project is just an implementation of TD(lambda).

The only exam was the final exam, which I expected to be like the one in ML (i.e Isbell's questions, which can be very hard for normal people, but in this case the exam was easier. As long as you review the material completely and understand the basic concepts you should make it. I hadn't watched more than 60% of the lectures for the semester and I reviewed them in the last week and still I think I did good.

You could dismiss most lectures and still have good grades on homeworks and projects, as long as you're fairly able to use Google effectively.

If you're expecting some interaction with Littman of Isbell, it is NONEXISTENT, NULL, ZERO. Although the TA's (at least Miguel Morales) seem to be very knowledgeable.

I don't agree with the exaggerated criticism from some students here. The lectures are pretty theoretical, but I believe they give a strong foundation if you really dedicate your time there. Be prepared to learn a lot by yourself and use other sources like the videos from Berkeley, DeepMind and David Silver on RL.

I have a ML/DL background from Udacity - Deep Learning, so I decided to take this as my first course and I don't regret it. If you don't have a ML background I suggest taking this course first.

There are 6 Homeworks and 3 projects. Project 1 and 3 are replication of papers and project 2 is the Lunar Lander from OpenAI Gym.

Due to my time zone, I always watched the recorded OH sessions in Blue Jeans, which helped a lot. This is where the TAs give advice on how to to complete the HW or the Projects.

TAs showed a lot of knowledge and where very active on Piazza. They would also show up on slack channel to help but perhaps also to monitor discussions.

I have a busy work, a wife and a kid which posed a challenge to attend this course due to the time required to watch lectures and read the papers, but it is absolutely doable if you have discipline and start the assignments as soon as they are released.

Interest in Reinforcement Learning has increased a lot in the last 3 years after the paper from DeepMind about a RL agent playing Atari Games and the launch of AlphaGo. Other players such as OpenAI, Unity are investing heavily on it.

Everyone interested in ML/AI should definitely take this course!

I have mixed feeling about this course, unlike many other reviews unfortunately. On one hand, the material is really interesting and the projects and homework are also quite engaging and I've learnt a lot from doing those. The same can't be said about the lectures...which has been said by some other reviews. Most of times I find it quite difficult to follow the lectures and gain a deep understanding of the content. There's also too much readings than can be realistically done on a part-time basis. After this course has been completed, I find that the course is more like a survey of topics in RL rather going into details on any of the materials.

This is a solid class. Couple of things to note:

Taking ML before this class is bound to be helpful, as others have noted. I did not take ML before this class, so picking up on some of the ML ideas was difficult. Second, the lectures aren't that great. Littman and Isbell have great chemistry, but a lot of the time, the importance or central idea of a concept is not clearly explained (sometimes terms aren't even defined), and is covered up in a lot of snark, which while amusing, does not help elucidate a lot of the topics. David Silver lectures should be used to supplement your understanding.

HW's are way easier than the projects. The projects are pretty brutal, but overall enjoyable. Make sure you have extra time to do them, especially Project 2, the Lunar Lander.

The final is garbage, but that's true of most OMSCS classes. There's really no idea what concepts are going to be focused on in the exam, and there's a ton of lectures, so it's easily possible to be asked a question about something you barely even focused on because it didn't seem that important. I understand the need for an exam like this to encourage students to watch the lectures, but they just aren't a really good gauge of how much you understand, so be sure to boost your grade up by doing well on the projects.

Overall, TA's were fantastic, and professor wasn't very involved, which is pretty much the norm in OMSCS.

Overall I though this was a good course. It demanded a large amount of work, but I do feel as though I got out what I put in. The first and last project revolved around implementing research papers. The middle project was based on the LunarLander OpenAI Gym environment. I'd describe the projects as such.

• P1 [Whitepaper Implementation]: Relatively straightforward, implementable using simple logic. Read the paper carefully (probably more than once), but all the information is there.
• P2 [OpenAI Gym Lunar Lander]: Straightforward, difficult to implement, computationally intensive to find the right hyperparameters (your code may be right... but still fail if you don't have the right hyperparameters). Online sources are helpful.
• P3 [Whitepaper Implementation]: Extremely difficult, linear programming required, not documented online at all, so there really isn't any external source to lean on. Utilize the TAs heavily for this project in office hours... you'll need them.

Difficulty: P3 > P2 > P1

Homeworks: Easier of course than the projects and they help build the fundamentals. They provide immediate feedback which is nice.

TAs: ridiculously excellent. BlueJeans is really nice for office hours. All the TAs are very helpful and are active on Slack a lot (and Piazza of course). The supplementary Linear Programming office hours that was done was very helpful and "above and beyond" what most TAs would be expected to do.

Professor: Prof Isbell was fairly absent from this course. The TAs ran it very well, so perhaps that's reflective of his confidence in them.

Lectures: Leave a bit to be desired. Apparently they were done in 2015, and RL has been getting a lot of love since then. They may need an update. This is more a reflection of the current progress that's been made in the industry than an issue with the professors. The lectures are a bit proof/math heavy which isn't my thing... but I didn't personally commission the lectures myself, so it's hard to complain that they are addressing a wider audience.

This is a great course, very demanding in terms of time commitment, effort, complexity of the subject matter, and so on; but the TAs were great, up there with the best I've seen in all the courses I've taken up to now (this is my eighth course). At the end of the course, after the pain and the sweat, you'll have learnt a lot and feel a genuine satisfaction with your ability to reason and solve RL problems; of course, this is subject to you putting in the effort.

I don't get the recent hate for this class on omscentral. The course IS very math heavy, but its not like the lectures aren't useful. They are just more focused on building strong foundations of understanding for RL, rather than learning the deep RL algorithm du jour. I personally think that is way more useful for a degree-based program, as you will develop a critical understanding of the subject matter, rather than mastering how to use tools that could be picked up in your spare time on your own anyway.

The TAs are very helpful! Also, piazza tended to be fairly quiet, but the slack channel was very active, including TAs.

The comments on the projects and homeworks in previous comments are pretty accurate. You get them in batches, with the deadlines staged across a month, but you shouldn't wait to start the project after the two homeworks' deadlines have passed. Start early, because just like ML, you could work on the projects forever if you really wanted to.

If you don't have experience with linear programming, take GA before this. If you don't have ML experience, take ML first. You don't need those to pass this class, but they will make your life easier.

Probably the best class I have taken yet in the OMSCS program. You will have 3 challenging but very rewarding projects in the class; 2 where you will reproduce seminal RL papers, and another Deep RL project. It's hard work but very rewarding and you will learn a ton. There are also 6 homework assignments that are challenging but well designed to drive home some understanding of the material. Unless you are experienced in ML, I would recommend taking this class after ML4T (where you will get an intro to numpy, pandas, and RL) and DVA. You may also want to do ML first, which also provides an intro to RL for the last 1/3 of the class. You probably don't want to pair this class with another.

The TAs in this class are the best I've seen in the entire OMSCS program. They understand the material thoroughly (many work in RL, one is writing a book for Manning), they understand the expectations for each homework assignment and project, and they put a lot of extra effort into the class. Each office hour is run very smoothly and well attended (both students and TAs). Usually, one TA will provide a brief overview of what you should be working on, and then usually a little talk about some of the material. There were also some extra study sessions conducted to try and help students with homework and prep for projects. Questions on Piazza are answered clearly and promptly. I've found the grading be fair and objective. They also tend to jump in a lot on the Slack channel (which has been a lot of fun).

Bottom line; this is a great class with instructors who care about the subject and want to help others learn it. It is a lot of work, but it's also very rewarding. You will learn a lot! And RL is awesome.

The worst course ever probably. Feel know very little in Reinforcement learning even studying very hard for a semester. This is my fourth postgraduate degree and I have taken over 40 courses from Georgia Tech. This is one of the worst ones IMO. The teaching is very theoretical, without anything truly useful. Projects are useless without learning anything from them. Hope everyone can skip this course. Compared to this course, David Silver's Reinforcement Learning is way better.

TA's are pretty bad. A lot of them are just undergraduate students who have never gone to graduate school. (I checked their linkedin profiles and found that.) They are extremely unprofessional and not capable of grading and understanding your project, nor they are clear about the objective of the project. Many of them probably never read or know how to read research papers or do research. You will get very frustrated working with them.

Overall, be very careful of this course. Probably like in a pitfall of some waste. Totally waste time and money to take it. Skip this course definitely.

This is a tough course but doable one. The projects are fun & very stimulating. Prof Isbell is not as active as in case of ML. But the TAs are very engaged so you won’t miss the guidance needed.Miguel Morales needs a special mention. He is not only very knowledgeable but very humble too; he’s very keen to teach as well so that makes it even better. Overall a very enjoyable class with lots of learning...

Very interesting course. Lectures were sort of useful. Thank god for David Silver lectures

The final exam of this course is very hard. The mean is below 60...other than that, the homework and projects are not that difficult. I agree this course is mainly about self-learning, and I don't like the style of the lectures. However, I really liked the homework and project design. It's not about hours' tedious work but focused on your thinking. I learnt a ton from the assignments.

Back to the lectures, I think the two professors are more focused on the details of RL. This is not friendly to beginners, but when I finished the whole course, I found the lectures not that bad.

Tips:

1. Take David Silver's RL course prior to this one. Very helpful.
2. Read the given papers carefully. Project 1 and 3 will be much easier if you understand the details in the paper. Same for the homework.
3. Some TA sessions are not helpful. So don't waste your time if you feel that at the first session.
4. No GPU needed for project 2. I wasted too much time on that and found out CPU is even faster. Although TA warned this....

The recommended flow for ML courses are DV -> ML4T -> ML -> RL. I took the course in summer as my 2nd course and had to finally drop the course. My maths is pretty good and always scored very high in math but this course was dealing with probability. I had extreme difficulty understanding this course and also didn't get much help (maybe because it was summer) . The course does not give lecture slides and taught on whiteboard. The recommended readings are thesis/dissertations. Didn't get much TA help. I would never recommend this course as a first course or as a summer course. Depending on your interest the course may or may not be useful for ML track.

Good:

-TA's are very responsive on piazza and active on slack.
-Guest office hours with Professor Leslie Kaelbling was a really nice opportunity.
-Project 2 was exciting and fun and helped to learn the material in an interesting way.
-HW's were reasonable and helped to reinforce the course material.
-Professor Littman and Isbell do a good job of working together and making sure questions that people might have are answered as they go through the material.
-Reproducing results from papers in the field helped to internalize the material and develop a level of academic maturity and confidence.

Bad:

-The office hours format does not provide interaction. Just ask a single question and wait for a response.
-Course load is too heavy. Less would give more time to absorb the material. Could probably be split into 2 courses with markov games as a separate class.
-One or two office hours with the course instructor per semester would be very helpful.
-Grading is a mystery, especially for projects.
-Better rig gives you an advantage especially on projects 2 & 3.

Good course. Covered contemporary RL. Theory focused.

I'm really on the fence about this class, even almost two weeks after being done. A slant towards dislike due to the poor grading policies. This is my 5th class in the program. My only ML background is ML4T and in retrospect I kind of wish I took ML first.

I didn't watch David Silver's lectures until I started reviewing for the final. I only watched a few and I'm not really sure how helpful they were. I enjoyed the lecture format for the Udacity RL lectures, but I don't really know how much I actually learned from them. I'm in the same boat as the other reviewer who finished with 90% with a cursory understanding of the material. My % was less but still an A, thanks to a very generous curve.

6 homeworks (30% total), 3 individual projects (45% total), 1 final exam (25%). Final is 25 T/F questions with explanations for each, whether we think T or F, and everything is fair game, including all lectures and all 30ish papers (a small number of which were more like books) we were assigned to read.

I thought HW 1 was the hardest one by far. Each is challenging in its own way but once you have that "aha" moment they're very simple. I never got that with HW1 but I did find another way to do it. And since we run our code on provided test cases and submit answers to an autograder (limit of 10 tries per problem, which I found to be way more than needed, which is good), it doesn't matter whether we code it the "right" way or not. Overall I had a lot of fun with the HWs.

Projects are coding with a report. Project 1 was about replicating results in a research paper. I enjoyed it since it was a perfect mix of everything. Challenging, but not so challenging that I just sat there wondering what to even do. It was rewarding to generate results and interesting to compare with classmates' results. Then came the grades. I lost nearly 10% for not doing something that wasn't even mentioned in the project description. Well, I actually did do it, but I didn't write about it because our reports had a hard limit of 5 pages. Given everything we had to write about and the types of graphs we had to include, it was a real struggle to stay in the limit, so the amount of extra stuff we can put in there is tiny, if anything at all. Based on class feedback it seemed like a common occurrence to lose points for things not specifically required. Speaking of feedback, many students said their feedback on P1 was extremely vague. I didn't have this problem on P1, but this was fixed in general for P2 and P3.

Several students asked for full rubrics to clarify the P1 grades, and the TAs were very adamant about not giving those out. I completely understand why they don't want to do that (and I was surprised that ML4T essentially does do that), but it would have been nice for them to meet us halfway. It's very frustrating to be docked points for something you (1) have no idea you're supposed to do, and (2) can't fit into the report anyway because you're squeezing everything you were told to do in the 5-page limit. I teach and this is something I would never do to my students, which it makes it all the more frustrating.

P2 was the OpenAI gym lunar lander project. I had fun with this at first but then it turned into a game of "find the parameters", which I find tedious, dull, and not educational. Parameter tuning is a big part of ML, RL, CV, etc., and it is why I'm heavily disinterested in these subjects now. I honestly don't know if I even want to take ML anymore.

P3 was replicating research results again, this time using game theory and linear programming. This was challenging in an unpleasant way (sitting there wondering what to even do) but once I got over that hurdle it was a little more enjoyable, even if it was a little tedious. In the end I lost a few points for reasons similar to P1.

This was the first course I've taken where we had absolutely no starter code for projects or homeworks. I like that since it's generally more rewarding to code something functional completely from scratch.

Overall, this is definitely my least favorite class so far. But I highly enjoyed the other ones I've taken, so that's not necessarily saying this course is terrible. I guess I'm glad to have taken it, but... I don't know, it just wasn't what I was expecting I guess. I absolutely do not recommend this as a first course unless you have some hardcore ML/RL experience. I don't know if I recommend it at all, but you will need to take it if you're doing the ML spec and you want to avoid group projects.

This course extends what is taught in ML (even using the same video lectures) and consists of 6 homeworks (5% each, normally taking 1-5 hours), 3 projects (15%, each, taking anywhere from 20-60 hours for me) and a final (25%, 2 hours, T/F with explanation).

The homeworks are generally fairly easy once you get the core problem, which most people do. They are autograded and you get a few submissions so you know your grade immediately. The first project is replicating a research paper and is relatively easy and not that interesting. Project 2 was a lot of fun, but much harder - it's the lunar lander problem from openAI gym. Project 3 was the hardest of the lot, involving linear programming to replicate another research paper. The grading was pretty haphazard and depended heavily on which TA you got. One tip is that the code does not matter at all - it doesn't matter if you replicate the research or not, it only matters that you talk about the algorithms used in the research paper. The exam was similar to the T/F questions from ML, and takes about 45 minutes to an hour (no rush).

The lectures are comically bad. Unfortunately you do need to watch them to know what is going to be on the exam. As everyone has mentioned, watch David Silver's lectures first, then just skim through the Udacity lessons and try not to let them disrupt the things you learned from Silver. I also used a course from Udemy which helped.

Pros:

• The TAs seem like nice people. I didn't find the office hours super helpful, but Miguel and Tim were capable head TAs, and Miguel held special 'help sessions' for the homeworks.
• The slack channel was great.

Cons:

• Abysmal lectures. If this was a paid Coursera MOOC I would ask for my money back and then some more for damages.
• Projects aren't really related to the material.
• No rubric for projects.
• Haphazard grading. If you get generous TAs you will get an A.

With all the good, free resources on ML and RL online, it's amazing that this course hasn't yet been Joyner-ified to catch up with them. Do this course because you need it for the ML specialization, not because you want to learn RL.

This was a fairly interesting course. I enjoyed watching most of the lectures and a lot of the papers were useful. I don't think I learned as much as I hoped to, however, and the projects, while informative, did not engage me as much as I had hoped and were generally fairly easy, with the hardest part being the length of time required to run the scripts (even when vectorized) on my laptop. The homeworks were too easy to reinforce any of the learning from the lectures. Overall though, this gave me good insight into the algorithms that can be used for RL and also which researchers to follow in the future.

The topics are very interesting. But I dont think the lectures explain them well enough. Compared with Sutton's book, David Silver's lecture, and lecture notes else where I found online, sometimes I feel our lectures explain the same concept in the weirdest way (maybe just not in line with my logic). As a result, I dont feel confident about what I've learnt. I participated in the first two help sessions organized by the TAs/TA, I feel the host (sorry I could not match face and name) explain some concepts in a very clear way, yeah, he knows what he is talking about, but after taking this class, I don't have that confidence (although I got over 90% in total).

Reflecting in the exam, (25T/F with one or two sentence explanation), I reviewed quite a lot but I still feel the questions are very confusing to me. I knew the questions are not supposed to be easy but I feel my problem is simply that I do not understand the topics well.

There are six assignments, they help you understand some basic theories mentioned in the lectures/papers. Seems most students got full marks.

There are three projects. Two are replicating research papers. One is a deep learning project, which gives me a very good introduction to DQN. One of the papers to replicate is very vague and I guess we are not really expected to fully replicate it.

Lessons I learnt:

1. regarding the projects, follow the requirements strictly and make it very clear in your paper that you are following them.
2. for the reports, i feel the thinking process is more important than the actual results. As the TAs mentioned, we can write a good report without great experiment results.
3. spend sometime learn linear programming.
4. start to prepare/google some lecture notes for yourself earlier. It will help a lot for the review. During the last week some google docs with lecture notes prepared by old students were circulated in the slack channel, some of them are very helpful high-level abstraction of the lectures.

Note: I rate it as medium difficulty because I feel the projects are manageable if you aim at being reasonable instead of being perfect in terms of the results.

This was a tough course, but I enjoyed it. 6 homeworks were relatively easy to get perfect scores on and the autograder worked well. The 3 projects were a challenge. Make sure you look at the directions really carefully and hit all the things they are asking for. They will take a long time to do and to do well. Make sure you start on them sooner rather than later. Grading of the projects are not super transparent, and it is a little slow, but I learned from the projects for sure. TAs are generally helpful with office hours and on piazza (apart from the slowish grading). There is only 1 test worth 25% of the grade. It is closed book closed notes closed everything. Takes the form of 25 t/F questions where you have to justify why you put T or F. It was a ton of information from the semester to take in for the one test. All the Udacity lectures and about 25 journal articles. I did not do super well on the exam, but liked the course overall and I feel like I learned a lot about Reinforcement Learning. I especially enjoyed the lectures and the banter between Isbell and Littman in the lectures.

I would not take this as a first course, but I am glad I took it and would recommend it to others.

This was my 8th class in the OMSCS program which included taking ML4T, AI, and ML prior to this one. Having some of that prior knowledge certainly made this class easier with respect to the reports and general flow of the course. If you plan to take this course, I do recommend the following:

• Really try to watch the lectures and do the weekly readings during the period they are assigned. This will make reviewing all that material again for the final easier to handle.
• What I saw a lot of people do, including myself, was stop watching the lecture videos after the first project. The lecture videos didn't seem to provide much help with the homework and projects after that point but trying to absorb those hours of lectures before the final can be daunting.
• Expect a really hard final but fairly graded given its difficulty. Professor Isbell is known for his tough and broad tests. When they tell you that you will be tested on any of the information that is given (lectures and all external readings) it is true.
• Consider not doing this one over the Summer. You are given fewer weeks than a Fall or Spring semester but have the same amount of work.

Overall, this class was good, and IMO necessary if you are going to get into the field of ML. The TAs were very good and really want you to succeed, but don't expect a whole lot coming from Professor Isbell in terms of Piazza.

This is a hard course to take in summer. Most of the students would have taken ML before RL know what to expect, but for other students like me who has NOT taken ML (yet) and RL is your first Dr. Isbell course, you are in for a ride. There is NO difference between taking RL in a summer or in a full semester. Unlike other courses, in RL there are NO cut-downs of assignments, projects. You just have less time to complete everything.

Coursework: 3 projects and 6 assignments/homeworks and 1 exam.
Extra credit is offered in 1 homework, but it's a very very small contribution to your overall grade.

You can kinda front-load work because the assignments and projects are released in batches. It's 2 assignments+1 project released at once with different deadlines for each deliverable (project having the most time).

The assignments/homeworks are just graded on the output of your program. No need for reports, but you need good understanding of readings and lectures to solve these problems.

The grading of the project is primarily focused on your report (5 pages total MAX including everything) - unlike ML, you CAN'T steal code - you have to implement RL algorithms yourself and write a detailed technical report describing all the why's/how's/if's of the observations you saw in your experiments. P1 and P3 are paper replication+analysis and P2 is the most interesting one where you teach an agent to play a game. Hardest one for me was P3. Make sure you have enough time for it since you will also have to study for your exam.

The final part (25% of your grade) depends on the final closed-book ProctorTrack exam. If you are taking the course in the summer, you better catch up with weekly reading or you are going to have one helluva time during the finals. EVERYTHING in the lectures+reading is a fair game for the final exam. If you don't do well on exams, better get high marks on the 3 projects and the homeworks you have (which account for 75% of your grade) and hope for the best in the exam.

You really don't need any resources other than the ones provided by TAs and the lectures. Many suggest watching Silver's lectures, but they are not really needed IMHO. I did not watch them, but the slides were indeed helpful. If you have extra time, no harm in watching them! :)

Taking notes was very useful to me in this class. Lot's of small facts are told here and there when the profs (Dr. Littman and Dr. Isbell) are conversing in the lectures. Make sure to note them as they will be very helpful for the exam!

Overall, a very useful and well-organized course. Although, I am not sure how readily the concepts you learn here can be applied in real-life, it is still a good skillset to have.

Finally, the TAs played a huge role in running the course smoothly even in summer! The TAs of this course have set my bars higher for future course TA expectations! :D The weekly office hours held by TAs are very useful for doing the projects and homeworks so don't miss them.

Don't take this as a summer course, the workload is very heavy because they don't trim down anything.

The lectures were useless. I get it that they are trying to make this fun and have banter, but I'd also like to learn. I highly recommend David Silver's lectures instead, they will take you through MDP's, Policy Iteration, Value Iteration, TD-learning, Q-learning, and DQN (which sets you up nicely for project 2).

Project 1 was ok - really serves as a barometer for what they will be looking for in your papers going forward. Take the feedback seriously!

Project 2 was great - lunar lander from openai gym. Implemented DQN, got to use tensorflow. Best project I've done in the program so far!

Project 3 was very difficult - there are a lot of missing details from the paper whose result you are being asked to reproduce. A huge component here is Linear Programming, so if you are not familiar with it you are in a tough spot, because they explain it VERY POORLY in the lectures/office hours.

Head TA was hard to understand. Frequently office hours left me more confused than before.

Interesting material, decent teaching style, lots of self study required.

The class starts off easy. I read many of these reviews speaking to the difficulty and workload and was 4 homeworks and a project completed sitting nearly at 100% thinking I must be some sort of genius.

I definitely was not. Once the second project hit, there was no way I could catch up to all of the required reading/lectures afterwards. Then the 3rd project continued to batter us when we were down and the final was a Mortal Kombat "Finish him" situation.

That being said, this was still one of the better classes I have taken in the program and it could only be made better if the instructors planned homeworks and projects with an appropriate amount of time between. I did take it in the summer so the compressed schedule didn't help. However, I would suggest that if the homeworks were brought down to only 3 while keeping it at 3 projects, it would lead to a better experience for all.

One last thing - I have seen many discussions that say it doesn't matter whether you took ML (CS7641) prior to this class. I have to say that I would STRONGLY recommend that you do if you want a good pedagogical experience.

TL;DR: Hard but good class, P2 and P3 are 4-5X harder than P1 or other homeworks, take ML beforehand.

This is a difficult course, mostly because you are on your own, and have to read a ton of advanced scientific papers, and make sense of what is being presented in them. Assignment 1, Project 3, and the final were tough. Project 2 was awesome, and probably the highlight of the course. The final was brutal, and probably should be removed from the course entirely. Overall, you are probably better off watching David Silver's course on RL through YouTube, as he does a better job at covering RL than how it is presented in this course.

The course is a great introductory course to Reinforcement Learning. Its recommended, although not necessary to have some background in ML.

The material is quite rigorous and covers all the theory, and is still kept quite interesting by the professors. Prof. Isbell and Littman, who teach the lectures, cover so much, and with such ease, that you'll wish they took all the classes. There is a lot of reading, that can be hard to keep up with, but some of the projects require you to implement those, and thus, you'll be forced to read some of the papers anyways. I did not consult any other material than the one already provided, and it worked just fine for me.

The homeworks and projects are fun, and challenging. You will get to learn a lot from them. Final is T/F based, and will test the concepts taught in the lectures.

RL is one of the frontiers in the field of ML/AI and has tremendous applications from Robotics, to even understanding human decision making and game theory. I'd strongly recommend this course for anyone who is interested in all that.

Well, RL is a great course and I took this as a successor to ML and it made some parts such as game theory easy as it was already covered in ML. Homeworks are kind of dicey and tricky and you might end up spending less or more time and some homework I had kind of a hard time to crack it. TAs are great and active piazza/slack channels help to understand the course lot better than what is covered in the videos. Videos are, well, hard to grasp and not very much interesting and not well taught. I always wondered why the content should be so tough to understand and why can't it be taught in an easier and understandable way. Projects are good, though I did not like the reproducing paper's projects. Final is a little hard though they are T/F type. You need to really get the concepts to crack finals. Overall, I liked the course tough some of the aspects mentioned could be improved.

This was my 7th course in the program. Taking this class in the summer is pretty relentless. 2 homework assignments and a project are released in 3 week intervals with the first 2 homework assignments due in the first 2 weeks and the project due the 3rd week of that interval. The homework assignments are autograded so you can see your grade immediately, and each one takes only 1-2 days to complete. For all of the assignments/projects, you have to start with a blank slate (no skeleton code given). Two of the projects were replicating research papers. I would say the first project was not that exciting, but project 2 was the coolest project I have done in OMSCS so far.

About half of the lectures were not very well done -- It feels like you are watching the professors nerd out instead of explaining the material. They also assign multiple research papers to read in addition to the lectures that you have to watch every week. I would say its not feasible in the summer to read all of those papers (24 in total), although some students were able to fit it in.

Miguel was a very helpful TA since he offered a few in depth help sessions in which he went through some of the assignments in detail. However, outside of the help session and regular office hours, the TAs avoid helping that much with the assignments/projects and usually give you more research papers to refer to if you are confused. I therefore recommend discussing on slack among your peers when you are in a bind.

Despite some of the flaws, the subject material is extremely interesting, and I am very glad I chose to take this course. 4/5

I think the class is very good. It contains most of the important aspect of RL. The lecturers are interesting. There are 6 homework and 3 projects and a final for summer semesters. I only feel tough on the project #3 and the final exam. The final exam is 25 T/F questions, but you have to explain why your choose True or False. The final exam is quite hard to prepare... But overall, I think the class is great! I learned a lot !!!

An extremely poor course that does an awful job at explaining what might very well be the most exciting field in Tech right now. One of the worst courses in the ML specialization. The first 3 to 4 lectures are okay but after that it's just downhill. Final exam is awful and tests you on nothing useful. Wasted opportunity.

We basically teach ourselves watching videos from other universities. I barely watched any lectures until the last week. TAs themselves know nothing about the material save for maybe the head TA. The professor is absent and not very friendly. He also has disrespectful grading policies as well in addition to making poor distracting jokes in the lectures.

Projects are tough. P1 is okay. P2 is time consuming to train but easiest of the 3 to implement. P3 is very hard. You replicate papers without any help.

You will walk out with only superficial understanding of RL as most of the important material is never tested in any meaningful way. There are other resources online that will teach RL better.

For context I got 90% (mean was 80) overall but I still feel i know very little about RL

You'll enjoy this class if you enjoy self-study.

The course lectures are almost useless. They are more a catalog of topics than an explanation. Expect to spend lots of time deciphering obscure details of papers and trying to get things to match the behavior of the papers with no help.

The projects are ridiculous. They take an inordinate amount of effort to complete and many people have trouble meeting the objective. Especially the last project was very poorly specified and required a lot of implementation and lots of debugging. Not a lot of code, but a lot of hair pulling. This is the exact opposite of say Computer Vision a class that has an insane workload but very well designed projects. This is by far the hardest class in the program, but not in a good way. I'd add that its probably the worst class in the program in spite of having some remarkably good TAs.

Tips to succeed: do all the Silver lectures ahead of time. Take GA for better understanding of Linear Programs (before you do this class). Take ML for Trading to get a good handle on Reinforcement Learning before taking this class. Take ML to get a handle on the Professor's teaching style.

The only thing good I can say about this class is that there are good alternatives on Coursera that you should prefer over this one. The TAs are very good too. But the way the class and projects are designed the class is a pretty horrible experience.

Tips:

• Focus on writing a really good, complete and detailed paper in your projects. That will influence your grade the most.
• Historical grade information shows that the final grade has been somewhat lenient. So in spite of all your suffering you can end up with a good grade.
• Consider skipping

Wish I had taken ML first, but this course was still fascinating and opened my mind to a lot of new ideas I hadn't encountered before. Project can be very tough and take a lot of time between iterations - start them early! I hate writing papers, but it was worth it for this class.

A fascinating class. You get to dive deep into reinforcement learning and actually reproduce important research papers in that domain. In particular, one of the projects involves deep reinforcement learning! I actually went on to reproduce some of DeepMind's recent papers on deep learning for Atari games, something which I would never have guessed I could do before this class.

The video material is quite engaging, but I found that it left some things fairly vague (for instance, I don't think it presents Q-Learning in a very clear way, though you definitely get to understand it through the assignments). Still, it covers a lot of advanced RL which is very exciting.

I haven't taken machine learning course, but directly this RL course. I have taken HPC, HPCA and database course and the declared as computer system specialization. To be honest, for me the homework and projects are not very hard, the most time-consuming part and most unexpected part is the report part. No matter how many efforts you put on the report, the score seems just a random number from 80-90 for me. It pretty frustrated me sometimes as in most computer system courses, if you got the code right, you can almost get 100% score. Overall, not a bad course, RL is very interesting although the lecture is elusive. The grade is generous. I got overall 84.5% but still an A.

I earned my first B in this course but I can rightly kick myself for it. When I recommend starting every project ASAP, please listen to avoid my mistakes. You might ace all the homeworks like me but they mean little (only 30% of your grade). The projects are worth the most (45%) and they take a LOOONG time to get right. First, it takes a long time to understand how to implement them. Then you have to test them, which being machine learning, also takes a long time. Then once you know they work, you need to run them for long periods of time to get the results you need for the analysis. Please don't wait. The analysis isn't bad for each project (only 5 pages each) but obviously you want your code to work properly first. I hate the fact that they didn't release them all at once. I really would have liked working ahead. They only give you 2 weeks for each project and that's a shame. Overall, the course was incredibly interesting and well taught. The TA's were very responsive and helpful. This was a great class but of the 6 courses I've taken so far, this one penalizes you the most for waiting on projects. Just don't do it.

Very deceptive course. Most of the materials for the course is from academic papers. The lectures do no justice in explaining the papers or setting the students up for success. The poor jokes of the professors make it even worse. The 3 projects in this course are mostly based on academic papers, which you have to read and replicated without much help. The TAs were kind enough to help on few of the homeworks and projects. Extremely hard final. I did take this course after taking ML (scoring an A in it). Personally I learned more from ML4T than ML and RL combined. Though the content taught in this course is interesting, there is not much practical applications possible (not a popular field at the moment). It would be better to take BD4H instead of RL as it will give you some hands-on experience for working in the industry.

This class is deceptive. The homeworks (6) are disproportionately easy, but the projects (at least two of them) require some advanced skills -- dissecting research papers by academic leaders in the field. Probably best to be taken in second year after you've developed some organizational discipline and should definitely be taken with a lighter-workload partner class or as a solo class. The final is extremely challenging. Be prepared to do a LOT of outside reading, be involved in Slack and on Piazza, and you will learn a lot.

Exellent course. There are many lectures to get through, and they get quite mathematical at times. But the homework assignments and projects are mostly practical-oriented. All the assignments really helped me test and cement my understanding of the theory. The course readings are all papers on reinforcement learning, many by the lecturers themselves, and I found there were too many to all read thoroughly. After taking this course I feel comfortable reading RL papers, and think I could write an interesting paper on the subject myself when I have some time.

For context, this was my second class after ML4T, so I had not taken ML.

The instruction format is Udacity lectures plus many many research papers to read. The lecturers (Isbell and Littman) are very engaging but the material itself can be quite dense. And the research papers are tough to get through if you are not used to reading academic CS papers. There are also weekly office hours with the TAs which were very helpful. The TAs were very accessible, hosting office hours and appearing in the Slack group.

The structure of the class is 2 homework assignments with a week to complete, then a project with 2 weeks to complete, for a total of 6 HW and 3 projects. There was an opportunity for an extra 10 points on one HW assignment. The assignments are released in 3 batches, so you can work ahead to some degree but couldn't finish the semester all at once.

The homework was challenging conceptually at times but usually did not require much code. I thought all of the homeworks did a good job of reinforcing concepts from the lectures. All 6 assignments were submitted through a web app that gave immediate feedback on your results. Each question on the assignments had a 10 submission limit to prevent brute-forcing solutions.

Two projects involved reproducing results from research papers, and one involved training an agent for an OpenAI Gym environment, using an algorithm from a research paper(s) of your choice. The deliverable was the code on Github and a 5-page report on your results.

I did not enjoy the projects based on reproducing papers and didn't find the topics very interesting. The OpenAI project was quite interesting but very time-consuming. Training time for the agent could take many hours so iterating on a concept was difficult.

Overall I thought the course was very well done even though I didn't enjoy it as much as I thought I would. It was more theory and less practice than I would have liked, but that's a matter of personal preference.

3 programming projects worth 50%. You need to read 3 academic papers and try to reproduce the figures. No more guidance, paper hard to read. Exam is 50% and very difficult to understand the requirement. If you want to take it, you better prepare with a strong heart and large time commitment. There are some changes this semester. The professor is not engaging and the TAs are very reluctant to give you any really useful guidance or hint.

Some students claimed it is fulfilling to learn RL, I dont disagree. But most of us end up watching youtube/google articles to learn the material. Yes, RL is fun, but not fun by taking this course to learn it.

If you look at the grade trend shown on the right, you can see the pattern. The course is spoiled by the popularity of the attendance and news article about AI

The projects were the crown jewels of the course. They were most of all fun, exciting, lengthy, full of pitfalls, horrifying, and on top of that you do learn quite a lot about the subject matter (as well as a lot of other subject matters outside of the purview of the course). Paraphrasing another classmate in the course - "I've only cried a handful of times in my adult life, Project 2 is one of those times."

TAs were a great team and put together a lot of excellent supplemental discussions and materials for the course.

Stick with Python, Linux, and get a start on projects and homeworks the very moment that they are assigned. The homeworks and projects are interweaved with each other in some sensible way. If you print out all the required paper readings, you'll find they are almost 2" thick.

The final exam required a very large breadth and depth of knowledge of everything presented in the semester.

I've previously taken only ML4T before taking RL, and I thought it was sufficient.

This was my first course in OMSCS (along with CSE6242) so I don't have much to compare it to. The course was nicely structured, with coursework relatively spread out. 1 week HW due, 1 week HW due, 2 weeks for completion of a project. Homework assignments were relatively easy, projects should be started well in advance. Reading external material helps a lot, and David Silver's lectures felt more detailed for the intro to the course. Recommendation: be intimate with Python; the later projects are much easier to complete in it. All around, the TAs were very helpful in office hours, and Slack becomes your best friend. I learned a ton, but definitely had to put the man hours in. Also, the final was rough... average was in the 50s.

This was my first course, hence, I don't have any reference to compare with. I really liked the course and has definitely given me knowledge required to do further research in application of RL. The most difficult part was learning Game theory. The lectures are really informative. I liked the conversational style of teaching. You may also have to listen David Silver's lecture to get a better understanding of certain topics. Piazza and Slack were really active. If it was not for slack and piazza, I don't know if I would have completed the last project. The final exam was really difficult. Overall i really enjoyed the course and I am all eager to apply RL at work.

This was my first OMSCS class. Overall, I think it's a good class. I learned a lot and certainly think I have a foundation for reinforcement learning now. I had to do all sorts of outside research, but I think that is expected. The T.A's did a great job, shoutout to all of them (especially Chris Serrano for his slack help during P2) they were very supportive and active on piazza and slack.

I have a background in ML so that was helpful (especially for P2, where you need to basically use DL). The videos are mostly good, but I highly recommend people to watch David Silver series. It seriously helps for the first 4 HW's and 2 projects. I would actually watch him first, then Charles and Michael. They are very deep and abstract, while he is really concrete and practical. I think it's better to have the "normal" stuff first before you get blasted into space and time.

My main issue was with the lectures actually, it's just very theoretical material and I could not always follow. Watching at 1.3x speed is a good idea to get broad strokes, especially if you don't love proofs.

The final is pretty hard - make sure to study for it.

Finally, I did well on the projects, but I somehow unlocked what the T.A's did not spell out. Which is that you need to write your report about what the project means, not what you did. Which may not be obvious, because doing the actual work involved a lot of head-banging and late nights, so by the time you are done with the work you may forget to think about why you did it at all.

I learned quite a bit in this class, but it is a challenging course. If you want to deep dive into a key aspect of Machine Learning, this class is for you. My biggest concern is that the professors were not around for the class, it was entirely run by the TA's. I wish they would update the content to be less on academic papers and more on the use of the latest RL libraries (ie DQN) / applications.

Excellent course with interesting topics. The TA's were amazing; highly responsive and genuinely trying to improve the class. There was very little interaction with the professor but the ta's made up for it in my opinion. The lectures were interesting and well-done. Grade based on both homeworks and projects, as well as a final; the homeworks and projects were interesting and effective teaching tools, but didn't always correspond to well to the lectures. Time commitment was fairly high especially for a summer class, but I'd highly recommend it to anyone who is interested in the field.

Great course. Cutting edge concepts. Literally piazza and TA's were godsend.

While I liked the conversational video lectures, they were really hard to correlate them to the home works and projects. Once I reached project 2, I literally stopped watching lecture videos and started youtube-ing Peter Norvig classes on Reinforcement learning.

Finals were brutal. Honestly I was clueless about at least 5 questions, like literally no idea what they were talking about.

Loved this course! Miguel (head TA) was awesome and held additional hands on sessions to help us with the projects and assignments. Fellow students were also very engaged and willing to walk each other through examples to better understand the concepts. The course was condensed for the summer and we only had a week to do the projects. I had to take days off from work for the last two projects and pulled an all-nighter to get the second one done. For the second project, you pretty much have to learn and apply DQN (not covered in the lectures or the readings at all) in order to get it to work. It was worth it though. This class prepares you to follow recent developments in reinforcement learning and is by far the most interesting class I have ever taken.

I loved the subject matter more than anything else. It's a really fascinating field. Very little participation by Prof Isbell, but the TAs were incredibly enthusiastic, involved, and had considerable knowledge of the subject.

The videos are well done. They're a bit chatty though as I learned going back through them for the final trying to take notes. I liked the blend of theoretical and practical, but if math isn't your think, the theoretical part is pretty dense.

The workload is quite high in the summer. After the first week, there's a homework (6 of them), project (3), or test (the final) due every Monday. I. e., [hw, hw, project]*3 + final. There are probably 20+ hours of lectures and 20+ papers. I don't think anyone really read all the papers in depth and most fell behind on the videos. We were always struggling to keep up with graded assignments. The second project (lunar lander) also required reading unassigned material (almost everyone who was successful used DQN, which wasn't explicitly covered). You should aim to get the two homeworks done in a week and leave yourself two weeks for the project, if you're taking it in the summer. The extra weeks in a regular semester would make a very big difference in the pace. You could do one hw per week and have two weeks for the project.

Personally, I'd prefer to see the course a little less project-oriented and split the final into a mid-term plus a final. That would force folks to stay up with the material and keep it up from being a semester of projects followed by a week of cramming. You could remove a project or make them a little easier. For instance, I would provide the soccer environment for project 3 and maybe give hints on some hyperparameters for the lunar lander project -- aspects of the projects that don't really have pedagogical value.

It's a lot of work, but you'll learn a lot and feel like you accomplished something.

Positive: Great material Great TAs Great "extra" videos to help explain parts of the projects The autograder site

Negatives: WAY too much workload for summer.

This was by far the heaviest class I have taken so far. Starting from week 4 through the end of the semester I didn't have a single night off from either reading papers, watching lectures, doing the homework, or completing a project. The class tried to cram the entire semester (16 weeks) into the shortened summer semester (12) weeks. We had our assignments released in sets of 3 with 2 homeworks and 1 project. The projects were longer and required a write up. The homeworks were typically turned in online and we would get instant feedback. That was awesome.

Good course overall !!! Projects are interesting... you get to know about latest research - hard work but you learn a ton.. Get grip over some algebra - it will only help.. One pro tip - focus on the report you submit - even if you are not able to accomplish end goal - just put in arguments about the subject matter and what all you tried.. you will be rewarded for your efforts and reports and not about the actual result - makes total sense - but something I was not aware of..

I'm pretty torn on this class. The subject matter is fascinating, but I didn't love the way the course was taught. Many of the videos were too theoretical for my taste (focus on things like algorithm convergence, etc), though as usual the instructors make a great team and are entertaining. There was a strong focus on technical papers (a large percentage the instructor's own), and I generally find them an awful way to learn new concepts. The 1st and 3rd projects were replicating results from old papers, which was tedious and dull - I don't think I learned much from them. Project 2, which used an openAI game was amazing and made the entire class worth it in my book. The TA's did a great job, but at least for summer session there was essentially no instructor involvement. I'm still somewhat glad I took it for the overview of some of the cooler topics, but if you really loved Machine Learning's practical focus, you might be disappointed with this class.

This was a extremely difficult class when I took it. The fact that they did not truncate it for summer makes it even more difficult, but I am glad they chose to do it that way.

Now, project two was a real killer. There was very minimal guidance and connection to class material. It was drop dead hard. Though the TA stepped in and helped a week before it's due, it was too late for many of us.

Project 3 was hard too, but more manageable.

The only saving grace was that grading was somewhat lenient. I ended up overkill the class, but I was so stressed out most of the time mainly because of project 2. But at the end I learned a lot, so it was worth it.

Overall, I really enjoyed the material, but completing the class (at least for the summer) was exhausting. The course requires a fast and consistent pace to keep up with the homework assignments and projects; I would not advise having any vacation or work responsibilities that take you away from the course for more than 2-3 days at a time. It definitely seems like this course has gotten quite a bit harder over time, as the workload indicated by the early reviews were quite lower than what I experienced. I would put this one and ML (7641) at roughly the same required time commitment, though RL is more focused on implementation and achieving a certain result, while ML is more focused on analysis and writing.

Pros: --Interesting material that really covers the seminal developments of the field --TAs do a strong job managing the class --The fastest grading I've experienced in the OMSCS program (tied with ML) --The autograder for the homework assignments was helpful. (You'll want to max out the points on homework to offset the inevitable point losses on the projects and final exam. ) --The projects are interesting and challenging, with the second one being the best.

Cons: --No interaction at all with the professor --The style of the lectures isn't concise and sometimes sacrifices the content for entertainment. Tighter scripting would be helpful. (Many others, though, love the style. ) --The oral presentations for the projects seem pointless. I never received any feedback on them, and they seemed to have no impact on my grades. --At least for the summer, there is a short window (less than a week) after finishing the last project to study for the final, which covers the whole semester. --Feedback isn't very detailed, and the specific point deductions for the projects and final exam are not revealed. --Just keeping up with the homework and projects can leave little time for reading the assigned papers

The material is great (very interesting and challenging) and the professors are excellent. Taking this in the summer can be a bit overwhelming; be prepared for a significant time commitment. I always managed to get the work done on time but I usually finish assignments early and noted many students struggled to keep up.

The good: I learned a lot and was challenged. Professor Littman made an appearance at the end, which was wonderful and appreciated. Project 2 was a big hit with the students.

The bad: The Professor wasn't (visibly to the students) involved in the class. It would have been good to have had some interactive sessions with him or even some piazza participation.

The mixed: The TAs ran weekly office hours. I felt those started off well and 2 of the TAs did some hands on sessions, which were great. However, they lacked consistency and did not engage in depth with the students about theoretical concepts, which would have been very useful. Questions were often ignored or not completely answered. It's great to have so many papers and resources to absorb but to have a discussion about it and how the student/TA interpreted the material would have facilitated learning even further. The interaction with fellow students in the class on slack helped fill in the gaps but it would have been good to have had the TAs (or the Professor) helping to drive discussion/learning a little more.

The course is pretty challenging but doable with good amount of effort. Through the semester, I had to constantly work on projects/hws and on top of reading papers but it's definitely worth the effort and you realized how much you learned after finishing the course and there is a lot more. I personally felt project and hw had enough time but it was too short to prepare for final for summer. You will be very busy completing the project and had only 1 week to cover all the materials covered through the semester.

Great course. TA group is the best I've had after 3 semesters. Lectures average 1-2 hours a week. 2 weeks were high at 3. 5ish Papers are ~25 in total. Most are 6-7 pages. Some are more. None are less. It's a lot, but not that bad compared to some other courses (looking at you AOS... ). They connect very well to the lectures.

3 projects (15% each). 6 home works (5% each). 1 final (25%).

It's a ton of work for the summer. It's a grind with something due every week. Even if you're ahead you'll get behind with the projects. You should definitely take ML before this. If not, you're in for a party. Highly recommend folks getting familiar with Barto and Sutton's book (it's free on Sutton's website), David Silver's RL lectures (YouTube), and numpy (vectorization for the win).

HW 1-4, Project 1-2, and the final are fair and very related to the content (project 2 is awesome). HW 5-6 and Project 3 need to be reworked. HWs 5-6 are relatively easy, but are 'eh. ' Project 3 is a massive step up in complexity with little resources available if you don't 'get it. ' They were such an adrenaline kill after how great project 2 was. It sounds like the TAs are going to review and revamp.

Isbell was absent, which was not a surprise due to past reviews. He was on piazza day 1, and that's essentially it. He missed the one office hour he was going to attend due to a storm.

TAs were able to get Michael Littman on a special office hours, which was awesome. Last semester they got Richard Sutton. Littman is a genius, and it was an awesome experience.

One note, after reading some negative reviews, Michael Littman is a professor at Brown. He doesn't have access to GA Tech systems (Piazza). Fine to down vote the course for Isbell not showing up, but it's ignorant to down vote for a Browns prof not being active at GA Tech.

TAs are knowledgeable. They will answer your questions. They sometimes need to review specifics (if referencing a paper). You will not find a better group of TAs.

As others have mentioned, the workload was very high over the summer. I believe there were only 2 projects for Summer 2016 - we had 3 projects, 6 homeworks, and a final for Summer 2017. This meant something was due every week, with projects due every 3 weeks.

I'd say this course in the summer is at least as much work as ML in a full semester, if not more. I wish I had taken it in a full semester, as even doubled up with another class I think the slower pace for the projects would have been extremely helpful.

Regarding the course overall, I liked it, but not as much as I was expecting to. The lectures were great, the volume of expected readings was overwhelming, and I have mixed feelings about the projects. I really enjoyed the second project (Lunar Lander), where we used cutting-edge RL techniques to solve an OpenAI Gym problem. The first and third projects were both replicating results from papers, and I found I don't enjoy that very much. I think it's a valuable skill to have, but the emphasis was too heavy on it. My ideal would be to keep the first project as a paper replication project, but replace the third project with one similar to the second (where we read papers to learn the techniques and apply them to solve a challenging problem). I actually quite enjoyed the material the third project focused on.

I also didn't really see the point of submitting both a 3-page paper and a 5-minute recorded presentation (with an exception for project 2, where it was neat to be able to show videos of my agent playing Lunar Lander). I think this is a common complaint, as the TAs polled the class on our preferred submission format, so I think this might change going forward.

One last thing, Dr. Isbell was almost completely absent from Piazza (I knew from reviews here to expect this, but it's a big departure from ML). The TA team was exceptional and did a great job of running the course, especially Miguel and Tim, but I wish Dr. Isbell was more active.

I found the workload to be pretty high in Reinforcement Learning @ summer. I took ML last semester. and I had one Master of Mathematics. I took 30+ and almost 40 hours for project 2 and project 3. You need run some jobs. and it may take a one day, sometimes two days. Also, you need better computer with GPU. Please buy a computer with GPU. It really helps a lots.

Also there are lots of papers need read. Some video I did not understand, I watched David Silver (youtube version RL Course by David Silver) or Andrew Ng (youtube version from lesson 16 to lesson 20) first. then watch video again. it helps a lot. But for summer, it was short time. and if you had full time job. I think you need sleep at 2:00 or 3:00 AM for paper and video. and get up at 7:00 AM for job from Monday to Friday. and no weekends, because you need finish your HWs or Projects every weeks. So i think no life if you plan to take this course at summer. BTW, TA is best. helpful and very great.

In continuation to my below post.....

Miguel and Tim rocked throughout undoubtedly!! Apart from that....

Things that I did not like about TAs is that they did not play any role in guiding the folks into the right direction or helping them out in understanding the concepts better. Office Hours were not help at all and, I gained nothing from those, they laughed at queries asked by people and what not, did not discuss the prior submitted assignments as if the folks who did not get those right, will get to know about their mistakes eventually (didn't allow us to share anything on piazza regarding submitted assignments post deadlines as well). At one point I really thought, if you know it then you got it else drop the class and that was the worst feeling ever.

Note: I have seen TAs doing better and great job in order to help folks and in making understand assignments better in other courses e. g. CV (King Pedro rocks!!). Felt like Miguel, Tim and Chris were the only TAs who were proactively participating and other TAs were nothing more than project graders.

I read somewhere in below comments that TAs were as busy as we were but I don't understand what made TAs busier than us as Hws were autograded and only thing that TAs were busy with Project grades (as well finals and HW5) and just one office hour per week (ignoring the 2-3 hands on session given by Miguel/Chris). Moreover, that was part of their job to support all these activities.

6 TAs joining same Office hour per week, could not understand that thing. I think that was waste of time for most of them as mostly Miguel and Tim drove those oHs and they could have divided the hands on and query sessions among themselves and could have provided more value to the folks like me.

In the end, I am satisfied with what I learned from this course but if it were not for my peers, I would not have made it this far in this course.

Happy Reinforcement Learning!!

I found the workload to be pretty high in Reinforcement Learning and one of the reason could be the summer semester, but I did love this course and could not have asked anything better than this. I specially loved the Lunar Lander Open AI gym project, it was great to watch code to land the lander.

HWs: I did find HWs easy but were quite tricky at the same time, and the submission & instant grades for the home works made it super awesome.

Projects: As mentioned by TA's before hand, I did also find P1<P2<P3 in increasing difficulty level. Recreating the results of the research papers was quite difficult for me as missing any point in the paper or making wrong assumption made it really difficult to replicate the results. Apart from that I loved the projects and now, after this course, I got bit comfortable reading the research papers.

To be continued.... (see above comment - ran out of space)

This was an interesting course where you get to learn about how machines/agents can learn about their environments and learn to interact with the environment using different learning techniques. The course was very challenging, but part of the challenge is tied into taking the course in the summer. With 6 homeworks, 3 projects and a final added on top of 20+ hours of lecture and 25+ papers to read, it's a lot of content for an 11 week (really, nothing much is available the first week, so 10-week) semester.

The TA group was top-notch - the best group of TA's I've encountered.

The projects were challenging, but fulfilling to get them done, similar to ML in that respect, but definitely a lot more hands-on. Also, you are expected to both write a short (3-page) paper and complete a short (5 minute) video presentation for each project.

The homeworks can be hit or miss. Of the 6, 3 were somewhat time consuming, and 1 of the 3 was extremely time consuming, however that may change in future semesters as they have been adjusting the HW to move towards a more auto-graded style. 5/6 were auto-graded and the one that was not was the extremely time consuming one.

The final exam is challenging, but fair, in it's current format. This is speculation, but given the expected 2 weeks to study in a normal semester, it would likely have felt a bit more manageable to review all of the content.

Overall, I thought the course was worthwhile, but it is definitely not one to take lightly with regards to workload or effort required.

Great course with interesting and cutting edge material. Professors are passionate about material in the lectures and it shows, but the lectures can be hard to follow as the are very math and theory heavy. Too many readings, which do not necessarily support the lectures of hw/projects at the time. As others have said, HW assignments are not that useful to understanding the material. 2 of the 3 projects are reproducing published research results, while the other is applying what you've learned to an OpenAI environment, which to me was much more interesting and useful. I would like to see more applied assignments like project 2.

A little too much material to cover in a summer semester. Projects 2 and 3 take several weeks at the least, but in summer something is due every week. HWs seem like an annoyance to get past to get started on the projects (start early!)

TAs this semester were great, very active on piazza and recommended or provided extra materials for help.

RL is tough but fun. The lecture videos were produced a couple of year ago. thanks to the efforts of DeepMind and OpenAI folks, quite a few foundational theories / assumptions in our lectures are no longer sufficient nor accurate enough. They have just added a new assignment with OpenAI and will be updating the readings to reflect the fast changing industry. Lectures are math heavy but you do learn a lot. Taking this in summer might be a bad idea, unless weekly assignment and cramming an exam within a week is your ideal summer. TAs office hours are not really helpful (except the hands-on sessions) and profs are no-shows; you will need discipline and motivation to finish the course and be active in reaching out to TA and Piazza for help.

I found the workload to be pretty high in RL, not unexpected for this much graduate-level content in a summer session, but I did find it to be enjoyable. That said, I haven't taken the final yet, and I'm in the period that I dreaded all summer knowing that I didn't have enough time to watch many lectures or read papers outside of what was required for the homeworks and projects. Unfortunately, right now, I'm cramming facts for the exam (25 T/F with "twitter style" answer justifications). During the summer session you have something due every week, with 2 homeworks and 1 project released simultaneously every 3 weeks and I found myself not having much time to spend on anything else other than getting those done. I did the best I could to get the homework done as early as possible to maximize time spent on the projects.

I disagree with criticisms of the TAs this summer. These guys were great, worked hard, and likely felt the pressure of the compressed schedule more than we did, especially since the profs were absent - the TAs completely ran the course (and did a pretty good job IMO).

The only ML background I had coming in was ML4T and I didn't feel that I needed any more than that as preparation. Its a good class, if you take it in the summer, know that it will be a lot of work. I spent a lot of late nights/early mornings and missed a couple of family events as a result of taking it, but have no regrets

I'm like the person below. I wanted to like this course a bit as the content really goes into cutting edge subjects (AI). Point of perspective, the whole Deep Mind / Alpha Go think is built on top of the content of (RL). In general though, there is no teaching and it's more like a sink or swim class, more like you are generally expected to know the content up front (90%) and the remaining should be easily learnable. I definitely did not come in with any RL knowledge at all. I made it, but the way the course is structured made it a lot more difficult than it should be.

Negatives: Too many lecture readings per week. Who has the time to read 3-5 academic papers a week and absorb the content? I understand maybe 1-2, but to ask students to read 50-100 pages a week is a bit much. Homeworks not useful. No coordination between HWS and Projects. HWs usually prep you for a project. This is not the case at all in this class. They are almost a unnecessary distraction. Projects were graded on how accurate can you reproduce a paper. I think this type of grading is poor, considering the people that wrote the paper, have usually have significantly higher qualifications then an OMSCS student and probably spent a couple months creating the results of the paper where OMSCS students have around two weeks to hack around and get the same results even though yes the equations are in the paper but they take a long time to understand and figure out. TA office hours not helpful. Lectures need to be cut down. Who really has time to watch 3+ hours of lectures a week. Streamline to 1-2 hours.

Positives: TA hands on sessions are useful. Project's content are really interesting.

In general, the course has potential. The problem is it's really meant for folks right know with a basic knowledge of RL versus none. HWs need to be revamped to intro to a project, example, a HW could be done on linear programming with probabilities as a trivial example as basically project 3 that's all it's about.

I really wanted to like this course. I really expected it to be educating and interesting. Instead what we got was a 3 month long race to complete the assignments on time.

The bad:

• There's no academic staff in this course. It's completely outrageous that a graduate level in CS is run by TAs. Isbell and Littmann are NOT following this course at all. Office hours are held by TAs which have no clue how to answer deeper questions on the subject.

• Assignments and course load are horrible. "go reproduce this and that paper's graph", "and you should also read 4 papers this week". Vague instructions, no help, no guidance whatsoever. If you have a full time job, you'll struggle. Constant banging your head against the wall, no learning there.

• Lectures ARE BAD. Yes, Isbell and Littmann are entertaining. But educating? No. The lectures are an incoherent, unordered mess. Anyone who watched David Silver's RL videos would agree with me - no contest.

• TA staff is not helpful nor attentive, they are not on Slack and they usually don't answer questions on Piazza but wait for a student to answer.

The good: Well, the subject matter itself is interesting.

Bottom line: Don't take this course. No ifs. Just don't.

Great course overall.

Lectures: Not bad, but get seriously math intensive by the middle and they don't really tie back to the projects that well. I skipped the last few to prioritize other things. There are also ~30 readings you should cover, and the textbook that you'll definitely need to refer to.

Assignments: There are a bunch of assignments plus 3 big projects. The assignments are generally very easy, and only sometimes require programming. The projects are huge time sinks. I really enjoyed the OpenAI Gym assignment as I used it as an opportunity to learn deep Q networks (not covered in the class) and thus had fun with the project. I still burned at least 60 hours on it though. The other projects involved implementing old RL papers, and I didn't enjoy these very much.

Exams: Just the final. Pretty hard to study for as there's a lot of material covered in the course.

Teaching staff: Professors are completely AFK, but TAs do a pretty good job keeping the class on track. Office hours are fun though of questionable value beyond entertainment. You'll have to figure out most things on your own.

Grading: Fair. I put in a lot of effort, learned more than I have in any class before, and came out with an A. I could have spent much less time and still gotten at least a B.

I did my undergrad in Computer Science. I never experienced a course as rigor as RL. TA's were great and assignments, Home Works were really challenging. For Projects we are expected to understand research Papers and implement some sections. In that process you will learn a lot and get good understanding of the Subject. For Project -1 we implemented a Sutton Paper on TD(lamdba). Taught in TD and friends Project -2 A paper published by Google Brain on DQN (We solved a problem in openAI using this paper. ). Taught in Function Approximation Lectures Project -3 - Markov Games (Q-learning, friend and foe Q-learning, Correlated Equilibrium ). Taught in last part of Markov Games and Game Theory.

I disagree with some of the post that says there is no link between course content and projects.

Isbell and Michel did a great job in making the course content simpler to understand.

Kudos to TA's for guest lecture with Richard Sutton (Father of Modern RL).

I have mixed feelings about this course. On the one hand it covers interesting material, and had one of my favorite projects in my academic career where we taught an AI agent to learn to solve a scenario on the OpenAI Gym. On the other hand I detest academic papers and this course puts a HEAVY emphasis on them.

In many ways this course felt catered to people who are pursuing careers in academia and not toward people who want to gain knowledge to solve real world problems, my assumption is most OMS students fall in the later. Two of the projects involved recreating results from academic papers. In addition, it is recommended that students read through 22 academic papers throughout the course. While you can survive the course without reading most of the academic papers, it shows where the focus is.

Homework felt completely unrelated to lecture material, the time needed to do homework assignments varied wildly and some of the homework and project timelines were unrealistic unless you can stay way ahead of schedule.

Overall, the course felt in some ways like a disservice to the material it covers. The course material is interesting and can be used to solve very cool problems like those seen on OpenAI, however it only spent a small portion of the time solving interesting problems.

The class concept is very cool. Don't kid yourself though, this is a tough course. Extremely fast-paced with heavy reading of 20-50 pages of technical journal articles or the instructor's PhD Thesis chapters every week. In addition, there is a project or homework due almost every week. Sometimes you get two weeks to complete something and in that case, you really need it. Many of the applications are reproducing journal articles' suggested methods and comparing results. Better know RLPY or the Java Burlap before you take the class or you'll start out behind. Best if you have had some form of linear programming before. This is not an easy course unless you work in the field. I do not. I ended up dropping it.

Good course, that delves into one part from 7641 deep. Many fun assignments, especially the one that uses Open AI gym. Lots of paper to read that gives a pretty good insight into the workings are reinforcement learning. Overall a fun, medium difficulty level course

This was a fantastic class; a great follow-up to Machine Learning (7641). Dr. Isbell and Dr. Littman return to dive into the details of the field of reinforcement learning in a fun and engaging way. Fall 2016 upgraded the difficulty level of this class both homework-wise and project-wise, in addition to adding a final exam. There was a homework every week or two that usually took under an hour to complete, and three projects throughout the semester that took 40+ hours to complete. Two of those projects were replicating paper results, and the other project was actually implementing a RL algorithm to play a video game to land a spaceship. This was a pretty cool class and definitely worth taking, but don't expect it to be any easier than ML.

The class was interesting and delved into reinforcement learning much more than the Machine Learning class. The video lectures were engaging and explained concepts well. Reproducing results from papers was a new exercise for me and helped increase my understanding of the difficulties of doing that when the paper publisher doesn't provide enough detail. I was a bit disappointed that Prof. Isbell wasn't as active during the class as he had been in Machine Learning.

I loved this course. The lectures were great - informative and engaging and the projects, which were rarely super hard, helped give you some subtle insights into the course content that I don't think you could have gotten any other way. I wish there were more courses on this... I'd take them in a heartbeat.

The content was very interesting and the TAs were very accommodating. The weekly projects we very light compared to the final project, but overall, the course is great. There were a lot of reading materials, but I learned a lot about Reinforcement learning!

For me, this was neither an "easy" nor a CCA-like (see the reviews if you don't know) class. It was somewhat challenging, but managable. I took it over the summer and was disappointed that Dr. Isbell was not very interactive in the class... the TAs ran the show daily/weekly. They were extremely helpful and responsive (above average). As always, start everything early. There were two projects and a single (final) exam over the summer, but I understand the 'regular' semesters have had a 3rd project and an extra exam. New developments for the summer: The class was language agnostic and you didn't have to submit code... just auto-graded for the weekly assignments. No code submission for the projects... just a report and video presentation. Some of my grades on the projects and exam were definitely not that great, but overall grading was fair given the class changes and compressed summer schedule. I'd say it's a must-take if you're interested in RL. There's a ton of reading, though, and the lectures are fairly dense... not a class you can run the videos on 2x the first time through!

Weekly assignments were easily accomplished with a few lines of code in Python without having to think too much in terms of MDPs. The final project required many more hours of reading and coding than the first.

The RL problem: How does an autonomous system learn from experience to achieve approximately optimal behavior without necessarily having any knowledge of its world dynamics or even what its goal is? You start with very small toy problems that can be solved directly using dynamic programming. From there, you gradually introduce the complications of the real world - infinite state spaces, stochastic reward functions, the inclusion of other agents that may be cooperative, adversarial, or indifferent, all the while learning of research from the past few decades resulting in tractable ways to induce human-or-better levels of performance in domains such as game play, autonomous drone flight, logistics scheduling, teaching robots to walk.

The idea is absurdly simple. Start with effectively random prediction of how well particular primitive actions might work and successively refine those predictions from whatever feedback you eventually receive, which may or may not be accurate feedback. Somehow this works, and this course gives you a mix of the mathematics to prove when and why it works, why it might work in practice even if you can't prove it works, and some of the practical research leading to hacks and tricks that make it work faster, better, or both.

The pros are the lectures and readings are engaging, insightful, among the best you'll get in this program. The projects, reproducing published research, give you the valuable real-world experience of just hacking away and making educated guesses until success magically happens. Then you have to figure out how it happened and be able to explain it.

The con is the weekly homeworks are too easy, simple algebra problems that might take a few lines of code but sometimes you can just do in your head, none of which take more than a couple minutes, basically free points. That time could be devoted to more projects. Function approximation (deep RL) especially is critical and hot and in the headlines and we barely covered it.

This course is a survey on Reinforcement Learning and Decision Making techniques. There are weekly assigned lectures and readings (papers provided by the instructor).

On Summer, an assignment was due every week, I'd say their goal was assessing that the students understood concepts explained on the lectures. You do not submit code, only results, so you could solve them with Java, Python, Excel, pencil and paper. There's an auto grader available! Some assignments were more interesting than others.

In Summer we only had one final exam (25 true/false questions, and you had to explain your answer). It was hard to cram all that information in one week (assuming you started to study once you submitted the final project). It seems that in full semesters, a midterm was also part of the evaluation, that would have made the load more manageable.

For both projects assigned (midterm and final project) we had to replicate the results of published RL papers. We had to explain about our implementation, the assumptions we made, the difficulties we had, etc. You can use any language/library since code was not submitted, only the report and a 3-minute video presentation. You learn a LOT doing that. Start working as soon as you can in the projects!

At the end, the grading was generous, since people who were on the C (70-80%) or B (80-90%) brackets, ended up earning the upper grade (B, A, respectively)

I highly recommend this course if you are interested on the topic. You can go as deep as you want in the material. The resources shared by the instructor are gold, and will give you many ideas for future work or research.

The course is very good, however the hw/assignments are not good at all. The hw/assignments did not relate to what we learned.

On the other hand, the lectures were amazing. I learn a lot ( theoretically ). If there were more application related hw/assignments it would have been perfect.

Professor Isbell and Littman did another excellent job with this course. The lectures were extremely engaging and the material was very interesting. In my opinion this course is a good bit easier than Machine Learning. The projects were very effective (the second one is quite challenging). For the summer semester there was only a final (25 T/F) which was not too difficult.

The reason I only "Liked" this course is because some of the assignments are fairly weak. While I appreciated not having to spend excessive amounts of time on them (a few took approximately 30 minutes), they did not always relate back clearly to the course material. However even with that, I would highly recommend this course.

If you liked ML, then I highly recommend this course. Isbell and Littman are a great team and produce excellent lectures. A good breadth of material is covered, and it leaves you with enough leads and a strong enough foundation to do your own exploration after the course is over. It left me wishing that there was a more advanced offering for RL in this program.

The work portion of this course consisted of eight assignments and two projects (for the summer semester; there are three in the normal offering, I think). The assignments were kind of hit-or-miss. There were a few that were excellent and engaging, and there were a few that were a bit too easy or disconnected from the material. Additionally, there were a few problems with the autograder site, either in terms of what it would accept or where there was a marked disconnect between what the assignment description on the site said versus what was actually expected. Just be sure to check the descriptions on Piazza and attempt submission early.

The projects were reproductions of academic papers accompanied by a short written report and a brief video. The projects were interesting, but not as open-ended or as tough as ML. Make sure that you have some ability to create a video (e. g. slides and voice-over) and don't leave it for the last day.

This course covered a lot of information, and I thought the lectures were excellent at inspiring curiosity and giving a high-level overview. The readings lined up very well with the lectures (it's wonderful to be able to read in more detail exactly what was presented at a high level), and it gave good exposure to research in this area. I thought the two projects were meaningful too - they left the implementation up to us, but required that we intimately understood the details of particular papers.

However, I thought the assignments were sometimes a bit lacking in relevance and rigor, and the feedback and involvement left a bit to be desired. It's difficult to have such a breadth of abstract information presented without much concrete practice to back it up.

This is a great course with a lot of good theory, but is still very abstract. A few more practical projects/HW could be beneficial for the course. It's language agnostic, which is great (Java and Python most common - but beware of buggy RL packages!). Some assignments can even be done with pen & paper - it's pure math and probability.

The two course projects were the places where you actually learn a ton! You read academic papers thoroughly and try to reproduce results from them. So, you're reading the cutting edge research that is being published. I wish there were more such courses. But some people may not like this overly academic scenario. It is likely that subsequent semesters will have more paper-reproduction projects and less MS-style homework assignments.

Finally, this course doesn't have as much instructor involvement as ML does. But our TAs Miguel and Tim were very responsive.

This was a great class - the return of pair lecturing with Smoov & Curly, an interesting subject, and a far lower workload than ML (as advertised). The big thing to keep in mind is that there is an easy way and a hard way to complete the assignments. If you find yourself spending more than a few hours on a problem, you're doing it the hard way. Go back to the lectures & the assigned readings, and think about how you would solve the problem without any RL tools.

This class I've been looking forward to taking, and I haven't been disappointed. Now when I read research papers on reinforcement learning I actually understand what I'm reading! The HW is often challenging, and fun too. I had been a bit worried about having to use Java for the class, as I'm a novice. Over the break I spent some time studying Java to prepare, but was pleasantly surprised that I've been able to do all the assignments using Python! :-) Now the course is Python-friendly, so Pythonistas are welcome. In short, another excellent OMSCS course, taught by reinforcement learning experts. I highly recommend it!!

Do NOT walk into this class thinking it will be easier than ML.

The midterm project was highly interesting; the midterm itself was cake (so long as you took good notes and studied).

The homeworks.... can be difficult. Most of them have a 'trick', and if you dont know this trick you will spend hours looking for answers. I had a hard time seeing how some of them related to the lecture for that week, so at times I wondered why I was doing them (outside of the grade earned). There are SEVERAL homeworks too (expect 1 a week if a project or test is not due that week), so its pretty much an unrelenting stream of work.

The professor himself is pretty awesome though, and the TAs are fabulous and very helpful.

The class is billed as much easier than ML, but make no mistake - this is far from reality. Approach this course like you did with ML (prepare to dig in to the material and set aside a massive chunk of time) and you should be more or less fine.

Take this course, but be prepared.

I loved the material, lectures and final project. Assignments were a mixed bag for me. communication on piazza was sorely lacking. Student study group a life saver.

The material is dense and complex, with a lot of math (like ML). The assignments are difficult to get good grades on because they are evaluated using a lot of edge cases. It's probably overall less work than ML but, in my opinion, far more coding intensive because there are no papers for most of the course, just difficult programming projects that have to not only output the right answers but do it fast (or slow) enough.

I've gotten A's on everything so far, but it's been a lot of work and by no means easy... Same or greater difficulty than Machine Learning, but what do you expect, it's taught by two Reinforcement Learning specialist. The good is that the content is cool, and the TAs/professors are fairly active. I would only recommend taking this class if you took the Machine Learning class and thoroughly enjoyed the last few lectures on reinforcement learning. Edit: Just had the 8th homework assignment. Literally sat in a chair all weekend working on it for 20+ hours, to only get close to the right answer. Be wary of what you're getting yourself into. The homeworks aren't as monolithic as CS7641, but there are more of them and ultimately they definitely add up to more work.

Mostly echoing comments above: There's a lot of math here. I really enjoyed the RL section of 7641, especially the last project. This is a bit more theoretical than I would like; very useful for people who plan to go farther with RL, but I was hoping for more 'practical stuff' (including lots of algorithmic detail) and projects that are more oriented towards interesting RL scenarios and less about examining some edge case. (All that said, the profs are fantastic, and it's fun to learn from such leaders in the field. )

There is a lot of math and a lot of homework, I would pair this one with an easy course if not sure what you are doing. I haven't taken ML, yet I'm doing pretty well on this class. One thing though, I work on it over 15h a week, every week. So it could be stressful if you are not use to having deadlines that you do not know how to solve, right on your head. Having a deadline every week, of homework that are even difficult to read and understand what is being asked is very challenging. I suggest you come prepared for some fun.

I agree with the above. I feel like the math is even more complicated than the maths in ML. I tried vey hard but I am not getting good grades for most of the assignments. Also unlike Intro to Computer Vision, you don't receive partial credits for assignments. If all your test cases give wrong answers, you will get 0 point for the assignment. Doesn't make a different if your overall program strcuture/though process is mostly correct.