Setup

You can run this code on your own machine or on Google Colab.

1. Local option: If you choose to run locally, you will need to install MuJoCo and some Python packages; see installation.md from homework 1 for instructions. There are two new package requirements (opencv-python and gym[atari]) beyond what was used in the previous assignments; make sure to install these with pip install -r requirements.txt if you are running the assignment locally.

2. Colab: The first few sections of the notebook will install all required dependencies. You can try out the Colab option by clicking the badges below:

Part I (Q-learning)

Part II (Actor-critic)

Complete the code

The following files have blanks to be filled with your solutions from homework 1. The relevant sections are marked with TODO: get this from hw1 or hw2.

• infrastructure/rl_trainer.py
• infrastructure/utils.py
• policies/MLP_policy.py

You will then need to implement new routines in the following files for homework 3 part 1 (Q-learning):

• agents/dqn_agent.py
• critics/dqn_critic.py
• policies/argmax_policy.py

and in the following files for part 2 (actor-critic):

• agents/ac_agent.py
• critics/bootstrapped_continuous_critic.py
• policies/MLP_policy.py

The relevant sections are marked with TODO.

You may also want to look through run_hw3_dqn.py and run_hw3_actor_critic.py (if running locally) or run_hw3_dqn.ipynb and run_hw3_actor_critic.ipynb (if running on Colab), though you will not need to edit this files beyond changing runtime arguments in the Colab notebook.

See the assignment PDF for more details on what files to edit.