Learning Experience (LX): Object Detection

About these activities

This learning experience will take you through the process of collecting data from the Duckietown simulator and formatting it to be used to train a neural network to perform object detection using the robot's camera image. We will use one of the most popular object detection neural networks, called YOLO (v5). You will also have to integrate this trained model into the autonomy stack. For now we will just stop whenever an object (duckie) is detected in the road.

This learning experience is provided by the Duckietown team and can be run on Duckiebots. Visit us at the Duckietown Website for more learning materials, documentation, and demos.

For guided setup instructions, lecture content, and more related to this LX, see the Self Driving Cars with Duckietown course on EdX.

Grading challenge

Your submissions will be sent to the lx22-objdet challenge.

You can verify the scores of your submissions on the Challenge Leaderboard after your submission is evaluated.

Instructions

NOTE: All commands below are intended to be executed from the root directory of this exercise (i.e., the directory containing this README).

1. Make sure your exercise is up-to-date

Update your exercise definition and instructions,

git pull upstream mooc2022

NOTE: to pull from upstream, you need to have completed the instructions in the duckietown-lx repository README to fork this repository.

2. Make sure your system is up-to-date

• 💻 Always make sure your Duckietown Shell is updated to the latest version. See installation instructions

• 💻 Update the shell commands: dts update

• 💻 Update your laptop/desktop: dts desktop update

• 🚙 Update your Duckiebot: dts duckiebot update ROBOTNAME (where ROBOTNAME is the name of your Duckiebot chosen during the initialization procedure.)

3. Work on the exercise

Launch the code editor

Open the code editor by running the following command,

dts code editor

Wait for a URL to appear on the terminal, then click on it or copy-paste it in the address bar of your browser to access the code editor. The first thing you will see in the code editor is this same document, you can continue there.

Walkthrough of notebooks

NOTE: You should be reading this from inside the code editor in your browser.

Inside the code editor, use the navigator sidebar on the left-hand side to navigate to the notebooks directory and open the first notebook.

Follow the instructions on the notebook and work through the notebooks in sequence.

💻 Testing in simulation

To test in simulation, use the command

$ dts code workbench --sim

There will be two URLs popping up to open in your browser: one is the direct view of the simulated environment. The other is VNC and only useful for some exercises, follow the instructions in the notebooks to see if you need to access VNC.

This simulation test is just that, a test. Don't trust it fully. If you want a more accurate metric of performance, continue reading to the Perform local evaluation section below.

ℹ️️ Check Robot Compatibility

While we try our best to support running these exercises on all versions of the Duckiebot, some activities require special hardware and are only supported on specific robots. Please use this section to ensure the compatibility of the exercise and your robot.

The support matrix of this exercise is as follows:

Duckiebot Type	Configuration	Support Level
DB21-J4	Jetson 4GB	✔️ Full Support
DB21-J2	Jetson 2GB	❌ Not Supported

🚙 Testing on a physical robot

You can test your agent on the robot using the command,

dts code workbench --duckiebot YOUR_DUCKIEBOT

This is the modality "everything runs on the robot".

You can also test using

dts code workbench --duckiebot YOUR_DUCKIEBOT --local 

This is the modality "drivers running on the robot, agent runs on the laptop."

📽 Perform local evaluation

We suggest you evaluate your work locally before submitting your solution. You can do so by running the following command,

dts code evaluate

This should take a few minutes. This is not supposed to be an interactive process: just let it run, and when you return, you will find the output in a folder, including videos, and trajectories, and all the statistics you would usually find on the website.

📬 Submit your solution

When you are ready to submit your homework, use the following command,

dts code submit

This will package all your code and send it to the Duckietown servers for evaluation.

Troubleshooting

If an error of this form occurs

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_cli/cli.py", line 76, in dt_challenges_cli_main
    dt_challenges_cli_main_(args=args, sections=sections, main_cmd="challenges")
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_cli/cli.py", line 203, in dt_challenges_cli_main_
    f(rest, environment)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_cli/cli_submit.py", line 165, in dt_challenges_cli_submit
    br = submission_build(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_cli/cmd_submit_build.py", line 41, in submission_build
    raise ZException(msg, available=list(credentials))
zuper_commons.types.exceptions.ZException: Credentials for registry docker.io not available
available:

you need to log into docker using dts. Use this command:

dts challenges config --docker-username <USERNAME> --docker-password <PASSWORD>

Retire obsolete submissions

Note that you can "retire" submissions that you know are wrong. You can do this through the Duckietown Challenges website.

To do so, login using your token, then find the submission you want to retire from the list of submission in your user profile page. Use the button "retire" to the right of the submission record line.