AWS DeepRacer League 2021 Update #1

It’s started! The first races of 2021 have begun with a wealth of new features. On behalf of the AWS Machine Learning Community I’m pleased to present the latest report from the race tracks!

If you need a bit of help catching up, here are a few articles that may help you:

• AWS DeepRacer is now available with new training algorithm Soft Actor Critic and a continuous action space
• AWS DeepRacer League 2021 has two divisions, Open for everyone who is starting their adventure with DeepRacer and Pro for those who qualify through the top 10% places of Open Division

AWS DeepRacer

AWS DeepRacer is a 1/18th scale autonomous race car but also much more. It is a complete program that has helped thousands of employees in numerous organizations begin their educational journey into machine learning through fun and rivalry.

Visit AWS DeepRacer page to learn more about how it can help you and your organization begin and progress the journey towards machine learning.

Join the AWS Machine Learning Community to talk to people who have used DeepRacer in their learning experience.

Pre-Season race

Let’s quickly summarize the Pre-Season race which took place in February, and that was no small race.

The racers took part in Time Trial race on European Seaside Circuit aiming for the the fastest three laps time. Each time the car went off the track it got reset onto nearest position and resumed the race after a three seconds penalty. The top 10% of racers qualified into the Pro Division and also won a cool DeepRacer racing jacket!

Let’s have a look at some stats for all submissions:

All submissions

Racers: 752
Total submission attempts: 75895
Best total time: 00:00:55.788000 (JJ)
Mean total time: 00:04:50.443291223
Best lap: 00:00:18.196000 (JJ)
Mean best lap: 00:01:32.344817819
Mean average lap: 00:01:36.814091755

With a whopping 752 participants we were sure to see some interesting entries. On average each racer made a 100 attempts. Unsurprisingly the first place has been claimed by the Community’s very own JJ with a total time of just under 56 seconds. The gap between him and an average racer is pretty impressive at 3 minutes and 55 seconds.

Now let’s go over to the top 20 results:

Top 20 submissions

Racers: 20
Total submission attempts: 31772
Best total time: 00:00:55.788000 (JJ)
Mean total time: 00:01:01.671450
Best lap: 00:00:18.196000 (JJ)
Mean best lap: 00:00:20.088850
Mean average lap: 00:00:20.556750

Nearly 42% of overall submissions come from the top 20 racers. An average top 20 racer completed the race in just 1:01.67 which makes the top of the leaderboard look much more competitive.

Unsurprisingly many of the top racers have been present in the finals of AWS DeepRacer League 2020. Having the same track to compete in meant they had a bit of a head-start. Let’s have a look at the final leaderboard:

752 racers in the competition means 75 racers in the Pro Division in March. It also means that those 75 racers cannot take part in Open Division anymore.

Pro Division

Let’s have a look at what our freshly baked professionals are dealing with:

This incredibly tricky track has been named after the winner of 2020 AWS DeepRacer League. 89.24 meters long, 107 centimeters wide and with some soul-crushing turns including the sharpest one exiting a chicane past the hairpin in the upper-right corner. Roger Logan from the community recommends that we call it Yellow Brick Road since it looks like the beginning of Yellow Brick Read from The Wizard of Oz.

The race is a head-to-bot format with four bot cars cruising at 2 m/s without lane changes. Racers are trying to complete three laps. Each collision and each driving off the track results in a five seconds penalty.

Let’s also add that racers compete for sixteen places in the final race which will be streamed live in early April. Top three from that race will qualify into the finals which will hopefully happen during the 2021 AWS re:Invent in Las Vegas. If it happens, it will be a great opportunity for a big DeepRacer community meeting with all the finalists from 2020 having their travel to the conference carried over to this year. Aside from that top ten racers will win an AWS DeepRacer EVO.

A quick glance at overall stats:

All submissions
Update time: 2021-03-07 21:00:00 UTC

Racers: 40
Total submission attempts: 6444
Best total time: 01:35.600 (JJ)
Mean total time: 04:18.505
Best lap: 00:31.191 (JJ)
Mean best lap: 01:19.833
Mean average lap: 01:26.168

40 out of 75 racers have submitted at least once to the race so far. This gives the Pro division a rather slow start despite its impressive speed. Historically speaking it always felt like the start of the month was rather slow so I am expecting those numbers to go up as we come closer to the end of month.

Again, JJ is holding the lead pretty and his results stand out well above the average racers results. Towards the end of month we should see average total time to get closer to JJ, maybe we could even get below 2 minutes?

Top 20 racers have done absolute majority of submissions which confirms that levels of activity in this week have been rather low so far. Mean total time of 02:24:985 will definitely go down and reflect more of the one in pre-season race where there was a difference of only 6 seconds.

Top 20 submissions
Update time: 2021-03-07 21:00:00 UTC

Racers: 20
Total submission attempts: 5946
Best total time: 01:35.600 (JJ)
Mean total time: 02:24.985
Best lap: 00:31.191 (JJ)
Mean best lap: 00:44.797
Mean average lap: 00:48.328

In the table Ernesto is really close behind JJ. Interestingly in the top 10 seven racers have had at least one reset. Most of them were off-track and not collisions which makes me suspect that Yellow Brick Road is taking its toll. This track is much more narrow and turns are sharper than in the 2020 finals which might cause racers to train using too high action spaces. I’m curious to see how it plays out.

Open Division

Open races will always be slightly easier than the Pro ones. Let’s see what’s on in March:

Po-Chun Speedway is a simplified version of the Pro track but don’t be fooled. While it is missing the Yellow Brick Road, that turn right after is still there and should by no means be underestimated. 68.68 meters long, 107 centimeters wide has some challenging features for the racers. In the Time Trial format competitors need to complete three laps and total time counts for their classification. Each off-track results in five seconds penalty.

Racers in the Open Division fight for qualification to the Pro group. They will also receive a Pro Welcoming Kit which is a big mystery to us, but I’m sure it will be fun.

Within the first week we’ve already had 303 entries – not bad at all. Comparing to last year’s opening racer it’s already half the racers. I’m curious to see how many participants we reach.

VaughanDiesel has been in a seemingly comfortable lead since 21:00 UTC on Wednesday. He’s also holding the best lap time of the group. Each racer has been submitting six time on average which is a pretty high number. The gap between the best and mean total time is pretty big and I am wondering where it will settle toward the end of the month.

All submissions

Update time: 2021-03-07 21:00:00 UTC
Racers: 303
Total submission attempts: 1987
Best total time: 01:27.471 (VaughanDiesel)
Mean total time: 05:28.429029
Best lap: 00:27.727 (VaughanDiesel)
Mean best lap: 01:43.462432
Mean average lap: 01:49.476019

Top 20 Open racers aren’t as active as those in the Pro division but there is still quite some movement on the first page of standings. With 20 submissions on average it’s clear that these folks just keep going and trying. It’s a very reasonable tactic even with one model – if you perform multiple submissions you will quickly notice that times histogram is reflecting a normal distribution with a mean value and two tails, one of which I call the happy tail. There is always a bit of variance in the performance of a model.

The difference between the best and mean total time appears to be pretty big but it’s only an illusion, I will cover that in a bit. Small difference between the mean best lap and mean average lap suggest that the models submitted are performing relatively consistently.

Top 20 submissions

Update time: 2021-03-07 21:00:00 UTC
Racers: 20
Total submission attempts: 452
Best total time: 01:27.471 (VaughanDiesel)
Mean total time: 02:05.312
Best lap: 00:27.727 (VaughanDiesel)
Mean best lap: 00:38.627
Mean average lap: 00:41.770

At first glance VaughanDiesel appears to be dominating the race, but this changes when you look at satussy777 and GregRacing (you can watch videos of all entries on the race page). They both have pretty nice racing performance. Also add to it that if they manage to eliminate those resets at a similar speed, they will outpace the leader.

There’s also one more thing that I can see from the data. VaughanDiesel showed up on Wednesday, improved his result an hour later and hasn’t improved since despite making more attempts every day. Having raced for a while I’m suspecting some frustration building up from lack of improvement. I don’t know if these are new models every day or the same over and over but I’m curious to see what outcome it will bring. While GregRacing appears to have submitted a few times in one go and not returned since, satussy777 keeps pushing. We will see in the future updates.

That’s it for today. If you’d like to learn some Machine Learning skills while racing for glory, be sure to check out the AWS DeepRacer Page for hints on how to get started. Be sure to join the AWS Machine Learning Community on Discord for more tips and tricks on how to improve your model. See you next week!