RAM-DQN.mp4
- CARLA provides open digital assets (urban layouts, buildings, vehicles) that were created for this purpose and can be used freely. The simulation platform supports flexible specification of sensor suites, environmental conditions, full control of all static and dynamic actors, maps generation, and much more.
- This project shows how to train a Self-Driving Car agent by segmenting the RGB Camera image streaming from an urban scene in the CARLA simulator to simulate the autonomous behavior of the car inside the game, utilizing the Reinforcement Learning technique Deep Q-Network [DQN].
- A reinforcement learning task is about training an AGENT which interacts with its ENVIRONMENT.
- The agent arrives at different scenarios known as STATES by performing ACTIONS.
- Actions lead to REWARDS which could be positive or negative.
- The agent has only one purpose here – to maximize its total reward across an EPISODE.
- This episode is anything and everything that happens between the first state and the last or terminal state within the environment.
- We reinforce the agent to learn to perform the BEST ACTIONS by experience. This is the STRATEGY or POLICY.
- Q-Learning builds a Q-table of State-Action values, with dimension (s, a), where s is the number of states and a is the number of actions.
- The Q-table maps state and action pairs to a Q-value.
- However, in a real-world scenario, the number of states could be huge, making it computationally intractable to build a table.
- In deep Q-learning, we use a Neural Network to approximate the Q-value function, which is called a Deep Q Network [DQN].
- This function maps a state [Input] to the Q-values of all the actions that can be taken from that state [Output].
DQN Algorithm: [1] Feed the DQN Agent with the preprocessed segmented urban scene image (state s)
- and it returns the Q-values of all possible actions in the state [different values for throttle, steer].
[2] Select an Action using the Epsilon-Greedy Policy.
- With the probability epsilon, we select a random action a.
- With the probability 1-epsilon, we select an action that has a maximum Q-value, such as a = argmax(Q(s,a,w)).
[3] Perform this Action in a state s and move to a new state s’ to receive a reward.
- This state s’ is the next image.
- We store this transition in our Replay Buffer as <s,a,r,s’>
[4] Next, sample some Random Batches of transitions from the Replay Buffer.
[5] Calculate the Loss which is just the squared difference between target-Q and predicted-Q.
[6] Perform Gradient Descent with respect to our actual network parameters [DQN Agent] in order to minimize this loss.
[7] After every C iteration, copy our actual network weights to the Target network weights.
[8] Repeat these steps for M number of episodes.
✔️ Training the model.
✔️ Evaluatation using pretrained model.
The following tools you will need to run this repo:
Before starting, you need to have Git and Anaconda installed. Also, you will need to run each python command in different terminal as they will be running at the same time during the training.
# Clone this project
$ git clone [email protected]:mohamedelmesawy/DQN-Agent-in-Calrla-Simulator
# Create Conda Environment
$ conda env create -f carla.yml
$ conda activate carla
# Run the Carla Simulator - default port 2000
$ ./CarlaUE4.sh -quality-level Low -fps 20
or
$ ./CarlaUE4.exe -quality-level Low -fps 20
# Generate Vehicles and Walkers inside Carla, [-n] for num_vehicles and [-w] for num_walkers
$ python ./generate_traffic.py -n 70 -w 100
# Run the DQN-Agent at Test Mode using the pretrained model
$ python ./loadmodel.py
# Run the DQN-Agent at Train Mode
$ python ./DQN_in_Carla.py
# Check the updated logs
$ tensorboard --logdir logsBy RAM-Team: MOhamed ElMesawy, Afnan Hamdy, Rowan ElMahalawy, Ali Hassanin, and MOhamed Samaha
This code was built on this repo CARLA RL and this book Hands-On Reinforcement Learning with Python.