Robotic_Arm_DQN

Work in Progress (including the README.md file)

Trains a Kuka LBR iiwa robotic arm through the reinforcement learning algorithm, deep Q-network, within the PyBullet Simulation Environment.

Yet to exhibit intelligent motion. Steps forward:

• Upgrade DQN learning algorithm to Double-DQN, Duelling-DQN, or both.
• Upgrade CNN to CapsNet.
• Tile joint state before the concatenate, as done in this paper.

Getting Started

I had some difficultly getting PyBullet up and running on my Windows machine, but here are the steps I took to make it work:

1. Download built tools, found here. During the installation process, be sure to check:
   • Windows 10 SDK (10.0...)
   • Visuals C++ tools for CMake
   • C++/CLI Support
   • VC++ 2015.3 v14.00 (v40) toolset for desktop
2. According to this article, copy the rc.exe and rcdll.dll files from the C:/Program Files (x86)/Windows Kits/8.1/bin/x86 directory to C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin.
3. Upgrade python set up tools using pip install --upgrade setuptools.
4. Install PyBullet using pip install pybullet.

Note: Please take these steps with a grain of salt. What may work for me might not for you. Also, I am sure I have a few of the dependencies already installed on my machine, so this is not a complete setup guide. Please refer to the documentation.

With PyBullet installed, the best place to get started is through the PyBullet Quick Start Guide.

urdf_utils.py

A collection of utility functions that can be used to test *.urdf robots. The original use case was to assure that the development of novel Kuka LBR iiwa end effectors were working as intended in the simulation environment.

Functions

The functions currently available in the urdf_utils.py file are demonstrated using the Kuka LBR iiwa robot arm. Full examples of how to use the functions are shown below.

Note: these functions have only been tested with the kuka_iiwa/model.urdf file and a handful of other *.urdf files.

find_joint_count()

Counts the number of joints in robot, as defined in the *.urdf file.

Arguments	Description	Default
ID	An integer representing a robot, as return from the p.loadURDF() method.

Example

import pybullet as p
import pybullet_data

import urdf_utils

p.connect(p.GUI)
p.setAdditionalSearchPath(pybullet_data.getDataPath())
kuka = p.loadURDF('kuka_iiwa/model.urdf', [0, 0, 0], useFixedBase=True)

kuka_joint_count = urdf_utils.find_joint_count(ID=kuka)

print(kuka_joint_count)

Returns:

7

test_movement_extents()

Iterates through each joint and rotates them to the extent of its movement. Returns a list of tuples representing the minimum and maximum extents of rotation for each joint.

Arguments	Description	Default
ID	An integer representing a robot, as return from the p.loadURDF() method.
joint_count	An integer representing the number of joints as defined in the *.urdf file. Can be calculated with the find_joint_count() function.
speed	A number representing the rate at which each link rotates.	0.2
precision	An integer representing the number of decimal places to which the extents will be calculated to.	5
delay	A number that artificially delays the simulation steps with time.sleep(delay).	0.001

Example

import pybullet as p
import pybullet_data

import urdf_utils

p.connect(p.GUI)
p.setAdditionalSearchPath(pybullet_data.getDataPath())
kuka = p.loadURDF('kuka_iiwa/model.urdf', [0, 0, 0], useFixedBase=True)

kuka_joint_count = urdf_utils.find_joint_count(ID=kuka)

kuka_extents = urdf_utils.test_movement_extents(
    ID=kuka,
    joint_count=kuka_joint_count,
    speed=0.5,
    precision=7,
    delay=0.002
)

print(kuka_extents)

Returns:

[
    (-169.9999682239179, 169.9999530056643),
    (-119.9999695404722, 119.9999487814800),
    (-169.9999682080224, 169.9999595191940),
    (-119.9999589600828, 119.9999484192288),
    (-169.9999686172800, 169.9999593911397),
    (-119.9999536575531, 119.9999615657730),
    (-174.9999600597941, 174.9999496216755)
]

More to come (when needed)...