ROBOT VISION IN ROBOT OPERATING SYSTEM (ROS2)

Course Overview

FUNDAMENTAL COURSE (2-Days)

This course provides a comprehensive introduction to the principles and techniques of computer vision, specifically tailored for robotics applications. It covers the essential concepts, algorithms, and tools necessary to equip students with the knowledge to design and implement robust vision systems for robots using ROS2.

​ADVANCED COURSE (3-Days)

This advanced course delves into the application of deep learning techniques to solve complex robot vision problems. It builds upon the fundamentals of computer vision and robotics, focusing on state-of-the-art deep learning models and their integration with robotic systems using ROS2.

Course Objectives

Upon completion of this course, students will be able to:

• Understand the basics of camera systems, image generation, and pixel processing

• Be able to calibrate cameras, enhance images, and reconstruct scenes based on 2D and 3D vision information

• Understand the concepts of Machine Learning and Deep Learning as applied to robot vision problems

• Be able to program cameras and build a 2D or 3D vision system to make a robot see its environment in ROS2 platform

Course Curriculum

​

DAY 1 (FUNDAMENTAL)

​

Module 1: Introduction to Computer Vision

• Image Formation and Sensors:

  • Pinhole camera model

  • Camera calibration

  • Types of sensors

• Image Processing Fundamentals:

  • Image acquisition and representation

  • Image enhancement techniques

  • Geometric transformations

Module 2: Feature Detection and Description

• Interest Point Detection:

  • Harris corner detector

  • Shi-Tomasi corner detector

  • Scale-invariant feature transform (SIFT)

  • Speeded Up Robust Features (SURF)

  • Oriented FAST and Rotated BRIEF (ORB)

• Feature Description:

  • Local binary patterns (LBP)

  • Histogram of Oriented Gradients (HOG)

Module 3: Stereo Vision and 3D Reconstruction

• Stereo Matching:

  • Disparity estimation techniques (e.g., block matching, dynamic programming)

  • Depth map generation

• Structure from Motion (SfM):

  • 3D reconstruction from multiple images

  • Bundle adjustment

• Simultaneous Localization and Mapping (SLAM)

  • Building 3D maps of an environment

  • Tracking robot’s position and orientation

  • Bundle adjustment

• Simultaneous Localization and Mapping (SLAM):

  • Building 3D maps of an environment

  • Tracking robot's position and orientation

DAY 2 (FUNDAMENTAL)

Module 4: Object Detection and Tracking

• Traditional Object Detection:

  • Template matching

  • Haar cascades

  • Histogram of Oriented Gradients (HOG) + Linear SVM

• Deep Learning for Object Detection:

  • Convolutional Neural Networks (CNNs)

  • Region-based Convolutional Neural Networks (R-CNN)

  • You Only Look Once (YOLO)

• Object Tracking:

  • Template matching

  • Kalman filters

  • Particle filters

  • DeepSORT

​

Module 5: Running OpenCV in ROS2

• Installing OpenCV in ROS2

• Running your vision code in ROS2

• Bonus Activity:

  • Mount a camera onto a robot arm

  • Stream its data, and

  • Process the information for object detection or calibration

​

DAY 3 (ADVANCE)

​

Module 1: Deep Learning Fundamentals for Computer Vision

• Convolutional Neural Networks (CNNs):

  • Architecture and components

  • Feature extraction and classification

  • Transfer learning and fine-tuning

• Recurrent Neural Networks (RNNs):

  • Sequence modeling and time series data

  • Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU)

• Transformers:

  • Self-attention mechanism and encoder-decoder architecture

  • Vision Transformer (ViT) and its variants

​

Module 2: Object Detection and Instance Segmentation

• Two-stage detectors:

  • Region-based Convolutional Neural Networks (R-CNN)

  • Faster R-CNN

  • Mask R-CNN

• One-stage detectors:

  • YOLO

  • EfficientDet

• Instance segmentation:

  • Mask R-CNN

  • Detectron2

​

Module 3: Semantic and Instance Segmentation

• Fully Convolutional Networks (FCNs):

  • Pixel-wise classification

• Encoder-Decoder architectures:

  • U-Net

  • DeepLabv3+

• Instance segmentation:

  • Mask R-CNN

​

DAY 4 (ADVANCE)

​

Module 4: Optical Flow and Motion Tracking

• Optical flow estimation:

  • Traditional methods (Lucas-Kanade)

  • Deep learning-based methods (RAFT, PWC-Net)

• Motion tracking:

  • Tracking-by-detection

  • DeepSORT

​

Module 5: 3D Object Detection and Pose Estimation

• Monocular 3D object detection:

  • Deep learning-based methods (Mono3D, PVRCNN) 

• Stereo 3D object detection:

  • Stereo R-CNN

• LiDAR-based 3D object detection:

  • PointPillars, PointRCNN

​

​

DAY 5 (ADVANCE)

​

Module 6: Integration with Robotics (Stretch Goals)

• Object detection and tracking: Implement a system to detect and track objects in real-time video streams.

• Semantic segmentation: Segment different objects and scenes in images.

• 3D object detection and pose estimation: Detect and estimate the pose of 3D objects from RGB-D or LiDAR data.

• Object Detection and Tracking: Implement an object detection and tracking system using traditional or deep learning methods.

• Robot Vision Application: Integrate a vision system with a robot to perform tasks like object grasping or autonomous navigation.

Course Fees

• FUNDAMENTALS ONLY (DAYS 1-2) : MYR 2,500 per person

• ADVANCED ONLY (DAYS 3-5) : MYR 4,000 per person

• FUNDAMENTALS + ADVANCED (Days 1-5) : MYR 5,990 per person (recommended!)

Notes:

• Only participants with ROS knowledge are allowed to enrol to the advanced course unless they also take the Fundamentals

• EMAIL US FOR IN-HOUSE / BESPOKE RATE!

• Payment Method: Wire or Bank Transfer (Maybank)

​