Welcome to Stretch Mapping, a semantic mapping framework built on Khronos[1] and ORB-SLAM2[2] for creating a semantic map using the Hello Robot Stretch mobile manipulator. Khronos is a semantic mapping pipeline that relies on visual SLAM, and semantic segmentation from RGB-D perception, and creates a hierarchical 3-D map of the world. We utilize the Stretch manipulation platform, which is equipped with a Intel RealSense D435i. We use semantic inference codebase which offers ROS integration for semantic segmentation and ORB-SLAM2[2] codebase with ROS integration for robot localization. This mapping framework is capable of creating hierarchical object-centric maps, tracking moving objects, and maintaining long-term object changes, and provides a rich contextual understanding for household navigation and mobile manipulation.

Dynamic human tracking	Metric semantic mapping	Topological mapping

[1] Lukas Schmid, Marcus Abate, Yun Chang, and Luca Carlone. Khronos: A unified approach for spatio-temporal metric-semantic slam in dynamic environments. In Proc. of Robotics: Science and Systems (RSS), Delft, Nether- lands, July 2024.

[2] Montiel J. M. M. Mur-Artal, Raúl and Juan D. Tardós. ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Transactions on Robotics, 31(5):1147–1163, 2015.

Table of Contents

• Installation 🚀 and Usage 🎮
• Quantitative Evaluation ✍🏼
• Support and Contribution 🤝

---

Installation 🚀 and Usage 🎮

This repository is tested on Ubuntu 20.04 and ROS Kinetic, but would potentially work on any ROS1 distribution. Refer to Installation.md for installation instructions, and Usage.md for instructions on how to run the system, both online on a robot and offline using ROS bags, and also a guide on how to tune various parameters.

---

Quantitative Evaluation ✍🏼

We assess the performance of our proactive mapping pipeline in a household environment with a person interacting with objects both in and out of the Stretch robot's field of view. We systematically evaluate our framework's ability to track moving objects and people within the robot's field of view and map objects displaced outside the robot's field of view by deploying the robot in the following four apartment scenarios:

1. Static Objects + No Person In View: No people are present in the apartment and all objects remain still.
2. Static Objects + Person Moving in View: A person moves inside the aparment within the robot's field of view but does not move any objects.
3. Displaced Objects + No Person in View: A person moves an object but remains outside the robot's field of view.
4. Moving Objects + Person Moving in View: A person moves an object within the robot's field of view.

For each scenario, we measure the object recall scores of both Khronos using the robot's odometry and Khronos using localization measurements from ORBSLAM2. Recall measures the mapping framework's ability to correctly capture all object instances in the environment. Please refer to each individual scenario for detailed results.

---

Support and Contribution 🤝

If you have any questions, feel free to post an issue on this repo or please reach out to our contributers. We're happy to help! To contribute to this codebase, please open a pull request or reach out to us via email.

Notice for contributing changes to Submodules ⚠️

DON'T directly make changes to the submodules in the repository and expect them to be updated. Since they are submodules, updates require a specific workflow. Instead, follow these steps:

1. Clone the submodule repository (e.g., for Khronos) and make changes:

   git clone git@github.com:<githubaccount>/Khronos.git
   cd Khronos
   # Make your changes here
   git add .
   git commit -m "Your changes"
   git push origin main

2. Navigate back to the StretchMapping repository:

   cd /path/to/StretchMapping

3. Update the submodule reference:

   git submodule status
   git submodule update --init --recursive
   cd Khronos
   git remote -v

   If the URL is incorrect or missing, set the correct remote:

   git remote set-url origin git@github.com:<githubaccount>/Khronos.git
   git fetch
   git checkout main

4. Update the submodule pointer in StretchMapping:

   cd ..
   git add Khronos
   git commit -m "Update Khronos submodule to latest revision"
   git push origin main
   git submodule update --remote --merge

Contributors:

1. Harsh Muriki: harshsuhith@gmail.com
2. Nhi Nguyen (Leona): yennhi1908hcm@gmail.com
3. Rahul Rustagi: rustagirahul24@gmail.com