Robots are great at running their repetitive autonomous tasks - security & inspection routes, bin picking, fruit harvesting, forklift robots all execute repetitive tasks in a flexible way. We have also heard a repeated need to save images for reporting in security and inspection systems, machine learning dataset generation or logging to determine when a fault has occurred, a mission needs to be triggered, or you need a custom interaction based on data from many robots.
We realized that this can be done quite elegantly with Freedom Robotics API - it was built to do these kinds of tasks both on one robot and at scale, We’ve been playing around with it and decided to write a tutorial - it's just a few lines of code and we're pretty excited about it.
Autonomy should be Operational from Day One
The assumption in robotics oftentimes is that you need to have a fully autonomous, tested and error-free robot to provide value. As a result, robotics companies require a huge amount of upfront development costs and don’t get the chance to iterate with the customer early on. We believe that you can provide value to your customer, learn their needs, and iterate on your product without fully building out your stack. What we’re releasing today is the key to that - a drop-in replacement for autonomy that also helps you develop autonomy over time. Let’s take a look at some real-world examples.
Example: Capturing Images of Harvested Fruit
Capturing images is really valuable for managing robots that interact with the outside world (as opposed to pre-defined movements on an assembly line) where you may need to trigger a mission, save key images or change how the robot performs based on dynamic changes in the environment (ex: YOLOV3 or other SSD object detection methods, a proximity sensor identifying a safety situation and remotely sending a new path, remotely reporting on success/failure of missions, etc), actions of a user or other external triggers.
.webp)
Build an operational orange picking robot from day one while gathering data for a machine learning pipeline. An operator remotely labels oranges and their ripeness after which the labeled images are stored to a server.
When harvesting fruit with a robot, both auditing the quality/ripeness of the fruit and also improving your computer vision algorithms require mass-recording of every fruit you pick. By being able to easily record hundreds or millions of images from any number of robots, you can simplify the data collection system significantly.
Example: Validating Events on a Security Robot
Let’s look at another interesting scenario. A security robot detects a new person roaming through the building at night. The image immediately gets saved on a remote database and an alert is sent to the operator on call. The person disappears before the security operator can handle the situation, but he can look at the pictures the robot took to identify who it was. A report automatically is generated the next morning detailing all incidents of the night.
.webp)
Being able to automatically identify if there is a disturbance and alert the operator, save an image and call the right people automatically allows for a much simpler on-robot system.
Current solution: A recipe for headaches
Some people SCP from the robot with a bash script (and need server keys), others try to save locally, some build systems to interface with AWS S3 and many more just record a ROS Bag and copy it off and then grep through it for their images in post-processing. All of these have significant challenges for stability, reliability and scalability.
When you have one robot, you could manage this all by hand with scripts. You can trigger things with Playstation controllers or other tethered buttons/keystrokes. A key challenge is what to do when the robot leaves the lab and goes into the world - what happens if you will never see it again, don’t have stable connectivity or you don’t control the SSH keys anymore? This quickly becomes unmanageable the moment you start growing your fleet.
A Tutorial: Autonomous/Operator Data Labeling and Image Capture
On real distributed robots, you need the ability to identify a state across the fleet in real-time, trigger an action (Capture an image, send a message, launch a script, etc) and do so efficiently.
We’ve released an example of how to do this in the form of a tutorial. Take a look at how you can quickly create a machine learning data collection pipeline for detecting chairs and tables in a few lines of Python.
When the operator sees something happen on the robot, or the robot’s CV algorithms flag a tag, then the script in the tutorial will automatically record the current video frames for all cameras so they can be reviewed later.
The python script running remotely watches for a trigger and grabs an image (or lidar scan, joint state data, or even a custom message) when a triggering message comes in. This triggering message can also originate from anywhere: from the operator or from the robot. While testing your object detection algorithm, you could even have both: the robot recognizes the objects and the operator flags those that the robot incorrectly classified or missed.
Using the Freedom Pilot interface, operators can trigger remote captures by adding a memorized command. This allows for easy data tagging and classification or triggering remote actions outside of the robot.
Head over to our tutorial to see how you can do this on your robot and modify the code for your application. We’ve been amazed by what people have been creating with the Freedom API and can’t wait to see what you build next!
