ZSPMS

An autonomous package-sorting cell in the Webots R2023b simulator: a UR10e arm picks packages off a conveyor, reads each destination label, and drops the package into the matching city crate. Built for the Advanced Robotics and Artificial Intelligence (ARAI) module of my MSc.

Interactive 3D animation — open on desktop to orbit and zoom.

The problem#

A sorting cell has to close the loop from pixels to placement: notice a package, work out where it's going, pick it up without dropping it, put it in the right crate, and not lose track of the packages still arriving. Each step has its own failure mode, and the failures compound — a misread label sends a package to the wrong crate no matter how well the arm moves.

Two constraints shaped the whole design. There was no pretrained detector for generic cardboard packages that worked inside Webots, and the Webots UR10e has no built-in inverse kinematics.

Perception: from pixels to a locked destination#

A custom detector, trained on simulated data#

Detectors trained on COCO or OpenImages have no "package" class suitable for logistics, so I built the dataset inside the simulator. A dedicated Webots controller captures RGB frames from the camera along with their labels and exports them to disk in the Ultralytics package-segmentation format.

One practical snag: Ultralytics doesn't create its dataset directories and cache until the first training run. I ran a single throwaway epoch to trigger that setup, added my Webots-generated images and labels into the training split, and then trained the YOLOv8 segmentation model for real. The result, best.pt, is a custom model trained on data from this environment, not a generic pretrained package detector.

At runtime the detector keeps detections above 0.45 confidence and 2,000 px² in area, and the pipeline follows the largest package — it is deliberately designed to process one package at a time.

Reading the label, then checking the reading#

The detected package is cropped, resized, converted to greyscale, blurred, morphologically processed, and adaptively thresholded before EasyOCR reads it. The raw text is lowercased and stripped to letters, then fuzzy-matched against the five valid destinations — London, Manchester, Sheffield, Leeds, and Lagos — to absorb OCR errors.

Two safeguards stop a single bad read from deciding where a package goes:

• Confirmation hits. The same destination has to be read twice before it locks.
• CLIP verification. OCR and fuzzy matching propose a destination; CLIP (openai/clip-vit-base-patch32) checks it against prompts like "A cardboard package with a shipping label for London" and rejects candidates below 0.20 probability. CLIP verifies — it isn't the primary classifier.

Manipulation: joint-space control without IK#

The Webots UR10e PROTO in R2023b exposes no inverseKinematics or endEffector field, so every motion is driven directly through the arm's six joint motors, with three finger motors on the gripper. Destination poses are calibrated six-joint targets per crate, loaded from config/bins.json when present.

What went wrong first#

The early state machine decided when a movement had finished by watching one joint: it opened the gripper once wrist_1_joint passed a threshold, and treated the return home as complete when the same joint came back near zero. One wrist angle says nothing about where the other five joints are, so the package could be released before the arm had actually reached the crate.

Gripper timing had the mirror-image problem. Commanding the arm to move in the same step as closing the gripper meant the arm could leave before the fingers had closed around the package.

The fix: a joint-sensor-driven FSM#

The final controller replaces timers and single-joint heuristics with the arm's own sensors:

1. IDLE — wait for a locked destination and a package in reach.
2. GRASP — close the gripper and wait for it to settle.
3. MOVE_TO_BIN — wait until all six joints reach the bin target.
4. RELEASE — open the gripper.
5. WAIT_RELEASE_CLEAR — wait for the distance sensor to confirm the package has gone.
6. MOVE_HOME — wait until all six joints reach home, then return to IDLE.

A movement is complete only when every one of the six joint position sensors is within tolerance of its target. Release is confirmed by the distance sensor rather than assumed after a delay.

Keeping the conveyor in step#

The conveyor runs while packages are being scanned and detected, stops once the robot has actually picked a package — not merely when the destination locks — and resumes only after the arm is back home. That keeps other packages from drifting past while the arm is busy, which avoids missed packages and keeps OCR stable.

Packages are spawned dynamically by the controller, which also acts as a Webots Supervisor: it imports new package nodes onto the conveyor with randomly chosen label textures, keeps them about 1.5 m apart, and caps the scene at 20.

Limitations and next steps#

• Calibrated destinations. Each crate needs manually calibrated joint angles. The next step I was investigating was runtime IK: use the Supervisor to read the destination crate's pose, compute a safe drop pose above it, and solve for the six joint angles with an external Python IK solver. It isn't implemented.
• One package at a time. Tracking follows the single largest detection; multi-package tracking is future work.
• No live overlay yet. A GUI showing per-destination counts and the last OCR read is planned, not built.
• Simulation only. A real-robot deployment would need a ROS 2 bridge.

Code#