Dagmawi Babi
Kinda cool ngl! With OpenAI, Figure 01 can now have full conversations with people. OpenAI models provide high-level visual and language intelligence and added to Figure's neural networks that deliver fast, low-level, dexterous robot actions it looks great.…
The entire architecture of Figure's robot is impressive. It's not even about OpenAI's multimodal LLMs. Any other model can be configured and control the bot so well.
Because all behaviors are driven by neural network visuomotor transformer policies, mapping pixels directly to actions. These networks take in onboard images at 10hz, and generate 24-DOF actions (wrist poses and finger joint angles) at 200hz.
This is a useful separation of concerns:
• Internet-pretrained models do common sense reasoning over images and text to come up with a high-level plan.
• Learned visuomotor policies execute the plan, performing fast, reactive behaviors that are hard to specify manually, like manipulating a deformable bag in any position.
• Meanwhile, a whole body controller ensures safe, stable dynamics. For example, maintaining balance.
#Figure #Robot
@Dagmawi_Babi
Because all behaviors are driven by neural network visuomotor transformer policies, mapping pixels directly to actions. These networks take in onboard images at 10hz, and generate 24-DOF actions (wrist poses and finger joint angles) at 200hz.
This is a useful separation of concerns:
• Internet-pretrained models do common sense reasoning over images and text to come up with a high-level plan.
• Learned visuomotor policies execute the plan, performing fast, reactive behaviors that are hard to specify manually, like manipulating a deformable bag in any position.
• Meanwhile, a whole body controller ensures safe, stable dynamics. For example, maintaining balance.
#Figure #Robot
@Dagmawi_Babi