. 24/7 Space News .
INTERNET SPACE
A one-up on motion capture
by Lauren Hinkel | MIT-IBM Watson AI Lab
Boston MA (SPX) May 01, 2022

MIT researchers used the RISP method to predict the action sequence, joint stiffness, or movement of an articulated hand, like this one, from a target image or video.

From "Star Wars" to "Happy Feet," many beloved films contain scenes that were made possible by motion capture technology, which records movement of objects or people through video. Further, applications for this tracking, which involve complicated interactions between physics, geometry, and perception, extend beyond Hollywood to the military, sports training, medical fields, and computer vision and robotics, allowing engineers to understand and simulate action happening within real-world environments.

As this can be a complex and costly process - often requiring markers placed on objects or people and recording the action sequence - researchers are working to shift the burden to neural networks, which could acquire this data from a simple video and reproduce it in a model. Work in physics simulations and rendering shows promise to make this more widely used, since it can characterize realistic, continuous, dynamic motion from images and transform back and forth between a 2D render and 3D scene in the world. However, to do so, current techniques require precise knowledge of the environmental conditions where the action is taking place, and the choice of renderer, both of which are often unavailable.

Now, a team of researchers from MIT and IBM has developed a trained neural network pipeline that avoids this issue, with the ability to infer the state of the environment and the actions happening, the physical characteristics of the object or person of interest (system), and its control parameters. When tested, the technique can outperform other methods in simulations of four physical systems of rigid and deformable bodies, which illustrate different types of dynamics and interactions, under various environmental conditions. Further, the methodology allows for imitation learning - predicting and reproducing the trajectory of a real-world, flying quadrotor from a video.

"The high-level research problem this paper deals with is how to reconstruct a digital twin from a video of a dynamic system," says Tao Du PhD '21, a postdoc in the Department of Electrical Engineering and Computer Science (EECS), a member of Computer Science and Artificial Intelligence Laboratory (CSAIL), and a member of the research team. In order to do this, Du says, "we need to ignore the rendering variances from the video clips and try to grasp of the core information about the dynamic system or the dynamic motion."

Du's co-authors include lead author Pingchuan Ma, a graduate student in EECS and a member of CSAIL; Josh Tenenbaum, the Paul E. Newton Career Development Professor of Cognitive Science and Computation in the Department of Brain and Cognitive Sciences and a member of CSAIL; Wojciech Matusik, professor of electrical engineering and computer science and CSAIL member; and MIT-IBM Watson AI Lab principal research staff member Chuang Gan. This work was presented this week the International Conference on Learning Representations.

While capturing videos of characters, robots, or dynamic systems to infer dynamic movement makes this information more accessible, it also brings a new challenge. "The images or videos [and how they are rendered] depend largely on the on the lighting conditions, on the background info, on the texture information, on the material information of your environment, and these are not necessarily measurable in a real-world scenario," says Du.

Without this rendering configuration information or knowledge of which renderer is used, it's presently difficult to glean dynamic information and predict behavior of the subject of the video. Even if the renderer is known, current neural network approaches still require large sets of training data. However, with their new approach, this can become a moot point. "If you take a video of a leopard running in the morning and in the evening, of course, you'll get visually different video clips because the lighting conditions are quite different. But what you really care about is the dynamic motion: the joint angles of the leopard - not if they look light or dark," Du says.

In order to take rendering domains and image differences out of the issue, the team developed a pipeline system containing a neural network, dubbed "rendering invariant state-prediction (RISP)" network. RISP transforms differences in images (pixels) to differences in states of the system - i.e., the environment of action - making their method generalizable and agnostic to rendering configurations. RISP is trained using random rendering parameters and states, which are fed into a differentiable renderer, a type of renderer that measures the sensitivity of pixels with respect to rendering configurations, e.g., lighting or material colors.

This generates a set of varied images and video from known ground-truth parameters, which will later allow RISP to reverse that process, predicting the environment state from the input video. The team additionally minimized RISP's rendering gradients, so that its predictions were less sensitive to changes in rendering configurations, allowing it to learn to forget about visual appearances and focus on learning dynamical states. This is made possible by a differentiable renderer.

The method then uses two similar pipelines, run in parallel. One is for the source domain, with known variables. Here, system parameters and actions are entered into a differentiable simulation. The generated simulation's states are combined with different rendering configurations into a differentiable renderer to generate images, which are fed into RISP. RISP then outputs predictions about the environmental states. At the same time, a similar target domain pipeline is run with unknown variables.

RISP in this pipeline is fed these output images, generating a predicted state. When the predicted states from the source and target domains are compared, a new loss is produced; this difference is used to adjust and optimize some of the parameters in the source domain pipeline. This process can then be iterated on, further reducing the loss between the pipelines.

To determine the success of their method, the team tested it in four simulated systems: a quadrotor (a flying rigid body that doesn't have any physical contact), a cube (a rigid body that interacts with its environment, like a die), an articulated hand, and a rod (deformable body that can move like a snake). The tasks included estimating the state of a system from an image, identifying the system parameters and action control signals from a video, and discovering the control signals from a target image that direct the system to the desired state.

Additionally, they created baselines and an oracle, comparing the novel RISP process in these systems to similar methods that, for example, lack the rendering gradient loss, don't train a neural network with any loss, or lack the RISP neural network altogether. The team also looked at how the gradient loss impacted the state prediction model's performance over time. Finally, the researchers deployed their RISP system to infer the motion of a real-world quadrotor, which has complex dynamics, from video. They compared the performance to other techniques that lacked a loss function and used pixel differences, or one that included manual tuning of a renderer's configuration.

In nearly all of the experiments, the RISP procedure outperformed similar or the state-of-the-art methods available, imitating or reproducing the desired parameters or motion, and proving to be a data-efficient and generalizable competitor to current motion capture approaches.

For this work, the researchers made two important assumptions: that information about the camera is known, such as its position and settings, as well as the geometry and physics governing the object or person that is being tracked. Future work is planned to address this.

"I think the biggest problem we're solving here is to reconstruct the information in one domain to another, without very expensive equipment," says Ma. Such an approach should be "useful for [applications such as the] metaverse, which aims to reconstruct the physical world in a virtual environment," adds Gan. "It is basically an everyday, available solution, that's neat and simple, to cross domain reconstruction or the inverse dynamics problem," says Ma.

This research was supported, in part, by the MIT-IBM Watson AI Lab, Nexplore, DARPA Machine Common Sense program, Office of Naval Research (ONR), ONR MURI, and Mitsubishi Electric.

Research Report:RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation


Related Links
MIT-IBM Watson AI Lab
Satellite-based Internet technologies


Thanks for being there;
We need your help. The SpaceDaily news network continues to grow but revenues have never been harder to maintain.

With the rise of Ad Blockers, and Facebook - our traditional revenue sources via quality network advertising continues to decline. And unlike so many other news sites, we don't have a paywall - with those annoying usernames and passwords.

Our news coverage takes time and effort to publish 365 days a year.

If you find our news sites informative and useful then please consider becoming a regular supporter or for now make a one off contribution.
SpaceDaily Monthly Supporter
$5+ Billed Monthly


paypal only
SpaceDaily Contributor
$5 Billed Once


credit card or paypal


INTERNET SPACE
Apple reports solid Q2, but warns of $4-$8 bn hit from Covid, supply chain
New York (AFP) April 28, 2022
Apple reported better-than-expected profits Thursday amid continued robust consumer demand, but warned that the China Covid-19 lockdown and ongoing supply chain woes would dent June quarter results by $4 to $8 billion. The iPhone maker enjoyed another solid performance for the period ending March 26, registering record revenues for the quarter. But executives said the difficulties of the pandemic have returned with a vengeance since the reporting period ended. "Supply constraints caused by Covid ... read more

Comment using your Disqus, Facebook, Google or Twitter login.



Share this article via these popular social media networks
del.icio.usdel.icio.us DiggDigg RedditReddit GoogleGoogle

INTERNET SPACE
Astronaut crew returning to Earth after six months on ISS

NASA chooses small businesses to continue exploration tech development

NASA's new solar sail system to be tested on-board NanoAvionics satellite bus

New standard will aid in classification of commercial spaceflight safety events

INTERNET SPACE
Musk secures $7.1 bn to finance Twitter deal

NASA's Crew-3 astronauts splash down in Atlantic Ocean

Aphelion Aerospace completes rocket engine development test

British rocket company calls for Iceland to grant licence for landmark launch

INTERNET SPACE
Emirates Mars mission discovers new mysterious aurora

China's Zhurong travels over 1.9 km on Mars

Farewell to the Torridon Quad - Sols 3459-3461

Enigmatic Rock Layer in Mars' Gale Crater Awaits Measurements by the Curiosity Rover

INTERNET SPACE
China opens Shenzhou-13 return capsule

NASA Chief slams China's refusal to cooperate with US

Xi Focus: Invigorating China's space exploration dream

Tianzhou-3 docks with Tianhe's front docking port

INTERNET SPACE
SSi Canada contracts SES to meet Canadian Government broadband goals

FCC grants experimental license to AST SpaceMobile for BlueWalker 3 cell phone tests

AST SpaceMobile announces collaboration with Globe Telecom

Nanoavionics builds first nanosatellite for Promethee's EO constellation

INTERNET SPACE
Unpacking black-box models

'Like family': Japan's virtual YouTubers make millions from fans

Cosmic Shielding to test Plasteel radiation shielding aboard Space Forge satellite

How can we reduce the carbon footprint of global computing?

INTERNET SPACE
Discovery of 30 exocomets in a young planetary system

Origin of complex cells started without oxygen

The instability at the beginning of the solar system

Scientists study microorganisms on Earth to gain insight into life on other planets

INTERNET SPACE
Juno captures moon shadow on Jupiter

Greenland Ice, Jupiter Moon Share Similar Feature

Search for life on Jupiter moon Europa bolstered by new study

Abundant features on Europa bodes well for search for extraterrestrial life









The content herein, unless otherwise known to be public domain, are Copyright 1995-2024 - Space Media Network. All websites are published in Australia and are solely subject to Australian law and governed by Fair Use principals for news reporting and research purposes. AFP, UPI and IANS news wire stories are copyright Agence France-Presse, United Press International and Indo-Asia News Service. ESA news reports are copyright European Space Agency. All NASA sourced material is public domain. Additional copyrights may apply in whole or part to other bona fide parties. All articles labeled "by Staff Writers" include reports supplied to Space Media Network by industry news wires, PR agencies, corporate press officers and the like. Such articles are individually curated and edited by Space Media Network staff on the basis of the report's information value to our industry and professional readership. Advertising does not imply endorsement, agreement or approval of any opinions, statements or information provided by Space Media Network on any Web page published or hosted by Space Media Network. General Data Protection Regulation (GDPR) Statement Our advertisers use various cookies and the like to deliver the best ad banner available at one time. All network advertising suppliers have GDPR policies (Legitimate Interest) that conform with EU regulations for data collection. By using our websites you consent to cookie based advertising. If you do not agree with this then you must stop using the websites from May 25, 2018. Privacy Statement. Additional information can be found here at About Us.