Adobe researchers present an AI framework that trains a neural network to learn data-driven input for mesh segmentation with distortion detection
Extracting a patch of considerable area surrounding a point that can be accurately mapped on the 2D plane is required for many interactive workflows such as decaling, texturing, or painting on a 3D model. Because they are inherently user-interactive, can achieve less distortion than their global counterparts, and are computationally more efficient, local parameterizations are desirable in some modeling contexts. But until now, methods for finding locally parameterizable surface patches have mostly relied on algorithms that strike a balance between compactness, patch size, and developability background. This study focuses on segmenting a small subregion around a point of interest on a mesh for parameterization rather than global parameterization techniques that map the entire mesh to 2D while introducing as few cuts as possible.
Instead, in this study we learn distortion-aware local segmentations that are best for local parameterization using a data-driven methodology. Their suggested system predicts a patch surrounding a point and its accompanying UV map using a single differentiable parameterization layer. This enables self-supervised training, allowing us to avoid the scarcity of parameterized labeled datasets by encouraging your network to forecast patches that maximize area and minimize distortion via a properly designed sequence of prior data. Their approach, which they call the Distortion-Aware Wand (DA Wand), produces smooth segmentation probabilities from an input mesh and an initial triangular selection. By creating a weighted variant of the traditional LSCM parameterization technique, which they refer to as wLSCM, they include these probabilities in their parameterization layer.
From this adaptation a probability-guided parameterization results, on which the distortion energy can be calculated to allow self-supervised training. They show the direct relationship between probabilities and binary segmentation in the context of parameterization by proving the theory that the wLSCM UV map converges on the LSCM UV map when soft probabilities converge on a binary segmentation mask. Since UV distortion increases monotonically with patch size, reducing UV map distortion and increasing the segmentation area are competing goals. They accomplish these objectives in harmony by creating a unique threshold distortion loss that penalizes triangles with distortion above a user-specified threshold. Simply adding these targets results in poor optimization with unwanted local minima.
They create a completely new segmentation dataset that is nearly developable, along with an automated build technique that can be used out-of-the-box and pre-trained to set your segmentation network weights. The network is then trained end-to-end using its parameterization layer with distortion and compactness backgrounds on an unlabeled natural shapes data set. They use a MeshCNN backbone to learn directly from input data triangulation, allowing for sensitivity to sharp features and a large receptive field allowing for patch expansion. Furthermore, his approach maintains rigid transformation invariance by using intrinsic mesh properties as input. In addition, they promote compactness through the use of a loss of softness modeled after the technique of graphic cuts.
A user can interactively pick a triangle on the mesh using the DA Wand to obtain a sizable and meaningful region around the selection that can be UV parameterized with little distortion. Unlike current heuristic approaches, which stop at the boundaries of high curvature zones, they demonstrate that the neural network can prolong segmentation with the least amount of distortion gain. Their approach outperforms competing approaches by producing user-driven targeting at interactive speeds. In Figure 1 above, an attractive and interactive application of the DA Wand is shown in which various areas of the sorting hat mesh are successively selected and labeled. The framework code is freely available on GitHub.
review the Project. All credit for this research goes to the researchers of this project. Also, don’t forget to join our reddit page Y discord channelwhere we share the latest AI research news, exciting AI projects, and more.
Aneesh Tickoo is a consulting intern at MarktechPost. She is currently pursuing her bachelor’s degree in Information Science and Artificial Intelligence at the Indian Institute of Technology (IIT), Bhilai. She spends most of her time working on projects aimed at harnessing the power of machine learning. Her research interest is image processing and she is passionate about creating solutions around her. She loves connecting with people and collaborating on interesting projects.