Filling Simple Holes by Using Enhanced Advancing Front Mesh Method (EAFM): Application to a Real Object

Nowadays, the physical object can be constructed in digital representation by reverse engineering process. The process started by collecting the point cloud data point from the surface of 3D object. The non-touch approaches from 3D laser scanner are advantageous because of their high scanning speed, high accuracy, real time application, etc. Object complexity such as glossy surface and shining can lead to deficiencies in the quality of data measure using a 3D scanner which leads to missing data and the formation of holes within the constructed 3D model object. A hole is referred to as simple if it does not have feature vertices except other point which have same connection of data points. This paper utilised the advanced front method (AFM) to create triangle into the simple holes area. The paper then improved the hole filling process for simple holes found on a 3D object or real-world object. In this paper, the enhanced method proposed two new AFM methods to the original method. The method was applied to the 3D object and the problem of finding new point for each triangle creation is solved.


Introduction
Recently, more interest has been rendered to the polygonal representation of three dimensional objects in various applications, specifically for triangular meshes. Polygonal meshes offer good visualization, simplicity, and flexibility in computer graphics and computer vision applications. One can obtain polygonal models from reverse or direct engineering processes. Direct engineering techniques create virtual models using CAD and modellers. On the other hand, reverse engineering methods use data provided by 3D sensors to build models. Reverse engineering (RE) is a powerful method for constructing a digital representation for an existing physical object. Reverse engineering was known with three main processes which are collection of data points, processing of data points and construction of surface [1]. Essentially, the RE process starts by acquiring a point cloud from the outer surface of the physical object. Then, data was acquired from both the technology of laser scanning and object complexity. One can measure the 3D laser scanner by using either the object in touch approach or other non-touch approaches. The non-touch approaches are advantageous because of their high scanning speed, high accuracy, real time application, etc. [2].
For cases of reverse and direct engineering, various kinds of mesh defects can be created. Therefore, designed (or virtual) models will typically have gaps, degenerated elements, self-intersections, and singularities in their geometry. However, the most typical defects observed in digitised models are noise, topology noise, holes, and aliasing. From these defects, holes and gaps are considered as the most vital flaws. They have also been studied in detail in the past few years. Holes refer to well defects in meshes that digitisation tasks generate -for instance, when laser scanners are used. In such a case, certain portions of the object may experience occlusion from others. The scanner will then fail to reach certain regions. These missing data produce empty areas within the mesh.
Object complexity such as glossy surface and shining can lead to deficiencies in the quality of data measure using a 3D scanner. One could easily interrupt scanning results, which leads to missing data and the formation of holes within the constructed 3D model object.Holes can be represented in triangle mesh and point cloud data. The appearance of different kinds of holes help user to separate the hole into simple and complex holes. Example of complex holes are islands, gaps, holes with high curvature, incomplete topology and a hole with degenerate elements. Disadvantage of holes to 3D representation is the lack of representation 3D models and longer time for post-processing process [3]. These disadvantages have the direct effects on time consuming process and high cost of operation [4].
Recently, numerous studies have been conducted about filling holes. Reference [5] did a study that utilised an exemplar-based framework for hole filling in 3D point clouds. This method makes use of non-local self-similarity to offer plausible reconstruction even for complex surfaces and large holes. Hole-filling is iteratively performed. This process also utilises templates located near the hole boundary to determine the best matching regions found elsewhere in the cloud. These regions are where current points are moved to the hole. Reference [6] suggested a method that can be used to automatically fill and identify occlusion holes. It includes occlusions or self-occlusion (the object has an invisible back side) on planar surfaces in 3-D scenes. Examples of this are indoor or man-made scenes, which includes floors and walls. Spatial relations between the background and foreground objects can be used to determine if an identified hole is a result of occlusion. These kinds of holes are filled through common inpainting algorithms for 2-D images.
In triangle mesh models, holes can be filled by recovering feature lines that pass through them. The feature lines that were recovered can then be used to split complex holes into smaller and simpler ones. It takes two kinds of splits into consideration: corner split and curve split. For the curve split, Euler spirals are built to serve as feature lines. One can automatically identify the required end-points and their corresponding tangent vectors so that the curve can be initialised. In terms of the corner split, one can recover the corner point contained in the hole through the optimisation of a quadric equation. One can then automatically construct multiple feature lines [7]. Using Kinect, the depth image obtained will typically contain occlusions and cavities at the edge of the object. A pixel filtering method will fill the small cavities. Then, an enhanced fast joint bilateral filtering algorithm is designed so that missing data in the large cavities can be filled in [8].
The focus of the work by [9] is on filling holes with islands and small holes while keeping up curvature continuity throughout the holes' boundaries. It first constructs a composite surface patch bound by the hole's boundary. This is then tessellated so that it conforms with the original mesh in order for the repaired mesh model. It then utilised a smooth Powell-Sabin spline for estimating the function's volume inside the hole by utilising just the data for the function where it is known. In other words, it refers to data found outside the hole. Furthermore, there is a need to fulfil specific global geometric considerations, as well as other local volume constraints on the hole [10].
Depth information is represented by image of depth value. Then, image of colour is used to represent colour of surfaces. Then, those images are inpainted. The resulting images are used to rebuild the holes of the model. Both textural inpainting and structural inpainting were also utilised. Here, structural inpainting is utilised to reconstruct depth values and texture of surface is reconstructed using textural inpainting [11]. Reference [12] conducted work that proposed a fast MRF-based hole filling method for view synthesis. The formulation of this is done as an energy minimization problem. Its solution is performed using loopy belief propagation (LBP). For the traditional ones, some foreground textures might be introduced to the hole regions. The traditional ones also suffer from high computational complexity. Optimization of the energy function is done by using the depth information to avoid the foreground textures filling holes.
A triangular mesh is a set of vertices as well as triangles which join these vertices. If two triangles have a common edge, then they are adjacent triangles. An edge generally connects two triangles. If it links only one, the edge is considered a boundary edge and the resultant triangle is known as a boundary triangle; a boundary edge refers to the edge on the boundary edge. When a seed boundary edge is found, a set of linked boundary edges can be drawn from it. If certain boundary edges develop a closed loop, then they comprise a hole [13].
A hole is referred to as simple if it does not have feature vertices except other points which have the same connection of data points. The newest study performed by [3] utilised varying hole filling methods on flat and simple holes of bunny model. After conducting a comparison with various hole filling techniques, it concluded that a method by [14] can perfectly repair small holes with the excellent geometry reconstruction and topology information. Alternatively, [15] wrote another paper that suggested a method that will identify and extract hollow edge. It then makes use of a triangle growth way that has its basis on boundary edge angle to fill the empty void.Reference [16] conducted a paper that splits complex holes into numerous simple holes in terms of the 3D shape of the hole boundary. Then, every divided simple hole is filled with planar triangulation method until it firmly closes all holes.
On the other hand, mesh have two types either structured or unstructured mesh. A structured mesh is one which all inner points have the same number of elements around and topologically pattern which look alike. An unstructured mesh is one in which does not have uniform pattern and vertices may have arbitrarily varying local neighborhoods. [17]. Unstructured mesh generation by using advanced front method (AFM) involve generation of mesh points and their connectivity. Mesh element was build element by element, adding new elements to previously generated elements [18]. AFM or MAFM will be used to directly fill a simple hole. Advantage of AFM is the connection of new triangle is co-instantaneous with other points. The size and shape of element can be adjusted by changing the location of point [19]. Another advantage of AFM is its tendency to generate high quality triangles and nicely graded meshes that are typically close in shape to equilaterals [20].
For any advancing front method, the start of the algorithm is a set of boundary loop segments that can be defined as the initial front. Construction of the triangles is done from the front segments. The triangle was created by growing towards interior of holes ( [19], [20]). AFM inserts new points in holes area and continues to add triangle until the whole holes were closed. The concept of AFM was simple and without re-implementing the whole algorithm, advanced functionality and heuristics can be added [21].
Reference [22] wrote a paper that adapted the AFM method and extended the current AFM so that a patching mesh can be created to fill the holes. Liu wrote another paper that built a smooth patch through the modification of the advancing front mesh and the fitting of the quadric surface to add details. This method is advantageous because it can circulate the fine-scale details within the surrounding holes [23]. Another research used the AFM method to refine the boundary edges of holes. During the refining process, small angles between adjacent boundary edges are removed and concave boundary edges become almost convex. Later, the Delaunay triangulation was applied to construct new meshes for filling holes [24].Using the advantage and benefit of AFM as motivation, the AFM method was adapted in this paper. The paper then improved the hole-filling process for simple holes found on a 3D object or real-world object. The EAFM method uses mentos 3D object to represent the enhanced method which later been applied to the other four object.

Methods
In this work, we used the advancing front mesh (AFM) technique to generate an initial triangle over the hole. In this research, the AFM technique construct the triangle in the hole starting by the initial patch by following the six steps below [25].
Step 1: Identify the first three points in the boundary of hole Step 2: Calculate the angle i θ between the edges 1 − i e and 1 + i e between the point A, B and C.
Step 3: The new triangle was formed according to three different angle criteria as shown in Fig. 1  (1) Step 5: Update the control points Step 6: Repeat step 1 through step 4 until the whole region of holes has been created and fill with triangle The recommended hole-filling algorithm has been deployed with MATLAB2016. 3D object known as mentos have been used to gauge the robustness, precision and accuracy of the approach. The holes in mentos were triangulated using the original AFM method and two EAFM methods were introduced for the problem. Fig. 2 and Fig. 3 shows mentos object and mentos 1 view data points that were used in this research.  However, in some cases, there exists a condition which does not satisfy all normal cases of AFM. Figure 4 illustrate simple holes problem that do not satisfy the threshold case in the normal AFM algorithm. The problem occurs when the newly constructed point does not connect to any nearest active node, and during construction of the triangle, no node lies within the threshold value that is appropriate to replace the new point as in Fig. 4.

Results and Discussion
The raw point data are typically disordered, unorganized, and noisy. Pre-processing the raw input is necessary to clean the noise, homogenize raw points, remove outliers, correct point normals to construct a foundation of 3D surface reconstruction, and organize disordered points [26]. The dense unorganized data points collected from the laser scanning were then utilised for the computation of a triangle mesh, a subdivision surface, or a set of B-spline/NURBS surfaces. The point cloud's triangulation is often a preliminary step to the generation of more complicated surfaces. Thus, it is vital to be able to generate a triangle mesh that fits the object outer surface best. Once pre-processing is done, one can repair the scattered point data by directly producing interpolation points in the hole region or by filling polygonal holes after the unorganized points are triangulated. To achieve the pre-processing and hole-filling processed into the 3D object, the following methods of the research need to be implemented: Step 1: To identify real boundary of object by using edge detection and approximation.
Edge detection is one of the most important stages in digital image processing and medical image processing. Canny edge detection is one of the algorithms that is widely used to locate sharp intensity changes and to find boundaries in an image [27]. Then by using distance formula, the smallest distance from 1 view data point and Canny boundary detection was calculated. The new boundary data point was then arranged in a sequence order according to nearest point. Result was displayed as in Fig.  5 below. Fig. 5(a) shows the result of approximate data from edge detection and the data points was highlighted in black colour while the red data points was the point that consider outlier and need to be deleted. Similar algorithm was applied to identify the boundary of holes in mentos as in Fig. 6(a). The red data point was those data that need to be deleted since it was not in the range of approximation. Different from the boundary of mentos, the outlier data points was inside the holes. After deleted the points, distance between each data points in hole boundary was calculated and then the data point was arranged according to the minimum distance and the result was represented in Fig. 6(b).  Step 2: To estimate distance between points and construct algorithm to delete and add points.
The original point on the boundary of holes has a problem that the data points are too near and there are not enough points to represent the holes. Data point which is too near will lead to the creation of smaller and skinny triangle which is bad triangle during triangle creation process. On the other hand, data point which is not enough also will contribute to the failure during introducing triangle in the holes. Therefore, the mean and standard deviation was used as a threshold to delete and add point to represent new boundary of holes as in Fig. 7. Step 3: To construct triangular mesh of scattered data by using Delaunay Triangulation (Fortune's algorithm).
Triangular mesh on the scattered data point of mentos was constructed by using Delaunay Triangulation. Fortune's algorithm which is based on sweep-line algorithm was chosen as a method to create Delaunay Triangulation. The method was chosen since it is simple and easy to implement [28]. Fig. 8 shows the result of Delaunay Triangulation with data points. Step 4: To highlight boundary of object and boundary of holes of 3D scattered data.

Boundary of object
Constrained edges for boundary of object need to be defined in order for Delaunay Triangulation to be constructed on the constrained area only. This was done by calculating the centre point for all triangle in Delaunay Triangulation by using mean of (x, y). According to Fig.  9(a), the interior boundary object was defined by blue data point while the outer boundary object was defined by green data point. Then by deleting the outer centre point, delaunay triangulation cover only on the bounded area as in Fig. 9(b).

Boundary of holes mentos
Similar algorithm was applied to constrain only on the boundary of holes for mentos. The center point for all triangles in Delaunay triangulation was calculated then the interior point was defined by green data points while other point was in blue data point. Then the interior data points were deleted in order to keep the outerior area only. The result is shown in Fig. 10(a) and Fig. 10

Final constrained DT
In order to represent both boundary of object and boundary of holes, centre point that needs to be deleted was defined to be outer boundary and interior of holes boundary. Then the Delaunay triangulation was constructed with those constraints as in Fig. 11 below. During triangle creation, by using original AFM method there was a conflict where the two new points were not connected to any nearest point as in Fig. 12. This occurs because there is no nearest point in a threshold radius of new point. Therefore new algorithm need to be proposed in order for those points to be connected.

First new point
According to Fig. 13(a), the two circles represent the threshold circle used to identify the nearest point to be connected. However, both new points cannot be connected to any point since there is no any point in the circle to be chosen. Therefore, Enhanced Advancing Front Mesh Method (EAFM) was proposed by considering the three shortest distance data points defined as point A, B and C as checkpoints for intersection. The equation of line was calculated for each connection of data point. The original circle was tested for any location of line in order to check for the intersection point.
The intersection point can be either tangent point, secant point or no intersection. In figure 13(a), line BC was not intersect with the circle, thus there is no intersection so cannot be used to find point intersection. Line AC and AB both intersect with line AB and AC with secant point. The distance between both secant points to all active points was calculated. The minimum distance was considered as the replacement for the new point. The first new point was connected as in Fig. 13(b).

Second new point
EAFM method was applied for the second new point where the three nearest point was labeled as A, B and C ( Fig. 14(a). Line CB and AC does not yield any intersection point since the circle does not intersect to the line. On the other hand, line AB intersects to the circle and has a secant point. Both points were calculated and minimum distance between the secant points to all active points was calculated and considered as a second new point. The result for EAFM for second new point was displayed in Fig. 14(b). According to Figure 15, there are two triangle (white) created in the final result. This condition was proposed as the second criterion in EAFM. The two triangles were automatically constructed since during formation of triangle, the three point and edges were connected to themselves. Lastly, the final result of EAFM on mentos was illustrated in Fig. 16. The holes were filled with EAFM Civil Engineering and Architecture 7(5): 251-261, 2019 259 for simple holes. This criterion was the similar criteria used by other researcher for cases of AFM [29].

Final Result
The final result of filling holes in mentos by using AFM and EAFM was shown in Fig. 16. Four other objects were used to represent both criteria in EAFM and result of the study showed that the enhanced method is able to fill holes in simple holes problems. The other object was represented in Fig. 17, Fig. 18, Fig. 19 and Fig. 20 below. Table 1 below shows the result of before and after filling holes. There is a change in a number of vertices and number of triangles. Among all five object, mentos shows the highest number of increase in the number of vertices and number of triangles value.

Conclusions
As a conclusion, this paper proposed two new AFM methods to solve problem during triangle creation. The result of the study shows that more points were introduced in the hole to represent the features of the object in better 3D representation. In this research, the enhanced method was also applied to other four different objects for comparison. The result of the study shows that a more number of vertices were introduced in the holes to represent the features of the object. The number of triangles was also increased with the increased in the number of vertices. In our next research, other view in 3D object will also be used to complete the filling-holes process to build the complete 3D object.