3D reconstruction of underwater objects using NeRF

2.1 NeRF (Neural Radiance Fields)

In this study, the shape reconstruction technique employed is Neural Radiance Fields (NeRF), a neural network-based method that learns color information and volume density representing the interaction between objects and light in 3D space. A key feature of NeRF is its ability to take a small set of multi-view images as input and generate novel views of the object from unseen angles, effectively reconstructing a 3D shape. As illustrated in Figure 1, NeRF estimates the color and volume density at any point in 3D space based on the input 3D position (x, y, z) and the viewing direction (θ, ϕ) of the camera. Volume density is defined as the inverse of transparency, where higher values indicate more opacity and lower values indicate greater transparency. To generate the training data, NeRF uses rays that pass-through pixels in the input images, which are cast into the 3D scene.

Figure 1:

An overview of neural radiance field scene representation: (a) Sampling 5D coordinates (spatial location and viewing direction) from input images; (b) Feeding the sampled coordinates into an MLP to estimate color and volume density.

As shown in Equation (1), a ray is defined as a set of points extending from the camera’s focal origin (o) in a specific direction (d), scaled by a distance (t). One ray is generated per pixel, and each ray is uniformly divided into n segments. Sample points along each ray are selected as input, and the 3D position and viewing direction at each point are used for training.

Once the input data is generated through rays, the shape reconstruction is learned using an MLP (Multi-Layer Perceptron). As illustrated in Figure 2, the MLP consists of a total of nine fully connected (FC) layers with ReLU (Rectified Linear Unit) activation functions. ReLU is a nonlinear function that outputs zero for negative inputs and returns the input value itself for positive inputs. It introduces nonlinearity into the network, enabling the learning of more complex patterns. Due to its simplicity, ReLU offers high computational efficiency and helps mitigate the vanishing gradient problem, thereby accelerating the training process.

Figure 2:

Network architecture of NeRF: Volume density is predicted using spatial location information, followed by color estimation based on viewing direction.

During training, the spatial position information is first passed through eight FC layers to predict the volume density. Then, using the viewing direction, the color value is estimated. This separation accounts for non-Lambertian effects, where an object’s appearance varies depending on the viewing angle. As shown in Equation (2), each point along a ray contributes to the final pixel color based on its volume density: points with higher density are assigned higher weights, while those with lower density receive lower weights. These weights are used to modulate the color values, which are then aggregated to determine the final pixel value.

Each ray is uniformly divided into n segments, and specific points along the ray are selected as input data. These sample points may lie within the object but can also fall into empty space where no object exists. Training the model with more focus on regions where objects are present, rather than on empty space, can lead to more accurate results. To address this, NeRF employs a two-stage training strategy as shown in Figure 3: a coarse network first processes the full ray uniformly, and then a fine network refines the learning by focusing on regions with high volume density inferred from the coarse stage.

Figure 3:

Two-stage training: Initial sampling along the full ray, followed by fine sampling in high-density regions; losses from both stages are computed and combined.

The loss function is defined in Equation (3). For both the coarse and fine networks, the rendered colors are compared with the ground truth pixel colors, and the respective losses are computed. The final loss is then obtained by summing the losses from both stages.

However, training the network directly with the 5D input (x, y, z, θ, ϕ) — comprising spatial location and viewing direction—can be challenging due to the limited amount of input data. To address this issue, positional encoding is employed. As shown in Equation (4), this technique transforms the spatial location into a 60-dimensional vector and the viewing direction into a 24-dimensional vector for training.

sampled point along the ray, the model learns to estimate its color and volume density. In other words, a single ray determines the color of a single pixel. By aggregating the results of all rays, a complete 3D shape is reconstructed.

2.2 Instant NGP(Neural Graphics Primitives)

Unlike the original NeRF, which uses a positional encoding scheme, Instant-NGP adopts a Multi-Resolution Hash Encoding approach to significantly accelerate training. While NeRF is suitable for generating high-quality 3D models—such as converting object image data into detailed 3D models for inspection and analysis—Instant-NGP is better suited for real-time 3D model generation. For instance, it is ideal for scenarios where the 3D structure of an object must be visualized instantly to support rapid decision-making or planning. Instant-NGP achieves over 1,000 times faster training speed compared to the original NeRF. Furthermore, it supports transfer learning by utilizing pre-trained data, allowing effective training even with a smaller number of input images. To leverage these advantages, this study adopts Instant-NGP—one of the NeRF-based approaches—for 3D shape reconstruction of underwater data.

Instant-NGP reduces training time by using Multi-Resolution Hash Encoding, which divides the 3D scene into grids of varying resolution depending on its spatial complexity. As illustrated in Figure 4, each vertex of the grid cells that contain a given point has a learnable feature vector of dimension F. These feature vectors are stored in a hash table, and each vertex coordinate is hashed to retrieve its corresponding feature vector. The vectors from the surrounding grid vertices are then linearly interpolated to generate an encoded input, which is passed into the neural network.

Figure 4:

Input generation process: Feature vectors are retrieved by hashing grid vertex coordinates around a location, linearly interpolated, and combined with view direction to form the network input.

Additionally, the viewing direction of the camera is provided as a supplementary input. Thus, the final input to the network consists of the spatial position and viewing direction, while the output includes the volume density and radiance (color) of each pixel.

3.1 Data Acquisition

In this study, data were collected using a monocular camera after placing the same anchor, shown in Figure 5, in both underwater and terrestrial environments. The anchor measures 51 cm in width and 68 cm in height. Underwater data were acquired using a remotely operated underwater vehicle (ROV) in coastal waters near Korea Maritime & Ocean University. Figure 6 (Top) shows the area where the underwater data were collected, and Figure 6 (Bottom) presents a photograph of the actual sea site. Terrestrial data were collected indoors under controlled conditions.

Figure 5:

Anchor used as the target object for 3D reconstruction in both underwater and terrestrial environments.

Figure 6:

Underwater anchor data acquisition at Korea Maritime and Ocean University: (Top) Data acquisition area; (Bottom) Photograph of the actual marine environment.

The remotely operated underwater vehicle (ROV) used for data acquisition was BlueROV2 from Blue Robotics, as shown in Figure 7, and its specifications are listed in Table 1. The ROV was equipped with six T200 thrusters, also from Blue Robotics. Additional onboard sensors included a monocular camera, a barometer, a 3-degree-of-freedom gyroscope, and pressure, depth, and temperature sensors. However, this study utilized only the monocular camera data for the 3D reconstruction process.

Figure 7:

Remotely operated underwater robot used for data acquisition

Table 1:

Specifications of the Blue Robotics BlueROV2

3.2 Training and Performance Evaluation

Underwater training data of the anchor were acquired using a monocular camera mounted on the remotely operated underwater vehicle. Video footage was recorded at a depth of approximately 1.2 meters, from which 190 frames were extracted and used as training images. Using the underwater dataset, Instant-NGP was trained. The top image in Figure 8 shows the original input at a specific time step, while the bottom image shows the reconstruction result at the same viewpoint.

Figure 8:

Comparison between original underwater images and NeRF reconstruction results for four scenes using Instant-NGP

To quantitatively evaluate the underwater reconstruction results, the same anchor was placed in an on-land environment. The terrestrial dataset was then used to train Instant-NGP, following the same procedure as the underwater case. The top image in Figure 9 shows the original input at a specific time step, while the bottom image shows the corresponding reconstruction result at the same viewpoint.

Figure 9:

Comparison between original terrestrial images and NeRF reconstruction results for four scenes using Instant-NGP

A purely visual comparison between the underwater and terrestrial NeRF reconstruction results has inherent limitations. Therefore, this study conducted a quantitative evaluation using objective performance metrics. Inspired by the original NeRF paper, three commonly used evaluation metrics were adopted to assess the similarity between images: PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index Measure), and LPIPS (Learned Perceptual Image Patch Similarity). These metrics were used to compare the original and reconstructed images at the same viewpoints, as illustrated in Figure 8 and Figure 9.

PSNR is a traditional metric used to assess the quality degradation in reconstructed or compressed images, expressed in decibels (dB). It typically ranges from 0 to above 30, where higher values indicate better reconstruction fidelity. Values above 30 dB suggest differences that are hardly perceptible to the human eye. SSIM improves upon PSNR by considering luminance, contrast, and structural similarity between images, ranging from 0 to 1. A higher SSIM value indicates greater structural similarity. LPIPS, on the other hand, is based on human visual perception. It computes feature similarity between image patches using a pre-trained VGG network. LPIPS ranges from 1 to 0, where lower values represent better perceptual similarity.

Twelve scenes were evaluated in both underwater and terrestrial environments, with the results summarized in Table 2. On average, the terrestrial data achieved better reconstruction performance across all metrics: PSNR (28.51), SSIM (0.97), and LPIPS (0.07), compared to the underwater data with PSNR (23.23), SSIM (0.71), and LPIPS (0.15). These results suggest that NeRF achieves higher reconstruction quality in terrestrial environments. This discrepancy is likely due to the physical characteristics of underwater imaging. In underwater conditions, scattering and absorption of light often degrade image sharpness and color information. Additionally, light scattering varies over time due to water movement, which violates NeRF’s assumption that the scene remains static during capture, potentially introducing temporal inconsistency errors.

Table 2:

Quantitative comparison of reconstruction performance on terrestrial and underwater images

The degradation in reconstruction accuracy can also be observed through artifacts present in the underwater results. As shown in Figure 10, the fog-like region above the anchor is an artifact generated during NeRF reconstruction. Artifacts refer to visually distorted shapes or colors that do not exist in the original input images but are erroneously produced by the model. These tend to appear more frequently when image resolution is low or when viewpoint sampling is sparse. In the underwater case, the limited visibility makes object boundaries less clear and distorts light paths, disrupting the ray continuity assumed during NeRF training. This, in turn, reduces the model's ability to estimate accurate volume densities and colors. In contrast, terrestrial conditions offer sharper object boundaries and more consistent lighting, which likely contributed to more stable and reliable reconstruction performance.

Figure 10:

Artifacts observed in underwater images reconstructed using NeRF

Author Contributions

Conceptualization, Y. E. Lim and J. H. Woo; Methodology, Y. E. Lim and J. H. Woo; Software, Y. E. Lim; Formal Analysis, Y. E. Lim and J. H. Woo; Investigation, Y. E. Lim; Resources, N. W. Kim and J. H. Woo; Data Curation Y. E. Lim; Writing-Original Draft Preparation, Y. E. Lim; Writing-Review & Editing, Y. E. Lim and J. H. Woo; Visualization, Y. E. Lim; Supervision, J. H. Woo; Project Administration, J. H. Woo; Funding Acquisition, N. W. Kim and J. H. Woo.