FeEcho4D

Motivation

Fetal echocardiography is the primary imaging modality for prenatal assessment of cardiac anatomy and function, yet most clinical analysis and existing deep learning (DL) approaches remain fundamentally two-dimensional. Conventional 2D views suffer from several important limitations:

Partial cardiac observations that limit accurate characterization of complex 3D cardiac geometry.
High sensitivity to probe misalignment and operator variability, often causing apical foreshortening where the true ventricular apex is not fully captured.
Inaccurate volumetric and functional assessment, including underestimation of left ventricular (LV) volumes and cardiac function.
Geometry-dependent functional estimation, where 2D Ejection Fraction (EF) estimation relies on Simpson’s rule and circular cross-sectional assumptions, potentially introducing errors in cases with asymmetric wall thickness or congenital cardiac malformations.

In contrast, 3D echocardiography captures the full cardiac volume without requiring precise imaging-plane alignment, making it less susceptible to foreshortening and more suitable for accurate anatomical and functional assessment. Although 3D/4D fetal echocardiography is clinically available through Spatio-temporal Image Correlation (STIC) acquisitions, its adoption remains limited by challenges in visualization, functional quantification, and automated analysis. While DL-based 3D/4D cardiac reconstruction offers a promising solution, progress has been constrained by the lack of publicly available 4D datasets and the complexity of volumetric DL algorithms.

Preparing 4D ground-truth annotations for fetal echocardiography is highly challenging, particularly for temporally coherent 4D analysis. Several key factors contribute to this difficulty:

High annotation complexity, requiring accurate and spatially consistent labeling across slices and time.
Poor fetal image quality caused by anisotropic resolution, acoustic shadowing, and signal dropout.
Ambiguous anatomical appearance in conventional views, where myocardial (MYO) structures transition inconsistently from U-shaped contours to irregular circular blobs near lateral regions in the short-axis (SAX) and long-axis (LAX) annotations.
Difficulty maintaining temporal coherence across 4D cardiac sequences during manual annotation.

Fig. 1: Example radial slices at different angular positions (θ ∈ [0°, 180°]) demonstrating the consistent U-shaped MYO appearance produced by radial slicing, enabling easier and more anatomically consistent annotation across views.

To overcome the challenges of 3D/4D annotation, we introduce radial slicing (see Fig. 1), which yields consistent and interpretable U-shaped MYO contours, unlike traditional SAX and LAX views. Our radial slicing extracts 2D slices by rotating the slicing plane around the LV longitudinal axis at fixed angular intervals (θ ∈ [0, π]), providing structured coverage of the MYO from multiple orientations. This cylindrical-coordinate acquisition enables consistent LV visualization around the heart, producing stable U-shaped MYO appearances across slices, as shown in Fig. 1.

Building upon this radial representation, our framework introduces two key innovations, as shown in Fig. 2 (details in the paper):

FeEcho4D: The first public benchmark dataset for 4D fetal echocardiography with temporally coherent 4D LV meshes.
Reconstruction: A 3D LV reconstruction framework over time based on radial slicing. The framework efficiently transforms the 3D problem into structured 2D tasks while preserving anatomical plausibility. Importantly, radial slicing introduces a built-in geometric inductive bias that can be explicitly modeled as a spatial anatomical prior within the network, unlike purely data-driven DL models that lack anatomical priors.

Fig. 2: Proposed full-stack framework from data preparation to mesh reconstruction. (A) Radial FeEcho4D data preparation. (B) Segmentation network incorporating geometric inductive bias. (C) Sparse radial slice-to-3D reconstruction. A purely data-driven UNet produces anatomically implausible meshes (ground truth: red wireframe and estimated: blue wireframe), leading to poor EF estimation.

Annotation

Our FeEcho4D dataset development from raw 4D volumes has six key steps:

Data extraction from raw STIC acquisitions using 4DView software (GE Healthcare, Chicago, IL, USA).
Radial slice extraction at each time frame.
Manual delineation of myocardium at clinically important time points—end-diastole (ED) and end-systole (ES).
Motion tracking to generate masks for intermediate frames.
Manual refinement of the propagated masks to correct tracking errors.
Slice-to-mask reconstruction for all time frames, producing complete 4D segmentations of each fetal echocardiography volume.

All steps are described in detail below, including technical and implementation specifics, to ensure reproducibility.

1. Data Extraction Step

Raw STIC acquisitions were retrieved directly from the fetal echocardiography system and processed with 4DView software (GE Healthcare, Chicago, IL, USA) to export temporally resolved 3D volumes. For each temporal extraction, slice number and spacing were kept constant to preserve the spatial–temporal integrity of the original 4D ultrasound data for subsequent analysis. The following Fig. 3 shows examples of the 3D extracted volume at different times for the same patient.

Fig. 3: Raw STIC fetal echocardiography volume exported via 4DView software and corresponding radial slices at selected time frames.

2. Radial Slice Extraction Step

All volumes were radially resampled using Algorithm 1, as illustrated in Fig. 4. This procedure generates a set of uniformly sampled radial slices (H×W) at fixed angular intervals, which together form structured radial 3D volumes. Compared to conventional SAX and LAX views, where the left ventricular shape often transitions from an apparent U-shape to blob-like contours, radial slices consistently preserve a U-shaped anatomy across depths. This structural consistency greatly facilitates accurate and efficient manual annotation, as shown in the segmentation of all S slices (see Fig. 1).

Algorithm 1 — Radial Slice Extraction

Inputs: 3D volume \( \mathbf{V}\in \mathbb{R}^{H \times W \times D} \), spacing \( \mathbf{S}=[s_x,s_y,s_z] \), angle set \( \Theta \).
Outputs: Radial slices \( \{ I_\theta \mid \theta \in \Theta \},\; I_\theta \in \mathbb{R}^{H \times W} \).

Select middle slice index \( \mathbf{M} \) along z and rotation center \( \mathbf{C}=[C_x,C_y] \) on that slice (LV center).
Generate coordinate mesh \( \mathbf{X}=\{(x_i,y_j,z_k)\} \) with voxel spacing: \[ \begin{aligned} x_i &= i\cdot s_x,\quad i=1,\dots,W \\ y_j &= j\cdot s_y,\quad j=1,\dots,H \\ z_k &= k\cdot s_z,\quad k=1,\dots,D \end{aligned} \]
Translate to center: \[ \mathbf{X}^C_p = \mathbf{X}_p - [\,C_x s_x,\; C_y s_y,\; M s_z\,],\quad p\in\{i,j,k\}. \]
For each \( \theta \in \Theta \):
1. Define y-axis rotation \[ R_y(\theta^{\mathrm r})= \begin{bmatrix} \cos\theta^{\mathrm r} & 0 & \sin\theta^{\mathrm r}\\ 0&1&0\\ -\sin\theta^{\mathrm r}&0&\cos\theta^{\mathrm r} \end{bmatrix},\quad \theta^{\mathrm r}=\theta\cdot\frac{\pi}{180}. \]
2. Rotate and inverse-translate: \[ \mathbf{X}^{\mathrm{rot}}=R_y(\theta^{\mathrm r})\,\mathbf{X}^C,\qquad \mathbf{X}^{\mathrm{target}}_p=\mathbf{X}^{\mathrm{rot}}_p+[\,C_x s_x,\; C_y s_y,\; M s_z\,]. \]
3. Interpolate: \[ V^{\mathrm{rot}}(i,j,k)=\mathrm{interp3}\!\left(\mathbf{V},\;\mathbf{X}^{\mathrm{target}}\right). \]
4. Slice at \( k=M \) to obtain \( I_\theta = V^{\mathrm{rot}}(:,:,M) \).

Fig. 4: Radial slice extraction at a given rotation angle (θ) with corresponding ultrasound image and myocardium mask.

3. Manual Delineation Step

Using a custom-built annotation tool (details in our SegmentationApp), experts manually delineated the LV's MYO and cavity by marking points along their boundaries (see Fig. 5). The tool supports 3D volume loading, interactive contour drawing, and real-time quality control through mask overlay. It also provides temporal navigation to capture cardiac motion, particularly valuable in cases affected by acoustic dropout, and annotation toggling for iterative refinement. To ensure accuracy and consistency, all annotations underwent multiple rounds of expert review followed by cardiologist validation.

Fig. 5: Manual delineation of myocardium at ED and ES by SegmentationApp.

4-5. Motion Tracking & Manual Refinement Step

FeEcho4D generates dense cardiac masks by propagating sparse ED and ES annotations using B-spline Fourier (BSF) motion model (Algorithm 2), which blends local and global deformations and regularizes trajectories to capture periodic motion (as shown in Fig. 6). This ensures temporally coherent, anatomically consistent masks, further refined with skeleton-based width correction (Algorithm 3) to maintain uniform myocardial thickness, and all results are manually reviewed for accuracy.

Algorithm 2 — BSF-Based Temporal Mask Propagation

Inputs: Radial slices \( \{x_\theta^t\}_{t=1}^T \), ED/ES masks \( M_{\mathrm{ED}}, M_{\mathrm{ES}} \), spacing \( s \), Fourier order \( N \), B-spline level \( L \).
Outputs: Dense masks \( M_t \in \mathbb{R}^{S\times H\times W},\ t=1,\dots,T \).

(Registration) For \( t=0 \) to \( T-2 \):
- \(\mathrm{Register}(x_\theta^t, x_\theta^{t+1}) \Rightarrow \mathbf{d}_{t\to t+1}\)
- \(\mathrm{Register}(x_\theta^0, x_\theta^{t+1}) \Rightarrow \mathbf{d}_{0\to t+1}\)
(Grid Initialization): \[ \mathbf{X}_0 = \mathrm{GridFromMask}(M_{\mathrm{ED}}) \]
(Grid Propagation) For \( t=0 \) to \( T-2 \): \[ \mathbf{X}_{t+1} = w(t)\cdot(\mathbf{X}_t + \mathbf{d}_{t\to t+1}) + (1-w(t))\cdot(\mathbf{X}_0 + \mathbf{d}_{0\to t+1}) \]
(BSF Fitting): \[ \mathbf{X}^{\mathrm{BSF}}_t = \sum_{f=0}^N \mathbf{F}_f \cos\!\left(\frac{2\pi f t}{P}\right) + \mathbf{F}_{f+N+1} \sin\!\left(\frac{2\pi f t}{P}\right) \] \[ \mathbf{F}_f(x,y) = \sum_{i,j} B_i(x) B_j(y)\cdot \mathbf{C}_{i,j,f} \]
(Trajectory Correction): Repeat until cost < threshold:
- For each control point \( u \): \[ \mathbf{C}_u = \arg\min \sum_{t,\Delta t} \left\| \mathbf{X}^{\mathrm{BSF}}_{t+\Delta t} - \mathbf{X}^{\mathrm{BSF}}_t - \mathbf{d}_{t\to t+\Delta t} \right\|^2 \]
(Mask Synthesis): For \( t=0 \) to \( T-1 \), for each \( p\in M_{\mathrm{ED}} \):
- \( p_t = \mathbf{X}^{\mathrm{BSF}}_t(p) \), set \( M_t(p_t) \leftarrow 1 \)

Return: \( \{M_t\}_{t=1}^T \)

Algorithm 3 — Skeleton-Based 2D Mask Refinement

Inputs: Raw propagated mask \( M_t \in \mathbb{R}^{S\times H\times W} \); tail–ratio \( \tau\in(0,1) \); opening radius \( r \ge 0 \).
Outputs: Refined mask \( \widetilde{M}_t \in \mathbb{R}^{S\times H\times W} \) with consistent width.

For \( s = 1 \) to \( S \):
1. (Component Selection): \( M'_s \gets \mathrm{LargestComponent}(M_t[s]) \)
2. (Skeleton Extraction): \( S_s \gets \mathrm{Skeletonize}(M'_s) \)
3. (Tail Extrapolation): Let \( S_{\mathrm{tail}} \) be the \( \lceil \tau |S_s| \rceil \) farthest points from \( \mathrm{centroid}(S_s) \); Fit line \( \ell_s \) via least squares and set \( S_s \gets S_s \cup (\ell_s \setminus S_s) \)
4. (Mean Half-Width Estimation): \[ d(x) = \min_{y \in \overline{M'_s}} \| x - y \|_2,\quad x \in \Omega \] \[ h_s = \left\lfloor \frac{1}{|S_s|} \sum_{x \in S_s} d(x) + \frac{1}{2} \right\rfloor \]
5. (Uniform Mask Synthesis): \[ \widetilde{M}_t[s] = \{ x \in \Omega \mid \mathrm{dist}(x, S_s) \le h_s \} \] If \( r > 0 \): \[ \widetilde{M}_t[s] \gets \mathrm{Opening}(\widetilde{M}_t[s],\ \mathrm{disk}(r)) \]

Return: \( \widetilde{M}_t \)

3D Reconstruction

Prior to mesh reconstruction (details in the paper), radial segmentations (manual or SCOPE-Net predictions) are embedded into a common 3D coordinate system. Each slice is rotated around the LV center according to its acquisition angle, producing a sparse cylindrical representation for geometrically consistent mesh reconstruction via Graph Harmonic Deformation (GHD) (details in our teamwork: GHD).

3D Reconstruction with GHD

The 3D over time left ventricular meshes are reconstructed into smooth, temporally consistent meshes using GHD. The meshes are further processed with Differentiable Voxelization & Slicing (DVS) (as shown in the following Fig. 7) (details in Algorithm 4), enabling differentiable conversion to voxel grids and slices for precise dynamic metrics and seamless integration with learning-based models. The resulting 4D meshes of ten different example patients are shown in the following Fig. 8.

Algorithm 4 — GHD-Based 3D Mesh Reconstruction

Inputs: Binary mask sequence \( \{M_t\}_{t=1}^T, \; M_t \in \mathbb{R}^{S\times H\times W} \); canonical mesh \( \mathcal{M}_0 \in \mathbb{R}^{V\times 3} \); graph basis \( U \in \mathbb{R}^{V\times B} \); radial slices \( \{x_\theta^t\}_{t=1}^T \).
Outputs: Deformed mesh sequence \( \{\mathcal{M}_t\}_{t=1}^T,\; \mathcal{M}_t \in \mathbb{R}^{V\times 3} \).

(Point Cloud Sampling): For \( t = 1 \) to \( T \):
- Sample myocardium and background point clouds from \( M_t \): \( P_{\mathrm{myo}}^t, \; P_{\mathrm{bg}}^t \subset \mathbb{R}^{N\times 3} \)
(Mesh Deformation): For \( t = 1 \) to \( T \):
- Deform canonical mesh using spectral basis: \( \mathcal{M}_t = \mathcal{M}_0 + U \cdot \phi_t \)
- Optimize \( \phi_t \in \mathbb{R}^{B\times 3} \) to minimize total loss
(DVS Supervision): For \( t = 1 \) to \( T \):
- Project mesh via DVS: \( \hat{x}_\theta^t = \mathrm{DVS}(\mathcal{M}_t) \)
- Compute Dice loss: \( \mathcal{L}_{\mathrm{DVS}}^t = 1 - \mathrm{Dice}(x_\theta^t, \hat{x}_\theta^t) \)
(Objective Optimization): \[ \min_{\phi_t} \; \mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{DVS}} + \lambda_1 \mathcal{L}_{\mathrm{lap}} + \lambda_2 \mathcal{L}_{\mathrm{norm}} + \lambda_3 \mathcal{L}_{\mathrm{thick}} \]

Return: \( \{\mathcal{M}_t\}_{t=1}^T \)

Fig. 7: Slice-to-mesh reconstruction using DVS with GHD mesh fitting.

Fig. 8: Reconstructed 4D left ventricular meshes from the FeEcho4D dataset.

Clinical Metrics Calculation

Clinical biomarkers, such as Stroke Volume (SV), Ejection Fraction (EF), Global Longitudinal Strain (GLS), and Global Circumferential Strain (GCS), are derived from the reconstructed left ventricular meshes, as explained in detail in Algorithm 5 and the following Fig. 9. SV and EF are computed from endocardial volumes, while GLS and GCS are obtained by averaging changes in apex-to-base lengths and mid-ventricular circumferences across representative planes or cross-sections, excluding extreme outliers for robustness.

Algorithm 5 — Clinical Metrics Computation

(Volume-Based Metrics):
- Measure endocardial volumes \( V_{\mathrm{ENDO,ED}} \) and \( V_{\mathrm{ENDO,ES}} \) at end-diastole (ED) and end-systole (ES).
- Compute: \[ SV = EDV - ESV,\quad EF = \frac{EDV - ESV}{EDV} \times 100\%. \]
(Global Longitudinal Strain — GLS):
- Slice the LV epicardial mesh into \( N \) radial planes aligned with the long axis, indexed by \( \theta \). (In our study: \( N = 30 \))
- For each plane, measure apex-to-base lengths \( L_\theta^{\mathrm{ED}}, L_\theta^{\mathrm{ES}} \).
- Exclude the 5 longest and 5 shortest planes to retain \( N' \) representative planes. (In our study: \( N' = 20 \))
- Compute: \[ GLS = \frac{\frac{1}{N'}\sum_{\theta\notin OL} L_\theta^{\mathrm{ES}} - \frac{1}{N'}\sum_{\theta\notin OL} L_\theta^{\mathrm{ED}}}{\frac{1}{N'}\sum_{\theta\notin OL} L_\theta^{\mathrm{ED}}} \times 100\%. \]
(Global Circumferential Strain — GCS):
- Extract \( N \) short-axis cross-sections within the mid-ventricular region, indexed by \( i \). (In our study: \( N = 6 \))
- Measure circumferences \( C_i^{\mathrm{ED}}, C_i^{\mathrm{ES}} \) using ellipse fitting.
- Exclude the 1 largest and 1 smallest circumferences to retain \( N' \) representative cross-sections. (In our study: \( N' = 4 \))
- Compute: \[ GCS = \frac{\frac{1}{N'}\sum_{i\notin OL} C_i^{\mathrm{ES}} - \frac{1}{N'}\sum_{i\notin OL} C_i^{\mathrm{ED}}}{\frac{1}{N'}\sum_{i\notin OL} C_i^{\mathrm{ED}}} \times 100\%. \]

Outlier removal (OL) ensures robust estimation by excluding extreme geometric deviations caused by noise or irregular contours.

Dataset Access & Licensing

The FeEcho4D dataset comprises fetal 4D echocardiography sequences from 52 annotated subjects, including both healthy and congenital heart disease (CHD) cases such as Tetralogy of Fallot (ToF) and dextro-transposition of the great arteries (dTGA). The dataset provides spatiotemporal annotations of the left ventricle (LV) and myocardium (MYO), together with corresponding temporally coherent 4D LV meshes for cardiac reconstruction and functional analysis.

Each patient folder (Patient0xx) follows the filename convention:

Patient0xx_slice0yytime0zz.png

xx = patient ID
yy = slice ID
zz = time frame ID

Annotated Patients


Patient0xx/
├── image/
│   ├── Patient0xx_slice0yytime0zz.png
│   ├── ...
│   └── Radially extracted 2D fetal echocardiography slices
│
├── mask/
│   ├── Patient0xx_slice0yytime0zz.png
│   ├── ...
│   └── Manual LV/MYO segmentations
│       ├── LV cavity   = 127
│       └── MYO         = 255
│
├── mesh/
│   ├── mesh_time0zz.obj
│   ├── ...
│   └── Reconstructed 3D LV meshes over time
│
├── scribble/
│   ├── Patient0xx_slice0yytime0zz_scribbleTypeN.png
│   ├── ...
│   └── Weak scribble prompts (N = 1 ... 10)
│
├── Patient0xx_bbox.csv
│   └── Bounding-box prompts:
│       ├── filename
│       ├── xmin
│       ├── ymin
│       ├── xmax
│       └── ymax
│
├── Patient0xx.mp4
│   └── 3D LV mesh visualization over time
│
└── Patient0xx_info.cfg
    ├── diagnosis = healthy / ToF / dTGA
    ├── x_mm
    ├── y_mm
    ├── z_mm
    ├── num_frames
    ├── ED_time
    └── ES_time

Unannotated Patients


Patient0xx/
├── image/
│   ├── Patient0xx_slice0yytime0zz.png
│   ├── ...
│   └── Radially extracted 2D fetal echocardiography slices
│
└── Patient0xx_info.cfg
    ├── diagnosis
    ├── x_mm
    ├── y_mm
    ├── z_mm
    └── num_frames

These additional 50 unannotated 4D studies are provided to support semi-supervised learning approaches and facilitate future methodological research and development.

The dataset is available via Zenodo for non-commercial research use. [Download]