Spatial Frequency Loss

Overview

Pixel-wise losses tend to prioritize the learning of low-frequency components, which can impede the learning of high-frequency components. Spatial Frequency Loss (the SFL) complements the pixel-wise reconstruction loss by measuring discrepancies between corresponding frequency-selective responses of output and target images.

Definitions of the SFL and the IER

Let \(I_k^o(c,h,w)\) and \(I_k^t(c,h,w)\) denote the output and target images of sample \(k\), respectively, where \(c\), \(h\), and \(w\) are the channel, vertical-coordinate, and horizontal-coordinate indices. Their responses in the \(s\)-th Difference-of-Gaussians subband are denoted by \(I_k^o(c,h,w \mid \sigma_s)\) and \(I_k^t(c,h,w \mid \sigma_s)\), respectively, where \(\sigma_s\) is the scale of the Difference-of-Gaussians filter for subband \(s\). The Spatial Frequency Loss is the weighted sum of the corresponding subband losses:

Here, \(E_{\mathrm{SFL}}(k \mid \sigma_s)\) is the subband loss at scale \(\sigma_s\), and \(w_{\mathrm{SFL}}(\sigma_s)\) is its weight. Smaller Gaussian scales correspond approximately to higher spatial frequencies. A large loss at a smaller scale therefore indicates insufficient reconstruction of high-spatial-frequency components.

The Inverse Error Ratio compares the pixel-wise L2 error over the entire image with the error in each subband:

A subband with a smaller reconstruction error receives a larger IER value. In natural-image datasets, high-frequency subbands generally have lower energy and smaller losses, so the IER tends to assign larger values to these subbands. The subband weights are obtained by averaging the IER over the calibration data:

This data-driven weighting gives greater emphasis to subbands with smaller initial errors and helps balance learning across spatial frequencies. In the public implementation, the calibration workflow estimates these weights before normal optimization and then keeps the calibrated weights fixed during training.

Example results

The SFL was evaluated using a VQ-VAE trained on the FFHQ and COCO2017 datasets. Compared with the baseline pixel-wise loss, incorporating the SFL produced sharper output images with better-preserved fine-scale structures.

VQ-VAE: FFHQ example

VQ-VAE: COCO2017 example

Validation image-quality metrics

The quantitative results are consistent with the visual comparisons. Incorporating the SFL improved PSNR, SSIM, and LPIPS on the validation data of both datasets.

Spatial-frequency subband losses

Lower values indicate smaller discrepancies between the corresponding frequency-selective responses of the output and target images. SB1–SB3 correspond approximately to high spatial frequencies, while the subsequent subbands cover progressively lower spatial-frequency ranges. The results show that the SFL substantially reduces the losses in the high-frequency subbands. In addition to reducing the overall L2 error, the SFL redistributes the remaining error across subbands to prioritize the learning of high-frequency components.

Subband-loss values are reported in units of ×10⁻⁵. The FFHQ baseline row reflects the author-identified correction to the published Table 2.

Image inpainting example

The SFL was also evaluated on an image-inpainting task using the DIV2K dataset and a convolutional network substantially different from the VQ-VAE used in the main experiments. The input image contains a central missing region covering 25% of the image area. Compared with the pixel-wise L2 loss alone, incorporating the SFL recovered sharper and more detailed structures within the missing region. This example corresponds to Fig. S9 in the Supplementary Material.

Computation time

The paper measured FFHQ training with 40,000 images resized to 256 × 256, batch size 4, and three complete epochs of 10,000 iterations each. The interval included GPU transfer, forward propagation, loss computation, backward propagation, and the optimizer update.

The first four rows are from Table 6 of the paper, where FFL and FRL stand for Focal Frequency Loss and Feature Reconstruction Loss, respectively. The original SFL implementation used the conv2d backend, which was retained throughout the paper to ensure fair comparisons across experiments. Because separable filtering is mathematically equivalent and substantially faster, the separable backend was implemented after publication and is used as the default in the public software. A post-publication measurement under the same measurement procedure found an approximately 4.01-fold speed-up over the original SFL implementation and an approximately 75.05% reduction in SFL training time.

Training code example using the public software

This example shows the simplest workflow in which the SFL automatically determines the subband weights using the Inverse Error Ratio (IER) before training. The model architecture, data loader, and optimizer do not require modification.

1. Import the SFL

from spatial_frequency_loss import SpatialFrequencyLoss

2. Initialize the SFL

sfl = SpatialFrequencyLoss(
    num_image_channels=3,
    base_scale=0.6,
    num_subbands=7,
    ier_update_mode="calibration",
).to(device)

mse_loss = torch.nn.MSELoss()

The calibration mode is specified explicitly for clarity, although it is the default update mode for the IER. The default filtering backend is separable.

3. Determine the subband weights using the IER

model.eval()
sfl.eval()

with torch.no_grad():
    for inputs, targets in calibration_loader:
        inputs = inputs.to(device)
        targets = targets.to(device)

        outputs = model(inputs)

        sfl.accumulate_calibration_ier(
            outputs,
            targets,
        )

sfl.finalize_ier_calibration()

The existing model and data loader are used for a forward-only calibration pass. The SFL accumulates the subband errors and automatically determines the weights based on the IER. No model parameters are updated during this step.

4. Train the model with the SFL

model.train()
sfl.train()

for epoch in range(num_epochs):
    for inputs, targets in train_loader:
        inputs = inputs.to(device)
        targets = targets.to(device)

        optimizer.zero_grad()

        outputs = model(inputs)

        l2_loss = mse_loss(
            outputs,
            targets,
        )

        (
            subband_losses,
            weighted_sfl,
            _,
            _,
            _,
        ) = sfl(
            outputs,
            targets,
        )

        total_loss = l2_loss + weighted_sfl

        total_loss.backward()
        optimizer.step()

Paper and post-publication information

Naoyuki Ichimura, “Spatial frequency loss for mitigating spectral bias in deep neural networks,” Machine Vision and Applications 37, 173 (2026). DOI: 10.1007/s00138-026-01927-5.