RandAffined does not work for differently sized images

[Spenhouet]

Describe the bug

When the images in a pipeline entry have different sizes, then the RandAffined does not work properly.
The resulting images no longer match.

To Reproduce

Steps to reproduce the behavior:

1. Load these files: example_files.zip. You can view them with Slicer or MRIcroGL (itk-snap seems to have a bug which prevents it to load these files).
2. Execute this pipeline:

from pathlib import Path

import numpy as np
from monai.transforms.compose import Compose
from monai.transforms.io.dictionary import LoadImaged, SaveImaged
from monai.transforms.spatial.dictionary import RandAffined
from monai.transforms.utility.dictionary import (EnsureChannelFirstd, ToTensord)
from monai.utils.enums import GridSampleMode

path = Path('scratch/rand_affine')

T1_KEY = 't1'
LABEL_KEY = 'labels'
FILE_KEYS = [T1_KEY, LABEL_KEY]
SAMPLE_MODES = [GridSampleMode.BILINEAR, GridSampleMode.NEAREST]

data = {
    T1_KEY: path / 't1.nii.gz',
    LABEL_KEY: path / 'labels.nii.gz',
}

process = Compose([
    LoadImaged(FILE_KEYS),
    ToTensord(FILE_KEYS),
    EnsureChannelFirstd(FILE_KEYS),
    RandAffined(
        FILE_KEYS,
        rotate_range=5.0 * np.pi / 180.0,  # type: ignore
        prob=1.0,
        mode=SAMPLE_MODES,
        allow_missing_keys=True,
    ),
    SaveImaged(
        FILE_KEYS,
        output_dir=path,
        output_postfix=f"out",
        resample=False,
        output_dtype=np.float32,
        separate_folder=False,
        print_log=False,
    )
])

process([data])

Expected behavior
The same random affine transformation is applied to all images.
After the transform the labels still match the image.

Screenshots

Given the following image and label:

When applying RandAffined, the image and labels no longer match:

Additional context

We are using torchio's RandomAffine transform instead.

This issue was previously closed (#3316) and moved to a discussion (#3319) due to a bug in itk-snap. Since this does not have anything to do with this bug report, I'm reopening it.

[Nic-Ma]

Hi @Spenhouet ,

Thanks for your feedback!
I think that because our current implementation shares both affine and spatial_size for all the keys.
Let me try to improve it to only share the same affine between keys.

Thanks.

[wyli]

Hi @Spenhouet, Without proper handling of the metadata, it doesn't make sense to compare the different shapes of the raw data arrays because they can be in different coordinate systems.

This goes back to the feature request #2424. And the core team is fully aware of the ticket. It is a major feature and it takes time to analyse different use cases and take care of all the implementation details, please be patient and do not repeatedly create new bug reports or suggest fractional patches on the same topic.

[Spenhouet]

do not repeatedly create new bug reports or suggest fractional patches on the same topic.

@wyli @Nic-Ma @ericspod @rijobro The project management feels rather arbitrary to me. I had the same with #2721. There I suggested a multiple of transforms to be implemented. There was some disagreement on if this should be tracked in a single issue or in individual issues. This disagreement does not seem to have been resolved. Instead the issue was silently changed to only the "add example images" part and then closed while the majority of the raised feature requests were not addressed. This suggests, that individual / fractional issues are preferred. Here I'm still about to create all the individual feature requests. The issue on #2424 is also rather broad. To get this done, there are many changes to be made. While we are working with MONAI, we are bumping into many issues which might relate to this. But they are very specific and individual issues which can be addressed independently. Based on my experience with #2721 I thought that individual bug reports and feature requests are more welcome. But apparently not. So what is it? What should I do. Please advise and maybe come to an internal decision on how you want to handle project management first.

[Nic-Ma]

Hi @Spenhouet ,

Thanks very much for your great feedback and suggestions.
We discussed these medical-image processing related feature requests last week, the main missing part in current codebase is how to maintain the metadata during transforms.
It would be a very big task for us to update metadata in every transform, but we already started to investigate the details.

Thanks.

[Spenhouet]

Hi @Nic-Ma,

good to hear. I believe this is a worth while undertaking. I think the current way the transforms and meta_dict are handled via extended key string is not great. If this could be unified and simplified this would help immensely for this. If this information would always be at the same place or always part of every image, this would remove quite some boilerplate.

Best regards,
Sebastian

Edit: Basically, adding a form of Image object which then always requires certain information like an affine or spacing.