Skip to content

Include metadata variables for (arrays of) structures in to_xarray#112

Merged
olivhoenen merged 5 commits intoiterorganization:developfrom
maarten-ic:feature/to-xarray-export-structure-metadata
Apr 2, 2026
Merged

Include metadata variables for (arrays of) structures in to_xarray#112
olivhoenen merged 5 commits intoiterorganization:developfrom
maarten-ic:feature/to-xarray-export-structure-metadata

Conversation

@maarten-ic
Copy link
Copy Markdown
Collaborator

@maarten-ic maarten-ic commented Mar 23, 2026

This allows reading back data in the following scenario:

ids = imas.IDSFactory().core_profiles()
... # fill IDS
xrds = imas.util.to_xarray(ids)
xrds.to_netcdf("data.nc", group="core_profiles/0")

with netCDF4.Dataset("data.nc", "a") as ds: 
    # Set global DD version metadata:
    ds.data_dictionary_version = imas.util.get_data_dictionary_version(ids)

with imas.DBEntry("data.nc", "r") as entry:
    ids2 = entry.get("core_profiles")

More readable code, and less repetition:
```python
# before
self.ncmeta.get_dimensions(..., self.homogeneous_time)
# after
self.get_dimensions(...)
```
This allows reading back data in the following scenario:

```python
ids = imas.IDSFactory().core_profiles()
... # fill IDS
xrds = imas.util.to_xarray(ids)
xrds.to_netcdf("data.nc", group="core_profiles/0")

with imas.DBEntry("data.nc", "r") as entry:
    ids2 = entry.get("core_profiles")
```

N.B. `to_xarray()` doesn't include metadata for inhomogeneously sized arrays of structures, so the two IDSs in above example could be different.
@maarten-ic
Copy link
Copy Markdown
Collaborator Author

N.B. I want to further close the gap between writing an IDS to a netCDF file (with DBEntry.put) and the xarray route. Marking this PR as draft until then.

- Centralize attribute metadata logic
- Include :shape arrays in imas.util.to_xarray
- Test that netCDF files written with xarray can be read as DBEntry
@maarten-ic maarten-ic marked this pull request as ready for review March 27, 2026 13:10
@maarten-ic
Copy link
Copy Markdown
Collaborator Author

@prasad-sawantdesai @olivhoenen I've updated the logic so that:

  • An IDS written to netCDF and loaded with xarray.load_dataset equals to an ids converted to xarray with imas.util.to_xarray()
  • An IDS converted to xarray and then store to netCDF can be read with DBEntry.get() (see example in description above)

Can you please review the PR?

Thanks!

@olivhoenen
Copy link
Copy Markdown
Collaborator

Works nicely, thanks @maarten-ic !
Could we describe the step

with netCDF4.Dataset("data.nc", "a") as ds: 
    # Set global DD version metadata:
    ds.data_dictionary_version = imas.util.get_data_dictionary_version(ids)

somewhere in the documentation (maybe after the section on converting directly IDS to xarray)?

@maarten-ic
Copy link
Copy Markdown
Collaborator Author

Works nicely, thanks @maarten-ic ! Could we describe the step

with netCDF4.Dataset("data.nc", "a") as ds: 
    # Set global DD version metadata:
    ds.data_dictionary_version = imas.util.get_data_dictionary_version(ids)

somewhere in the documentation (maybe after the section on converting directly IDS to xarray)?

Thanks for the suggestion! Please check if the documentation is clear like this: https://imas-python--112.org.readthedocs.build/en/112/netcdf.html#store-xarray-datasets-in-imas-compatible-netcdf-file

@olivhoenen olivhoenen merged commit abd7b6c into iterorganization:develop Apr 2, 2026
9 checks passed
@maarten-ic maarten-ic deleted the feature/to-xarray-export-structure-metadata branch April 2, 2026 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants