Refactor ensemble generator for flexibility#940
Open
matthewhoffman wants to merge 13 commits intoMPAS-Dev:mainfrom
Open
Refactor ensemble generator for flexibility#940matthewhoffman wants to merge 13 commits intoMPAS-Dev:mainfrom
matthewhoffman wants to merge 13 commits intoMPAS-Dev:mainfrom
Conversation
Generalize ensemble generator to support multiple model configurations used for different studies. * Introduced a new configuration module to handle model configurations for ensemble generation. * Updated `BranchRun` and `EnsembleMember` classes to accept a `resource_module` parameter for dynamic configuration loading. * Created default configuration files for branch and spinup ensembles, including necessary namelists and stream definitions. * Modified the main ensemble generator configuration to streamline the setup process and improve clarity. * Enhanced error handling for missing configuration sections and options. * Updated the `SpinupEnsemble` class to utilize the new configuration methods for improved modularity and maintainability.
Variations on the word configuration are already too widespread so this change should reduce confusion.
The primary new functionality is the ability to support any namelist option as a parameter rather than only pre-defined parameters. The refactor also simplifies how parameter values are specified and puts all parameters in a dedicated cfg section. More details on the format are included in the updated docs.
Remove unnecessary extra section level, Separate spinup_ensemble options from options general to the whole test group.
There was a problem hiding this comment.
Pull request overview
This PR refactors the landice/ensemble_generator test group to support template-based model configurations and a more flexible parameter-perturbation workflow (including generic float namelist perturbations configured entirely from cfg files), with corresponding documentation updates.
Changes:
- Introduce
ensemble_templates/<name>/spinupand.../branchpackages to manage per-study cfg/namelist/streams/(Albany) resources. - Replace hard-coded sampled-parameter logic with ordered definitions in
[ensemble.parameters], including genericnl.*namelist float perturbations. - Update User’s/Developer’s Guide docs and Landice API docs to reflect the new workflow and helper functions.
Reviewed changes
Copilot reviewed 12 out of 21 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/users_guide/landice/test_groups/ensemble_generator.rst | Documents new parameter types, template workflow, and updated cfg structure |
| docs/developers_guide/landice/test_groups/ensemble_generator.rst | Describes new template package layout and updated configuration behavior |
| docs/developers_guide/landice/api.rst | Adds new ensemble_template.* functions to the API docs list |
| compass/landice/tests/ensemble_generator/spinup_ensemble/init.py | Loads template cfg and parses ordered [ensemble.parameters] including nl.* handling |
| compass/landice/tests/ensemble_generator/ensemble_template.py | New helpers for selecting and loading template packages/files |
| compass/landice/tests/ensemble_generator/ensemble_member.py | Applies generic namelist option overrides; loads resources from selected template |
| compass/landice/tests/ensemble_generator/branch_ensemble/init.py | Loads branch template cfg and passes template resources into branch runs |
| compass/landice/tests/ensemble_generator/branch_ensemble/branch_run.py | Uses template-provided namelist/streams; makes albany_input.yaml optional |
| compass/landice/tests/ensemble_generator/plot_ensemble.py | Updates cfg section usage to match new split between ensemble_generator and spinup_ensemble |
| compass/landice/tests/ensemble_generator/ensemble_templates/default/** | Adds default template cfg/namelist/streams/(Albany) resources for spinup and branch |
Comments suppressed due to low confidence (2)
compass/landice/tests/ensemble_generator/ensemble_templates/default/spinup/ensemble_generator.cfg:5
- The comment above
ensemble_templatesays this is the subdirectory wherebranch_ensembleoptions are located, but this file is the spinup template (.../spinup/ensemble_generator.cfg). This is misleading—update the comment to refer to spinup/ensemble_generator options (or to templates in general).
compass/landice/tests/ensemble_generator/ensemble_templates/default/spinup/ensemble_generator.cfg:108 - The template uses backslashes for line continuation in
nl.von_mises_threshold.option_name. With the current parser (_split_entries()), the backslashes will be treated as literal entries and can produce an invalid namelist option name. Prefer ConfigParser-style multiline values (indented continuation lines without\) or keep the list on one line.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
compass/landice/tests/ensemble_generator/spinup_ensemble/__init__.py
Outdated
Show resolved
Hide resolved
docs/developers_guide/landice/test_groups/ensemble_generator.rst
Outdated
Show resolved
Hide resolved
This function was flagged as too complex, so Copilot helped me break it into smaller functions.
Update ensemble generator parsing and docs based on outdated but still relevant Copilot review feedback from PR MPAS-Dev#940. - Sanitize multiline option parsing in spinup_ensemble._split_entries to remove continuation backslashes before tokenization. - Use importlib.resources.as_file() when handling optional albany_input.yaml in ensemble_member setup. - Clarify in developer docs that albany_input.yaml is copied only when present for Albany-based configurations. - Fix users-guide cfg block indentation and multiline .option_name examples to match ConfigParser behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR refactors the landice/ensemble_generator test case to support two new major enhancements:
Checklist
api.rst) has any new or modified class, method and/or functions listedTestingin this PR) any testing that was used to verify the changes