ModelComparisonSimulator: handle different outputs from individual simulators
#452
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
|
|
Thanks for the PR, I skimmed it and like the idea behind the changes. I'll try to conduct a proper review some time this week. |
LarsKue
reviewed
Apr 29, 2025
add newlines to correctly render lists, make reference to other class a link
vpratz
reviewed
Apr 30, 2025
| def _determine_key_conflicts(self, sims): | ||
| # determine only once | ||
| if self._keys is not None: | ||
| return self._keys |
There was a problem hiding this comment.
Can this return "wrong" results if some simulators had n=0 in line 120 when the function first runs, and n>0 later on? Is this something we want to safeguard against?
There was a problem hiding this comment.
I can imagine this function to be quite cheap to compute, would it make sense to run it completely every time (but only logging the info once)?
LarsKue
requested changes
May 3, 2025
| def _determine_key_conflicts(self, sims): | ||
| # determine only once | ||
| if self._keys is not None: | ||
| return self._keys |
| assert set(samples) == {"x", "model_indices", "c", "w"} | ||
| assert np.sum(np.isnan(samples["c"])) + np.sum(np.isnan(samples["w"])) == batch_size | ||
| elif multimodel_key_conflicts.key_conflicts == "error": | ||
| with pytest.raises(Exception): |
There was a problem hiding this comment.
This is too broad of a check. Use specific exception types.
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #441
As per #441 (comment), this PR implements
However, by default the simulator will just drop (with an info warning) keys that are not common for all simulators, since in most situations we would not need those outputs in the first place.