Skip to content

Gibbs test | Fix dynamic model test in Gibbs sampler suite #2579

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

AoifeHughes
Copy link

The existing dynamic model test was problematic:

  • Used a complex Dirichlet Process mixture model that was slow (10,000 samples)
  • Only performed regression testing against "old" values with a TODO indicating it wasn't ideal
  • Failed to properly test the core dynamic model functionality that Gibbs sampling should support

Here I've changed this with:

a new dynamic_bernoulli_normal model that:

  • Has varying dimensions: A Bernoulli variable b controls whether the model has 1 or 2 parameters
  • Has analytical ground truth: Marginal likelihoods and posterior for b can be computed exactly
  • Is fast to sample: Only 1,000 samples vs 10,000, runs in ~7-12 seconds
  • Tests core functionality: Validates that Gibbs sampling correctly handles dimension changes without errors

@AoifeHughes AoifeHughes requested a review from Copilot June 4, 2025 09:23
@AoifeHughes AoifeHughes self-assigned this Jun 4, 2025
Copilot

This comment was marked as outdated.

@AoifeHughes
Copy link
Author

Leave this as a draft for a teeny bit, I need to properly test and fully understand how julia tests work so I'm not breaking what I think I understand - no need for any reviews yet.

@AoifeHughes
Copy link
Author

This should address: #2402 when completed.

@mhauru
Copy link
Member

mhauru commented Jun 4, 2025

As discussed on Slack, feel free to change the base branch of this to main. Since it's not a user-interface breaking change no need to make a breaking release for it and make it go through the breaking branch.

@AoifeHughes AoifeHughes changed the base branch from breaking to main June 4, 2025 10:01
@AoifeHughes
Copy link
Author

Results from my local testing:

Test Summary:                                          | Pass  Total      Time
Turing                                                 | 2613   2613  19m50.9s
  Aqua                                                 |           0      0.1s
  AD                                                   |           0      0.0s
  essential                                            |           0      0.0s
  samplers (without AD)                                |           0      0.0s
  inference with samplers                              | 2613   2613  19m50.8s
    GibbsContext                                       |  868    868     27.7s
      type stability                                   |  868    868     27.7s
    Invalid Gibbs constructor                          |    8      8      0.1s
    Sampler call order                                 |    1      1     53.9s
    Equivalence of RepeatSampler and repeating Sampler |    1      1      4.5s
    Gibbs warmup                                       |    8      8      2.3s
    Testing gibbs.jl                                   | 1727   1727  18m19.3s
      Gibbs constructors                               |   13     13     19.7s
      Gibbs inference                                  |   22     22   6m06.7s
      transitions                                      |           0      2.1s
      dynamic model with analytical posterior          |    5      5      8.4s
      dynamic model with ESS                           |    6      6     33.4s
      dynamic model with dot tilde                     |           0     11.3s
      Demo model                                       | 1564   1564   8m32.2s
      multiple varnames                                |   20     20      1.5s
      non-identity varnames                            |    2      2      9.2s
      submodels                                        |    2      2     11.3s
      CSMC + ESS                                       |   42     42     38.9s
      CSMC + ESS (usage of implicit varname)           |   42     42     51.8s
      externalsampler                                  |    8      8     28.2s
      linking changes dimension                        |    1      1      4.5s
  variational algorithms                               |           0      0.0s
  mode estimation                                      |           0      0.0s
  variational optimisers                               |           0      0.0s
  stdlib                                               |           0      0.0s
  utilities                                            |           0      0.0s
───────────────────────────────────────────────────────────
                                Time          Allocations
                          ───────────────   ───────────────
     Total measured:           1191s             534GiB

Section           ncalls     time    %tot     alloc    %tot
───────────────────────────────────────────────────────────
inference              1    1191s  100.0%    534GiB  100.0%
  mcmc/gibbs.jl        1    1191s  100.0%    534GiB  100.0%
───────────────────────────────────────────────────────────     Testing Turing tests passed

@AoifeHughes AoifeHughes marked this pull request as ready for review June 5, 2025 10:02
Copy link
Contributor

github-actions bot commented Jun 5, 2025

Turing.jl documentation for PR #2579 is available at:
https://TuringLang.github.io/Turing.jl/previews/PR2579/

Copy link

codecov bot commented Jun 5, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.58%. Comparing base (7ebde76) to head (771934c).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2579   +/-   ##
=======================================
  Coverage   85.57%   85.58%           
=======================================
  Files          22       22           
  Lines        1456     1457    +1     
=======================================
+ Hits         1246     1247    +1     
  Misses        210      210           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coveralls
Copy link

coveralls commented Jun 5, 2025

Pull Request Test Coverage Report for Build 15822411978

Details

  • 2 of 2 (100.0%) changed or added relevant lines in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.01%) to 85.646%

Totals Coverage Status
Change from base Build 15765001949: 0.01%
Covered Lines: 1247
Relevant Lines: 1456

💛 - Coveralls

@AoifeHughes AoifeHughes requested a review from Copilot June 9, 2025 07:50
Copilot

This comment was marked as outdated.

@yebai yebai requested a review from penelopeysm June 20, 2025 10:36
@AoifeHughes AoifeHughes requested a review from Copilot June 20, 2025 11:53
Copilot

This comment was marked as outdated.

@AoifeHughes AoifeHughes requested a review from Copilot June 23, 2025 08:36
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request refactors several model definitions and tests to adopt the new default-type parameter syntax and introduces a new dynamic model (dynamic_bernoulli_normal) with an analytical posterior for faster and more robust testing of the Gibbs sampler. Key changes include:

  • Updating function signatures across multiple tests to use the new parameter syntax with parentheses.
  • Introducing a dynamic model with a Bernoulli variable controlling parameter dimensionality.
  • Simplifying and speeding up tests while ensuring core Gibbs sampling functionality is exercised.

Reviewed Changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated no comments.

File Description
test/mcmc/hmc.jl Updated the default type parameter syntax in model definitions.
test/mcmc/gibbs.jl Updated function signatures and added a new dynamic model test.
test/mcmc/Inference.jl Applied similar parameter signature updates across model tests.
src/mcmc/external_sampler.jl Updated constructor and helper function parameter syntax.
Comments suppressed due to low confidence (2)

test/mcmc/gibbs.jl:564

  • [nitpick] The check for θ[2] values only verifies that one of the conditions is met. Consider splitting the test into two separate assertions to clearly validate that when b=0, θ[2] is missing, and when b=1, θ[2] exists.
            @test n_missing_theta2 > 0 || n_present_theta2 > 0  # One of these should be true

src/mcmc/external_sampler.jl:47

  • [nitpick] Consider adding a comment or type annotations to clarify the expected type and meaning of the 'unconstrained' flag, which would improve maintainability and readability of the code.
function requires_unconstrained_space(

@yebai
Copy link
Member

yebai commented Jun 23, 2025

Thanks @AoifeHughes -- good work!

Comment on lines +568 to 572
# Helper function for logsumexp
function logsumexp(x)
max_x = maximum(x)
return max_x + log(sum(exp.(x .- max_x)))
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor, but it might be better to use LogExpFunctions.logsumexp here rather than rolling our own version.

Copy link
Member

@yebai yebai Jun 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

Copy link
Member

@penelopeysm penelopeysm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise, if @yebai if happy, I am too

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants