FI-3762: Improve IG file loading #630

Jammjammjamm · 2025-02-21T15:34:05Z

Summary

This branch copies IG packages to data/igs to be used by the validator. It also prevents the routes from being hosted in the workers.

Testing Guidance

Add the following to the Gemfile:

gem 'carin_for_blue_button_test_kit',
    git: 'https://github.com/inferno-framework/carin-for-blue-button-test-kit.git',
    branch: 'main'

Require the carin test kit in the demo suite, bundle, and start inferno.

During startup you will see:

08:01:39 web.1     | I, [2025-02-21T08:01:39.993085 #35004]  INFO -- : Copying carin4bb-110.tgz to data/igs
08:01:39 web.1     | I, [2025-02-21T08:01:39.994850 #35004]  INFO -- : Copying carin4bb-200-dev-nonfinancial-fixed.tgz to data/igs
08:01:39 web.1     | I, [2025-02-21T08:01:39.996344 #35004]  INFO -- : Copying carin4bb-200.tgz to data/igs

You can check data/igs to see that the files are there, and run the carin tests to see that the validator works.

codecov · 2025-02-21T15:35:44Z

Codecov Report

Attention: Patch coverage is 64.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 84.67%. Comparing base (ee56a04) to head (5a29205).

Files with missing lines	Patch %	Lines
lib/inferno/config/boot/ig_files.rb	57.14%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #630      +/-   ##
==========================================
- Coverage   84.71%   84.67%   -0.05%     
==========================================
  Files         283      284       +1     
  Lines       12408    12431      +23     
  Branches     1503     1511       +8     
==========================================
+ Hits        10512    10526      +14     
- Misses       1888     1897       +9     
  Partials        8        8

Flag	Coverage Δ
backend	`92.19% <64.00%> (-0.14%)`	⬇️
frontend	`79.21% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

lib/inferno/config/boot/ig_files.rb

arscan · 2025-02-21T16:22:21Z

lib/inferno/config/boot/ig_files.rb

+    if File.exist? File.join(Dir.pwd, 'data', 'igs')
+      ig_files.each do |source_file_path|
+        destination_file_path = File.join(Dir.pwd, 'data', 'igs', File.basename(source_file_path))
+        Inferno::Application['logger'].info("Copying #{File.basename(source_file_path)} to data/igs")


would be nice if we captured the test kit name in ig_files so we could print it out here too, so you know where the files came from in case multiple test kits have the same name (e.g. multiple test kit just throws a package.tgz in there (ack, we should flat out block that or add a spec rule for that), or 2 decide to 'fix' the same version of us core in different ways). I'm not suggesting we necessarily try to proactively solve that problem, but having a tad more logged here might help us figure out where to look if there are multiple of the same name. Also, it would help to know which one 'won'.

decide to 'fix' the same version of us core in different ways

Any alterations to IG files could be problematic, depending on how the validator works. I could imagine that if a modified IG package is loaded and added to the package cache first, then any subsequent validators just relying on that package id and version would use the modified rather than published version.

Yeah, 'fixing' published IG files is a gross case but unfortunately something we need to deal with if we want to provide a decent experience to users. I suppose the more sane case is for 'unpublished' IGs, and hopefully its obvious where they come from.

~~If you won't want to add complexity to the code here to log where the IG comes from that's fine by me, leave it up to you.~~ N/m I see that you added this functionality.

I already updated the logging.

arscan · 2025-02-21T16:24:27Z

This branch copies IG packages to data/igs to be used by the validator.

At the risk of sounding pedantic, part of the point of this is we want these files to be available for broader use within inferno core beyond the validator and to do so we want more standardization and automation, right?

arscan · 2025-02-21T16:27:17Z

Would it be a good idea to force test kits to upgrade by providing a central spec test that looks for these missing things? It looks like this is created in a backwards compatible manner, but perhaps there is wisdom in having our stuff just be kept up to date so a test kit can just starting using those features without knowing they first have to figure out all the things that need to be updated.

dehall · 2025-02-21T16:31:55Z

lib/inferno/config/boot/ig_files.rb

+    # only once from the "web" process
+    next if Sidekiq.server?
+
+    target_container.start :logging


This means that logging has to start before this, right? Should we have a similar dependency that this has to run before the validator boot process? https://github.com/inferno-framework/inferno-core/blob/main/lib/inferno/config/boot/validator.rb

Yes, good catch.

arscan · 2025-02-21T17:37:49Z

lib/inferno/config/boot/ig_files.rb

+    end
+
+    if File.exist? File.join(Dir.pwd, 'data', 'igs')
+      test_kit_ig_files.each do |ig_files|


I think we should clear out the directory first so the it doesn't get filled with old ig files that aren't needed anymore. This isn't the place to manually dump igs and shouldn't be touched by the user, similar to the other content in ./data (though, its great to be able to inspect this folder manually... just don't mutate it and expect persistence)

Jammjammjamm · 2025-02-24T14:57:34Z

Ok, shared specs have been added.

bundle exec bin/inferno new my-test-kit
mv my-test-kit ..
Update the Gemfile in my test kit to point to this branch
bundle
Add the following spec to my test kit:

RSpec.describe MyTestKit, order: :defined do
  it_behaves_like 'platform_deployable_test_kit'
end

bundle exec rake and the validator-related specs should pass
alter one of the IG volume lines you you should get a failure
remove data/igs/.keep and you should get a failure
comment out the validator services and you should no longer get a failure about data/igs/.keep

Jammjammjamm requested review from arscan and dehall February 21, 2025 15:34

Jammjammjamm changed the title ~~FI-3762: improve ig file loading~~ FI-3762: Improve IG file loading Feb 21, 2025

arscan reviewed Feb 21, 2025

View reviewed changes

lib/inferno/config/boot/ig_files.rb Outdated Show resolved Hide resolved

arscan reviewed Feb 21, 2025

View reviewed changes

dehall reviewed Feb 21, 2025

View reviewed changes

Jammjammjamm force-pushed the fi-3762-improve-ig-file-loading branch from ef7d507 to 76134da Compare February 21, 2025 17:14

arscan reviewed Feb 21, 2025

View reviewed changes

Jammjammjamm added 12 commits February 24, 2025 09:52

add ig files startup provider

f9ca1f0

update template

9a5e830

prevent web/ig file initialization in workers

a31a86f

update .gitignore

26a60ba

remove commented code

b569dd4

update legacy validator igs path in docker compose template

c342a43

fix ig loading description

4f7d7e2

ensure ig files are loaded prior to initializing validator sessions

9a19379

update ig file loader to indicate source of ig files

13ff989

empty igs folder prior to copying igs

dd8d76c

add spec for proper ig volume paths

c1e9559

fix ig volume in template

5a29205

Jammjammjamm force-pushed the fi-3762-improve-ig-file-loading branch from 76134da to 5a29205 Compare February 24, 2025 14:53

Jammjammjamm requested a review from arscan February 24, 2025 14:57

Jammjammjamm self-assigned this Feb 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FI-3762: Improve IG file loading #630

FI-3762: Improve IG file loading #630

Jammjammjamm commented Feb 21, 2025

codecov bot commented Feb 21, 2025 •

edited

Loading

arscan Feb 21, 2025 •

edited

Loading

Jammjammjamm Feb 21, 2025

arscan Feb 21, 2025 •

edited

Loading

Jammjammjamm Feb 21, 2025

arscan commented Feb 21, 2025 •

edited

Loading

arscan commented Feb 21, 2025

dehall Feb 21, 2025

Jammjammjamm Feb 21, 2025

arscan Feb 21, 2025 •

edited

Loading

Jammjammjamm commented Feb 24, 2025

FI-3762: Improve IG file loading #630

Are you sure you want to change the base?

FI-3762: Improve IG file loading #630

Conversation

Jammjammjamm commented Feb 21, 2025

Summary

Testing Guidance

codecov bot commented Feb 21, 2025 • edited Loading

Codecov Report

arscan Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

Jammjammjamm Feb 21, 2025

Choose a reason for hiding this comment

arscan Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

Jammjammjamm Feb 21, 2025

Choose a reason for hiding this comment

arscan commented Feb 21, 2025 • edited Loading

arscan commented Feb 21, 2025

dehall Feb 21, 2025

Choose a reason for hiding this comment

Jammjammjamm Feb 21, 2025

Choose a reason for hiding this comment

arscan Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

Jammjammjamm commented Feb 24, 2025

codecov bot commented Feb 21, 2025 •

edited

Loading

arscan Feb 21, 2025 •

edited

Loading

arscan Feb 21, 2025 •

edited

Loading

arscan commented Feb 21, 2025 •

edited

Loading

arscan Feb 21, 2025 •

edited

Loading