Improved Duplicate Handling in Nightscout BG Data Processing

[bjorkert]

Resolves #357

This update fixes an issue where duplicate glucose readings (nightscout dexcom bridge and Loop or Trio) were were sometimes included, resulting in a delta of 0. The new logic ensures that readings are spaced at least 4 minutes apart, preventing duplicate values from affecting calculations.

Additionally, the previous method, which iterated through multiple time buckets and filtered the closest reading, has been replaced with a more efficient single-pass approach. This results in faster processing while maintaining accurate data selection.

Fixes:
• Eliminates near-duplicate readings that caused incorrect delta calculations.
• Improves performance by removing unnecessary filtering operations.

[marionbarker]

Code Review

This does what it says. But is this the right thing?
Do we need to worry about a Nightscout instance that holds 1 min intervals (like Libre can provide). In that case, LF only shows a new reading every 5 minutes. What is the Nightscout protocol? It appears to show correct 5-minute delta even when glucose are irregular or close together.

Test

Test dual uploads

Upload the same fixed glucose pattern from two different Terminal windows.
Start the second upload about 30 sec after upload 1 - repeat at several different time offsets.
Only one of the upload values is shown in LoopFollow (both are in Nightscout).
There are never instances of delta of 0 shown in LoopFollow.

Test one minute uploads

Now just one trace, but upload at 60 sec intervals.
Nightscout shows every upload, LF shows only some.

upload	NS & Delta	LF & Delta
122		122 +12
126
131	131 +12
135	135 +17
140	140 +19
144	144 +18	144 +18
149	149 +18
153	153 +18
163	163 +20
173	173 +23
183	183 +29	183 +34
193	193 +33
205	205 +37
215	215 +40
230	230 +44
240	240 +45	240 +47

[bjorkert]

The problematic double uploads are probably very close in time, would you like to re-do the test having
let minInterval: Double = 4 * 60
set to
let minInterval: Double = 50
instead?

[marionbarker]

Modify minInterval

Built after changing minInterval to 50 // sec

Not the right change. I believe func ProcessDexBGData needs to be rewritten.

If it gets one minute intervals, the 5 minute delta should be (val2-val1)/(timedelta) normalized to 5 minutes and reported as integer mg/dL.

Test one minute uploads

Now just one trace, but upload at 60 sec intervals.
Nightscout reports deltas that are at 5-min intervals, e.g. 144 - 126 = 18.
LoopFollow still only reports once per 5 minutes but now does the delta for last 2 readings, even though the previous one was not reported. I think the delta needs to be

upload	NS & Delta	LF & Delta
111	111 -1	111 +1
116	116 +2	116 +5
122	122 +6
126	126 +11	rebuild again
131	131 +15	131 +5
135	135 +17
140	140 +19
144	144 +18
149	149 +18	149 +5

[bjorkert]

Adjusted the duplicate check method for glucose readings.
• Now ensures that a reading is only added if its sgv differs from the last added value or if at least 30 seconds have passed since the last addition.
• Previously, the check was based solely on a 4-minute interval.

[marionbarker]

Test one minute uploads again

One glucose trace, upload at 60 sec intervals.
Nightscout reports deltas that are converted to 5-min intervals, e.g. 144 - 126 = 18.
LoopFollow will not auto-update at 1-min intervals, so started doing the pull down when NS updated (after the blank LF row) and every CGM reading was updated with successive deltas reported.

upload	NS & Delta	LF & Delta
122	122 +6	122 +6
126	126 +11
131	131 +15	131 +5
135	135 +17	135 +4
140	140 +19	140 +5
144	144 +18	144 +4
149	149 +18	149 +5
153	153 +18	153 +4

IMO this is good enough. LoopFollow - reports successive deltas for any numbers more than 30 sec apart. It does not convert to a a 5 minute effective delta but I think that's fine too.

Test with dual uploads, started a few sec apart

upload 1 started at: 2025-01-30T15:44:58.000Z
upload 2 started at: 2025-01-30T15:45:04.000Z

Let it run for for 4 pairs of uploads and observe only one of each pair shows up in LoopFollow and the delta is always appropriate.

[marionbarker]

One last test - confirm that if there really is a repeated glucose value, it is reported appropriately

Create a new trace with constant CGM value (120 mg/dL) and upload it at 5 minute intervals.

mins	upload	NS & Delta	LF & Delta
0	120	120 +0	120 +0
5	120	120 +0	120 +0
10	120	120 +0	120 +0

[marionbarker]

Reviewed, tested and approved.