Skip to content

Problem on timestamp extraction #69

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Closed
hugokernel opened this issue May 18, 2018 · 4 comments
Closed

Problem on timestamp extraction #69

hugokernel opened this issue May 18, 2018 · 4 comments

Comments

@hugokernel
Copy link

Hi,

I work on a tool to extract data from Swimming Strap HRM from fit files and when I try to display data as chart, I see big difference between real chart (from Garmin Connect) and data extracted from fit-parse.

After investigation, I found a difference in the fitparse timestamp calculation compared to same data extracted from the Garmin FitCSVTool.

First 4 lines of Strap HRM data extracted with Garmin FitCSVTool (format: filtered_bpm, event_timestamp) :

71|71|71|71|71|71|71|71,2224910.171875|2224911.158203125|2224912.220703125|2224913.0419921875|2224913.7802734375|2224914.5126953125|2224915.34765625|2224915.7939453125
71|71|71|71|71|71|71|71,2224916.271484375|2224916.982421875|2224917.2734375|2224919.5859375|2224920.556640625|2224921.078125|2224921.45703125|2224921.990234375
71|71|72|72|72|73|73|74,2224924.7451171875|2224925.24609375|2224925.7421875|2224926.2412109375|2224927.1728515625|2224927.73828125|2224928.2197265625|2224928.6328125
74|74|75|77|83|84|87|89,2224929.126953125|2224929.619140625|2224930.1142578125|2224930.62890625|2224931.1552734375|2224931.67578125|2224932.1982421875|2224932.72265625

I use this code to extract and format data :

from fitparse import FitFile
fitfile = FitFile('lagny_fim.fit')
for _, msg in enumerate(fitfile.get_messages('hr')):

    if msg.header.local_mesg_num == 12:
        startat = round(msg.get('event_timestamp').value)
        continue

    timestamps = []
    for field in msg.fields:
        if field.name == 'event_timestamp':
            timestamps.append(startat + float(field.value))

    print('%s,%s' % (
        '|'.join(str(bpm) for bpm in msg.get('filtered_bpm').value),
        '|'.join(str(t) for t in timestamps)
    ))

Data extracted with this script :

71|71|71|71|71|71|71|71,2224912.17188|2224913.1582|2224914.2207|2224915.04199|2224915.78027|2224916.5127|2224917.34766|2224917.79395|2224918.0|2224918.0                                                                        
71|71|71|71|71|71|71|71,2224918.27148|2224918.98242|2224919.27344|2224921.58594|2224922.55664|2224923.07812|2224923.45703|2224923.99023|2224926.0|2224926.0                                                                     
71|71|72|72|72|73|73|74,2224926.74512|2224927.24609|2224927.74219|2224928.24121|2224929.17285|2224929.73828|2224930.21973|2224930.63281|2224934.0|2224934.0                                                                     
74|74|75|77|83|84|87|89,2224935.12695|2224935.61914|2224936.11426|2224936.62891|2224937.15527|2224937.67578|2224938.19824|2224938.72266|2224942.0|2224942.0   

As you ca see, there are an error of +2 seconds on calculated timestamp and maybe other errors that occur in the extraction that would explain the overall errors in extracting heart rate data.

Do you have an idea of what can happen?
Am I making a mistake in extracting data?

I can provide the original fit file for investigation.

Thank you.

@polyvertex
Copy link
Contributor

polyvertex commented May 30, 2018

Same issue here with an HRM-Tri strap.

I stumbled upon a working example after some head scratching:

from fitparse import FitFile

fitfile = FitFile('lagny_fim.fit')
for _, msg in enumerate(fitfile.get_messages('hr')):

    if msg.header.local_mesg_num == 12:
        last_event_timestamp = msg.get('event_timestamp').raw_value
        continue

    timestamps = []
    octets = msg.get('event_timestamp_12').raw_value
    idx = 0
    odd = 0

    while octets and idx < len(octets) - 1:
        last_event_timestamp12 = last_event_timestamp & 0xFFF;

        if not odd % 2:
            next_event_timestamp12 = octets[idx] + ((octets[idx + 1] & 0xF) << 8)
            last_event_timestamp = (last_event_timestamp & 0xFFFFF000) + next_event_timestamp12
        else:
            next_event_timestamp12 = 16 * octets[idx + 1] + ((octets[idx] & 0xF0) >> 4)
            last_event_timestamp = (last_event_timestamp & 0xFFFFF000) + next_event_timestamp12
            idx += 1

        if next_event_timestamp12 < last_event_timestamp12:
            last_event_timestamp += 0x1000

        timestamps.append(last_event_timestamp / 1024.0)

        idx += 1
        odd += 1

    print('%s,%s' % (
        '|'.join(str(bpm) for bpm in msg.get('filtered_bpm').value),
        '|'.join(str(t) for t in timestamps)
    ))

The way I understand how event_timestamp_12 must be decoded, each one of its components is meant to "replace" the last 12 bits of the raw value of the startat timestamp. Hence the 0xfffff000 mask.

There seem to be 2 issues here:

  • While fitparse parses the records correctly, it does not handle event_timestamp_12 properly. Since I'm not familiar enough with the whole protocol nor with fitparse, I'm not sure whether this field and its logic should be considered a special case or if fitparse's code in handling field components should be revised.

  • Also, I'm not sure why Fit SDK's Profiles.xlsx would list 10 event_timestamp components for field event_timestamp_12 (hr message), whereas it can only hold 8 of them. Leading then fitparse to blindly output 10 event_timestamp fields. Not sure whether there is a mistake in Profiles.xlsx or in the way it is interpreted by fitparse, for the lack of proper FIT specifications.

EDIT: improved code to avoid overflow errors (logic from GoldenCheetah)

@hugokernel
Copy link
Author

I will try that as soon as possible.
Thx

@pdura
Copy link
Contributor

pdura commented May 31, 2018

Hi polyvertex,
Thanks a lot for your feedback on this as I was also wondering how to get hr from my swim strap.
I'm still missing something there. How do you convert the timestamp you've got from this code to a real epoch timestamp ? I know the fit parser does it well but I can't manage to find where :/
I'm trying to add the UTC_REFERENCE but I still don't have a valid one.
Thanks a lot for your help there.

@polyvertex
Copy link
Contributor

Hey @pdura, hr messages are appended to the file during the post-activity synchronization stage. E.g. on Garmin devices, it's when you Save your activity. Then it is up to your file reader to apply the content of these late hr messages to the record messages you previously read.

Regarding the timestamps, fitparse, makes them available to you. fitdump is invaluable for this kind of questions...

polyvertex added a commit to polyvertex/python-fitparse that referenced this issue Jul 18, 2018
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants