Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

bug: CustomStreamMap always convert datetime field to date #2739

Closed
1 task done
lngvietthang opened this issue Nov 4, 2024 · 1 comment · Fixed by #2741
Closed
1 task done

bug: CustomStreamMap always convert datetime field to date #2739

lngvietthang opened this issue Nov 4, 2024 · 1 comment · Fixed by #2741
Assignees
Labels
kind/Bug Something isn't working Stream Maps
Milestone

Comments

@lngvietthang
Copy link

Singer SDK Version

0.41.0

Is this a regression?

  • Yes

Python Version

3.9

Bug scope

Mapping (stream maps, flattening, etc.)

Operating System

Linux

Description

In my Meltano configuration, I use meltano-map-transformer (latest version with singer-sdk@0.41.0) with config as below:

stream_maps:
  xxx:
    creation: datetime.datetime.strptime(creation, '%Y-%m-%dT%H:%M:%S.%f%z')
...

With new update from singer-sdk v0.41, this field (creation) is mapped with date type in schema message.

class CustomStreamMap(StreamMap):
    def _eval_type(  # noqa: PLR0911
        self,
        expr: str,
        default: th.JSONTypeHelper | None = None,
    ) -> th.JSONTypeHelper:
        ...
        if expr.startswith("datetime.date") or expr.endswith(".date()"):
            return th.DateType()

        if expr.startswith("datetime.datetime"):
            return th.DateTimeType()
        ....

I think we need to swap the position of 2 if clause to check the datetime data type firstly.

Related PR:
#2665

Code

class CustomStreamMap(StreamMap):
def _eval_type( # noqa: PLR0911
self,
expr: str,
default: th.JSONTypeHelper | None = None,
) -> th.JSONTypeHelper:
...
if expr.startswith("datetime.date") or expr.endswith(".date()"):
return th.DateType()

    if expr.startswith("datetime.datetime"):
        return th.DateTimeType()
    ....

Link to Slack/Linen

No response

@edgarrmondragon
Copy link
Collaborator

I think we need to swap the position of 2 if clause to check the datetime data type firstly.

That makes sense @lngvietthang. PRs welcome!

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
kind/Bug Something isn't working Stream Maps
Projects
None yet
2 participants