Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Faker not available in stream api #493

Closed
maxmarcon opened this issue Sep 2, 2024 · 7 comments · Fixed by #496
Closed

Faker not available in stream api #493

maxmarcon opened this issue Sep 2, 2024 · 7 comments · Fixed by #496

Comments

@maxmarcon
Copy link
Contributor

maxmarcon commented Sep 2, 2024 •
edited
Loading

I would like to add a stream_maps -based transformation and use faker in it. The transformation fails because "'fake' is not defined" . Here's my stream_maps config:

 config:
    stream_maps:
       data_platform_raw-tbii_daily_gmv_nmv:
           at_channel_id: fake.pyint()
       faker_config:
           seed: 0

And here's the error's stacktrace:

2024-09-02 15:20:38,215 | ERROR    | tap-postgres.data_platform_raw-tbii_daily_gmv_nmv | An unhandled error occurred while syncing 'data_platform_raw-tbii_daily_gmv_nmv'
Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 554, in _eval_name
    return self.names[node.id]
KeyError: 'fake'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 350, in _eval
    result: str | int | float = self.expr_evaluator.eval(
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 664, in eval
    return super(EvalWithCompoundTypes, self).eval(expr, previously_parsed)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 428, in eval
    return self._eval(previously_parsed or self.parse(expr))
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 443, in _eval_expr
    return self._eval(node.value)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 527, in _eval_call
    func = self._eval(node.func)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 589, in _eval_attribute
    node_evaluated = self._eval(node.value)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 567, in _eval_name
    raise NameNotDefined(node.id, self.expr)
simpleeval.NameNotDefined: 'fake' is not defined for expression 'fake.pyint()'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1214, in sync
    for _ in self._sync_records(context=context):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1139, in _sync_records
    self._write_record_message(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 889, in _write_record_message
    for record_message in self._generate_record_messages(record):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 873, in _generate_record_messages
    mapped_record = stream_map.transform(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 285, in transform
    transformed_record = self._transform_fn(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 596, in transform_fn
    result[prop_key] = self._eval(
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 356, in _eval
    raise MapExpressionError(msg) from ex
singer_sdk.exceptions.MapExpressionError: Failed to evaluate simpleeval expressions fake.pyint().
Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 554, in _eval_name
    return self.names[node.id]
KeyError: 'fake'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 350, in _eval
    result: str | int | float = self.expr_evaluator.eval(
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 664, in eval
    return super(EvalWithCompoundTypes, self).eval(expr, previously_parsed)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 428, in eval
    return self._eval(previously_parsed or self.parse(expr))
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 443, in _eval_expr
    return self._eval(node.value)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 527, in _eval_call
    func = self._eval(node.func)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 589, in _eval_attribute
    node_evaluated = self._eval(node.value)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 567, in _eval_name
    raise NameNotDefined(node.id, self.expr)
simpleeval.NameNotDefined: 'fake' is not defined for expression 'fake.pyint()'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(TapPostgres.cli())
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/plugin_base.py", line 82, in invoke
    return super().invoke(ctx)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 507, in invoke
    tap.sync_all()
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 466, in sync_all
    stream.sync()
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1214, in sync
    for _ in self._sync_records(context=context):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1139, in _sync_records
    self._write_record_message(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 889, in _write_record_message
    for record_message in self._generate_record_messages(record):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 873, in _generate_record_messages
    mapped_record = stream_map.transform(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 285, in transform
    transformed_record = self._transform_fn(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 596, in transform_fn
    result[prop_key] = self._eval(
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 356, in _eval
    raise MapExpressionError(msg) from ex
singer_sdk.exceptions.MapExpressionError: Failed to evaluate simpleeval expressions fake.pyint().

The docs say:

Config for the Faker instance variable fake used within map expressions. Only applicable if the plugin specifies faker as an addtional dependency (through the singer-sdk faker extra or directly).

I tried installing the faker extra dep:

- name: tap-postgres
    variant: meltanolabs
    pip_url: meltanolabs-tap-postgres[faker] git+https://github.com/MeltanoLabs/tap-postgres.git@v0.0.11

But I get the same error.

How can I access faker for the stream_maps transformation with tap-postgres?

Meltano: 3.4.1
Python: 3.9.18
@edgarrmondragon
Copy link
Member

v0.0.12 was published

@edgarrmondragon
Copy link
Member

Related:

@edgarrmondragon edgarrmondragon mentioned this issue Sep 2, 2024
Closed
@maxmarcon
Copy link
Contributor Author

maxmarcon commented Sep 3, 2024 •
edited
Loading

Thanks for publishing the new version. I tried 0.0.12 but I'm unfortunately getting the same error, nothing changed 😞

Not a meltano or poetry expert by any means, but shouldn't the faker extra dep be listed here perhaps?

@edgarrmondragon
Copy link
Member

Thanks for publishing the new version. I tried 0.0.12 but I'm unfortunately getting the same error, nothing changed 😞

Not a meltano or poetry expert by any means, but shouldn't the faker extra dep be listed here perhaps?

@maxmarcon can you confirm that MeltanoLabs/target-postgres#424 has what you need?

@maxmarcon
Copy link
Contributor Author

@edgarrmondragon I'm confused. I need tap-postgres but you sent me a link to target-postgres ? Thanks

@maxmarcon maxmarcon mentioned this issue Sep 4, 2024
Merged
@maxmarcon
Copy link
Contributor Author

maxmarcon commented Sep 4, 2024 •
edited
Loading

This is the one-line change we need to fix the problem:
#496
I verified that it works

@edgarrmondragon
Copy link
Member

@edgarrmondragon I'm confused. I need tap-postgres but you sent me a link to target-postgres ? Thanks

Woops, I got confused 😅. Let me take a look at your PR.

edgarrmondragon pushed a commit that referenced this issue Sep 5, 2024
@maxmarcon
packaging: Add faker extra dependency to singer-sdk (#496)
8cc55cd 
Fixes: #493

---------

Co-authored-by: mmarcon <massimiliano.marcon.external@tradebyte.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
MeltanoLabs Overview
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

packaging: Add faker extra dependency to singer-sdk maxmarcon/tap-postgres
2 participants
@maxmarcon @edgarrmondragon