Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Adds resource field reference syntax to template strings #2210

Open
wants to merge 12 commits into
base: devel
Choose a base branch
from

Conversation

burnash
Copy link
Collaborator

@burnash burnash commented Jan 14, 2025

Resolves #2190
Implements #1880 by introducing incremental context object.

Copy link

netlify bot commented Jan 14, 2025

Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit 719136d
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/67afec435582720008e7a8f2

@burnash burnash force-pushed the feat/2190-simplify-dependent-resource-declaration branch from 40d5201 to 5c1e6b2 Compare January 14, 2025 16:45
@burnash burnash force-pushed the feat/2190-simplify-dependent-resource-declaration branch from 5c1e6b2 to 6ac4fdc Compare February 3, 2025 02:39
@burnash burnash requested a review from rudolfix February 3, 2025 09:01
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some comments to the code. Tests look good though. What is still missing:

  1. examples (github/demo pipeline - incremental) not converted to a new notation
  2. I do not see tests that use invalid placeholders
  • {unknown.attr}
  • {incremental.unknown}
  1. a test where we define several parent resources ie. {resources.issue.id} and {resources.posts.id} in the same resource. I think that there's universal check for those but it makes sense to check it

dlt/sources/rest_api/__init__.py Outdated Show resolved Hide resolved
dlt/sources/rest_api/__init__.py Outdated Show resolved Hide resolved
dlt/sources/rest_api/config_setup.py Outdated Show resolved Hide resolved
if isinstance(obj, str):
return obj.format(**placeholders)

if isinstance(obj, dict):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could replace recursive code with utils.map_nested_in_place and just provide lambda obj: obj.format() as mapping function

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer keeping the non-mutating way here rather than changing the dict in place.

dlt/sources/rest_api/config_setup.py Show resolved Hide resolved
dlt/sources/rest_api/config_setup.py Outdated Show resolved Hide resolved
dlt/sources/rest_api/config_setup.py Show resolved Hide resolved
dlt/sources/rest_api/config_setup.py Outdated Show resolved Hide resolved
- Add support for interpolating in query and json parameters
- Stricter validation for resource params
- Refactor path and parameter binding - Error handling for invalid resource params
- Generalize context validation for path and parameter interpolation
- Add parameterized test for incorrect interpolation scenarios
@burnash burnash requested a review from rudolfix February 15, 2025 01:34
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
2 participants