feat(config/toml): support jinja templates in toml files #33123

jiaweikho · 2024-12-15T15:51:20Z

Changes

Massage toml content before parsing by removing jinja delimiters according to the specification: jinja
Add removeTemplate option to the parseToml function
Add removeTemplate: true config to poetry and pep621 pyproject.toml parsers.

Context

This addresses the feature request raised in #33105 by applying a similar method in the yaml parser in #26682.

Documentation (please check one with an [x])

I have updated the documentation, or
No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

Code inspection only, or
Newly added/modified unit tests, or
No unit tests but ran on a real repository, or
Both unit tests + ran on a real repository

Following the specification in https://jinja.palletsprojects.com/en/stable/templates/#synopsis, there are three types of delimiters: 1. Statements 2. Expressions 3. Comments

lib/util/toml.ts

Co-authored-by: Michael Kriese <michael.kriese@gmx.de>

jiaweikho · 2024-12-15T17:51:34Z

@viceice thanks for the review. I haven't documented this new functionality yet. Do you think I should add a note in the poetry/pep621 docs about jinja templates?

jiaweikho · 2024-12-16T10:46:23Z

@viceice I've added tests for the pep621 and poetry managers based on the CI coverage feedback. I've tried my best to follow the precedent but the style of extract.spec.ts in pep621 and poetry are quite different so you may notice some style differences between them.

rarkins

Please inline the fixtures as part of the tests

jiaweikho · 2024-12-23T12:13:32Z

Hi @viceice @rarkins, is there anything else you would like me to do before this PR can be merged?

viceice

we're currently not merging features because all maintainer are in Christmas holiday and we don't want to break anything now.

jiaweikho · 2024-12-23T12:27:38Z

we're currently not merging features because all maintainer are in Christmas holiday and we don't want to break anything now.

Thanks for letting me know. Have a wonderful holiday!

sigma67 · 2025-01-15T17:12:02Z

Is this still being considered? Seems it got lost over New Year's

jiaweikho · 2025-01-17T21:34:28Z

@viceice could you please have another look at this and let me know if I can help in any way.

viceice · 2025-01-31T09:01:11Z

lib/util/schema-utils.ts

@@ -258,7 +258,7 @@ export const MultidocYaml = z.string().transform((str, ctx): JsonArray => {

 export const Toml = z.string().transform((str, ctx) => {
  try {
-    return parseToml(str);
+    return parseToml(str, { removeTemplates: true });


I don't like to do this by default. @zharinov how to best handle such options 🤔

Maybe we should export a toml function which allows to pass parder options? would this help for yaml on gitlab too?

~~I think it isn't necessary to couple with TOML parsing.~~

~~Let's use stripTemplates(parseToml(...)) or StripTemplates.pipe(Toml).pipe(z.object(...)) instead.~~

Yes, these transforms could lead to errors that are hard to detect.
Let's keep it localized to the Python ecosystem:

feat: String utility for stripping templates #34447

sigma67 · 2025-02-24T08:34:33Z

How can we proceed here? Is this waiting for response from @zharinov ?

zharinov · 2025-02-25T11:26:17Z

Superseded by feat: String utility for stripping templates #34447

sigma67 · 2025-02-25T13:08:38Z

@zharinov I tried latest renovate with our pyproject.toml.jinja file, but unfortunately it still fails

{
    "err":
    {
        "column": 1,
        "index": 709,
        "lineNumber": 39,
        "message": "Unexpected token",
        "stack": "SyntaxError: Unexpected token
		    at Context.reportParseError (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/toml-eslint-parser/lib/toml-parser/context.js:114:15)
		    at TOMLParser.TABLE (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/toml-eslint-parser/lib/toml-parser/index.js:107:20)
		    at TOMLParser.parse (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/toml-eslint-parser/lib/toml-parser/index.js:77:51)
		    at parseForESLint (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/toml-eslint-parser/lib/parser.js:11:24)
		    at parseTOML (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/toml-eslint-parser/lib/index.js:46:40)
		    at parse (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/util/toml.ts:4:24)
		    at parsePyProject (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/modules/manager/pep621/utils.ts:140:30)
		    at Object.extractPackageFile (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/modules/manager/pep621/extract.ts:25:29)
		    at extractPackageFile (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/modules/manager/index.ts:75:9)
		    at getManagerPackageFiles (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/workers/repository/extract/manager-files.ts:58:43)
		    at /opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/workers/repository/extract/index.ts:57:28
		    at async Promise.all (index 3)
		    at extractAllDependencies (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/workers/repository/extract/index.ts:54:26)
		    at extract (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/workers/repository/process/extract-update.ts:160:28)
		    at extractDependencies (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/workers/repository/process/index.ts:158:26)
		    at Object.renovateRepository (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/workers/repository/index.ts:71:9)
		    at attributes.repository (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/workers/global/index.ts:206:11)
		    at start (/opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/workers/global/index.ts:191:7)
		    at /opt/containerbase/tools/renovate/39.180.2/22.11.0/node_modules/renovate/lib/renovate.ts:19:22"
    },
    "hostname": "runner-cqva6w8oe-project-1694-concurrent-0",
    "level": 20,
    "logContext": "XBuHFJk1JOOxotnc2vNXs",
    "msg": "Failed to parse and validate pyproject file",
    "name": "renovate",
    "packageFile": "templates/pyproject.toml.jinja",
    "pid": 17,
    "repository": "pes/common/pes-python-module-template",
    "time": "2025-02-25T12:58:37.776Z",
    "v": 0
}

this is the file

[project]
name = "{{ distribution_pkg_name }}"
dynamic = ["version"]
description = "{{ module_short_description }}"
readme = "README.md"
keywords = [
    "python",
    "module"
]
classifiers = [
    "Topic :: Software Development :: Libraries :: Python Modules",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12"
]
maintainers = [
    {name = "{{ author_first_name }} {{ author_surname }}", email = "{{ author_mail }}"},
]
authors = [
    {name = "{{ author_first_name }} {{ author_surname }}", email = "{{ author_mail }}"},
]
license = {file = "LICENSE"}

dependencies = [
    "pycowsay>=0.0.0.1",
    {%- if is_application %}
    "click>=8.1.7",
    {%- endif %}
]
requires-python = ">=3.10"

[project.urls]
Repository = "{{ gitlab_url }}"
{%- if is_application %}

[project.scripts]
{{ distribution_pkg_name }} = "{{ import_pkg_name }}.__main__:cli"
{%- endif %}

[build-system]
requires = ["pdm-backend==2.3.3", "pdm-build-locked==0.3.2"]
build-backend = "pdm.backend"


#########
# TOOLS #
#########

[tool.pdm.version]
source = "scm"

[tool.pdm]
plugins = [
    "pdm-build-locked"
]

[tool.pdm.build]
package-dir = "src"
excludes = ["tests/_temp"]
{%- if is_application %}
locked = true
{%- endif %}

sigma67 · 2025-03-03T20:30:39Z

Fixed with 39.185.6

jiaweikho changed the title ~~feat(config/toml) Support jinja templates in toml files~~ feat(config/toml) support jinja templates in toml files Dec 15, 2024

jiaweikho added 5 commits December 15, 2024 15:52

Add template toml test

069e313

Remove jinja delimiters

d912396

Following the specification in https://jinja.palletsprojects.com/en/stable/templates/#synopsis, there are three types of delimiters: 1. Statements 2. Expressions 3. Comments

Add removeTemplate option

87a1410

Remove jinja templates in pep62 pyproject

5561282

Remove jinja templates in poetry pyproject

0099a48

jiaweikho force-pushed the feat/toml-template-support branch from fa5430b to 0099a48 Compare December 15, 2024 15:52

jiaweikho changed the title ~~feat(config/toml) support jinja templates in toml files~~ feat(config/toml): support jinja templates in toml files Dec 15, 2024

viceice reviewed Dec 15, 2024

View reviewed changes

lib/util/toml.ts Outdated Show resolved Hide resolved

lib/util/toml.ts Outdated Show resolved Hide resolved

jiaweikho and others added 2 commits December 16, 2024 00:44

Export TomlOptions interface

82157c6

Co-authored-by: Michael Kriese <michael.kriese@gmx.de>

Use replaceAll instead of regex global modifier

21aea6a

Co-authored-by: Michael Kriese <michael.kriese@gmx.de>

jiaweikho added 4 commits December 16, 2024 09:10

Add regex global modifier

025d60b

Add poetry pyproject with template test

1b30b21

Remove templates when parsing toml schema

8ae8b92

Add pep621 pyproject template test

5c7a126

jiaweikho requested a review from viceice December 16, 2024 10:46

rarkins requested changes Dec 17, 2024

View reviewed changes

jiaweikho added 2 commits December 17, 2024 21:09

Inline poetry fixture

099efbb

Inline pep621 fixture

c8233e9

jiaweikho requested a review from rarkins December 17, 2024 21:12

Merge branch 'main' into feat/toml-template-support

2dc582b

viceice reviewed Dec 23, 2024

View reviewed changes

viceice self-requested a review December 23, 2024 12:18

viceice reviewed Jan 31, 2025

View reviewed changes

zharinov mentioned this pull request Feb 24, 2025

feat: String utility for stripping templates #34447

Merged

6 tasks

zharinov mentioned this pull request Feb 25, 2025

fix(poetry): Template keys handling #34460

Merged

6 tasks

viceice closed this Mar 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(config/toml): support jinja templates in toml files #33123

feat(config/toml): support jinja templates in toml files #33123

jiaweikho commented Dec 15, 2024

jiaweikho commented Dec 15, 2024

jiaweikho commented Dec 16, 2024

rarkins left a comment

jiaweikho commented Dec 23, 2024

viceice left a comment

jiaweikho commented Dec 23, 2024

sigma67 commented Jan 15, 2025

jiaweikho commented Jan 17, 2025

viceice Jan 31, 2025

zharinov Feb 24, 2025 •

edited

Loading

zharinov Feb 24, 2025

sigma67 commented Feb 24, 2025

zharinov commented Feb 25, 2025

sigma67 commented Feb 25, 2025 •

edited

Loading

sigma67 commented Mar 3, 2025

feat(config/toml): support jinja templates in toml files #33123

feat(config/toml): support jinja templates in toml files #33123

Conversation

jiaweikho commented Dec 15, 2024

Changes

Context

Documentation (please check one with an [x])

How I've tested my work (please select one)

jiaweikho commented Dec 15, 2024

jiaweikho commented Dec 16, 2024

rarkins left a comment

Choose a reason for hiding this comment

jiaweikho commented Dec 23, 2024

viceice left a comment

Choose a reason for hiding this comment

jiaweikho commented Dec 23, 2024

sigma67 commented Jan 15, 2025

jiaweikho commented Jan 17, 2025

viceice Jan 31, 2025

Choose a reason for hiding this comment

zharinov Feb 24, 2025 • edited Loading

Choose a reason for hiding this comment

zharinov Feb 24, 2025

Choose a reason for hiding this comment

sigma67 commented Feb 24, 2025

zharinov commented Feb 25, 2025

sigma67 commented Feb 25, 2025 • edited Loading

sigma67 commented Mar 3, 2025

zharinov Feb 24, 2025 •

edited

Loading

sigma67 commented Feb 25, 2025 •

edited

Loading