[TRIS-722] Custom Tagging #2144

RikishK · 2024-11-13T23:25:51Z

Options to add custom tags are in the order:

environment variable dictionary,
step functions cli create flag,
batch decorator variable dictionary

This PR builds ontop of this PR: #1628 - this original PR did not work when testing but I would like to credit it here as some of the changes are still used.

RikishK · 2024-11-25T04:24:23Z

Testing

Setup:

Environment variable:

{ name: 'METAFLOW_BATCH_EMIT_TAGS', value: 'True' },
{ name: 'METAFLOW_BATCH_DEFAULT_TAGS', value: '{"default_greeting": "default_world"}' }

Step functions create command:

step-functions create --aws-tags "hello=world"

Flow:

from metaflow import (
    FlowSpec,
    step,
    batch,
    retry,
    schedule,
    project,
    conda,
    current
)
import logging
import logging_setup

logging_setup.configure()


custom_step_tags = {
    "goodbye": "world",
    "hello": "universe",
   #"inv@lid": "t@g"
    }

@project(name="ml_apac_trisolaris")
@schedule(hourly=True)
class CanarySimpleAdhoc(FlowSpec):
    @step
    def start(self):
        self.logger = logging.getLogger(self.__class__.__name__)
        self.logger.info(f"Canary Hello")
        self.next(self.hello)

    @batch(cpu=1, memory=500, tags=custom_step_tags)
    @retry
    @step
    def hello(self):
        self.logger.info("Canary Hello World in Batch!")
        self.next(self.end)

    @step
    def end(self):
        self.logger.info("HelloAWS is finished.")


if __name__ == "__main__":
    CanarySimpleAdhoc()

Output/Result:

Start step batch tags:

Hello step batch tags:

Additional notes:
Tag validation was tested with "inv@lid": "t@g" which caused an error as expected.

saikonen

notes for testing, the config variable

METAFLOW_BATCH_EMIT_TAGS=True

needs to be set in order for tags to have any effect (even validation).

For a better UX, there could be a check somewhere in the path that if aws tags are being set, but emit tags is set to false, we print a warning on this.

Validation could also maybe be run despite the emit_tags feature flag, in order to catch issues.

metaflow/plugins/aws/batch/batch_decorator.py

saikonen · 2025-02-04T19:15:01Z

metaflow/plugins/aws/batch/batch_decorator.py

we need to add similar logic that is present in f.ex. the kubernetes_decorator.py#runtime_step_cli which json.dumps some attributes to the cli_args.commands, namely the new aws_tags in this case.

otherwise the passed in value will be not handled correctly in the CLI

for example replacing the cli_args.commands.update(self.attributes) with the following should work:

# dictionaries need to be json dumped for the CLI json_types = ["aws_tags"] for k, v in self.attributes.items(): if k in json_types: cli_args.command[k] = json.dumps(v) else: cli_args.command_options[k] = v

also for the batch_cli.py side, the --aws_tags option needs to be

@click.option("--aws-tags", multiple=False, type=JSONTypeClass(), default=None, help="AWS tags.")

in order to convert types correctly. note the multiple=False also, was there a reason to have the plural option act as multiple?

We wanted it to be a multiple arg so --aws-tags can be used like: --aws-tags "hello=world" --aws-tags "something=else"

This makes it a bit easier to generate the command based on specs. But I'm open to changing if it makes it easier to setup.

saikonen · 2025-02-04T19:41:44Z

metaflow/plugins/aws/batch/batch.py

+
+            if not isinstance(BATCH_DEFAULT_TAGS, dict):
+                raise BatchException(
+                    "The BATCH_DEFAULT_TAGS config option must be a dictionary of key-value tags."
+                )
+
+            for name, value in BATCH_DEFAULT_TAGS.items():
+                aws_tag = {'key': name, 'value': value}
+                validate_aws_tag(aws_tag)
+                job.tag(name, value)
+
+            if step_function_tags is not None:
+                aws_tags_list = []
+                for tag in step_function_tags:
+                    key_value = tag.split("=", 1)
+                    if len(key_value) == 2:
+                        aws_tags_list.append({
+                            'key': key_value[0],
+                            'value': key_value[1]
+                            })
+                for tag in aws_tags_list:
+                    validate_aws_tag(tag)
+                    job.tag(tag['key'], tag['value'])
+
+            if aws_tags is not None:
+                if not isinstance(aws_tags, dict):
+                    raise BatchException("tags must be a dictionary of key-value tags.")
+                decorator_aws_tags_list = [
+                    {'key': key,
+                        'value': val} for key, val in aws_tags.items()
+                ]
+                for tag in decorator_aws_tags_list:
+                    validate_aws_tag(tag)
+                    job.tag(tag['key'], tag['value'])
+


I didn't test this part thoroughly, but there is one significant caveat with running validations in the batch setup phase, namely: when validation fails, the metadata for the run/step/task have already been registered in this case, leading to a failed task with no logs.

An alternative approach with a better UX would be to perform batch tag validation inside the batch_decorator.py#step_init, which will happen before job submission. This allows for erroring out fairly quickly due to malformed tags.

Is there any issue with moving the validation and the BATCH_DEFAULT_TAGS usage into the decorator completely?

saikonen · 2025-02-04T19:44:51Z

also can you verify after the changes that the following flows run as expected? both with batch, and step-functions

expected to fail

from metaflow import step, FlowSpec, batch

class BatchTags(FlowSpec):
    @batch(aws_tags={"inv@lid": "t@g"})
    @step
    def start(self):
        print("Starting 👋")
        self.next(self.end)

    @step
    def end(self):
        print("Done! 🏁")


if __name__ == "__main__":
    BatchTags()

expected to run

from metaflow import (
    FlowSpec,
    step,
    batch,
)

custom_step_tags = {
    "goodbye": "world",
    "hello": "universe",
    }

class BatchTags2(FlowSpec):
    @step
    def start(self):
        self.next(self.hello)

    @batch(aws_tags=custom_step_tags)
    @step
    def hello(self):
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    BatchTags2()

saikonen · 2025-02-11T21:34:59Z

metaflow/plugins/aws/batch/batch_decorator.py

+        if self.attributes["aws_tags"] is not None:
+            if not isinstance(self.attributes["aws_tags"], dict) and not all(isinstance(k, str) and isinstance(v, str) for k, v in self.attributes["aws_tags"].items()):
+                raise BatchException("aws_tags must be Dict[str, str]")
+            decorator_aws_tags_list = [
+                {'key': key,
+                    'value': val} for key, val in self.attributes["aws_tags"].items()
+            ]
+            for tag in decorator_aws_tags_list:
+                validate_aws_tag(tag)
+


Can you move these under step_init instead to get cleaner formatting for exceptions. Unless there is some restriction that requires them to be in __init__

saikonen · 2025-02-11T21:35:54Z

metaflow/plugins/aws/batch/batch_decorator.py

 from ..aws_utils import (
    compute_resource_attributes,
    get_docker_registry,
    get_ec2_instance_metadata,
 )
+from .batch import BatchException
+from metaflow.tagging_util import validate_tags, validate_aws_tag


validate_tags is unused.

saikonen · 2025-02-11T21:42:50Z

metaflow/plugins/aws/batch/batch.py

+            for name, value in BATCH_DEFAULT_TAGS.items():
+                aws_tag = {'key': name, 'value': value}
+                validate_aws_tag(aws_tag)
+                job.tag(name, value)


merging the default values from BATCH_DEFAULT_TAGS could be moved into the batch_decorator. This way we have a single source to read tags from, and validation can be performed there as well.

saikonen · 2025-02-11T21:53:17Z

metaflow/plugins/aws/batch/batch.py

+        aws_tags=None,
+        step_function_aws_tags=None,


do we need the step_function_aws_tags? as a comparison, the way custom labels/annotations are implemented for argo workflows&kubernetes we made a choice to treat the kubernetes decorator as the single source of truth.

right now it seems that step-functions create --aws-tags "test=tag" will only apply the tags to the batch jobs spun up by the state machine. There already exists a mechanism for applying tags to those without the new CLI option though, so keeping things simple might be appropriate at this point.

# tagging all steps python test.py --with batch:aws_tags='{"test":"tag"}' step-functions create # or set the environment variable METAFLOWBATCH_DEFAULT_TAGS='{"envtag":"testvalue"}' python test.py step-functions create # tagging a specific step, simply add a batch deco to the step before deploying @batch(aws_tags={"tracking":"test_step"})

removing the step-function level --aws-tags option would reduce possible confusion about the scope of the option. One might think that it would apply tags to the state machine as well, which at the moment is not the case.

My suggestion is that we should keep this PR scoped to tagging batch jobs and revisit the need to tag other AWS resources as required, unless you already have a need for that?
Candidates for tagging would include the state machine, and eventbridge rules, but deciding how to configure these on the Metaflow side is nontrivial

saikonen

Opened a stacked PR in your fork to address some of the low hanging fruit. zendesk#8

saikonen · 2025-02-18T10:48:39Z

metaflow/plugins/aws/batch/batch_decorator.py

@@ -71,6 +73,9 @@ class BatchDecorator(StepDecorator):
        A swappiness value of 0 causes swapping not to happen unless absolutely
        necessary. A swappiness value of 100 causes pages to be swapped very
        aggressively. Accepted values are whole numbers between 0 and 100.
+    aws_batch_tags: Dict[str, str], optional


calling this aws_batch_tags seems a bit excessive given that we're inside the Batch decorator. Could it simply be aws_tags as before, or was there some issue with that?

saikonen · 2025-02-18T10:53:07Z

metaflow/plugins/aws/batch/batch_decorator.py

+            self.attributes["aws_batch_tags"] = BATCH_DEFAULT_TAGS
+
+        if self.attributes["aws_batch_tags"] is not None:
+            if not isinstance(self.attributes["aws_tags"], dict) and not all(isinstance(k, str) and isinstance(v, str) for k, v in self.attributes["aws_batch_tags"].items()):


should be "aws_batch_tags" in the check. now its throwing an error.

saikonen · 2025-02-18T11:03:18Z

metaflow/plugins/aws/batch/batch_decorator.py

+                raise BatchException("aws_batch_tags must be Dict[str, str]")
+
+            batch_default_tags_copy = BATCH_DEFAULT_TAGS.copy()
+            self.attributes["aws_batch_tags"] = batch_default_tags_copy.update(self.attributes["aws_batch_tags"]) 


this is leading to the aws_batch_tags being None in the default case and failing in the following dict comprehension. Can it be simplified into

self.attributes["aws_batch_tags"] = { **BATCH_DEFAULT_TAGS, **self.attributes["aws_batch_tags"] }

pending ordering. Should env take precedence over decorator values, or the other way around?

saikonen · 2025-02-18T11:05:40Z

metaflow/plugins/aws/batch/batch_decorator.py

+            batch_default_tags_copy = BATCH_DEFAULT_TAGS.copy()
+            self.attributes["aws_batch_tags"] = batch_default_tags_copy.update(self.attributes["aws_batch_tags"]) 
+
+            decorator_aws_tags_list = [


why is this conversion needed? would a dict not suffice?

saikonen · 2025-02-18T12:09:15Z

metaflow/plugins/aws/batch/batch_decorator.py

@@ -179,6 +185,26 @@ def __init__(self, attributes=None, statically_defined=False):
        if self.attributes["trainium"] is not None:
            self.attributes["inferentia"] = self.attributes["trainium"]

+        if not isinstance(BATCH_DEFAULT_TAGS, dict) and not all(instance(k, str) and isinstance(v, str) for k, v in BATCH_DEFAULT_TAGS.items()):


isinstance, not instance

saikonen · 2025-02-18T12:58:59Z

metaflow/tagging_util.py

@@ -29,6 +30,12 @@ def is_utf8_decodable(x):
 # How long may an individual tag value be
 MAX_TAG_SIZE = 500

+def validate_aws_tag(tag):
+    validate_tags([tag['key'], tag['value']])


validate_tags is for metaflow specific metadata tags validation. Some of the rules, like max tag set size probably do not apply to aws tags. Does it make sense to reuse the metaflow tag validation here, or can we do without?

It makes more sense to me that the AWS tag validation be kept completely separate

was this the reason that the tags were turned into a more complex type?

Fixes Netflix#1627

This reverts commit 65fd888.

…ing (RikishK: Edited, only taking the decorator changes from here )

RikishK · 2025-03-12T00:52:57Z

Testing

Flow:

Here we set the decorator level tags. NOTE: the invalid tag is commented out for this test

from metaflow import (
    FlowSpec,
    step,
    batch,
    retry,
    schedule,
    project,
    conda,
    current
)
import logging
from common import (
    logging_setup
)

logging_setup.configure()


custom_step_tags = {
    "goodbye": "world",
    "hello": "universe",
    "batch_decorator_tag": "True",
    #"inv@lid": "t@g"
}

custom_step_tags_v2 = {
    "something": "else",
    "batch_decorator_tag": "True"
}

@project(name="ml_apac_trisolaris")
@schedule(hourly=True)
class CanarySimpleAdhoc(FlowSpec):
    @step
    def start(self):
        self.logger = logging.getLogger(self.__class__.__name__)
        self.logger.info(f"Canary Hello")
        self.next(self.hello)

    @batch(cpu=1, memory=500, aws_batch_tags=custom_step_tags)
    @retry
    @step
    def hello(self):
        self.logger.info("Canary Hello World in Batch!")
        self.next(self.goodbye)

    @batch(cpu=1, memory=500, aws_batch_tags=custom_step_tags_v2)
    @retry
    @step
    def goodbye(self):
        self.logger.info("Canary Goodbye World in Batch!")
        self.next(self.end)


    @step
    def end(self):
        self.logger.info("HelloAWS is finished.")


if __name__ == "__main__":
    CanarySimpleAdhoc()

Step functions creation command

Here we set the step functions cli level tags

poetry run python ${pipeline_full_path} --with retry --datastore=s3 step-functions create --aws-batch-tags "hello=world" --aws-batch-tags "cli-tag=true"

Env vars tags

Here we set the environment variables level tags

{ name: 'METAFLOW_BATCH_DEFAULT_TAGS', value: '{"default_greeting": "default_world", "env_var_aws_tag": "True"}' }

Expectations

We should see env vars tags and step functions cli tags being set on batch jobs. Batch jobs with decorator tags will apply their decorator tags last and override env vars and step functions cli tags.

Output Batch Jobs

Start Step:

Hello Step

Goodbye step

End step

RikishK · 2025-03-12T04:07:03Z

Testing Invalid Tags

Testing with the same flow as previous test but with the invalid tag enabled.

Flow

import logging
from common import (
    logging_setup
)

logging_setup.configure()


custom_step_tags = {
    "goodbye": "world",
    "hello": "universe",
    "batch_decorator_tag": "True",
    "inv@lid": "t@g"
}

custom_step_tags_v2 = {
    "something": "else",
    "batch_decorator_tag": "True"
}

@project(name="ml_apac_trisolaris")
@schedule(hourly=True)
class CanarySimpleAdhoc(FlowSpec):
    @step
    def start(self):
        self.logger = logging.getLogger(self.__class__.__name__)
        self.logger.info(f"Canary Hello")
        self.next(self.hello)

    @batch(cpu=1, memory=500, aws_batch_tags=custom_step_tags)
    @retry
    @step
    def hello(self):
        self.logger.info("Canary Hello World in Batch!")
        self.next(self.goodbye)

    @batch(cpu=1, memory=500, aws_batch_tags=custom_step_tags_v2)
    @retry
    @step
    def goodbye(self):
        self.logger.info("Canary Goodbye World in Batch!")
        self.next(self.end)


    @step
    def end(self):
        self.logger.info("HelloAWS is finished.")


if __name__ == "__main__":
    CanarySimpleAdhoc()

Output of trying to deploy stepfunction:

"app":"WaitForEnvoy","commit":"","level":"info","msg":"WaitForEnvoy starting up","pid":1,"time":"2025-03-12T03:56:12Z","version":"v1.1.1"}
[pipeline-deploy] Start
[pipeline-deploy] Run Environment [staging]
[pipeline-deploy] AWS_REGION [us-east-1]
[pipeline-deploy] AWS_DEFAULT_REGION [us-east-1]
[pipeline-deploy] Setting USERNAME environment variable to [staging]
[pipeline-deploy] Deploying [canary/canary_simple_adhoc.py] - full path [/app/canary/canary_simple_adhoc.py]
+ poetry run python /app/canary/canary_simple_adhoc.py --with retry --datastore=s3 step-functions create --aws-batch-tags hello=world --aws-batch-tags cli-tag=true
Skipping virtualenv creation, as specified in config file.
Metaflow 2.12.30 executing CanarySimpleAdhoc for user:staging
Project: ml_apac_trisolaris, Branch: user.staging
    Tagging error:
    Tags must match pattern: ^[\w\s._-]+$

BATCH_DEFAULT_TAGS: {'default_greeting': 'default_world', 'env_var_aws_tag': 'True'}
aws_batch_tags: {'goodbye': 'world', 'hello': 'universe', 'batch_decorator_tag': 'True', 'inv@lid': 't@g'}
Generating aws compatible list. Old aws_batch_tags: {'default_greeting': 'default_world', 'env_var_aws_tag': 'True', 'goodbye': 'world', 'hello': 'universe', 'batch_decorator_tag': 'True', 'inv@lid': 't@g'}, new generated list: [{'key': 'default_greeting', 'value': 'default_world'}, {'key': 'env_var_aws_tag', 'value': 'True'}, {'key': 'goodbye', 'value': 'world'}, {'key': 'hello', 'value': 'universe'}, {'key': 'batch_decorator_tag', 'value': 'True'}, {'key': 'inv@lid', 'value': 't@g'}]
BATCH_DEFAULT_TAGS: {'default_greeting': 'default_world', 'env_var_aws_tag': 'True'}
aws_batch_tags: {'something': 'else', 'batch_decorator_tag': 'True'}
Generating aws compatible list. Old aws_batch_tags: {'default_greeting': 'default_world', 'env_var_aws_tag': 'True', 'something': 'else', 'batch_decorator_tag': 'True'}, new generated list: [{'key': 'default_greeting', 'value': 'default_world'}, {'key': 'env_var_aws_tag', 'value': 'True'}, {'key': 'something', 'value': 'else'}, {'key': 'batch_decorator_tag', 'value': 'True'}]

joecorcoran · 2025-04-01T07:47:48Z

Love this PR – we could really benefit from it. Is there anything we can do to help get it merged? ❤️

saikonen · 2025-04-01T14:29:35Z

metaflow/plugins/aws/step_functions/step_functions_cli.py

@@ -97,6 +98,12 @@ def step_functions(obj, name=None):
    "with the given tag. You can specify this option multiple "
    "times to attach multiple tags.",
 )
+@click.option(
+    "--aws-batch-tags",


If we are to keep this, the flag itself should be singular in my opinion as it is used to set one tag at a time

--aws-batch-tag some=tag --aws-batch-tag another=tag

RikishK force-pushed the fsat/pr-1627--rebased--2.12.28 branch 6 times, most recently from 2810093 to 0d45ad5 Compare November 21, 2024 05:13

RikishK force-pushed the fsat/pr-1627--rebased--2.12.28 branch from 0d45ad5 to 4c5cebf Compare November 22, 2024 02:38

saikonen reviewed Jan 28, 2025

View reviewed changes

metaflow/plugins/aws/batch/batch_decorator.py Outdated Show resolved Hide resolved

metaflow/plugins/aws/batch/batch_decorator.py Outdated Show resolved Hide resolved

saikonen requested changes Feb 4, 2025

View reviewed changes

saikonen reviewed Feb 4, 2025

View reviewed changes

saikonen requested changes Feb 11, 2025

View reviewed changes

RikishK force-pushed the fsat/pr-1627--rebased--2.12.28 branch 2 times, most recently from bdf3930 to 2040a7c Compare February 16, 2025 23:25

saikonen reviewed Feb 18, 2025

View reviewed changes

RikishK force-pushed the fsat/pr-1627--rebased--2.12.28 branch 6 times, most recently from ddcfefb to 49793ff Compare February 24, 2025 23:25

Limess and others added 8 commits February 25, 2025 10:40

Add capability to add custom tags to batch

3af75fb

Fixes Netflix#1627

step functions aws_tags

c100fc7

type and debug help

765698e

Revert "better error message with dump (Netflix#2130)" (Netflix#2141)

ae8db7c

This reverts commit 65fd888.

bump version to 2.12.30 (Netflix#2142)

27841b4

import fix

81628ab

dont pass tags to create_job

aa5a109

no aws tags type

68a7f0d

RikishK added 3 commits February 25, 2025 10:40

Explicitly grab metaflow_config.BATCH_DEFAULT_TAGS

170d507

use metaflow_config.

87b5a8e

Debug

985974b

RikishK force-pushed the fsat/pr-1627--rebased--2.12.28 branch 5 times, most recently from f3da850 to b42e360 Compare February 25, 2025 03:22

Fixup

1442ecf

RikishK force-pushed the fsat/pr-1627--rebased--2.12.28 branch from b42e360 to 1442ecf Compare February 25, 2025 03:59

RikishK and others added 6 commits March 3, 2025 20:22

Better logging to identify issues

dc602db

correct keying. also apply black

268ad74

fix default case for batch deco tags

a69ab2a

tidy up errors

954a127

fixes to batch deco

801801d

clean up batch tag handling. add json serialization to CLI value pass…

6cafebe

…ing (RikishK: Edited, only taking the decorator changes from here )

RikishK force-pushed the fsat/pr-1627--rebased--2.12.28 branch from a6faf9a to e791a6d Compare March 10, 2025 05:33

Debug

7b89b81

RikishK force-pushed the fsat/pr-1627--rebased--2.12.28 branch from e791a6d to 7b89b81 Compare March 10, 2025 05:48

RikishK added 3 commits March 10, 2025 17:13

Default empty dict

806d593

Test no json serialisation of aws_batch_tags

7a05723

Logging during generation of aws compatible tags list

619a900

RikishK force-pushed the fsat/pr-1627--rebased--2.12.28 branch from 61ac55b to 25eae34 Compare March 11, 2025 05:40

Temporary isolate conversion for better debugging

475e282

RikishK force-pushed the fsat/pr-1627--rebased--2.12.28 branch from 25eae34 to 475e282 Compare March 11, 2025 23:25

savingoyal self-requested a review March 27, 2025 16:06

saikonen reviewed Apr 1, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TRIS-722] Custom Tagging #2144

[TRIS-722] Custom Tagging #2144

RikishK commented Nov 13, 2024 •

edited

Loading

RikishK commented Nov 25, 2024

saikonen left a comment

saikonen Feb 4, 2025

saikonen Feb 4, 2025

saikonen Feb 4, 2025

RikishK Mar 9, 2025

saikonen Feb 4, 2025

saikonen commented Feb 4, 2025

saikonen Feb 11, 2025

saikonen Feb 11, 2025

saikonen Feb 11, 2025

saikonen Feb 11, 2025

saikonen Feb 11, 2025

saikonen left a comment

saikonen Feb 18, 2025

saikonen Feb 18, 2025

saikonen Feb 18, 2025

saikonen Feb 18, 2025

saikonen Feb 18, 2025

saikonen Feb 18, 2025

saikonen Feb 18, 2025

RikishK commented Mar 12, 2025 •

edited

Loading

RikishK commented Mar 12, 2025

joecorcoran commented Apr 1, 2025

saikonen Apr 1, 2025

[TRIS-722] Custom Tagging #2144

Are you sure you want to change the base?

[TRIS-722] Custom Tagging #2144

Conversation

RikishK commented Nov 13, 2024 • edited Loading

RikishK commented Nov 25, 2024

Testing

saikonen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saikonen commented Feb 4, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saikonen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RikishK commented Mar 12, 2025 • edited Loading

Testing

Flow:

Step functions creation command

Env vars tags

Expectations

Output Batch Jobs

Start Step:

Hello Step

Goodbye step

End step

RikishK commented Mar 12, 2025

Testing Invalid Tags

Flow

Output of trying to deploy stepfunction:

joecorcoran commented Apr 1, 2025

Choose a reason for hiding this comment

RikishK commented Nov 13, 2024 •

edited

Loading

RikishK commented Mar 12, 2025 •

edited

Loading