Filter gradients of IndexedSlices in tf grad sim backend, fixes #828 #829

mauicv · 2022-11-30T14:11:33Z

What is this:

Bugfix for #828.

Implementation:

Having spoken to @RobertSamoilescu we've agreed on the following plan of action:

If the tensor is in a sparse format we try to convert it to dense and then to np.ndarray. This will also need to be done for sparse representations in pytorch.
Finally we filter layers with trainable=False
If a user passes a model that has some layers set to trainable=False then we raise warning to make it clear that those layers won't be used in the computation. Note this should only happen in the model init call!
Check non-trainable warning behaviour for batch layers...
Raise informative error if specific layer doesn't convert to numpy
Instead of filtering for batch norm layers in trainable layers check, just list non-trainable layers in warning.

More Todo:

Ensure correct formatting of module and variable names in docstrings: numpy -> 'numpy'
Ensure public methods all have correct docstrings with return types etc...
Add note to FAQ explaining warnings

Issues:

If we fail to convert a sparse tensor to dense or the gradient tensor has no numpy attribute the original plan was to raise a warning informing the user that some tensors grads are not used.
- Having thought further about this I'm not sure about this? Seems like we'd end up catching a broad range of errors. In principle any model passed should have gradients computable for all layers so it's not clear that we should discard errors that might arise here!?
- If an error due to something like the above arises we can suggest the user set trainable to false on the layer in question while we figure out a fix? I'm open to doing this differently though!
- UPDATE: having spoken to @RobertSamoilescu, we've agreed it makes sense to raise error here if we can't the gradient to a numpy array. Other errors will not be attempted to be caught. The error will also suggest tot he user they mark the gradinet as non-trainable so they can still get results.
tensorflow has two representations of sparsity, 1. IndexedSlice which arises as gradients of embedding layers as we've seen and 2. SparceTensor. They have different methods of converting to dense representations for instance the below needs to be used:
```
if isinstance(grad, tf.SparseTensor):
  grad = tf.sparse.to_dense(grad)
if isinstance(grad, tf.IndexedSlices):
  grad = tf.convert_to_tensor(grad)
```
The issue I have is that I can't find any indication that tensorflow layers would ever have sparse gradients other than the IndexedSlices that results from tf.gather in the embedding layer. So I'm not sure the above case is needed. You certainly can't take gradients w.r.t. SparseTensors. Doing so yeilds:
```
ValueError: Type SparseTensor is not supported as a gradient source or gradient target.
```
@jklaise and @RobertSamoilescu can either of you think of any situation in which this would arise? For now, I'm going to remove that logic. If an error due to something like the above arises we can suggest the user set trainable to false on the layer in question while we figure out a fix
Checking weather or not models contain non-trainable layers is a little non trivial. For instance the batch_normalization layer has non-trainable weights and so if a user includes this in there model naive solutions would trigger this warning. I've set it to filter the batch norm layer as according to the tensorflow docs this is the only layer that is non-trainable. A user might create a custom layer which would throw warnings but I think that's ok as they should be alerted in this case...
UPDATE:: having spoken to @RobertSamoilescu we've agreed to remove the checks for batch normalization as this kind of approach is inherently non-exhaustive and instead replace with a more informative warning that lists all layers not included in grad sim computation!

…nIO#828

codecov · 2022-11-30T15:03:06Z

Codecov Report

Merging #829 (4bd1f20) into master (91a2a88) will increase coverage by 0.62%.
The diff coverage is 100.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #829      +/-   ##
==========================================
+ Coverage   84.65%   85.28%   +0.62%     
==========================================
  Files          74       74              
  Lines        8722     8759      +37     
==========================================
+ Hits         7384     7470      +86     
+ Misses       1338     1289      -49

Flag	Coverage Δ
macos-latest-3.10	`?`
ubuntu-latest-3.10	`85.23% <100.00%> (+0.62%)`	⬆️
ubuntu-latest-3.7	`85.03% <100.00%> (+0.51%)`	⬆️
ubuntu-latest-3.8	`85.17% <100.00%> (+0.62%)`	⬆️
ubuntu-latest-3.9	`85.17% <100.00%> (+0.62%)`	⬆️
windows-latest-3.9	`82.81% <100.00%> (+0.54%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
alibi/explainers/tests/test_simiarlity/conftest.py	`100.00% <ø> (ø)`
...ibi/explainers/similarity/backends/pytorch/base.py	`95.74% <100.00%> (+1.80%)`	⬆️
.../explainers/similarity/backends/tensorflow/base.py	`100.00% <100.00%> (ø)`
alibi/explainers/similarity/grad.py	`95.65% <100.00%> (+0.33%)`	⬆️
alibi/explainers/permutation_importance.py	`97.09% <0.00%> (-0.02%)`	⬇️
alibi/prototypes/protoselect.py	`93.36% <0.00%> (+25.99%)`	⬆️

jklaise · 2022-12-09T14:13:02Z

Before merging this we should answer a few questions.

As a background, this error was raised when trying to calculate GradientSimilarity for a transformer model on text instances with respect to token ids (as opposed to embeddings). The error is to be expected since the layer mapping token ids to embeddings is not differentiable; however, this solution proved to be an effective workaround - just get rid of the results which correspond to non differentiable layers.

On the other hand, one could make the claim that the original (error) is expected behaviour as the user should be aware that the model they're passing in has non-differentiable layers. However, fixing the model so that gradients are only taken from the embedding layer onwards proved to be very difficult in the specific case of transformers models which included workarounds splitting the main model pipeline in several steps.

Therefore, the proposed solution here seems like a good alternative. To discuss and investigate:

Can there be any potential issues when simply discarding results of non-differentiable layers like in this PR?
Should we raise a warning if this happens?
Is there a more principled way of filtering the results other than looking at IndexSlices (e.g. looking at the existence of gradients)?
The pytorch backend should have the same behaviour if we are to implement this PR.
@RobertSamoilescu @mauicv

mauicv · 2022-12-13T10:22:36Z

Can there be any potential issues when simply discarding results of non-differentiable layers like in this PR? Should we raise a warning if this happens?

I can't think of any obvious ones. In terms of raising warnings, I think we'd only really be dealing with the case where the user expects the similarity to be taken with respect to some layer but doesn't realise that it's not differentiable and so it isn't. I'm not sure it's our responsibility to correct this sort of misunderstanding but it certainly wouldn't hurt so we might as well. Perhaps we'd prevent some number of open issues by doing the above.

Is there a more principled way of filtering the results other than looking at IndexSlices (e.g. looking at the existence of gradients)?

Yes, thinking about it after the fact, IndexSlices was not the correct approach. We should filter on hasattr(layer, 'grad') and getattr(layer, 'grad') is not None or whatever the tensorflow equivalent is!!

The PyTorch backend should have the same behaviour if we are to implement this PR.
@RobertSamoilescu @mauicv

I'll test and see how a non-differentiable layer behaves in the PyTorch case, but agree that the two back ends should match

mauicv · 2023-01-05T11:00:33Z

Having looked into this further it's not actually the case that the embedding layer is non-differentiable (see this explanation). It's just that the tf.keras.layers.Embedding represents the gradients as Sparse tensors in the form tf.IndexedSlices and this is where the error arises as these don't have .numpy() method.

Moving forward it makes sense to either

filter for IndexedSlices or on something like hasattr(grad, 'numpy').
Or find a way to convert them to the numpy gradinet tensor...

Note: pytorch allows you to set the embedding layer gradients as sparse or not using a kwarg in the Embedding layer this defaults to False however. But in the case that a user sets sparse=True we'll get the error:

TypeError: can't convert Sparse layout tensor to numpy. Use Tensor.dense() first.

jklaise · 2023-01-05T13:21:05Z

Nice find. Would it be more correct to convert all sparse layers to dense and take gradients into account rather than filtering them out? On the other hand, are we risking memory blowing up?

mauicv · 2023-01-05T13:24:12Z

Having spoken to @RobertSamoilescu we've agreed on the following plan of action:

If the tensor is in a sparse format we try to convert it to dense and then to np.ndarray. This will also need to be done for sparse representations in pytorch.
~~If the above fails or the tensor has no numpy attribute we raise a warning informing the user that some tensors grads are not used.~~
- Having thought further about this I'm not sure about this? Seems like we'd end up catching a broad range of errors. In principle any model passed should have gradients computable for all layers so it's not clear that we should discard errors that might arise here!?
- If an error due to something like the above arises we can suggest the user set trainable to false on the layer in question while we figure out a fix? I'm open to doing this differently if that's the preference though!
Finally we filter layers with trainable=False
If a user passes a model that has some layers set to trainable=False then we raise warning to make it clear that those layers won't be used in the computation. Note this should only happen in the model init call!
Check non-trainable warning behaviour for batch layers...
- Checking weather or not models contain non-trainable layers is a little non trivial. For instance the batch_normalization layer has non-trainable weights and so if a user includes this in there model naive solutions would trigger this warning. I've set it to filter the batch norm layer as according to the tensorflow docs this is the only layer that is non-trainable. A user might create a custom layer which would throw warnings but I think that's ok as they should be alerted in this case...

mauicv · 2023-01-05T13:27:38Z

Nice find. Would it be more correct to convert all sparse layers to dense and take gradients into account rather than filtering them out? On the other hand, are we risking memory blowing up?

w.r.t. memory usage, agreed this is potentially an issue if the vocabulary or the embedding dimension is very big. I think the correct solution to this is to maintain the sparse representation of the layer grads while computing the similarity. So the get_grads method would return sparse_grads and dense_grads. This is a bit trickier though so we decided to consider it for a later PR. In the meantime, if a user wants to reduce the memory footprint they can set trainable to False on layers...

mauicv · 2023-01-05T15:47:56Z

tensorflow has two representations of sparsity, 1. IndexedSlice which arises as gradients of embedding layers as we've seen and 2. SparceTensor. They have different methods of converting to dense representations for instance the below needs to be used:

if isinstance(grad, tf.SparseTensor):
  grad = tf.sparse.to_dense(grad)
if isinstance(grad, tf.IndexedSlices):
  grad = tf.convert_to_tensor(grad)

The issue I have is that I can't find any indication that tensorflow layers would ever have sparse gradients other than the IndexedSlices that results from tf.gather in the embedding layer. So I'm not sure the above case is needed. You certainly can't take gradients w.r.t. SparseTensors. Doing so yeilds:

ValueError: Type SparseTensor is not supported as a gradient source or gradient target.

Can either of you think of any situation in which this would arise? For now, I'm going to remove that logic and issue a warning if the layer can't be converted to numpy as planned.

RobertSamoilescu · 2023-01-06T15:33:09Z

alibi/explainers/similarity/backends/pytorch/base.py

+
+    @staticmethod
+    def get_non_trainable(model: nn.Module) -> List[Union[int, str]]:
+        """Checks that all layers in a model are trainable."""


Please update docstrings with something descriptive of what the function is actually doing.

RobertSamoilescu · 2023-01-06T15:45:41Z

alibi/explainers/similarity/backends/tensorflow/base.py

+
+    @staticmethod
+    def get_non_trainable(model: keras.Model) -> List[Union[int, str]]:
+        """Checks if all layers in a model are trainable.


Update docstring as for pytorch backend.

alibi/explainers/similarity/backends/tensorflow/base.py

RobertSamoilescu · 2023-01-06T15:56:33Z

alibi/explainers/similarity/backends/tensorflow/base.py

+
+        Note: batch normalization layers are ignored as they are not trainable by default.
+        """
+        return [getattr(layer, 'name', i) for i, layer in enumerate(model.layers) if not layer.trainable]


This approach does not work as for the pytorch backend.
For example, consider the model:

model = keras.Sequential([ keras.layers.Dense(10), keras.layers.Dense(20), keras.Sequential([ keras.layers.Dense(30), keras.layers.Dense(40), ]) ]) model.build((None, 10))

Calling model.layers will return

[<keras.layers.core.dense.Dense at 0x7f54d4332430>, <keras.layers.core.dense.Dense at 0x7f54d43323d0>, <keras.engine.sequential.Sequential at 0x7f54d44dea30>]

Setting on layer inside the sequential layer to non-trainable:

model.layers[-1].layers[0].trainable = False

If we call your function, it returns an empty list, which is incorrect.

get_non_trainable(model)

If we call it with the sequential layer, then it returns the correct answer.

get_non_trainable(model.layers[-1])

This shows that the function does not work for nested layers, which are very common. Not sure if I am missing something and if the problem is addressed somewhere else?

Yeah, good catch! 🤦‍♂️ It's better if I just iterate through the model.non_trainable_weights here!

It's a similar issue for tensorflow models which have nesting, e.g. any transformer model (see the model for our text app as an example). I think we should try a recursive approach.

~~I think @RobertSamoilescu misspoke no? The above is a Keras model?~~

Fixed by iterating through model.non_trainable_weights instead of layers and added tests.

alibi/explainers/similarity/backends/pytorch/base.py

jklaise · 2023-01-06T16:29:45Z

alibi/explainers/similarity/grad.py

+        non_trainable_layers = self.backend.get_non_trainable(self.predictor)
+        if non_trainable_layers:
+            layers_msg = 'The following layers are not trainable: '
+            layers = ", ".join([f"'{layer}'" for layer in non_trainable_layers if layer is not None])


if layer is not None -> are there instances where the returned list containe None elements?

I think eager tensors don't have names... Although I've forgotten to handle them properly here as I'm enumerating and returning the layer index instead of the name. Ill need to check how that behaves though!

I haven't been able to figure out this behaviour. EagerTensors don't have name attributes, so those layer names returned would be None. I'm leaving this in defensively so that if this behaviour does happen then an error won't occur but the user still receives an error message.

I've also added an error message for the case where no parameter in the model is trainable.

alibi/explainers/similarity/grad.py

jklaise · 2023-01-06T16:50:33Z

alibi/explainers/similarity/backends/pytorch/base.py

+                               if param.grad is not None])
+
+    @staticmethod
+    def _grad_to_numpy(grad: torch.Tensor, name: Optional[str] = None) -> torch.Tensor:


grad type should probably be a union here since it can be sparse?

Ignore this, was thinking of the tensorflow backend.

No your correct, torch also has SparseTensors

Ignore me, torch handles sparse tensors by setting a layout attribute on normal tensors!

jklaise · 2023-01-06T16:55:44Z

alibi/explainers/similarity/backends/pytorch/base.py

+            raise TypeError((f'Could not convert gradient to numpy array{name}. To ignore these '
+                             'gradients in the similarity computation use `requires_grad=False`.'))


Is it possible for the user to easily do this? I'm thinking of the situation where a layer has multiple gradient tensors, e.g. weights and bias. Here we would be reporting the tensors not the layers?

A related point might be that this could be annoying if many such layers are present, as the error would be thrown for the first one, and then once it's fixed the next one etc. Should we consider checking the numpy attribute of all tensors first (after converting to dense) and then raise one error if there are any that don't have that attribute? Perhaps this is a minor point if in realistic scenarios we don't expect there to be more than a handful of such layers?

to be honest I'm hoping there are no such layers. But it's hard to know without exhaustively checking all of them. If there is a model that does have lots of such layers then hopefully the user can figure out the pattern that identifies those layers and programmatically disable training on them but I agree it would be good to return a list instead, this way just seemed simple for an issue that it's not clear exists ... 😕 happy to do whatever though

So it sounds like this check is more of a fail-safe and shouldn't really happen unless something is quite wrong? In which case I think it's fine as we at least have a readable error rather than a non-readable one.

alibi/explainers/similarity/backends/tensorflow/base.py

alibi/explainers/tests/test_simiarlity/conftest.py

jklaise · 2023-01-06T17:20:53Z

alibi/explainers/tests/test_simiarlity/conftest.py

+        tf.keras.layers.Dense(output_shape),
+        tf.keras.layers.Dense(output_shape),


Assume the duplication was unintentional?

Ah I see the same for pytorch, was an extra layer needed for extra tests?

Yeah, the issue was there was only one layer that could be set to non-trainable and I felt we probably wanted some weights that were still trainable to make sure that the setting was doing what we want it to. I've replaced it in what I'm currently doing though because the model for checking for non-trainable layers is sufficiently complex now that it doesn't make sense to use the above fixture. i.e. I've added nested sequential layers etc...

alibi/explainers/tests/test_simiarlity/test_backends.py

…nings

mauicv · 2023-01-10T08:44:33Z

alibi/explainers/similarity/backends/pytorch/base.py

+
+        return np.concatenate([_PytorchBackend._grad_to_numpy(grad=param.grad, name=name)
+                               for name, param in model.named_parameters()
+                               if param.grad is not None])


Note that torch tensors have a grad attribute by default set to None so we don't have to check it exist first!

doc/source/overview/faq.md

alibi/explainers/similarity/backends/pytorch/base.py

alibi/explainers/similarity/backends/tensorflow/base.py

alibi/explainers/similarity/grad.py

jklaise

Looks good! Left a few minor comments, plus the one we discussed offline here:

Testing this on a language model with all the transformer layers set to non-trainable results in a very large warning off the screen as we list all non-trainable parameters by name. Although the names could potentially be useful, we're likely better off changing the warning to output the number of non-trainable parameters detected and adding a few short sentences about the 2 possibilities (user having set them manually or model having some in by default e.g. batch norm).

alibi/explainers/similarity/backends/pytorch/base.py

alibi/explainers/similarity/backends/tensorflow/base.py

alibi/explainers/similarity/backends/pytorch/base.py

alibi/explainers/similarity/grad.py

alibi/explainers/tests/test_simiarlity/test_grad_methods_integration.py

doc/source/overview/faq.md

alibi/explainers/tests/test_simiarlity/test_grad_methods_integration.py

Filter gradients of IndexedSlices in tf grad sim backend, fixes Seldo…

f15d7a8

…nIO#828

jklaise requested a review from RobertSamoilescu December 9, 2022 14:12

mauicv added 9 commits January 5, 2023 16:43

Add functionality to convert sparse gradients to nd.array

1fd9c38

Add warnings if non-trainable layers present in model

3dbc6c6

Fix linting and typing errors

ab16515

Add check for non-trainable layers in gradsim

87d4ca7

Add better docstrings

6cad12c

Add better docstrings for tests

12937d8

Fix minor linting error

219a6a0

Add check for tensor attribute

764bc56

Check for and log list of non trainable layers on GradSim __init__

4a89f48

RobertSamoilescu requested changes Jan 6, 2023

View reviewed changes

jklaise self-requested a review January 6, 2023 16:16

jklaise reviewed Jan 6, 2023

View reviewed changes

alibi/explainers/similarity/backends/pytorch/base.py Outdated Show resolved Hide resolved

jklaise reviewed Jan 6, 2023

View reviewed changes

alibi/explainers/similarity/grad.py Outdated Show resolved Hide resolved

jklaise reviewed Jan 6, 2023

View reviewed changes

alibi/explainers/similarity/backends/tensorflow/base.py Outdated Show resolved Hide resolved

jklaise reviewed Jan 6, 2023

View reviewed changes

alibi/explainers/tests/test_simiarlity/conftest.py Show resolved Hide resolved

jklaise reviewed Jan 6, 2023

View reviewed changes

alibi/explainers/tests/test_simiarlity/test_backends.py Outdated Show resolved Hide resolved

mauicv added 10 commits January 9, 2023 15:34

Revert minor changes

188e1d3

Minor spelling fix to test docstring

2af3367

Update test comment to make behavour clearer

a6e437c

Fix typeo in test docstring

ab23c05

Add comment to test to explain build method call

e95aaf1

Add note on batch norm layer in GradSim integration tests

6704c64

Fix minor linting errors

876ea61

Add backticks to modules names in docstrings

bf7fad3

Make get_not_trainable private

0e0c919

Add questions to the FAQ docs page detailing performance soln and war…

09a1674

…nings

mauicv commented Jan 10, 2023

View reviewed changes

mauicv added 2 commits January 10, 2023 09:03

Minor fixes

f443185

Fix linting error

8a99be5

jklaise reviewed Jan 10, 2023

View reviewed changes

doc/source/overview/faq.md Outdated Show resolved Hide resolved

jklaise reviewed Jan 10, 2023

View reviewed changes

alibi/explainers/similarity/backends/pytorch/base.py Outdated Show resolved Hide resolved

jklaise reviewed Jan 10, 2023

View reviewed changes

alibi/explainers/similarity/backends/tensorflow/base.py Outdated Show resolved Hide resolved

jklaise reviewed Jan 10, 2023

View reviewed changes

alibi/explainers/similarity/backends/tensorflow/base.py Outdated Show resolved Hide resolved

jklaise reviewed Jan 10, 2023

View reviewed changes

alibi/explainers/similarity/grad.py Outdated Show resolved Hide resolved

jklaise suggested changes Jan 10, 2023

View reviewed changes

RobertSamoilescu requested changes Jan 10, 2023

View reviewed changes

mauicv added 5 commits January 10, 2023 10:57

Add minor changes

f69227b

Fix Requested PR changes

42cf904

Update non-trainable params warning

57d04b4

Minor fix

f72d197

Remove backticks

308b51f

RobertSamoilescu approved these changes Jan 10, 2023

View reviewed changes

jklaise reviewed Jan 10, 2023

View reviewed changes

alibi/explainers/tests/test_simiarlity/test_grad_methods_integration.py Outdated Show resolved Hide resolved

Remove commented out lines from test

4bd1f20

jklaise approved these changes Jan 10, 2023

View reviewed changes

jklaise merged commit f29bc2f into SeldonIO:master Jan 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filter gradients of IndexedSlices in tf grad sim backend, fixes #828 #829

Filter gradients of IndexedSlices in tf grad sim backend, fixes #828 #829

mauicv commented Nov 30, 2022 •

edited

Loading

codecov bot commented Nov 30, 2022 •

edited

Loading

jklaise commented Dec 9, 2022

mauicv commented Dec 13, 2022

mauicv commented Jan 5, 2023 •

edited

Loading

jklaise commented Jan 5, 2023

mauicv commented Jan 5, 2023 •

edited

Loading

mauicv commented Jan 5, 2023

mauicv commented Jan 5, 2023 •

edited

Loading

RobertSamoilescu Jan 6, 2023

RobertSamoilescu Jan 6, 2023

RobertSamoilescu Jan 6, 2023

mauicv Jan 6, 2023

jklaise Jan 6, 2023

mauicv Jan 6, 2023 •

edited

Loading

mauicv Jan 9, 2023

jklaise Jan 6, 2023

mauicv Jan 6, 2023

mauicv Jan 9, 2023

jklaise Jan 6, 2023

jklaise Jan 6, 2023

mauicv Jan 6, 2023

mauicv Jan 9, 2023

jklaise Jan 6, 2023

mauicv Jan 6, 2023

jklaise Jan 6, 2023

jklaise Jan 6, 2023

jklaise Jan 6, 2023

mauicv Jan 6, 2023

mauicv Jan 10, 2023

jklaise left a comment

		raise TypeError((f'Could not convert gradient to numpy array{name}. To ignore these '
		'gradients in the similarity computation use `requires_grad=False`.'))

		tf.keras.layers.Dense(output_shape),
		tf.keras.layers.Dense(output_shape),

Filter gradients of IndexedSlices in tf grad sim backend, fixes #828 #829

Filter gradients of IndexedSlices in tf grad sim backend, fixes #828 #829

Conversation

mauicv commented Nov 30, 2022 • edited Loading

What is this:

Implementation:

More Todo:

Issues:

codecov bot commented Nov 30, 2022 • edited Loading

Codecov Report

jklaise commented Dec 9, 2022

mauicv commented Dec 13, 2022

mauicv commented Jan 5, 2023 • edited Loading

jklaise commented Jan 5, 2023

mauicv commented Jan 5, 2023 • edited Loading

mauicv commented Jan 5, 2023

mauicv commented Jan 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mauicv Jan 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jklaise left a comment

Choose a reason for hiding this comment

mauicv commented Nov 30, 2022 •

edited

Loading

codecov bot commented Nov 30, 2022 •

edited

Loading

mauicv commented Jan 5, 2023 •

edited

Loading

mauicv commented Jan 5, 2023 •

edited

Loading

mauicv commented Jan 5, 2023 •

edited

Loading

mauicv Jan 6, 2023 •

edited

Loading