Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Implement log_loss Function and Add Comprehensive Test Coverage #28800

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

Implement log_loss Function and Add Comprehensive Test Coverage #28800

wants to merge 6 commits into from

Conversation

muzakkirhussain011
Copy link
Contributor

PR Description

This PR introduces the following changes:

Feature Implementation:

  • log_loss Function:
    • Added the log_loss function to the metrics module.
    • The function computes the logistic loss (or binary cross-entropy loss) for binary classification tasks.
    • Supports clipping of predicted probabilities to avoid logarithm of zero and allows for optional sample_weight input to handle weighted samples.
    • Ensures y_true and y_pred inputs have the same shape and includes validation for edge cases such as improper input shapes or invalid values.

Test Implementation:

  • test_sklearn_log_loss:
    • Added a comprehensive test function to validate the log_loss function across various scenarios.
    • The test generates random binary classification data (y_true) and corresponding predicted probabilities (y_pred).
    • Ensures correct handling of sample_weight and validates the function's performance across different backends (e.g., TensorFlow, PyTorch).
    • The test compares the output of the Ivy log_loss implementation with the reference implementation from sklearn.metrics.log_loss to ensure correctness.

Additional Notes:

  • All methods and functions utilized in the implementation (ivy.clip, ivy.log, ivy.mean, etc.) have been cross-checked and confirmed to be available within the Ivy framework.
  • This PR enhances the robustness of the Ivy metrics module, particularly for binary classification tasks.

Testing:

  • Added test cases cover a range of input scenarios, including edge cases, to ensure that the log_loss function performs as expected under various conditions.
  • Tests have been verified across different backends to ensure compatibility and consistency.

Checklist

  • Did you add a function?
  • Did you add the tests?
  • Did you run your tests and are your tests passing?
  • Did pre-commit not fail on any check?
  • Did you follow the steps we provided?

@muzakkirhussain011
Create func_wrapper.py
d3c3de2 
func_wrapper.py is a Python module designed to streamline the integration of Hugging Face Transformers into your natural language processing (NLP) projects. It provides a set of input and output conversion wrappers to simplify the process of passing data between your custom functions and Transformers' data structures.

Input Conversion Wrappers:

inputs_to_transformers_tensors: This wrapper converts input data (text, tensors, etc.) into Transformers-compatible data structures. It is particularly useful when your custom functions expect diverse input types.
Output Conversion Wrappers:

outputs_to_pytorch_tensors: After your custom function returns data, this wrapper ensures that the output data is converted into PyTorch tensors or other appropriate formats.
Usage:

Import func_wrapper.py into your project.
Initialize a Hugging Face Transformers model and tokenizer.
Wrap your custom function with to_transformers_tensors_and_back. This wrapped function can now accept and return Transformers-compatible data.
Here's a simple example of how to use func_wrapper.py:


import torch
from transformers import BertForSequenceClassification, BertTokenizer
from ivy.functional.frontends.transformers.func_wrapper import to_transformers_tensors_and_back

# Initialize the model and tokenizer
model_name = "bert-base-uncased"
model = BertForSequenceClassification.from_pretrained(model_name)
tokenizer = BertTokenizer.from_pretrained(model_name)

# Wrap your custom function using the conversion wrappers
wrapped_function = to_transformers_tensors_and_back(your_function, model, tokenizer)

# Prepare sample input data
sample_input_text = "This is a sample input text."
sample_input_tensor = torch.rand((3, 3))

# Call your wrapped function with the sample input data
output = wrapped_function(sample_input_text, sample_input_tensor)

# The output is automatically converted to PyTorch tensors
print(output)
Please note that func_wrapper.py is still in development, and further enhancements and refinements are expected. Your feedback and contributions to improve its functionality are welcome.
@muzakkirhussain011
Merge branch 'unifyai:main' into patch-2
ea8612d
@muzakkirhussain011
feat(metrics): Implement log_loss function with sample weight support
c01177b
@muzakkirhussain011
test(metrics): Add test for log_loss function with various input scen…
c79effd 
…arios
@muzakkirhussain011
Delete ivy/functional/frontends/transformers/func_wrapper.py
e140942
@muzakkirhussain011 muzakkirhussain011 marked this pull request as draft August 9, 2024 12:02
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@muzakkirhussain011