Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[SPARKNLP-1052] Adding random suffix to avoid duplication in spark files #14340

Conversation

danilojsl
Copy link
Contributor

Description

This PR addresses the issue of file duplication by adding a random suffix when using sparkContext.addFile for ONNX and OpenVINO models. This enhancement ensures that each file added to Spark has a unique name, preventing conflicts and improving the robustness of file handling in distributed environments.

Motivation and Context

Check this jira issue

How Has This Been Tested?

Screenshots (if appropriate):

  • Databricks notebooks
  • Google Colab notebooks

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Code improvements with no or little impact
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING page.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@danilojsl danilojsl self-assigned this Jul 8, 2024
@maziyarpanahi maziyarpanahi merged commit a070adc into release/541-release-candidate Jul 14, 2024
6 checks passed
@maziyarpanahi maziyarpanahi mentioned this pull request Jul 14, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants