Enhance ATS keyword extraction and analysis

[DavidOsipov]

Refactor and Enhance ATS Keyword Optimizer Script

This PR introduces significant improvements and refactoring to the ATS Keyword Optimizer script, enhancing its performance, reliability, and configurability. The changes span across the core script, configuration, and testing framework, leading to a more robust and user-friendly tool.

Summary of Changes:

• Code Refactoring (keywords4cv.py):

  • Improved code structure for better readability and maintainability.
  • Enhanced text preprocessing with optimized caching and batch processing for performance gains.
  • Refined keyword extraction logic, including semantic validation and more flexible n-gram handling.
  • Implemented chunking mechanism to handle large job description datasets and prevent memory issues.
  • Enhanced error handling and logging for better debugging and monitoring.
  • Improved semantic categorization of keywords using spaCy embeddings.
  • Robust input validation and sanitization to ensure data integrity.

• Expanded Configuration (config.yaml):

  • Added new configuration options to provide greater control over the analysis process:
    • section_headings: Define custom section headings to improve keyword extraction context.
    • spacy_model: Allow users to specify different spaCy language models for customized NLP processing.
    • cache_size: Configure the size of the text preprocessing cache for performance tuning.
    • whitelist_ngram_range: Set n-gram range specifically for whitelist matching.
    • timeout: Implement a timeout mechanism for long-running analyses.
    • model_download_retries: Configure retries for spaCy model downloads to handle network issues.
    • auto_chunk_threshold, memory_threshold, max_memory_percent, max_workers, min_chunk_size, max_chunk_size: Fine-grained control over chunking behavior for memory management and performance optimization.
    • max_retries: Set the maximum number of retries for the entire analysis process in case of transient errors.
    • strict_mode: Enable strict mode to halt analysis on any exception, or disable for more lenient error handling.
    • semantic_validation: Toggle semantic validation of extracted keywords for improved accuracy.
    • similarity_threshold: Adjust the similarity threshold for semantic categorization.
    • text_encoding: Specify the text encoding for input job descriptions.
  • Updated default configuration values for better out-of-the-box performance.

• Comprehensive Unit Tests (test_keywords4cv.py):

  • Introduced a new test suite using pytest to ensure code reliability and prevent regressions.
  • Implemented unit tests for various components:
    • Utility functions (cosine_similarity, ensure_nltk_resources, load_job_data, parse_arguments, save_results).
    • EnhancedTextPreprocessor: Testing preprocessing steps, caching, and batch processing.
    • AdvancedKeywordExtractor: Testing keyword extraction, synonym generation, n-gram handling, section extraction, and semantic filtering.
    • ATSOptimizer: Testing core analysis workflow, configuration loading, input validation, chunking logic, and error handling.
    • CLI and end-to-end tests to validate command-line interface and overall script functionality.
    • Performance tests to monitor and ensure efficient execution.
  • Increased test coverage to improve confidence in the script's correctness.

• Dependency Updates (requirements.txt):

  • Updated dependencies to their latest versions to leverage bug fixes, performance improvements, and security patches.

Benefits of these changes:

• Improved Performance: Optimized text preprocessing, batch processing, and chunking significantly enhance the script's speed and efficiency, especially for large datasets.
• Enhanced Reliability: Robust error handling, input validation, and comprehensive unit tests make the script more stable and dependable.
• Increased Configurability: New configuration options provide users with finer control over the analysis process, allowing for customization based on specific needs and environments.
• Better Memory Management: Chunking implementation addresses memory limitations when processing large volumes of job descriptions, making the script more scalable.
• Enhanced Keyword Accuracy: Semantic validation and improved synonym generation contribute to more relevant and accurate keyword extraction.
• Improved Maintainability: Refactored code and comprehensive unit tests make the script easier to understand, maintain, and extend in the future.

Testing:

• All unit tests in test_keywords4cv.py have been executed and passed successfully, ensuring the stability and correctness of the changes.

Please review and merge this PR to incorporate these significant improvements into the main branch.