Enable Pyproject Inclusion (#28345)

* inactive packages are still assembled, but will not be automatically tested unless included using ENABLE_PACKAGE_NAME=true
* checks can now be individually disabled through the pyproject.toml
* added package classifiers to ParsedSetup properties
* add new function is_check_enabled which combines environment_exclusions, pyproject.toml, and environment overrides into one slick package
* all run_check scripts now honor is_check_enabled to enable/disable atomically
* update eng_sys_checks for new functionality
* add pyproject.toml for all core/ packages
* enable black on only track 2 core packages
* remove package build exclusion list in favor of honoring classifier
* update a couple packages with Inactive classifier so they are no longer built. azure-mgmt-scheduler, azure-mgmt-documentdb, azure-mgmt-regionmove

Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>

Author: scbedd

.vscode/cspell.json

@@ -27,7 +27,7 @@
     "sdk/**/_generated/**",
     "**/*requirement*.txt",
     "**/assets.json",
-
+    "**/pyproject.toml",
     "sdk/anomalydetector/**",
     "sdk/applicationinsights/azure-applicationinsights/**",
     "sdk/batch/azure-batch/**",
@@ -95,9 +95,13 @@
     ".gitignore",
     "tools/azure-sdk-tools/devtools_testutils/fake_credentials.py",
     "tools/azure-sdk-tools/packaging_tools/**",
-    "tools/azure-sdk-tools/setup.py"
+    "tools/azure-sdk-tools/setup.py",
+    "tools/azure-sdk-tools/tests/test_servicemetadata.py"
   ],
   "words": [
+    "pyprojecttoml",
+    "tomli",
+    "sdkrel",
     "qnamaker",
     "mindependency",
     "automl",

doc/eng_sys_checks.md

@@ -4,7 +4,9 @@
   - [Targeting a specific package at build queue time](#targeting-a-specific-package-at-build-queue-time)
   - [Skipping a tox test environment at build queue time](#skipping-a-tox-test-environment-at-build-queue-time)
   - [Skipping entire sections of builds](#skipping-entire-sections-of-builds)
+  - [The pyproject.toml](#the-pyprojecttoml)
   - [Environment variables important to CI](#environment-variables-important-to-ci)
+    - [Atomic Overrides](#atomic-overrides)
   - [Analyze Checks](#analyze-checks)
     - [MyPy](#mypy)
     - [Pylint](#pylint)
@@ -102,6 +104,28 @@ This is the most useful skip, but the following skip variables are also supporte
 - `Skip.VerifyDependencies`
   - Omit checking that a package's dependencies are on PyPI before releasing.
 
+## The pyproject.toml
+
+Starting with [this pr](https://github.com/Azure/azure-sdk-for-python/pull/28345), which checks apply to which packages are now **established** in a `pyproject.toml`, right next to each package's `setup.py`. This not only allows devs to fine-tune which checks that are applied at a package-level, but also seriously reduces confusion as to which checks apply when.
+
+We default to **enabling** most of our checks like `pylint`, `mypy`, etc. Due to that, most `pyproject.toml` settings will likely be **disabling** checks.
+
+Here's an example:
+
+```toml
+# from sdk/core/azure-servicemanagement-legacy/pyproject.toml, which is a legacy package
+# as a result, all of these checks are disabled
+[tool.azure-sdk-build]
+mypy = false
+type_check_samples = false
+verifytypes = false
+pyright = false
+pylint = false
+black = false
+```
+
+If a package does not yet have a `pyproject.toml`, creating one with just the section `[tool.azure-sdk-build]` will do no harm to the release of the package in question.
+
 ## Environment variables important to CI
 
 There are a few differences from a standard local invocation of `tox <env>`. Primarily, these differences adjust the checks to be friendly to parallel invocation. These adjustments are necessary to prevent random CI crashes.
@@ -114,6 +138,22 @@ There are a few differences from a standard local invocation of `tox <env>`. Pri
 
 The various tooling abstracted by the environments within `eng/tox/tox.ini` take the above variables into account automatically.
 
+### Atomic Overrides
+
+Packages with classifier `Development Status :: 7 - Inactive`, are **not** built by default and as such normal `checks` like `mypy` and `pylint` are also not run against them. Older "core" packages like `azure-common` and `azure-servicemanagement-legacy` are present, but excluded from the build due to this restriction.
+
+To temporarily **override** this restriction, a dev need only set the queue time variable: `ENABLE_PACKAGE_NAME`. The `-` in package names should be replaced by an `_`, as that is how the environment variable will be set on the actual CI machine anyway.
+
+- `ENABLE_AZURE_COMMON=true`
+- `ENABLE_AZURE_SERVICEMANAGEMENT_LEGACY=true`
+
+This same methodology also applies to _individual checks_ that run during various phases of CI. Developers can use a queue time variable of format `PACKAGE_NAME_CHECK=true/false`.
+
+The name that you should use is visible based on what the `tox environment` that the check refers to! Here are a few examples of enabling/disabling checks:
+
+- `AZURE_SERVICEBUS_PYRIGHT=true` <-- enable a check that normally is disabled in `pyproject.toml`
+- `AZURE_CORE_PYLINT=false` <-- disable a check that normally runs
+
 ## Analyze Checks
 
 Analyze job in both nightly CI and pull request validation pipeline runs a set of static analysis using external and internal tools. Following are the list of these static analysis.
@@ -122,15 +162,8 @@ Analyze job in both nightly CI and pull request validation pipeline runs a set o
 
 [`MyPy`](https://pypi.org/project/mypy/)  is a static analysis tool that runs type checking of python package. MyPy is an opt-in check for packages. Following are the steps to run `MyPy` locally for a specific package:
 
-1. Add the package name to the end of the [`mypy_hard_failure_packages.py`](https://github.com/Azure/azure-sdk-for-python/blob/main/eng/tox/mypy_hard_failure_packages.py) file:
-   ```python
-   MYPY_HARD_FAILURE_OPTED = [
-      ...,
-      "azure-my-package",
-   ]
-   ```
-2. Go to root of the package
-3. Execute following command: `tox -e mypy -c ../../../eng/tox/tox.ini`
+1. Go to root of the package
+2. Execute following command: `tox -e mypy -c ../../../eng/tox/tox.ini`
 
 ### Pylint
 

eng/pipelines/templates/steps/build-artifacts.yml

@@ -63,7 +63,7 @@ steps:
     condition: and(succeeded(),eq(variables['SetDevVersion'],'true'))
 
   - pwsh: |
-      sdk_build -d "$(Build.ArtifactStagingDirectory)" "$(TargetingString)" --service=${{parameters.ServiceDirectory}}
+      sdk_build -d "$(Build.ArtifactStagingDirectory)" "$(TargetingString)" --service=${{parameters.ServiceDirectory}} --inactive
     displayName: 'Generate Packages'
     condition: succeededOrFailed()
 

eng/pipelines/templates/steps/run_black.yml

@@ -11,7 +11,7 @@ steps:
     condition: succeededOrFailed()
 
   - script: |
-      pip install black==21.6b0
+      pip install black==21.6b0 tools/azure-sdk-tools["build"]
     displayName: 'Prep Environment'
     condition: succeededOrFailed()
 

eng/scripts/get_package_properties.py

@@ -17,6 +17,7 @@
             if "setup.py" in files:
                 try:
                     parsed = ParsedSetup.from_path(root)
+
                     print(
                         "{0} {1} {2} {3}".format(
                             parsed.name, parsed.version, parsed.is_new_sdk, os.path.dirname(parsed.setup_filename)

eng/tox/run_mypy.py

@@ -15,8 +15,7 @@
 
 from ci_tools.environment_exclusions import (
     is_ignored_package,
-    MYPY_OPT_OUT,
-    TYPE_CHECK_SAMPLES_OPT_OUT,
+    is_check_enabled
 )
 
 logging.getLogger().setLevel(logging.INFO)
@@ -36,7 +35,7 @@
 
     args = parser.parse_args()
     package_name = os.path.basename(os.path.abspath(args.target_package))
-    if package_name in MYPY_OPT_OUT or is_ignored_package(package_name):
+    if not is_check_enabled(args.target_package, "mypy", True) or is_ignored_package(package_name):
         logging.info(
             f"Package {package_name} opts-out of mypy check. See https://aka.ms/python/typing-guide for information."
         )
@@ -62,7 +61,7 @@
     except CalledProcessError as src_err:
         src_code_error = src_err
 
-    if package_name in TYPE_CHECK_SAMPLES_OPT_OUT:
+    if not is_check_enabled(args.target_package, "type_check_samples", True):
         logging.info(
             f"Package {package_name} opts-out of mypy check on samples."
         )

eng/tox/run_pylint.py

@@ -14,7 +14,7 @@
 import logging
 import sys
 
-from ci_tools.environment_exclusions import PYLINT_OPT_OUT
+from ci_tools.environment_exclusions import is_check_enabled
 from ci_tools.parsing import ParsedSetup
 
 logging.getLogger().setLevel(logging.INFO)
@@ -43,7 +43,7 @@
 
     top_level_module = pkg_details.namespace.split('.')[0]
 
-    if pkg_details.name not in PYLINT_OPT_OUT:
+    if is_check_enabled(args.target_package, "pylint"):
         try:
             check_call(
                 [

eng/tox/run_pyright.py

@@ -15,8 +15,7 @@
 
 from ci_tools.environment_exclusions import (
     is_ignored_package,
-    PYRIGHT_OPT_OUT,
-    TYPE_CHECK_SAMPLES_OPT_OUT,
+    is_check_enabled,
 )
 
 logging.getLogger().setLevel(logging.INFO)
@@ -36,7 +35,7 @@
 
     args = parser.parse_args()
     package_name = os.path.basename(os.path.abspath(args.target_package))
-    if package_name in PYRIGHT_OPT_OUT or is_ignored_package(package_name):
+    if not is_check_enabled(args.target_package, "pyright") or is_ignored_package(package_name):
         logging.info(
             f"Package {package_name} opts-out of pyright check. See https://aka.ms/python/typing-guide for information."
         )
@@ -46,7 +45,7 @@
         os.path.join(args.target_package, "azure"),
         os.path.join(args.target_package, "samples"),
     ]
-    if package_name in TYPE_CHECK_SAMPLES_OPT_OUT:
+    if not is_check_enabled(args.target_package, "type_check_samples"):
         logging.info(
             f"Package {package_name} opts-out of pyright check on samples."
         )

eng/tox/run_verifytypes.py

@@ -16,7 +16,7 @@
 import logging
 import sys
 
-from ci_tools.environment_exclusions import is_ignored_package, VERIFYTYPES_OPT_OUT
+from ci_tools.environment_exclusions import is_ignored_package, is_check_enabled
 
 logging.getLogger().setLevel(logging.INFO)
 
@@ -98,7 +98,7 @@ def get_type_complete_score(commands, check_pytyped=False):
     module = package_name.replace("-", ".")
     setup_path = os.path.abspath(args.target_package)
 
-    if package_name in VERIFYTYPES_OPT_OUT or is_ignored_package(package_name):
+    if not is_check_enabled(args.target_package, "type_check_samples") or is_ignored_package(package_name):
         logging.info(
             f"{package_name} opts-out of verifytypes check. See https://aka.ms/python/typing-guide for information."
         )

eng/tox/tox.ini

@@ -39,6 +39,7 @@ pkgs =
     wheel==0.37.0
     packaging==20.4
     urllib3==1.26.12
+    tomli==2.0.1
 
 
 [testenv]

scripts/devops_tasks/common_tasks.py

@@ -31,7 +31,7 @@
 from packaging.version import Version
 from packaging.version import parse
 
-from ci_tools.functions import MANAGEMENT_PACKAGE_IDENTIFIERS, lambda_filter_azure_pkg
+from ci_tools.functions import MANAGEMENT_PACKAGE_IDENTIFIERS, lambda_filter_azure_pkg, str_to_bool
 from ci_tools.parsing import parse_require, ParsedSetup
 
 DEV_REQ_FILE = "dev_requirements.txt"
@@ -41,8 +41,6 @@
 logging.getLogger().setLevel(logging.INFO)
 
 
-
-
 def log_file(file_location, is_error=False):
     with open(file_location, "r") as file:
         for line in file:
@@ -81,17 +79,6 @@ def clean_coverage(coverage_dir):
             raise
 
 
-def str_to_bool(input_string):
-    if isinstance(input_string, bool):
-        return input_string
-    elif input_string.lower() in ("true", "t", "1"):
-        return True
-    elif input_string.lower() in ("false", "f", "0"):
-        return False
-    else:
-        return False
-
-
 def run_check_call(
     command_array,
     working_directory,

scripts/devops_tasks/test_regression.py

@@ -23,8 +23,7 @@
     find_whl,
     find_tools_packages,
     get_installed_packages,
-    extend_dev_requirements,
-    str_to_bool,
+    extend_dev_requirements
 )
 
 from git_helper import (
@@ -34,7 +33,7 @@
     clone_repo,
 )
 
-from ci_tools.functions import discover_targeted_packages
+from ci_tools.functions import discover_targeted_packages, str_to_bool
 from ci_tools.parsing import ParsedSetup
 
 AZURE_GLOB_STRING = "azure*"

scripts/devops_tasks/tox_harness.py

@@ -323,12 +323,12 @@ def prep_and_run_tox(targeted_packages: List[str], parsed_args: Namespace, optio
         inject_custom_reqs(destination_dev_req, parsed_args.injected_packages, package_dir)
 
         if parsed_args.tox_env:
-            filtered_tox_environment_set = filter_tox_environment_string(parsed_args.tox_env, package_name)
+            filtered_tox_environment_set = filter_tox_environment_string(parsed_args.tox_env, package_dir)
 
             if not filtered_tox_environment_set:
                 logging.info(
                     f"All requested tox environments for package {package_name} have been excluded by the environment exclusion list."
-                    + " Check file /tools/azure-sdk-tools/ci_tools/environment_exclusions.py"
+                    + " Check file /tools/azure-sdk-tools/ci_tools/environment_exclusions.py and the pyproject.toml."
                 )
                 continue
 

scripts/devops_tasks/validate_formatting.py

@@ -12,29 +12,40 @@
 import subprocess
 
 logging.getLogger().setLevel(logging.INFO)
+from ci_tools.functions import discover_targeted_packages
+from ci_tools.environment_exclusions import is_check_enabled
 
 root_dir = os.path.abspath(os.path.join(os.path.abspath(__file__), "..", "..", ".."))
 sdk_dir = os.path.join(root_dir, "sdk")
 
 def run_black(service_dir):
+    results = []
     logging.info("Running black for {}".format(service_dir))
 
-    out = subprocess.Popen([sys.executable, "-m", "black", "-l", "120", "sdk/{}".format(service_dir)],
-        stdout=subprocess.PIPE,
-        stderr=subprocess.STDOUT,
-        cwd = root_dir
-    )
+    discovered_packages = discover_targeted_packages("azure*", os.path.join(root_dir, "sdk", service_dir))
+
+    for package in discovered_packages:
+        package_name = os.path.basename(package)
+
+        if is_check_enabled(package, "black", True):
+            out = subprocess.Popen([sys.executable, "-m", "black", "-l", "120", "sdk/{}/{}".format(service_dir, package_name)],
+                stdout=subprocess.PIPE,
+                stderr=subprocess.STDOUT,
+                cwd = root_dir
+            )
 
-    stdout,stderr = out.communicate()
+            stdout,stderr = out.communicate()
 
-    if stderr:
-        raise RuntimeError("black ran into some trouble during its invocation: " + stderr)
+            if stderr:
+                results.append((package_name, stderr))
 
-    if stdout:
-        if "reformatted" in stdout.decode('utf-8'):
-            return False
+            if stdout:
+                if "reformatted" in stdout.decode('utf-8'):
+                    results.append((package_name, False))
+            else:
+                print(f"black succeeded against {package_name}")
 
-    return True
+    return results
 
 
 if __name__ == "__main__":
@@ -51,7 +62,13 @@ def run_black(service_dir):
 
     args = parser.parse_args()
     if args.validate != "False":
-        if not run_black(args.service_directory):
+        results = run_black(args.service_directory)
+        
+        if len(results) > 0:
+            for result in results:
+                error = "Code needs reformat." if result[1] == False else error
+                logging.error(f"Black run for {result[0]} ran into an issue: {error}")
+
             raise ValueError("Found difference between formatted code and current commit. Please re-generate with the latest autorest.")
 
     else:

sdk/core/azure-common/pyproject.toml

@@ -0,0 +1,8 @@
+[tool.azure-sdk-build]
+type_check_samples = false
+verifytypes = false
+pyright = false
+mypy = false
+pylint = false
+regression = false
+black = false

sdk/core/azure-core-experimental/pyproject.toml

@@ -0,0 +1,5 @@
+[tool.azure-sdk-build]
+type_check_samples = false
+verifytypes = false
+pyright = false
+mypy = false

sdk/core/azure-core-tracing-opencensus/pyproject.toml

@@ -0,0 +1,5 @@
+[tool.azure-sdk-build]
+type_check_samples = false
+verifytypes = false
+pyright = false
+mypy = false

sdk/core/azure-core-tracing-opentelemetry/pyproject.toml

@@ -0,0 +1,5 @@
+[tool.azure-sdk-build]
+type_check_samples = false
+verifytypes = false
+pyright = false
+mypy = false