Rename multiple pipelines #1044 (#1053)

* Rename multiple pipelines #1044

Signed-off-by: tdruez <tdruez@nexb.com>

* Update the documentation with pipelines renaming #1044

Signed-off-by: tdruez <tdruez@nexb.com>

* Add backward compatibility support in API and CLI #1044

Signed-off-by: tdruez <tdruez@nexb.com>

---------

Signed-off-by: tdruez <tdruez@nexb.com>

CHANGELOG.rst

@@ -1,6 +1,22 @@
 Changelog
 =========
 
+Unreleased
+----------
+
+- Rename multiple pipelines for consistency and precision:
+   * docker: analyze_docker_image
+   * root_filesystems: analyze_root_filesystem_or_vm_image
+   * docker_windows: analyze_windows_docker_image
+   * inspect_manifest: inspect_packages
+   * deploy_to_develop: map_deploy_to_develop
+   * scan_package: scan_single_package
+
+  A data migration is included to facilitate the migration of existing data.
+  Only the new names are available in the web UI but the REST API and CLI are backward
+  compatible with the old names.
+  https://github.com/nexB/scancode.io/issues/1044
+
 v33.0.0 (2024-01-16)
 --------------------
 

docs/automation.rst

@@ -93,7 +93,7 @@ For instance, you can create a project and trigger it using the following comman
 crontab::
 
     docker compose exec -it web scanpipe create-project scan-$(date +"%Y-%m-%dT%H:%M:%S") \
-      --pipeline scan_package \
+      --pipeline scan_single_package \
       --input-url https://github.com/package-url/packageurl-python/archive/refs/heads/main.zip \
       --execute
 

docs/built-in-pipelines.rst

@@ -18,26 +18,26 @@ Pipeline Base Class
     :members:
     :member-order: bysource
 
-.. _pipeline_deploy_to_develop:
+.. _pipeline_analyze_docker_image:
 
-Deploy To Develop
------------------
-.. autoclass:: scanpipe.pipelines.deploy_to_develop.DeployToDevelop()
+Analyse Docker Image
+--------------------
+.. autoclass:: scanpipe.pipelines.docker.Docker()
     :members:
     :member-order: bysource
 
-.. _pipeline_docker:
+.. _pipeline_analyze_root_filesystem:
 
-Docker Image Analysis
----------------------
-.. autoclass:: scanpipe.pipelines.docker.Docker()
+Analyze Root Filesystem or VM Image
+-----------------------------------
+.. autoclass:: scanpipe.pipelines.root_filesystem.RootFS()
     :members:
     :member-order: bysource
 
-.. _pipeline_docker_windows:
+.. _analyze_windows_docker_image:
 
-Docker Windows Image Analysis
------------------------------
+Analyse Docker Windows Image
+----------------------------
 .. autoclass:: scanpipe.pipelines.docker_windows.DockerWindows()
     :members:
     :member-order: bysource
@@ -50,35 +50,35 @@ Find Vulnerabilities
     :members:
     :member-order: bysource
 
-.. _pipeline_inspect_manifest:
+.. _pipeline_inspect_packages:
 
-Inspect Manifest
+Inspect Packages
 ----------------
-.. autoclass:: scanpipe.pipelines.inspect_manifest.InspectManifest()
+.. autoclass:: scanpipe.pipelines.inspect_packages.InspectPackages()
     :members:
     :member-order: bysource
 
 .. _pipeline_load_inventory:
 
-Load Inventory From Scan
-------------------------
+Load Inventory
+--------------
 .. autoclass:: scanpipe.pipelines.load_inventory.LoadInventory()
     :members:
     :member-order: bysource
 
-.. _pipeline_populate_purldb:
+.. _pipeline_map_deploy_to_develop:
 
-Populate PurlDB
----------------
-.. autoclass:: scanpipe.pipelines.populate_purldb.PopulatePurlDB()
+Map Deploy To Develop
+---------------------
+.. autoclass:: scanpipe.pipelines.deploy_to_develop.DeployToDevelop()
     :members:
     :member-order: bysource
 
-.. _pipeline_root_filesystems:
+.. _pipeline_populate_purldb:
 
-Root Filesystem Analysis
-------------------------
-.. autoclass:: scanpipe.pipelines.root_filesystems.RootFS()
+Populate PurlDB
+---------------
+.. autoclass:: scanpipe.pipelines.populate_purldb.PopulatePurlDB()
     :members:
     :member-order: bysource
 
@@ -98,10 +98,10 @@ Scan Codebase Package
     :members:
     :member-order: bysource
 
-.. _pipeline_scan_package:
+.. _pipeline_scan_single_package:
 
-Scan Package
-------------
-.. autoclass:: scanpipe.pipelines.scan_package.ScanPackage()
+Scan Single Package
+-------------------
+.. autoclass:: scanpipe.pipelines.scan_single_package.ScanSinglePackage()
     :members:
     :member-order: bysource

docs/command-line-interface.rst

@@ -172,7 +172,7 @@ You can use more than one ``PIPELINE_NAME`` to add multiple pipelines at once.
 For example, assuming you have created beforehand a project named "foo", this will
 add the docker pipeline to your project::
 
-    $ scanpipe add-pipeline --project foo docker
+    $ scanpipe add-pipeline --project foo analyze_docker_image
 
 
 `$ scanpipe execute --project PROJECT`

docs/distros-os-images.rst

@@ -39,18 +39,19 @@ may be only used for certain pipelines:
 - **RPM-based** Linux distros: RHEL, Fedora, openSUSE/SUSE
 - **Alpine** Linux distros
 
-For the above three flavors, the :ref:`docker <pipeline_docker>` and
-:ref:`root_filesystems <pipeline_root_filesystems>` pipelines support comprehensive
-detection of installed system packages, their provenance, their license metadata,
-and their installed files.
+For the above three flavors, the
+:ref:`analyze_docker_image <pipeline_analyze_docker_image>` and
+:ref:`analyze_root_filesystem_or_vm_image <pipeline_analyze_root_filesystem>` pipelines
+support comprehensive detection of installed system packages, their provenance,
+their license metadata, and their installed files.
 
-- For **Windows**, the :ref:`docker_windows <pipeline_docker_windows>` pipeline supports
-  Windows Docker images with extensive detection of installed Windows packages,
-  programs, and the majority of installed files.
+- For **Windows**, the :ref:`analyze_windows_docker_image <analyze_windows_docker_image>`
+  pipeline supports Windows Docker images with extensive detection of installed Windows
+  packages, programs, and the majority of installed files.
 
 - **Distroless** Docker images system packages are detected with the
-  :ref:`docker <pipeline_docker>` pipeline; package and license metadata are also
-  detected.
+  :ref:`analyze_docker_image <pipeline_analyze_docker_image>` pipeline; package and
+  license metadata are also detected.
   However, some work needs to be done to achieve comprehensive support and fix
   the issue of system packages ot tracking their installed files. Check `this
   open GitHub issue <https://github.com/GoogleContainerTools/distroless/issues/741>`_

docs/faq.rst

@@ -23,22 +23,22 @@ Selecting the right pipeline for your needs depends primarily on the type of inp
 data you have available.
 Here are some general guidelines based on different input scenarios:
 
-- If you have a **Docker image** as input, use the :ref:`docker <pipeline_docker>`
-  pipeline.
+- If you have a **Docker image** as input, use the
+  :ref:`analyze_docker_image <pipeline_analyze_docker_image>` pipeline.
 - For a full **codebase compressed as an archive**, choose the
   :ref:`scan_codebase <pipeline_scan_codebase>` pipeline.
 - If you have a **single package archive**, opt for the
-  :ref:`scan_package <pipeline_scan_package>` pipeline.
+  :ref:`scan_single_package <pipeline_scan_single_package>` pipeline.
 - When dealing with a **Linux root filesystem** (rootfs), the
-  :ref:`root_filesystems <pipeline_root_filesystems>` pipeline is the appropriate
-  choice.
+  :ref:`analyze_root_filesystem_or_vm_image <pipeline_analyze_root_filesystem>` pipeline
+  is the appropriate choice.
 - For processing the results of a **ScanCode-toolkit scan** or **ScanCode.io scan**,
   use the :ref:`load_inventory <pipeline_load_inventory>` pipeline.
 - When you have **manifest files**, such as a
   **CycloneDX BOM, SPDX document, lockfile**, etc.,
-  use the :ref:`inspect_manifest <pipeline_inspect_manifest>` pipeline.
+  use the :ref:`inspect_packages <pipeline_inspect_packages>` pipeline.
 - For scenarios involving both a **development and deployment codebase**, consider using
-  the :ref:`deploy_to_develop <pipeline_deploy_to_develop>` pipeline.
+  the :ref:`map_deploy_to_develop <pipeline_map_deploy_to_develop>` pipeline.
 
 These pipelines will automatically execute the necessary steps to scan and create the
 packages, dependencies, and resources for your project based on the input data provided.
@@ -56,10 +56,11 @@ by running some of the following additional pipelines:
   Please ensure that you have set up
   :ref:`PurlDB <scancodeio_settings_purldb>` before running this pipeline.
 
-What is the difference between scan_codebase and scan_package pipelines?
-------------------------------------------------------------------------
+What is the difference between scan_codebase and scan_single_package pipelines?
+-------------------------------------------------------------------------------
 
-The key differences are that the :ref:`scan_package <pipeline_scan_package>` pipeline
+The key differences are that the
+:ref:`scan_single_package <pipeline_scan_single_package>` pipeline
 treats the input as if it were a single package, such as a package archive, and
 computes a **License clarity** and a **Scan summary** to aggregate the package scan
 data:
@@ -116,8 +117,9 @@ The following tools and libraries are used during the docker images analysis pip
  - Secondary libraries and plugins from
    `scancode-plugins <https://github.com/nexB/scancode-plugins>`_.
 
-The pipeline documentation is available at :ref:`pipeline_docker` and its source code
-at `docker.py <https://github.com/nexB/scancode.io/blob/main/scanpipe/pipelines/docker.py>`_.
+The pipeline documentation is available at :ref:`pipeline_analyze_docker_image` and
+its source code at
+`docker.py <https://github.com/nexB/scancode.io/blob/main/scanpipe/pipelines/docker.py>`_.
 It is hopefully designed to be simple and readable code.
 
 Am I able to run ScanCode.io on Windows?

docs/output-files.rst

@@ -69,7 +69,7 @@ as shown below
         ],
     "runs": [
       {
-        "pipeline_name": "docker",
+        "pipeline_name": "analyze_docker_image",
         "description": "A pipeline to analyze a Docker image.",
         "uuid": "5f1ec0c5-91ed-45c8-ab3d-beae44018716",
         "created_date": "2021-06-13T00:50:18.367560Z",
@@ -78,7 +78,7 @@ as shown below
         "task_end_date": "2021-06-13T01:20:56.486136Z",
         "task_exitcode": 0,
         "task_output": "",
-        "log": "2021-06-13 01:20:47.66 Pipeline [docker] starting\n2021-06-13 01:20:47.66 Step [extract_images] starting\n2021-06-13 01:20:47.72 Step [extract_images] completed in 0.05 seconds\n2021-06-13 01:20:47.72 Step [extract_layers] starting\n2021-06-13 01:20:47.84 Step [extract_layers] completed in 0.12 seconds\n2021-06-13 01:20:47.84 Step [find_images_linux_distro] starting\n2021-06-13 01:20:47.84 Step [find_images_linux_distro] completed in 0.00 seconds\n2021-06-13 01:20:47.85 Step [collect_images_information] starting\n2021-06-13 01:20:47.85 Step [collect_images_information] completed in 0.00 seconds\n2021-06-13 01:20:47.85 Step [collect_and_create_codebase_resources] starting\n2021-06-13 01:20:48.65 Step [collect_and_create_codebase_resources] completed in 0.79 seconds\n2021-06-13 01:20:48.65 Step [collect_and_create_system_packages] starting\n2021-06-13 01:20:50.89 Step [collect_and_create_system_packages] completed in 2.24 seconds\n2021-06-13 01:20:50.89 Step [flag_uninteresting_codebase_resources] starting\n2021-06-13 01:20:50.90 Step [tag_uninteresting_codebase_resources] completed in 0.00 seconds\n2021-06-13 01:20:50.90 Step [tag_empty_files] starting\n2021-06-13 01:20:50.91 Step [tag_empty_files] completed in 0.00 seconds\n2021-06-13 01:20:50.91 Step [scan_for_application_packages] starting\n2021-06-13 01:20:50.98 Step [scan_for_application_packages] completed in 0.07 seconds\n2021-06-13 01:20:50.98 Step [scan_for_files] starting\n2021-06-13 01:20:56.46 Step [scan_for_files] completed in 5.48 seconds\n2021-06-13 01:20:56.46 Step [analyze_scanned_files] starting\n2021-06-13 01:20:56.47 Step [analyze_scanned_files] completed in 0.00 seconds\n2021-06-13 01:20:56.47 Step [tag_not_analyzed_codebase_resources] starting\n2021-06-13 01:20:56.48 Step [tag_not_analyzed_codebase_resources] completed in 0.00 seconds\n2021-06-13 01:20:56.48 Pipeline completed\n",
+        "log": "2021-06-13 01:20:47.66 Pipeline [analyze_docker_image] starting\n2021-06-13 01:20:47.66 Step [extract_images] starting\n2021-06-13 01:20:47.72 Step [extract_images] completed in 0.05 seconds\n2021-06-13 01:20:47.72 Step [extract_layers] starting\n2021-06-13 01:20:47.84 Step [extract_layers] completed in 0.12 seconds\n2021-06-13 01:20:47.84 Step [find_images_linux_distro] starting\n2021-06-13 01:20:47.84 Step [find_images_linux_distro] completed in 0.00 seconds\n2021-06-13 01:20:47.85 Step [collect_images_information] starting\n2021-06-13 01:20:47.85 Step [collect_images_information] completed in 0.00 seconds\n2021-06-13 01:20:47.85 Step [collect_and_create_codebase_resources] starting\n2021-06-13 01:20:48.65 Step [collect_and_create_codebase_resources] completed in 0.79 seconds\n2021-06-13 01:20:48.65 Step [collect_and_create_system_packages] starting\n2021-06-13 01:20:50.89 Step [collect_and_create_system_packages] completed in 2.24 seconds\n2021-06-13 01:20:50.89 Step [flag_uninteresting_codebase_resources] starting\n2021-06-13 01:20:50.90 Step [tag_uninteresting_codebase_resources] completed in 0.00 seconds\n2021-06-13 01:20:50.90 Step [tag_empty_files] starting\n2021-06-13 01:20:50.91 Step [tag_empty_files] completed in 0.00 seconds\n2021-06-13 01:20:50.91 Step [scan_for_application_packages] starting\n2021-06-13 01:20:50.98 Step [scan_for_application_packages] completed in 0.07 seconds\n2021-06-13 01:20:50.98 Step [scan_for_files] starting\n2021-06-13 01:20:56.46 Step [scan_for_files] completed in 5.48 seconds\n2021-06-13 01:20:56.46 Step [analyze_scanned_files] starting\n2021-06-13 01:20:56.47 Step [analyze_scanned_files] completed in 0.00 seconds\n2021-06-13 01:20:56.47 Step [tag_not_analyzed_codebase_resources] starting\n2021-06-13 01:20:56.48 Step [tag_not_analyzed_codebase_resources] completed in 0.00 seconds\n2021-06-13 01:20:56.48 Pipeline completed\n",
         "execution_time": 8
       }
     ],

docs/rest-api.rst

@@ -94,7 +94,7 @@ Using cURL:
     data='{
         "name": "project_name",
         "input_urls": "https://download.url/package.archive",
-        "pipeline": "scan_package",
+        "pipeline": "scan_single_package",
         "execute_now": true
     }'
 
@@ -111,7 +111,7 @@ Using cURL:
         upload_file="/path/to/the/archive.zip"
 
         curl -F "name=project_name" \
-             -F "pipeline=scan_package" \
+             -F "pipeline=scan_single_package" \
              -F "execute_now=True" \
              -F "upload_file=@$upload_file" \
              "$api_url"
@@ -131,7 +131,7 @@ Using Python and the **"requests"** library:
     data = {
         "name": "project_name",
         "input_urls": "https://download.url/package.archive",
-        "pipeline": "scan_package",
+        "pipeline": "scan_single_package",
         "execute_now": True,
     }
     response = requests.post(api_url, data=data)
@@ -149,7 +149,7 @@ Using Python and the **"requests"** library:
         api_url = "http://localhost/api/projects/"
         data = {
             "name": "project_name",
-            "pipeline": "scan_package",
+            "pipeline": "scan_single_package",
             "execute_now": True,
         }
         files = {"upload_file": open("/path/to/the/archive.zip", "rb")}
@@ -279,7 +279,7 @@ Using cURL:
     api_url="http://localhost/api/projects/6461408c-726c-4b70-aa7a-c9cc9d1c9685/add_pipeline/"
     content_type="Content-Type: application/json"
     data='{
-        "pipeline": "docker",
+        "pipeline": "analyze_docker_image",
         "execute_now": true
     }'
 
@@ -434,7 +434,7 @@ The run details view returns all information available about a pipeline run.
 
     {
         "url": "http://localhost/api/runs/8d5c3962-5fca-47d7-b8c8-47a19247714e/",
-        "pipeline_name": "scan_package",
+        "pipeline_name": "scan_single_package",
         "status": "success",
         "description": "A pipeline to scan a single package archive with ScanCode-toolkit.",
         "project": "http://localhost/api/projects/cd5b0459-303f-4e92-99c4-ea6d0a70193e/",

docs/tutorial_api_analyze_package_archive.rst

@@ -15,7 +15,7 @@ Instructions:
 - First, let's create a new project called ``boolean.py-3.8``.
 - We'll be using this `package <https://github.com/bastikr/boolean.py/archive/refs/tags/v3.8.zip>`_
   as the project input.
-- We can add and execute the scan_package pipeline on our new project.
+- We can add and execute the scan_single_package pipeline on our new project.
 
 .. note::
     Whether you follow this tutorial and previous instructions using cURL or
@@ -33,7 +33,7 @@ Using cURL
     data='{
         "name": "boolean.py-3.8",
         "input_urls": "https://github.com/bastikr/boolean.py/archive/refs/tags/v3.8.zip",
-        "pipeline": "scan_package",
+        "pipeline": "scan_single_package",
         "execute_now": true
     }'
 
@@ -52,7 +52,7 @@ Using cURL
         {
             "name": "boolean.py-3.8",
             "input_urls": "https://github.com/bastikr/boolean.py/archive/refs/tags/v3.8.zip",
-            "pipeline": "scan_package",
+            "pipeline": "scan_single_package",
             "execute_now": true
         }
 
@@ -100,7 +100,7 @@ Using Python script
     data = {
         "name": "boolean.py-3.8",
         "input_urls": "https://github.com/bastikr/boolean.py/archive/refs/tags/v3.8.zip",
-        "pipeline": "scan_package",
+        "pipeline": "scan_single_package",
         "execute_now": True,
     }
     response = requests.post(api_url, data=data)