From f2d6f5c243068e26d602c3838a2b6a011683c021 Mon Sep 17 00:00:00 2001 From: Cody Bruno Date: Thu, 6 Feb 2025 16:09:05 -0500 Subject: [PATCH] Add row enirchment info and remove unsupported section from plugin management guide --- docs/manage/plugin.md | 27 ++++--------------- .../develop/plugin-release-checklist.md | 15 ++++++++++- 2 files changed, 19 insertions(+), 23 deletions(-) diff --git a/docs/manage/plugin.md b/docs/manage/plugin.md index 5200592..dea939d 100644 --- a/docs/manage/plugin.md +++ b/docs/manage/plugin.md @@ -26,7 +26,7 @@ For example, to install the 0.118.0 version of the aws plugin: $ tailpipe plugin install aws@0.118.0 ``` -This will download the aws plugin version 0.118.0 (the one with the `0.118.0` tag) from the Hub registry. +This will download the aws plugin version 0.118.0 (the one with the `0.118.0` tag) from the Hub registry. ## Installing from a SemVer Constraint @@ -34,7 +34,7 @@ Plugins should follow [semantic versioning](https://semver.org/) guidelines, and The intent of the version tag is that it is immutable - while it is technically possible to move the version tag to a different image version, this should not be done. -Installing with a semver constraint allows you to "lock" (or pin) to a specific set of releases which match the contraints. +Installing with a semver constraint allows you to "lock" (or pin) to a specific set of releases which match the contraints. If you install via `tailpipe plugin install aws@^1`, for example, `tailpipe plugin update` (and auto-updates) will only update to versions greater than `1.0.0` but less than `2.0.0`. @@ -72,7 +72,7 @@ $ tailpipe plugin install aws@2.x.x ``` - To install the latest version locked to a specific minor version: -```bash +```bash $ tailpipe plugin install aws@~2.1 # or $ tailpipe plugin install aws@2.1.x @@ -86,23 +86,6 @@ Tailpipe plugins are packaged in OCI format and can be hosted and installed from $ tailpipe plugin install us-docker.pkg.dev/myproject/myrepo/myplugin@mytag ``` -## Installing from a File - -A plugin binary can be installed manually, and this is often convenient when developing the plugin. Tailpipe will attempt to load any plugin that is referred to in a `connection` configuration: -- The plugin binary file must have a `.plugin` extension -- The plugin binary must reside in a subdirectory of the `~/.tailpipe/plugins/` directory and must be the ONLY `.plugin` file in that subdirectory -- The `connection` must specify the path (relative to `~/.tailpipe/plugins/`) to the plugin in the `plugin` argument - -For example, consider a `myplugin` plugin that you have developed. To install it: -- Create a subdirectory `.tailpipe/plugins/local/myplugin` -- Name your plugin binary `myplugin.plugin`, and copy it to `.tailpipe/plugins/local/myplugin/myplugin.plugin` -- Create a `~/.tailpipe/config/myplugin.tpc` config file containing a connection definition that points to your plugin: - ```hcl - connection "myplugin" { - plugin = "local/myplugin" - } - ``` - ## Viewing Installed Plugins You can list the installed plugins with the `tailpipe plugin list` command: @@ -120,7 +103,7 @@ To update a plugin to the latest version for a given stream, you can use the `t ``` tailpipe plugin update plugin_name[@stream] -``` +``` The syntax and semantics are identical to the install command - `tailpipe plugin update aws` will get the latest aws plugin, `tailpipe plugin update aws@1` will get the latest in the 1.x major stream, etc. @@ -136,7 +119,7 @@ You can uninstall a plugin with the `tailpipe plugin uninstall` command: ``` tailpipe plugin uninstall [plugin] -``` +``` ## Tailpipe Plugin Registry Support Lifecycle diff --git a/docs/reference/develop/plugin-release-checklist.md b/docs/reference/develop/plugin-release-checklist.md index e46cb31..7227d99 100644 --- a/docs/reference/develop/plugin-release-checklist.md +++ b/docs/reference/develop/plugin-release-checklist.md @@ -93,6 +93,17 @@ Every table and column has a description. These are consistent across tables. Th ## Table and Column Design + Row enrichment + +The table enriches the row with the following required [common fields](https://tailpipe.io/docs/manage/table#common-fields): +- `tp_date` - The date the event was originally generated. +- `tp_id` - A unique identifier for the row. In Turbot plugins, typically set to an [xid](https://github.com/rs/xid). +- `tp_index` - The index used to partition the data, e.g., AWS account ID, GitHub organization, hostname. +- `tp_ingest_timestamp` - The timestamp when the event was ingested into the system. +- `tp_timestamp` - The timestamp when the event was originally generated. + +Additional common fields, like `tp_ips` and `tp_source_location` should be added based on what is available in the log type. + Default file layout For each supported artifact source, the table defines a default `FileLayout` that uses the most common pattern. For instance, the `aws_cloudtrail_log` uses the standard AWS log format as the default `FileLayout` for the `aws_s3_bucket` source as part of its [GetSourceMetadata function](https://github.com/turbot/tailpipe-plugin-aws/blob/6b2620d49330aff84f8879e2740fabb39fc87b79/tables/cloudtrail_log/cloudtrail_log_table.go#L26-L28). @@ -144,7 +155,9 @@ The color matches the provider's brand guidelines, typically stated on a page li The description in `docs/index.md` is appropriate for the provider. The [AWS plugin](https://hub.tailpipe.io/plugins/turbot/aws), for example, uses: -> AWS provides on-demand cloud computing platforms and APIs to authenticated customers on a metered pay-as-you-go basis. +``` +AWS provides on-demand cloud computing platforms and APIs to authenticated customers on a metered pay-as-you-go basis. +``` The opening sentence of the Wikipedia page for the provider can be a good source of guidance here.