Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Cloud docs restructure #23319

Merged
merged 10 commits into from
Feb 28, 2023
2 changes: 1 addition & 1 deletion docs/archive/faq/data-loading.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ according to the frequency you set.

## **How frequently can Airbyte sync data?**

You can adjust the load time to run as frequent as every hour or as infrequent as once a year using [Cron expressions](https://docs.airbyte.com/cloud/managing-airbyte-cloud#edit-stream-configuration).
You can adjust the load time to run as frequent as every hour or as infrequent as once a year using [Cron expressions](https://docs.airbyte.com/cloud/managing-airbyte-cloud/edit-stream-configuration).

## **Why wouldn’t I choose to load all of my data more frequently?**

Expand Down
2 changes: 1 addition & 1 deletion docs/cloud/core-concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ After a sync is complete, Airbyte normalizes the data. When setting up a connect

* Raw data (no normalization): Airbyte places the JSON blob version of your data in a table called `_airbyte_raw_<stream name>`
* Basic Normalization: Airbyte converts the raw JSON blob version of your data to the format of your destination. *Note: Not all destinations support normalization.*
* [dbt Cloud integration](https://docs.airbyte.com/cloud/dbt-cloud-integration): Airbyte's dbt Cloud integration allows you to use dbt Cloud for transforming and cleaning your data during the normalization process.
* [dbt Cloud integration](https://docs.airbyte.com/cloud/managing-airbyte-cloud/dbt-cloud-integration): Airbyte's dbt Cloud integration allows you to use dbt Cloud for transforming and cleaning your data during the normalization process.

:::note

Expand Down
2 changes: 1 addition & 1 deletion docs/cloud/getting-started-with-airbyte-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ Verify the sync by checking the logs:
3. Check the data at your destination. If you added a Destination Stream Prefix while setting up the connection, make sure to search for the stream name with the prefix.

## Allowlist IP addresses
Depending on your [data residency](https://docs.airbyte.com/cloud/managing-airbyte-cloud#choose-your-default-data-residency) location, you may need to allowlist the following IP addresses to enable access to Airbyte:
Depending on your [data residency](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-data-residency#choose-your-default-data-residency) location, you may need to allowlist the following IP addresses to enable access to Airbyte:

### United States and Airbyte Default
#### GCP region: us-west3
Expand Down
2 changes: 1 addition & 1 deletion docs/cloud/managing-airbyte-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -339,7 +339,7 @@ If a connection’s source schema has breaking changes, it will stop syncing. Yo
### Enable schema update notifications

To get notified when your source schema changes:
1. Make sure you have [webhook notifications](https://docs.airbyte.com/cloud/managing-airbyte-cloud#manage-airbyte-cloud-notifications) set up.
1. Make sure you have [webhook notifications](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications#set-up-slack-notifications) set up.

2. On the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard, click **Connections** and select the connection you want to receive notifications for.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# Using the dbt Cloud integration
# Use the dbt Cloud integration

By using the dbt Cloud integration, you can create and run dbt transformations during syncs in Airbyte Cloud. This allows you to transform raw data into a format that is suitable for analysis and reporting, including cleaning and enriching the data.

## Step 1: Generate a service token

Expand All @@ -19,7 +21,7 @@ To set up the dbt Cloud integration in Airbyte Cloud:

2. Click **dbt Cloud integration**.

3. Paste the service token from [Step 1](#step-1-prerequisites) and click **Save changes**.
3. Paste the service token from [Step 1](#step-1-generate-a-service-token) and click **Save changes**.

4. Click **Connections** and select the connection you want to add a dbt transformation to.

Expand Down
94 changes: 94 additions & 0 deletions docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Edit stream configuration

By editing stream configurations, you can customize how your data syncs to the destination. This allows you to manage aspects of the sync, such as replication frequency, destination namespace, and ensuring your data is written to the correct location.

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Connections** and then click the connection you want to change.

2. Click the **Replication** tab.

The **Transfer** and **Streams** settings include the following parameters:

| Parameter | Description |
|--------------------------------------|-------------------------------------------------------------------------------------|
| Replication frequency | How often the data syncs |
| [Non-breaking schema updates](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes) detected | How Airbyte handles syncs when it detects non-breaking schema changes in the source |
| Destination Namespace | Where the replicated data is written |
| Destination Stream Prefix | Helps you identify streams from different connectors |

:::note

These parameters apply to all streams in the connection.

:::

If you need to use [cron scheduling](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html):
1. In the **Replication Frequency** dropdown, click **Cron**.
2. Enter a cron expression and choose a time zone to create a sync schedule.

:::note

* Only one sync per connection can run at a time.
* If cron schedules a sync to run before the last one finishes, the scheduled sync will start after the last sync completes.
* Airbyte Cloud does not allow schedules that sync more than once per hour.

:::

In the **Activate the streams you want to sync section**, you can make changes to any stream you choose.

To search for a stream:

1. Click the **Search stream name** search box.

2. Type the name of the stream you want to find.

3. Streams matching your search are displayed in the list.

To change individual stream configuration:

![Single Edit Gif 7](https://user-images.githubusercontent.com/106352739/187313088-85c61a6d-1025-45fa-b14e-a7fe86defea4.gif)

1. In the **Sync** column of the stream, toggle the sync on or off.

2. Click the dropdown arrow in the **Sync mode** column and select the sync mode you want to apply.

:::note

Depending on the sync mode you select, you may need to choose a cursor or primary key.

:::

3. If there is a dropdown arrow in the **Cursor** or **Primary key** fields, click the dropdown arrow and choose the cursor or primary key.

To change multiple stream configurations:

![Batch Edit gif 5](https://user-images.githubusercontent.com/106352739/187312110-d16b4f9a-9d43-4b23-b644-b64004f33b58.gif)

1. Click the first checkbox in the table header to select all streams in the connection.

2. Deselect the checkboxes of streams you do not want to apply these changes to.

3. In the highlighted header of the table, toggle the sync on or off.

4. Click the dropdown arrow in the **Sync mode** column and select the sync mode you want to apply to these streams.

5. If there is a dropdown arrow in the **Cursor** or **Primary key** fields of the highlighted table header, click the dropdown arrow and choose the cursor or primary key.

6. Click **Apply** to apply these changes to the streams you selected, or click **Cancel** to discard the changes.

To save the changes:
1. Click **Save changes**, or click **Cancel** to discard the changes.

2. The **Stream configuration changed** dialog displays. This gives you the option to reset streams when you save the changes.

:::caution

Airbyte recommends that you reset streams. A reset will delete data in the destination of the affected streams and then re-sync that data. Skipping a reset is discouraged and might lead to unexpected behavior.

:::

3. Click **Save connection**, or click **Cancel** to close the dialog.

To refresh the source schema:
1. Click **Refresh source schema** to fetch the schema of your data source.

2. If the schema has changed, the **Refreshed source schema** dialog displays them.
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Manage Airbyte Cloud notifications

This page provides guidance on how to manage notifications for Airbyte Cloud, allowing you to stay up-to-date on the activities in your workspace.

## Set up Webhook notifications

To set up Webhook notifications:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Settings**.

2. Click **Notifications**.

3. [Create an Incoming Webhook for Slack](https://api.slack.com/messaging/webhooks).

4. Navigate back to the Airbyte Cloud dashboard > Settings > Notifications and enter the Webhook URL.

5. Toggle the **When sync fails** and **When sync succeeds** buttons as required.

6. Click **Save changes**.

## Enable schema update notifications

To get notified when your source schema changes:
1. Make sure you have [Webhook notifications](#set-up-webhook-notifications) set up.

2. On the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard, click **Connections** and select the connection you want to receive notifications for.

3. Click the **Settings** tab on the Connection page.

4. Toggle **Schema update notifications**.
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Manage your Airbyte Cloud workspace

An Airbyte Cloud workspace allows you to collaborate with other users and manage connections under a shared billing account.

:::info
Airbyte [credits](https://airbyte.com/#) are assigned per workspace and cannot be transferred between workspaces.
:::

## Add users to your workspace

To add a user to your workspace:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Settings**.

2. Click **Access Management**.

3. Click **+ New user**.

4. On the **Add new users** dialog, enter the email address of the user you want to invite to your workspace.

5. Click **Send invitation**.

:::info
The user will have access to only the workspace you invited them to. They will be added as a workspace admin by default.
:::

## Remove users from your workspace​

To remove a user from your workspace:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Settings**.

2. Click **Access Management**.

3. Click **Remove** next to the user’s email.

4. The **Remove user** dialog displays. Click **Remove**.

## Rename a workspace

To rename a workspace:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Settings**.

2. Click **General Settings**.

3. In the **Workspace name** field, enter the new name for your workspace.

4. Click **Save changes**.

## Delete a workspace

To delete a workspace:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Settings**.

2. Click **General Settings**.

3. In the **Delete your workspace** section, click **Delete**.

## Single workspace vs. multiple workspaces

You can use one or multiple workspaces with Airbyte Cloud.

### Access
| Number of workspaces | Benefits | Considerations |
|----------------------|-------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
| Single | All users in a workspace have access to the same data. | If you add a user to a workspace, you cannot limit their access to specific data within that workspace. |
| Multiple | You can create multiple workspaces to allow certain users to access the data. | Since you have to manage user access for each workspace individually, it can get complicated if you have many users in multiple workspaces. |

### Billing
| Number of workspaces | Benefits | Considerations |
|----------------------|-------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
| Single | You can use the same payment method for all purchases. | Credits pay for the use of resources in a workspace when you run a sync. Resource usage cannot be divided and paid for separately (for example, you cannot bill different departments in your organization for the usage of some credits in one workspace). |
| Multiple | Workspaces are independent of each other, so you can use a different payment method card for each workspace (for example, different credit cards per department in your organization). | You can use the same payment method for different workspaces, but each workspace is billed separately. Managing billing for each workspace can become complicated if you have many workspaces. |

## Switch between multiple workspaces

To switch between workspaces:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click the current workspace name under the Airbyte logo in the navigation bar.

2. Click **View all workspaces**.

3. Click the name of the workspace you want to switch to.
72 changes: 72 additions & 0 deletions docs/cloud/managing-airbyte-cloud/manage-credits.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Manage credits

This page provides guidance on enrolling in Airbyte Cloud’s [Free Connector Program](https://airbyte.com/free-connector-program) and purchasing credits.

## Enroll in the Free Connector Program

The Free Connector Program allows you to sync connections with [alpha](https://docs.airbyte.com/project-overview/product-release-stages#alpha) or [beta](https://docs.airbyte.com/project-overview/product-release-stages/#beta) connectors at no cost.

:::note

You must be enrolled in the program to use alpha and beta connectors for free. If either the source or destination is in alpha or beta, the whole connection is free to sync. When both the source and destination of a connection become [generally available](https://docs.airbyte.com/project-overview/product-release-stages/#general-availability-ga) (GA), the connection will no longer be free. We will email you two weeks before both connectors in a connection move to GA.

:::

Before enrolling in the program, [set up](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-source) at least one alpha or beta connector and verify your email if you haven't already.

To enroll in the program:
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Credits** in the navigation bar.

2. Click **Enroll now** in the **Free Connector Program** banner.

3. Click **Enroll now**.

4. Input your credit card information and click **Save card**.

:::note

Credit card information is required, even if you previously bought credits on Airbyte Cloud. This ensures uninterrupted syncs when both connectors move to GA.

:::

Since alpha and beta connectors are still in development, support is not provided. For additional resources, check out our [Connector Catalog](https://docs.airbyte.com/integrations/), [Troubleshooting & FAQ](https://docs.airbyte.com/troubleshooting/), and our [Community Slack](https://slack.airbyte.io/).

## Buy credits

Airbyte [credits](https://airbyte.com/#) are used to pay for Airbyte resources when you run a sync. You can purchase credits on Airbyte Cloud to keep your data flowing without interruption.

To buy credits:

1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Credits** in the navigation bar.

2. If you are unsure of how many credits you need, click **Talk to Sales** to find the right amount for your team.

3. Click **Buy credits**.

4. The Stripe payment page displays. If you want to change the amount of credits, click Qty **200**. The **Update quantity** dialog displays, and you can either type the amount or use minus (**-**) or plus (**+**) to change the quantity. Click **Update**.

:::note

Purchase limits:
* Minimum: 100 credits
* Maximum: 999 credits

:::

To buy more credits or a subscription plan, reach out to [Sales](https://airbyte.com/talk-to-sales).

5. Fill out the payment information.

After you enter your billing address, sales tax is calculated and added to the total.

6. Click **Pay**.

Your payment is processed. The Credits page displays the updated quantity of credits, total credit usage, and the credit usage per connection.

A receipt for your purchase is sent to your email. [Email us](mailto:ar@airbyte.io) for an invoice.

:::note

Credits expire after one year if they are not used.

:::
56 changes: 56 additions & 0 deletions docs/cloud/managing-airbyte-cloud/manage-data-residency.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Manage data residency

In Airbyte Cloud, you can set the default data residency and choose the data residency for individual connections, which can help you comply with data localization requirements.

## Choose your default data residency

Default data residency allows you to choose where your data is processed.

:::note

Configuring default data residency only applies to new connections and does not affect existing connections.

:::

For individual connections, you can choose a data residency that is different from the default through [connection settings](#choose-the-data-residency-for-a-connection) or when you create a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection).

:::note

While the data is processed in a data plane in the chosen residency, the cursor and primary key data is stored in the US control plane. If you have data that cannot be stored in the US, do not use it as a cursor or primary key.

:::

To choose your default data residency:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Settings**.

2. Click **Data Residency**.

3. Click the dropdown and choose the location for your default data residency.

4. Click **Save changes**.

:::info

Depending on your network configuration, you may need to add [IP addresses](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud/#allowlist-ip-addresses) to your allowlist.

:::

## Choose the data residency for a connection
You can choose the data residency for your connection in the connection settings. You can also choose data residency when creating a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection), or you can set the [default data residency](#choose-your-default-data-residency) for your workspace.

To choose the data residency for your connection:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Connections** and then click the connection that you want to change.

2. Click the **Settings** tab.

3. Click the **Data residency** dropdown and choose the location for your default data residency.

4. Click **Save changes**

:::note

Changes to data residency will not affect any sync in progress.

:::
Loading