Skip to content

Commit 3ba5a3c

Browse files
sophia-wileyandyjihedmundito
authoredMar 28, 2023
Docs: streams table (#24326)
* Added updated text * edits * edits to text * removed header * edited links * edited some text * added new gifs to doc * typo fixes * Update docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md Co-authored-by: Andy Jih <andyjih@users.noreply.github.com> * Update docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md Co-authored-by: Andy Jih <andyjih@users.noreply.github.com> * Update edit-stream-configuration.md * Update docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> * Update edit-stream-configuration.md edited based on Edmundo's suggestions --------- Co-authored-by: Andy Jih <andyjih@users.noreply.github.com> Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com>
1 parent 25069a3 commit 3ba5a3c

File tree

2 files changed

+123
-41
lines changed

2 files changed

+123
-41
lines changed
 
Original file line numberDiff line numberDiff line change
@@ -1,94 +1,162 @@
1-
# Edit stream configuration
1+
# Manage syncs
22

3-
By editing stream configurations, you can customize how your data syncs to the destination. This allows you to manage aspects of the sync, such as replication frequency, destination namespace, and ensuring your data is written to the correct location.
3+
After you have created a connection, you can change how your data syncs to the destination by modifying the [configuration settings](#configure-connection-settings) and the [stream settings](#modify-streams-in-your-connection).
44

5-
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Connections** and then click the connection you want to change.
5+
## Configure connection settings
6+
7+
Configuring the connection settings allows you to manage various aspects of the sync, such as how often data syncs and where data is written.
8+
9+
To configure these settings:
10+
11+
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Connections** and then click the connection you want to change.
612

713
2. Click the **Replication** tab.
814

9-
The **Transfer** and **Streams** settings include the following parameters:
15+
3. Click the **Configuration** dropdown.
16+
17+
You can configure the following settings:
1018

11-
| Parameter | Description |
19+
:::note
20+
21+
These settings apply to all streams in the connection.
22+
23+
:::
24+
25+
| Setting | Description |
1226
|--------------------------------------|-------------------------------------------------------------------------------------|
1327
| Replication frequency | How often the data syncs |
28+
| Destination namespace | Where the replicated data is written |
29+
| Destination stream prefix | How you identify streams from different connectors |
1430
| [Non-breaking schema updates](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes/#review-non-breaking-schema-changes) detected | How Airbyte handles syncs when it detects non-breaking schema changes in the source |
15-
| Destination Namespace | Where the replicated data is written |
16-
| Destination Stream Prefix | Helps you identify streams from different connectors |
1731

18-
:::note
19-
20-
These parameters apply to all streams in the connection.
32+
To use [cron scheduling](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html):
2133

22-
:::
23-
24-
If you need to use [cron scheduling](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html):
2534
1. In the **Replication Frequency** dropdown, click **Cron**.
35+
2636
2. Enter a cron expression and choose a time zone to create a sync schedule.
2737

2838
:::note
2939

3040
* Only one sync per connection can run at a time.
31-
* If cron schedules a sync to run before the last one finishes, the scheduled sync will start after the last sync completes.
32-
* Airbyte Cloud does not allow schedules that sync more than once per hour.
41+
* If a sync is scheduled to run before the previous sync finishes, the scheduled sync will start after the completion of the previous sync.
42+
* Airbyte Cloud does not support schedules that sync more frequently than once per hour.
3343

3444
:::
3545

36-
In the **Activate the streams you want to sync section**, you can make changes to any stream you choose.
46+
## Modify streams in your connection
3747

38-
To search for a stream:
48+
In the **Activate the streams you want to sync** table, you can choose which streams to sync and how they are loaded to the destination.
3949

40-
1. Click the **Search stream name** search box.
50+
:::note
51+
A connection's schema consists of one or many streams. Each stream is most commonly associated with a database table or an API endpoint. Within a stream, there can be one or many fields or columns.
52+
:::
53+
54+
To modify streams:
55+
56+
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Connections** and then click the connection you want to change.
57+
58+
2. Click the **Replication** tab.
59+
60+
3. Scroll down to the **Activate the streams you want to sync** table.
61+
62+
:::note
4163

42-
2. Type the name of the stream you want to find.
64+
You can modify a single stream, or you can modify multiple streams at once.
4365

44-
3. Streams matching your search are displayed in the list.
66+
:::
4567

46-
To change individual stream configuration:
68+
Modify an individual stream:
4769

48-
![Single Edit Gif 7](https://user-images.githubusercontent.com/106352739/187313088-85c61a6d-1025-45fa-b14e-a7fe86defea4.gif)
70+
![gif-single-edit-march-2023](https://user-images.githubusercontent.com/106352739/226917693-068256da-c948-4f22-bdce-49f5bad95bf6.gif)
4971

50-
1. In the **Sync** column of the stream, toggle the sync on or off.
72+
1. In the **Activate the streams you want to sync** table, toggle **Sync** on or off for your selected stream.
5173

52-
2. Click the dropdown arrow in the **Sync mode** column and select the sync mode you want to apply.
74+
2. Click the **Sync mode** dropdown and select the sync mode you want to apply.
5375

5476
:::note
5577

5678
Depending on the sync mode you select, you may need to choose a cursor or primary key.
5779

5880
:::
5981

60-
3. If there is a dropdown arrow in the **Cursor** or **Primary key** fields, click the dropdown arrow and choose the cursor or primary key.
82+
3. Select the **Cursor** or **Primary keys** when they are required by the stream sync mode.
6183

62-
To change multiple stream configurations:
84+
:::note
6385

64-
![Batch Edit gif 5](https://user-images.githubusercontent.com/106352739/187312110-d16b4f9a-9d43-4b23-b644-b64004f33b58.gif)
86+
Source-defined cursors and primary keys are selected automatically and cannot be changed in the table.
6587

66-
1. Click the first checkbox in the table header to select all streams in the connection.
67-
68-
2. Deselect the checkboxes of streams you do not want to apply these changes to.
88+
:::
89+
90+
4. Click on a stream to display the stream details panel.
91+
92+
5. Toggle individual fields to include or exclude them in the sync, or use the toggle in the table header to select all fields at once.
6993

70-
3. In the highlighted header of the table, toggle the sync on or off.
94+
:::note
95+
96+
* You can only deselect top-level fields. You cannot deselect nested fields.
97+
* The Airbyte platform may read all data from the source (depending on the source), but it will only write data to the destination from fields you selected. Deselecting fields will not prevent the Airbyte platform from reading them.
98+
* When you refresh the schema, newly added fields will be selected by default, even if you have previously deselected fields in that stream.
7199

72-
4. Click the dropdown arrow in the **Sync mode** column and select the sync mode you want to apply to these streams.
100+
:::
73101

74-
5. If there is a dropdown arrow in the **Cursor** or **Primary key** fields of the highlighted table header, click the dropdown arrow and choose the cursor or primary key.
102+
6. Depending on the sync mode you chose for your connection, you can select the **Cursor** or **Primary keys** for individual fields in this table.
75103

76-
6. Click **Apply** to apply these changes to the streams you selected, or click **Cancel** to discard the changes.
104+
7. Click the **X** to close the stream details panel.
77105

78-
To save the changes:
79-
1. Click **Save changes**, or click **Cancel** to discard the changes.
106+
8. Click **Save changes**, or click **Cancel** to discard the changes.
80107

81-
2. The **Stream configuration changed** dialog displays. This gives you the option to reset streams when you save the changes.
108+
9. The **Stream configuration changed** dialog displays. This gives you the option to reset streams when you save the changes.
82109

83110
:::caution
84111

85112
Airbyte recommends that you reset streams. A reset will delete data in the destination of the affected streams and then re-sync that data. Skipping a reset is discouraged and might lead to unexpected behavior.
86113

87114
:::
88115

89-
3. Click **Save connection**, or click **Cancel** to close the dialog.
116+
10. Click **Save connection**.
117+
118+
Modify multiple streams:
119+
120+
![gif-batch-edit-march-2023](https://user-images.githubusercontent.com/106352739/226917994-c43941db-bb54-4a12-8270-f24fc4e2e6a7.gif)
121+
122+
1. In the **Activate the streams you want to sync** table, select the checkboxes of streams that you want to apply changes to.
123+
124+
:::note
90125

91-
To refresh the source schema:
92-
1. Click **Refresh source schema** to fetch the schema of your data source.
126+
To select or deselect all streams, click the checkbox in the table header. To deselect an individual stream, deselect its checkbox in the table.
127+
128+
:::
129+
130+
* In the highlighted footer of the table:
131+
132+
1. Toggle **Sync** on or off.
133+
134+
2. Click the **Sync mode** dropdown and select the sync mode you want to apply.
135+
136+
:::note
137+
138+
Depending on the sync mode you select, you may need to choose a cursor or primary key.
139+
140+
:::
141+
142+
3. Select the **Cursor** and **Primary keys** if there are dropdowns in those fields.
143+
144+
:::note
145+
146+
Source-defined cursors and primary keys cannot be changed while configuring multiple streams.
147+
148+
:::
149+
150+
4. Click **Apply** to apply these changes to the streams you selected, or click **Cancel** to discard the changes.
151+
152+
2. Click **Save changes**, or click **Cancel** to discard the changes.
153+
154+
3. The **Stream configuration changed** dialog displays. This gives you the option to reset streams when you save the changes.
155+
156+
:::caution
157+
158+
Airbyte recommends that you reset streams. A reset will delete data in the destination of the affected streams and then re-sync that data. Skipping a reset is discouraged and might lead to unexpected behavior.
159+
160+
:::
93161

94-
2. If the schema has changed, the **Refreshed source schema** dialog displays them.
162+
4. Click **Save connection**.

‎docs/cloud/managing-airbyte-cloud/manage-schema-changes.md

+14
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,17 @@ To review and fix breaking schema changes:
5555
If a connection’s source schema has breaking changes, it will stop syncing. You must review and fix the changes before editing the connection or resuming syncs.
5656

5757
:::
58+
59+
### Manually refresh the source schema
60+
61+
In addition to Airbyte Cloud’s automatic schema change detection, you can manually refresh the source schema to stay up to date with changes in your schema.
62+
63+
To manually refresh the source schema:
64+
65+
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Connections** and then click the connection you want to refresh.
66+
67+
2. Click the **Replication** tab.
68+
69+
3. In the **Activate the streams you want to sync** table, click **Refresh source schema** to fetch the schema of your data source.
70+
71+
2. If there are changes to the schema, you can review them in the **Refreshed source schema** dialog.

0 commit comments

Comments
 (0)