feat(datasource): remove deleted columns and update column type on metadata refresh #10619
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
villebro
commented
Aug 17, 2020
Comment on lines
+78
to
+100
| const columns = [ | ||
| { | ||
| name: 'ds', | ||
| type: 'DATETIME', | ||
| nullable: true, | ||
| default: '', | ||
| primary_key: false, | ||
| }, | ||
| { | ||
| name: 'gender', | ||
| type: 'VARCHAR(32)', | ||
| nullable: true, | ||
| default: '', | ||
| primary_key: false, | ||
| }, | ||
| { | ||
| name: 'new_column', | ||
| type: 'VARCHAR(10)', | ||
| nullable: true, | ||
| default: '', | ||
| primary_key: false, | ||
| }, | ||
| ]; |
There was a problem hiding this comment.
This is the correct schema coming back from the API (compare with extraColumn above).
villebro
commented
Aug 17, 2020
Comment on lines
+107
to
+116
| { | ||
| type: 'DATETIME', | ||
| description: null, | ||
| filterable: false, | ||
| verbose_name: null, | ||
| is_dttm: true, | ||
| expression: '', | ||
| groupby: false, | ||
| column_name: 'ds', | ||
| }, |
There was a problem hiding this comment.
This column is unchanged.
villebro
commented
Aug 17, 2020
Comment on lines
+117
to
+126
| { | ||
| type: 'VARCHAR(32)', | ||
| description: null, | ||
| filterable: true, | ||
| verbose_name: null, | ||
| is_dttm: false, | ||
| expression: '', | ||
| groupby: true, | ||
| column_name: 'gender', | ||
| }, |
There was a problem hiding this comment.
This type is updated from VARCHAR(16) to VARCHAR(32)
villebro
commented
Aug 17, 2020
Comment on lines
+127
to
+130
| expect.objectContaining({ | ||
| column_name: 'new_column', | ||
| type: 'VARCHAR(10)', | ||
| }), |
villebro
commented
Aug 17, 2020
Comment on lines
+133
to
+135
| expect(inst.state.databaseColumns).not.toEqual( | ||
| expect.arrayContaining([expect.objectContaining({ name: 'name' })]), | ||
| ); |
There was a problem hiding this comment.
This is one of the removed columns in the original metadata.
villebro
commented
Aug 17, 2020
| let { databaseColumns } = this.state; | ||
| let hasChanged; | ||
| const currentColNames = databaseColumns.map(col => col.column_name); | ||
| updateColumns(cols) { |
There was a problem hiding this comment.
Method was renamed to reflect that metadata is updated, not only merged.
villebro
commented
Aug 17, 2020
| dbcols = ( | ||
| db.session.query(TableColumn) | ||
| .filter(TableColumn.table == self) | ||
| .filter(or_(TableColumn.column_name == col.name for col in table_.columns)) |
There was a problem hiding this comment.
I'm not sure what the intent here was, but I don't see any scenario where we want to bring in columns from another table with the same column names.
villebro
changed the title
villebro
changed the title
willbarrett
approved these changes
Aug 17, 2020
There was a problem hiding this comment.
This looks like a good improvement/fix - I'd like to tag @lilykuang as well who's touched this code recently for a review.
lilykuang
approved these changes
Aug 17, 2020
amitmiran137
pushed a commit
to amitmiran137/incubator-superset
that referenced
this pull request
Aug 21, 2020
* master: (43 commits) feat: Getting fancier with Storybook (apache#10647) fix: dedup groupby in viz.py while preserving order (apache#10633) feat: bump superset-ui for certified tag (apache#10650) feat: setup react page with submenu for datasources listview (apache#10642) feat: add certification to metrics (apache#10630) feat(viz-plugins): add date formatting to pivot-table (apache#10637) fix: controls scroll issue (apache#10644) feat: Allow tests files in /src (plus Label component tests) (apache#10634) fix: remove duplicated params and cache_timeout from list_columns; add viz_type to list_columns (apache#10643) chore: splitting button stories into separate stories (apache#10631) refactor: remove slice level label_colors from dashboard init load (apache#10603) feat: card view bulk select (apache#10607) style: Label styling/storybook touchups (apache#10627) fix: removing unsupported modal sizes (apache#10625) feat(datasource): remove deleted columns and update column type on metadata refresh (apache#10619) improve documentation for country maps (apache#10621) chore: npm audit fix as of 2020-08-15 (apache#10613) feat: dataset REST API for distinct values (apache#10595) chore: bump react-redux to 5.1.2, whittling console noise (apache#10602) fixing console error about bad html attribute (apache#10604) ... # Conflicts: # superset-frontend/src/explore/components/ExploreViewContainer.jsx # superset-frontend/src/views/App.tsx # superset/config.py
Ofeknielsen
pushed a commit
to ofekisr/incubator-superset
that referenced
this pull request
Oct 5, 2020
…tadata refresh (apache#10619) * fix: remove missing columns on metadata refresh * add tests * lint and simplify * lint * reduce locals * fix label style
auxten
pushed a commit
to auxten/incubator-superset
that referenced
this pull request
Nov 20, 2020
…tadata refresh (apache#10619) * fix: remove missing columns on metadata refresh * add tests * lint and simplify * lint * reduce locals * fix label style
mistercrunch
added
🏷️ bot
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
Currently refreshing table metadata doesn't remove columns that aren't available on the table any longer. In addition, the React Table view doesn't update column types if they've changed. This PR:
AFTER
BEFORE
The type pills on the column tab were using an incorrect style, causing white foreground on white background.
TEST PLAN
Local testing + new tests
ADDITIONAL INFORMATION