feat: Add Excel sheet upload #9825

blcksrx · 2020-05-17T19:27:27Z

SUMMARY

Upload excel file to create table In addition to the csv

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

codecov-io · 2020-05-17T19:33:12Z

Codecov Report

Merging #9825 into master will increase coverage by 4.70%.
The diff coverage is 65.00%.

@@            Coverage Diff             @@
##           master    #9825      +/-   ##
==========================================
+ Coverage   66.16%   70.87%   +4.70%     
==========================================
  Files         585      585              
  Lines       30427    30443      +16     
  Branches     3152     3152              
==========================================
+ Hits        20133    21575    +1442     
+ Misses      10113     8757    -1356     
+ Partials      181      111      -70

Flag	Coverage Δ
#cypress	`53.61% <ø> (?)`
#javascript	`59.25% <ø> (ø)`
#python	`70.99% <65.00%> (-0.04%)`	⬇️

Impacted Files	Coverage Δ
superset/db_engine_specs/base.py	`85.47% <56.25%> (-1.64%)`	⬇️
superset/config.py	`89.66% <100.00%> (+0.08%)`	⬆️
superset/views/database/views.py	`89.71% <100.00%> (ø)`
superset/viz.py	`71.91% <0.00%> (-0.10%)`	⬇️
...rontend/src/SqlLab/components/AceEditorWrapper.tsx	`56.98% <0.00%> (+1.07%)`	⬆️
superset-frontend/src/components/EditableTitle.jsx	`81.69% <0.00%> (+1.40%)`	⬆️
...perset-frontend/src/components/CopyToClipboard.jsx	`36.36% <0.00%> (+1.51%)`	⬆️
...hboard/components/resizable/ResizableContainer.jsx	`71.87% <0.00%> (+1.56%)`	⬆️
...ashboard/components/gridComponents/ChartHolder.jsx	`81.35% <0.00%> (+1.69%)`	⬆️
superset-frontend/src/utils/common.js	`69.64% <0.00%> (+1.78%)`	⬆️
... and 139 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea9b7f2...c7d389b. Read the comment docs.

blcksrx · 2020-05-17T19:33:39Z

I need your help guys. With this PR the user is able to upload an Excel file in addition to the CSV files. but I don't know how to change it on the UI! I mean I don't know should I create a page for that like Upload excel file or should I handle uploading files e.g CSV and Excel only on the 1 page?
What about labels and existing translation! I only know English and Persian! and other questions :))

@villebro

villebro

Thanks for the contribution! First pass comments.

superset/config.py

codecov-commenter · 2020-05-19T10:16:09Z

Codecov Report

Merging #9825 into master will decrease coverage by 0.59%.
The diff coverage is 35.93%.

@@            Coverage Diff             @@
##           master    #9825      +/-   ##
==========================================
- Coverage   70.49%   69.89%   -0.60%     
==========================================
  Files         594      190     -404     
  Lines       31386    18474   -12912     
  Branches     3214        0    -3214     
==========================================
- Hits        22126    12913    -9213     
+ Misses       9144     5561    -3583     
+ Partials      116        0     -116

Flag	Coverage Δ
#cypress	`?`
#javascript	`?`
#python	`69.89% <35.93%> (-0.39%)`	⬇️

Impacted Files	Coverage Δ
superset/views/database/views.py	`60.11% <17.64%> (-29.64%)`	⬇️
superset/db_engine_specs/base.py	`83.70% <25.00%> (-3.06%)`	⬇️
superset/views/database/forms.py	`78.37% <61.29%> (-12.32%)`	⬇️
superset/app.py	`81.60% <80.00%> (-0.22%)`	⬇️
superset/config.py	`90.07% <100.00%> (+0.07%)`	⬆️
superset/views/database/mixins.py	`59.64% <0.00%> (-21.06%)`	⬇️
superset/utils/cache.py	`48.00% <0.00%> (-20.00%)`	⬇️
superset/db_engine_specs/mysql.py	`78.72% <0.00%> (-12.59%)`	⬇️
superset/views/database/validators.py	`78.94% <0.00%> (-5.27%)`	⬇️
superset/views/database/api.py	`84.09% <0.00%> (-3.41%)`	⬇️
... and 437 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 42a56e2...2483631. Read the comment docs.

villebro · 2020-05-19T10:29:24Z

@blcksrx at least the following files are involved in the CSV upload feature:

superset/app.py: here the menu item is defined.
superset/views/database/forms.py: here the actual form is defined.
superset/views/database/views.py: here the view is defined.

I think Uploading of Excel files probably needs to happen via a dedicated dialog, as many of the CSV fields are not applicable for Excel. Having said that, many are, hence it might make sense to have a parent form with the global options (fail, replace, append etc), and then only add/override the ones that are specific to the format in question.

blcksrx · 2020-05-19T12:34:36Z

@villebro So it needs to do copy and pates many same things for that. right? for example, it needs to add a column allow_excel_upload to the database and extra. do you okay with that?

villebro · 2020-05-25T06:34:33Z

@blcksrx I think we can assume CSV and Excel importing being equal here, so no need to add a new column in table metadata for that. We should perhaps change the name to "allow upload" or similar. But I don't think that needs to be done now.

blcksrx · 2020-05-25T07:52:39Z

@villebro I got it. thanks

villebro

First round comments

superset/app.py

superset/db_engine_specs/base.py

superset/views/database/forms.py

superset/views/database/views.py

villebro

Second pass comments

setup.py

superset/app.py

superset/config.py

villebro

Apart from one small fix, LGTM! I think there shouldn't be any changes to requirements.txt now that Excel is an optional dependency. Also, if you can add a note in UPDATING.md to say that uploading of Excel Sheets can be enabled by the optional excel flag, people upgrading will be aware of the new feature.

requirements.txt

villebro

A few changes requested to UPDATING.md, other than that LGTM

UPDATING.md

villebro

LGTM

cyw233 · 2020-07-21T00:32:54Z

Hi @blcksrx and @villebro thanks for bringing this feature to Superset! Today when I first tried this feature it gave me this error: "AttributeError: 'SupersetSecurityManager' object has no attribute 'database_access'", but when I changed security_manager.database_access(database) to security_manager.can_access_database(database) it worked. I don't quite understand why the change works, so could you guys help me with this? Cheers!

blcksrx · 2020-07-28T10:26:12Z

Hi @blcksrx and @villebro thanks for bringing this feature to Superset! Today when I first tried this feature it gave me this error: "AttributeError: 'SupersetSecurityManager' object has no attribute 'database_access'", but when I changed security_manager.database_access(database) to security_manager.can_access_database(database) it worked. I don't quite understand why the change works, so could you guys help me with this? Cheers!

Hi. it seems, there are some other commits that above my commit that changes the database_access to the can_access_database and the commiters forgot to change this on my commit

pull-request-size bot added the size/M label May 17, 2020

villebro requested changes May 19, 2020

View reviewed changes

superset/config.py Outdated Show resolved Hide resolved

blcksrx force-pushed the excel branch from c7d389b to 695e131 Compare May 19, 2020 10:04

blcksrx requested a review from villebro May 19, 2020 10:05

blcksrx force-pushed the excel branch 3 times, most recently from abfa7b7 to d74fb4d Compare May 25, 2020 18:06

pull-request-size bot added size/L and removed size/M labels May 25, 2020

blcksrx changed the title ~~WIP: Upload excel~~ Upload excel May 25, 2020

blcksrx force-pushed the excel branch from d74fb4d to 9e7784a Compare May 25, 2020 18:11

villebro changed the title ~~Upload excel~~ feat: Add Excel sheet upload feature May 26, 2020

villebro requested changes May 26, 2020

View reviewed changes

blcksrx force-pushed the excel branch 4 times, most recently from 8d7510a to 97c9950 Compare May 26, 2020 06:53

blcksrx requested a review from villebro May 26, 2020 07:11

villebro requested changes May 28, 2020

View reviewed changes

setup.py Outdated Show resolved Hide resolved

superset/app.py Outdated Show resolved Hide resolved

superset/config.py Outdated Show resolved Hide resolved

blcksrx force-pushed the excel branch from 97c9950 to b3058c9 Compare May 28, 2020 10:48

blcksrx force-pushed the excel branch 3 times, most recently from b3058c9 to 83c18eb Compare June 7, 2020 08:26

blcksrx force-pushed the excel branch from a7e6ff0 to 5789c7f Compare June 24, 2020 19:35

blcksrx requested a review from villebro June 24, 2020 19:44

villebro requested changes Jun 26, 2020

View reviewed changes

requirements.txt Outdated Show resolved Hide resolved

blcksrx force-pushed the excel branch 2 times, most recently from 9c541cf to 63dfc34 Compare June 26, 2020 19:51

pull-request-size bot added size/L and removed size/XL labels Jun 26, 2020

blcksrx force-pushed the excel branch 5 times, most recently from 7cd9506 to 2483631 Compare June 26, 2020 20:19

blcksrx requested a review from villebro June 26, 2020 20:20

villebro requested changes Jun 29, 2020

View reviewed changes

UPDATING.md Outdated Show resolved Hide resolved

UPDATING.md Outdated Show resolved Hide resolved

blcksrx force-pushed the excel branch 2 times, most recently from 87e8770 to 5b11937 Compare June 29, 2020 17:29

blcksrx requested a review from villebro June 29, 2020 17:37

villebro requested changes Jun 30, 2020

View reviewed changes

UPDATING.md Outdated Show resolved Hide resolved

Upload excel

a8b9d0e

blcksrx force-pushed the excel branch from 5b11937 to a8b9d0e Compare June 30, 2020 11:14

blcksrx requested a review from villebro June 30, 2020 11:15

villebro approved these changes Jul 3, 2020

View reviewed changes

villebro merged commit fdd28c1 into apache:master Jul 3, 2020

pphszx mentioned this pull request Jul 28, 2020

fix: excel sheet upload is not working #10450

Merged

6 tasks

blcksrx deleted the excel branch October 24, 2020 06:19

auxten pushed a commit to auxten/incubator-superset that referenced this pull request Nov 20, 2020

Upload excel (apache#9825)

35cf56d

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.37.0 labels Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Excel sheet upload #9825

feat: Add Excel sheet upload #9825

blcksrx commented May 17, 2020 •

edited

Loading

codecov-io commented May 17, 2020 •

edited

Loading

blcksrx commented May 17, 2020

villebro left a comment

codecov-commenter commented May 19, 2020 •

edited

Loading

villebro commented May 19, 2020

blcksrx commented May 19, 2020

villebro commented May 25, 2020

blcksrx commented May 25, 2020 •

edited

Loading

villebro left a comment

villebro left a comment

villebro left a comment

villebro left a comment

villebro left a comment

cyw233 commented Jul 21, 2020 •

edited

Loading

blcksrx commented Jul 28, 2020

feat: Add Excel sheet upload #9825

feat: Add Excel sheet upload #9825

Conversation

blcksrx commented May 17, 2020 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

codecov-io commented May 17, 2020 • edited Loading

Codecov Report

blcksrx commented May 17, 2020

villebro left a comment

Choose a reason for hiding this comment

codecov-commenter commented May 19, 2020 • edited Loading

Codecov Report

villebro commented May 19, 2020

blcksrx commented May 19, 2020

villebro commented May 25, 2020

blcksrx commented May 25, 2020 • edited Loading

villebro left a comment

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

cyw233 commented Jul 21, 2020 • edited Loading

blcksrx commented Jul 28, 2020

blcksrx commented May 17, 2020 •

edited

Loading

codecov-io commented May 17, 2020 •

edited

Loading

codecov-commenter commented May 19, 2020 •

edited

Loading

blcksrx commented May 25, 2020 •

edited

Loading

cyw233 commented Jul 21, 2020 •

edited

Loading