Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

v1.1 preliminary testing #164

Closed
51 of 72 tasks
craig-willis opened this issue Feb 15, 2022 · 2 comments
Closed
51 of 72 tasks

v1.1 preliminary testing #164

craig-willis opened this issue Feb 15, 2022 · 2 comments

Comments

@craig-willis
Copy link
Collaborator

craig-willis commented Feb 15, 2022 •
edited
Loading

Test Plan

Need to confirm:

  • Sometimes file manager doesn't change tab contents (e.g., workspace files show in external data)

Splash page

  • Splash page
  1. Goto https://dashboard.stage.wholetale.org
  2. Confirm Terms of Service link
  3. Confirm acknowledgements (NSF, XSEDE/Jetstream, Stata)
  4. Confirm partner logos

Authentication

  • Basic login flow
  1. Goto https://dashboard.stage.wholetale.org
  2. Select "Access Whole Tale" button. You should be prompted to login via Globus Auth.
  3. Select your organization and login
  4. You should be taken to the "My Tales" page
  5. Your Gravatar should display in the upper right
  • Basic logout flow
  1. Select the "User" menu
  2. Select the "Log out" menu item. You should be taken to the login page.
  3. Select "Access Whole Tale" button.
  4. Note that you are not prompted to login.
  5. Restart your incognito session
  6. Select "Access Whole Tale" button. You should be prompted to login via Globus Auth.
  • Return-route for non-logged in users
  1. Logout from Whole Tale
  2. Attempt to access https://dashboard.stage.wholetale.org/public
  3. You should be redirected to https://dashboard.stage.wholetale.org/#?rd=%252Fpublic
  4. Select "Access Whole Tale".
  5. You should be redirected to the "Public Tales" page.

Navigation

  • General navigation
  1. Select "Tale Dashboard" link. "My Tales" page should display
  2. Click "Create New Tale" > "Create New Tale". The Create Tale modal should display. Click "Cancel" to close the modal
  3. Select the "i" icon at the upper right corner; it should open the User Guide in new tab
  4. Select the "Notification" button next to the "i" icon; it should display the notification panel
  5. Select the "bug" icon; it should open https://github.com/whole-tale/whole-tale/issues in new tab (after authentication)
  6. Mouse over a Tale and select "View". The Run page should display
  7. Click the "user" icon at the top right
  • You should see a dropdown expand
  • The dropdown should contain the user's first/last name and gravatar image
  • The dropdown should provide a link to the Settings page
  • The dropdown should provide an option to Logout
  • Select the "Settings" option; it should bring you to the Settings page
  • Trivial: Tooltips on icons could be useful (per verifier feedback)

Tale Dashboard

Preconditions:

  • Assumes production Tales present (e.g., LIGO, materials, etc).

  • No running instances

  • General

  1. Select "My Tales" link. Confirm "My Tales" page displays
  2. Confirm that no Tales are running
  3. Select "Shared with Me" link. Confirm "Shared with Me" page displays

  • Notes: "No Public Tales have been shared" -- perhaps reword

  • Search

  1. Select "Public Tales" link. Confirm "Public Tales" page displays
  2. Enter "LIGO" as search term. Confirm only LIGO tale displays (by K. Kowalik)
  3. Clear search term. Confirm all tales display

  • Notes: Search is case sensitive; search only applies to selected tab; searches only subset of metadata?

  • View tale

  1. Select "View" button on LIGO Tale
  2. Confirm that Metadata page displays by default
  3. Confirm that you cannot delete or edit Tale properties
  4. Select Back button. Confirm that you are taken back to Public Tales page
  5. Select "View" button on Tale you own
    1. Confirm that the Run and Close buttons are present
    2. Confirm that you can edit Tale properties
    3. Select "Close"
    4. Confirm that you are taken back to "My Tales"
  • Launch instance
  1. Select "Run Tale" button on "Ligo" tale
    1. Confirm you are prompted to Copy and Run
    2. Select "Confirm"
    3. Confirm that you are redirected to the Tale's "Metadata" page
    4. Confirm that the "Edit" button is present
    5. Select "Run Tale"
    6. Confirm that the "Interact" page is displayed with spinner
    7. Confirm that the notification dialog appears
    8. Confirm that the step progression is correct
    9. Confirm that the "View Logs" button shows the build logs in a modal
    10. Navigate to the "My Tales" page
    11. Confirm that the Tale is shown under "Currently Running"
    12. Select "View" on the Tale card
    13. Confirm that "Interact" page is displayed
  2. Select "Stop" to end the Tale instance
    1. Confirm that the Tale has stopped
    2. Confirm that the button has changed to Run Tale
  3. Navigate to "My Tales"
    1. Confirm that there are no Tales under "Currently Running"
    2. Select the "Run Tale" button on a Tale you own
    3. After the instance is launched, navigate back to the My Tales page
    4. Confirm that you cannot select the delete button on the My Tales page
    5. Confirm that the "Run Tale" button has been replaced with "Stop Tale"
    6. Confirm that the Tale is shown at the top of the page under "Currently running"
  • Notes:
    • BL complained about copy and run wording.
    • Pop-out and tale history icons are not obvious (per verifier feedback). Tooltips or make more prominent (text).
    • Interactive pop-out only present on interact tab
    • "Cached image exists for this Tale" -- possibly indicate which image was used?
    • Ligo tale is old-style notebook

Managing Data

Preconditions

  • Empty home directory

  • No registered data

  • Register General

  1. Open an owned Tale
  2. Select 'External Data' > "+" > Web (DOI or URL)
  3. Confirm list of providers matches list of supported providers (Globus, DataONE, Dataverse, Zenodo)

  • Trivial: Need to add DERIVA to test plan and list

  • Register DataONE data

  1. Open an owned Tale
  2. Select 'External Data' > "+" > Web (DOI or URL)
  3. Confirm modal displays
  4. Search for DOI 10.5065/D6862DM8
  5. Confirm "Humans and Hydrology" dataset is found on DataONE.
  6. Select Register
  7. Confirm dataset appears in Manage Data WT Data Catalog list
  8. Confirm additional URL formats:
    1. https://search.dataone.org/view/doi:10.5063/F1Z60M87
    2. https://search.dataone.org/view/https://pasta.lternet.edu/package/metadata/eml/knb-lter-nwt/115/1
    3. https://search.dataone.org/view/doi:10.18739/A2CN6XZ7H
    4. https://raw.githubusercontent.com/whole-tale/dashboard/master/.travis.yml
  • Dataverse
  1. Register dataset doi:10.7910/DVN/SZ9YXZ
  2. Confirm "Replication Data for: Direct Democracy and Women's Political Engagement" from Dataverse
  • Globus/MDF
  1. Confirm DOI resolution for http://dx.doi.org/doi:10.18126/M2301J results in https://petreldata.net/mdf/detail/pub_30_shahani_twinmediated_v1.2
  2. Register dataset https://petreldata.net/mdf/detail/pub_30_shahani_twinmediated_v1.2

ADD

  • DERIVA
  1. Register dataset https://pbcconsortium.s3.amazonaws.com/wholetale/5ad7cdf55b0d5007601015b7ff1ea8d6/2021-11-09_21.47.58/Dataset_1-882P.zip

Run Tale

Preconditions:

  • No running Tale instances

  • General

  1. On "Public Tales", click "Run" on LIGO tale (Copy on Launch)
  2. Select "Run Tale" to start the instance
  3. Confirm you are redirected to the Run page
    1. Confirm that the Run panel displays the Tale icon, title, and author
    2. Confirm that the Run page displays the Interact, Files, Metadata and Share tabs
    3. Confirm that Interact tab is displayed by default
    4. Confirm notification panel displays and progression is correct
  4. Select the "..." menu.
    1. Confirm that menu options exist for "Rebuild Tale", "Restart Tale", "Save Tale Version", "Recorded Run", "Duplicate Tale", "Publish Tale", "Export Tale", "Connect to Git Repository...", View fullscreen, Read the docs
    2. Confirm that the "Read the docs" link displays https://wholetale.readthedocs.io/en/latest/users_guide/run.html in a new tab or window (stable is for production)
    3. Confirm that the "View Fullscreen" link displays the Run panel in full screen

  • Notes:

    • Notification panel components overlap when window is small
    • Test case missing "Share tab"
    • Test case missing Recorded run, view fullscreen, read the docs in menu list

  • Interact tab

  1. Select the "Interact" tab
  2. Confirm Jupyter environment is displayed in iframe
  3. Select "popout" icon. Confirm Jupyter is opened in new Tab
  • Files tab
  1. Select "Files" tab
  2. You should see a vertical navigation menu with three tabs: "Home", "External Data", and "Tale Workspace" under CURRENT and two tabs: Saved Versions and Recorded Runs under ARCHIVED
  3. Confirm Tale Workspace is selected by default

  • Note: Versions/recorded run missing from test case

  • Metadata tab - Owned Tale

  1. Open a Tale that you own (e.g., LIGO Tale copy)
  2. Navigate to the Metadata tab
  3. Confirm that your Tale has the CC4 license assigned to it
  4. Confirm that the Tale's Published Location reads This Tale has not been published
  5. Note "Datasets Used: No citable data"
  6. Add multiple dataset via External Datasets (Humans and Hydrology, Replication Data for Direct Democracy...)
  7. Navigate to Run > Metadata
  8. Confirm that there are citations for the external datasets
  9. Click the "Edit" button to edit the metadata fields
  10. Confirm editable fields (Title, Authors, Category, Environment, License, Description, Illustration)
  11. Valid environments should be listed in the Environment dropdown menu
  12. Valid licenses should be listed in the License dropdown (CC4 and CC0)
  13. Change the name of the Tale, Category, Description, Illustration, Public, License, and add multiple authors
  14. Select Save. Confirm that you see a notification telling you that the Tale was saved.
  15. Confirm that changes are reflected in the page.
  16. Refresh the page. Your changes should be preserved between page refreshes, confirming that they are being properly persisted.
  17. Confirm that the new Tale authors are displayed on the Tale card
  1. Open a Tale that you don't own
  2. Navigate to the metadata page
  3. Note that there isn't an 'Edit' button present
  4. Confirm read-only fields (Title, Created By, Authors, Category, Environment, License, Description, Datasets Used, Related Identifiers, Publised Location, Date Created, Last Updated)
  • Home
  1. View a tale that you own
  2. Select Files > Home. Confirm that your home directory displays
  3. Create a folder and upload a file to the folder
  4. Confirm file operations: Copy, rename, remove, download, Move To...
    1. Copy file/folder
    2. Rename file/folder
    3. Download file/folder
    4. Upload file / create folder with duplicate name

  • Issues:

    • Create folder test1. Copy the folder (test 1 (1)). Rename new folder to test1. Returns "OSError: OSError(39, 'Directory not empty')" and page is not refreshed -- folder appears to be renamed test1.
    • Create folder test2. Rename to test3. File menu no longer works without refresh? Happens also with files.

  • External data

  1. Precondition: Register one or more Datasets (e.g. doi:10.18739/A29G5GD0V)
    1. Confirm notification progress and success
  2. Create a new tale
  3. Confirm "External Data" list is initially empty
  4. Confirm (+) button ("Select Data modal") behavior
    1. Confirm that "WT Data Catalog" shows all registered datasets
    2. Confirm that "My External Data" filters to only show your registered datasets
    3. Select a dataset
    4. Confirm that clicking "Add Selected" adds the dataset to the "Selected Data" column
    5. Confirm that "Remove Selected" data to dissociate it with a Tale
    6. Using the dataset above, add the dataset to the Tale, by clicking "Save"
    7. Confirm that the Tale's "External Data" list matches what was saved
    8. Remove the dataset from the Tale
    9. Confirm "External Data" list matches what was saved
    10. Select a dataset but then click out of the modal or click "Cancel"
    11. Confirm that the changes weren't saved
  • Tale Workspace
  1. Confirm create folder
  2. Confirm upload file
  3. Select "+" button has options "New folder", "Upload file", "Select Data..."
  4. Folder operations
    1. Confirm context menu Copy, Rename, Download, Remove
    2. Confirm rename folder
    3. Confirm download folder (this should result in a zip file)
    4. Confirm remove folder
  5. File operations
    1. Upload a file
    2. Confirm context menu Copy, Rename, Download, Remove
    3. Confirm rename file
    4. Confirm download file
    5. Confirm remove file
  6. Select "Select Data..."
    1. Note vertical navigation with Tale Workspaces
    2. Copy to Workspace from another workspace
    3. Move to workspace from another workspace

  • Notes:

    • Move to within tale is annoying -- need to browse all tales...
    • Issue: Renaming to existing name results in Folder 'test2' already exists in wtlocal:L3RtcC9kYXRhL3dvcmtzcGFjZXMvNi82MjBiZGI3Y2ZkYmZkMjc0YTRlOGIwZDgvdGVzdDF8NjIwYmRiN2NmZGJmZDI3NGE0ZThiMGQ5
    • After rename context menu doesn't work (no console error) as above
    • Issue: Select Data -- Copy and move to workspace fails without error

  • Files - Non-Owned Tale

  1. Open a Tale that you don't have write permissions to
  2. Attempt to add data to the workspace and external data
  3. Note that the "+" icons are greyed out, preventing you from doing so
  4. Note that Home directory isn't present
  • Export Tale
  1. Open your copy of the LIGO Tale or import https://girder.stage.wholetale.org/api/v1/integration/zenodo?doi=10.5281%2Fzenodo.490449&resource_server=sandbox.zenodo.org
  2. Select "Export Tale" from Tale menu
  3. Confirm that a download starts for the ZIP archive
  4. Open the archive
  5. Confirm that exported ZIP is a valid Bag
    1. Must contain bagit.txt, bag-info.txt
    2. Optionally pip install bdbag and bdbag --validate full .
  6. Run the exported tale and confirm that it builds and runs as expected
    1. Follow instructions in README.md
bdbag --resolve-fetch all .
bdbag --validate full .
bdbag --validate fast .
python -m bagit --validate .
bdbag --validate-profile full . fails?
bdbag --validate-profile full 620bdd87fdbfd274a4e8b0e7.zip works
  • Profile fails to validate? Error: [ProfileValidationError] '620bdd87fdbfd274a4e8b0e7: Bag serialization is required but Bag is a directory.'
  • run-local could rely on image in registry (push to dockerhub?)

Settings

  • Default State
  1. Navigate to the Settings page
  2. Confirm there are empty settings for DataONE, Dataverse, and Zenodo
  • Connect to Zenodo
  1. Click "Connect Account" in the Zenodo section
  2. Confirm that the connection modal opens
  3. Confirm that sandbox.zenodo.org and zenodo.org are listed
  4. Select sandbox.zenodo.org from repository list
  5. Select "Get from Zenodo"
    1. Login to Zenodo if necessary
    2. Create token
    3. Copy token
  6. Enter your API key into WT modal
  7. Click "Connect"
  8. Confirm that you see "Authorized on "
  9. Select disconnect, confirmation window should appear
  10. Repeat test with invalid key, confirm error message appears
  11. Repeat for each Zenodo server
  12. Confirm "Get from" link matches selected repo

  • Notes: Do we need to update docs to define which capabilities need to be added for a token?

  • Connect to Dataverse

  1. Click "Connect Account" in the Dataverse section
  2. Confirm that the connection modal opens
  3. Enter your API key for one of the repositories
  4. Click "Connect"
  5. Confirm that you see "Authorized on "
  6. Select disconnect, confirmation window should appear
  7. Repeat test with invalid key, confirm error message appears
  8. Repeat for each Dataverse server
  9. Confirm "Get from" link matches selected repo

  • Notes:

    • Remove dev2 from list for deploy-dev

  • Connect to DataONE

  1. Click "Connect Account" under the DataONE repository
  2. Confirm that the connection modal opens
  3. Enter your JWT for one of the repositories
  4. Confirm that you see "Authorized on "
  5. Select disconnect, confirmation window should appear
  6. Repeat test with invalid key, confirm error message appears
  7. Repeat for each DataONE server
  • Confirm tokens retained across logins
    #370
  1. Connect one or more accounts as outlined above
  2. Logout and login again
  3. Using Swagger UI, hit the /user/me endpoint to confirm tokens are still present

Tale Creation

  • Create JupyterLab Tale
  1. Select the Create New Tale button on the 'Browse' page
  2. Enter title "Test Tale"
  3. Select JuptyerLab environment
  4. Select "Create New Tale"
  5. Confirm that you're brought to the 'Metadata' page and the "Run Tale" button is visible
  6. Click "Run Tale"
  7. Confirm notification dialog opens and progress order is correct
  8. Confirm that you are redirected to "Interact" tab with Tale launching spinner
  9. During build image step, select View Logs. Confirm logs are accessible.
  10. Wait for launch to complete
  11. Select "Return to Dashboard"
  12. Confirm the Tale is shown under "Currently running"
  13. Select "View", confirm that the interact tab is displayed with JupyterLab IDE
  14. Select popout icon and confirm that JupyterLab IDE displays
  15. Stop and delete the tale

  • Notes:

    • Need to remove password from URL at some point

  • Create RStudio Tale

  1. Create tale from Git repo https://github.com/whole-tale/rstudio-example/
  2. Select environment RStudio 4.0.2
  3. Run tale and confirm RStudio IDE displays in iframe and works in popout
  4. Open Code/1_credit_permissiveness.R
  5. Confirm outputs are created in Outputs without error
  • Compose Jupyter Notebook Tale
  1. Create tale from Git repo https://github.com/whole-tale/jupyter-example/
  2. Select environment Jupyter Notebook
  3. Run tale and confirm Jupyter Classic IDE displays in iframe and works in popout
  4. Add usco2005.xlsx from Humans and Hydrology dataset
  5. Run notebook, confirm it produces most glorious map of Texas
  • Compose JupyterLab Tale
  1. Create tale from Git repo https://github.com/whole-tale/jupyter-example/
  2. Select JupyterLab environment
  3. Run tale and confirm JupyterLab IDE displays in iframe and works in popout
  4. Confirm JupyterLab IDE displays
  5. Add usco2005.xlsx from Humans and Hydrology dataset
  6. Run notebook, confirm it produces most glorious map of Texas
  • Compose JupyterLab with Spark
  1. Create tale from Git repo https://github.com/whole-tale/spark-example/
  2. Select Jupyter with Spark environment
  3. Run tale and confirm JupyterLab IDE displays in iframe and works in popout
  4. Run notebook and confirm version is 2.4.3
  • Compose MATLAB Desktop Tale
  1. Create tale from Git repo https://github.com/whole-tale/matlab-example/
  2. Select MATLAB Desktop environment
  3. Run tale and confirm MATLAB Web Desktop IDE displays in iframe and works in popout
  4. Open and run multiplicative_arima_example_script.m, confirm outputs
  5. Open and run multiplicative_arima_example_livescript.m, confirm outputs
  • Compose MATLAB Jupyter Kernel Tale
  1. Create tale from Git repo https://github.com/whole-tale/matlab-example/
  2. Select MATLAB Jupyter Kernel environment
  3. Run tale and confirm JupyterLab IDE displays in iframe and works in popout
  4. Open and jun multiplicative_arima_example.ipynb, confirm outputs match multiplicative_arima_example.html
  • Compose MATLAB Linux Desktop Tale
  1. (Only test if present)
  2. Create tale from Git repo https://github.com/whole-tale/matlab-example/
  3. Run tale and confirm Xpra-based MATLAB IDE displays in iframe and works in popout
  4. Open and run multiplicative_arima_example_script.m, confirm outputs
  • Compose STATA Desktop Tale
  1. Create tale from Git repo https://github.com/whole-tale/stata-example/
  2. Run tale and confirm Xpra-based STATA IDE displays in iframe and works in popout
  3. Run example.do, confirm outputs
  • Compose STATA Jupyter Kernel Tale
  1. Create tale from Git repo https://github.com/whole-tale/stata-example/
  2. Run tale and confirm JupyterLab IDE displays in iframe and works in popout
  3. Run example.ipynb, confirm outputs match example.html
  • Compose OpenRefine Tale
  1. Create a new tale based on the OpenRefine environment
  2. Run tale and confirm OpenRefine IDE displays in iframe and works in popout
  • Too many instances
  1. Run 2 tales
  2. Attempt to run another Tale
  3. Confirm that you receive an error that you have the max limit

Analyze in Whole Tale

These test cases cover potential situations that can occur when importing datasets from Dataverse.

  • Import dataset from Dataverse
  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataverse?datasetPid=doi%3A10.7910%2FDVN%2F3MJ7IR&siteUrl=https%3A%2F%2Fdataverse.harvard.edu
  3. Confirm that the Tale title reads as Replication Data for: "Agricultural Fires and Health at Birth"
  4. Confirm that the only item in the Input data section matches the uri with Data Source appended
  5. Confirm that no environment is selected
  6. Confirm that the Create New Tale button is disabled
  7. Select an environment
  8. Confirm that READ/WRITE is selected
  9. Click Create New Tale
  10. Confirm registration notification progress and success
  11. Confirm that you are redirected to the Metadata page and "Run Tale" is enabled
  12. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  13. Confirm that the data exists under Tale Workspace
  14. Confirm that the Tale's category is science

Notes:

  • Test case: category is social science

  • Import dataset from DataONE: READ-ONLY

  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataone?uri=https%3A%2F%2Fsearch.dataone.org%2Fview%2Fdoi%3A10.18739%2FA2VQ2S94D&title=Fire%20influences%20on%20forest%20recovery%20and%20associated%20climate%20feedbacks%20in%20Siberian%20Larch%20Forests%2C%20Russia&environment=RStudio%20%28R%203.5.1%29
  3. Confirm that the Tale title reads as Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, Russia
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that Rstudio (3.5.1) is selected in the Environments widget
  6. Confirm that the Create New Tale button is enabled
  7. Confirm READ ONLY is selected
  8. Click Create New Tale
  9. Confirm that you are redirected to the Metadata tab
  10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  11. Confirm that the data exists in the Tale under External Data
  12. Confirm that the Tale's category is science
  13. Confirm that the Datasets Used and Related Identifiers are updated to reflect the dataset

  • Notes:

    • Test case: Environment RStudio is no longer valid

  • Import from DataONE: READ-WRITE

  1. Navigate to https://girder.local.wholetale.org/api/v1/integration/dataone?uri=https%3A%2F%2Fsearch.dataone.org%2Fview%2Fdoi%3A10.18739%2FA2VQ2S94D&title=Fire%20influences%20on%20forest%20recovery%20and%20associated%20climate%20feedbacks%20in%20Siberian%20Larch%20Forests%2C%20Russia&environment=RStudio%20%28R%203.5.1%29&asTale=true
  2. Confirm that the Tale title matches the dataset
  3. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  4. Confirm that the Create New Tale button is disabled
  5. Select READ/WRITE
  6. Click Create New Tale
  7. Confirm that the notification bar appears & properly updates
  8. Confirm that you are redirected to the run page
  9. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  10. Confirm that the data exists in the Tale Workspace
export GIRDER_TOKEN=<your token>
curl -X POST -H "Content-Type: application/zip" --data-binary "@<path-to-zip>.zip" --header "Girder-Token: ${GIRDER_TOKEN}" 'https://girder.local.wholetale.org/api/v1/tale/import'
  1. Confirm tale imports, builds and run
  2. Try with invalid token: (You must be logged in)
  3. Try with invalid zipfile (Bad bag)
  • Notes:
    • Test case: Could update to test versions, runs, bag validation, etc.
    • Q. Do we validate bags fully on import?

Tale metadata tests

The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.

  • manifest.json
  1. Launch a Tale that you own
  2. Add a dataset to the Tale
  3. Add at least one author
  4. Add at least one file to the workspace
  5. Create a version
  6. Export the Tale (as BagIt)
  7. Open the manifest.json file under metadata/
  8. Confirm wt context present
  9. Confirm @id references the correct Tale ID
  10. Confirm that the author is listed under schema:author
  11. Confirm that you are listed under createdBy
  12. Confirm that the file(s) in the workspace are properly listed in aggregates
  13. Confirm that the external dataset is referenced in aggregates
  14. Confirm that the dataset is also listed under wt:usesDataset
  15. Confirm dct:hasVersion is present and matches your created version
  16. Optionally confirm rdflib can parse:
from rdflib import Graph

import urllib.parse
urllib.parse.uses_relative.append('arcp')
urllib.parse.uses_netloc.append('arcp')

g = Graph().parse(source="manifest.json", format="json-ld")
print(g.serialize(format='turtle', indent=2).decode())

Notes:

  • Test case:

    • Add recorded run
    • Need a better test case for this/automate query

  • environment.json

  1. Open the archive from the manifest.json test
  2. Navigate to metadata/environment.json
  3. Open the girder web API
  4. Locate and GET the Image that the Tale used
  5. Confirm that the config section from girder matches the config section in environment.json

Notes:

  • Test case:

    • Need a better test case for this/automate query

  • LICENSE

  1. Open the archive from the manifest.json test
  2. Open the LICENSE file
  3. Navigate to Run > metadata
  4. Confirm that the opened LICENSE file matches the selected license on the metadata page
  5. Change the Tale's license
  6. Export the Tale
  7. Confirm that the LICENSE changed appropriately

Tale Sharing Tests

These tests are centered around the feature for sharing Tales with other Whole Tale
users. To run these tests, register at least one other account, referred to as User B.

  • Case 1: Sharing with 'Can View' permissions
    1. Create a new Tale
    2. Add data to the workspace and add at least one dataset
    3. Navigate to the 'Share' tab
    4. Share the Tale with User B
    5. Confirm that the default permission is 'Can View'
    6. Switch the account to User B
    7. Navigate to the 'Shared with Me' tab on the Browse page
    8. Confirm that you see the Tale from step 1
    9. Confirm that you can't edit the Workspace, External Data, or metadata
    10. Unshare tale with User B
    11. Note modal displays that tale has been unshared.

Notes:

  • Test case: User B on shared with me tab can confirm notification

  • User with view permissions cannot create /delete versions or runs

  • Case 2: Sharing with 'Can Edit' permissions

    1. Create a new Tale
    2. Add data to the workspace and add at least one dataset
    3. Navigate to the 'Share' tab
    4. Share the Tale with User B
    5. Set the permissions to 'Can Edit'
    6. Switch the account to User B
    7. Navigate to the 'Shared with Me' tab on the Browse page
    8. Confirm that you see the Tale from step 1
    9. Remove one of the files from the Workspace
    10. Add a dataset to External Data
    11. Change a field in the metadata (title, description, etc)
    12. Switch to User A
    13. Confirm that the changes made to the Tale persisted

Notes:

  • User B can create a recorded run and confirm user a can see results

  • Case 3: Edit sync modal

    1. Login as User A and User B in separate browser sessions
    2. As User B, select "Shared with me"
    3. As User A, share a tale with User B; as User B confirm the tale appears
    4. As User B, select "View" on the shared tale
    5. As User A and B edit the tale metadata
    6. As User A change title, author, category, description; save changes
    7. As User B confirm Tale edited modal, select Yes, confirm updates to metadata

Publishing tests

  • Case 1: No Accounts Configured

    1. Disconnect all of your external accounts
    2. Open a Tale that you own
    3. Click the ellipsis dropdown on the Run page
    4. Click Publish tale...
    5. Confirm that your are notified to link accounts on the settings page
    6. Click Go to Settings
    7. Confirm you are re-directed to settings

  • Case 2: Publishing to DataONE

    1. Connect to DataONE (dev server)
    2. Open a Tale that you own
    3. Add some data to it
    4. Click Publish tale...
    5. Confirm that the publishing modal opens
    6. Confirm that the third party that you are connected to appears in the dropdown
    7. Select the third party
    8. Click Publish
    9. Once complete, navigate to the metadata page
    10. Confirm that the published location is present
    11. Visit the published location
    12. Confirm that you see the following files plus yours
      1. metadata.xml
      2. LICENSE
      3. README.md
      4. manifest.json
      5. environment.json
    13. After publishing to DataONE, use girder to query the Tale
    14. Ensure that the 'pid' field in the 'publishInfo' object has an identifier in a doi format

  • Notes:

    • Not tested this cycle because of CN issues.

  • Case 3: Re-publish Tale to DataONE

  1. Launch a Tale that has been published before
  2. Take note of the Published Location in Run > metadata
  3. Re-publish the Tale to the third party
  4. Confirm that the Published Location has changed to the correct package landing page

Zenodo integration tests

  • Register Zenodo data
  1. On a new or existing tale, select "Files" > "External Data"
  2. Select "+" > "Web (DOI or URL)"
  3. Search for Concept DOI doi:10.5281/zenodo.16384
  4. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_1
  5. Search for URL https://zenodo.org/record/1172960#.XidIvlNKh24
  6. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
  7. Search for DOI doi:10.5281/zenodo.1172960
  8. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
  9. Search for Concept DOI doi:10.5281/zenodo.608984
  10. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
  11. Select Register
  12. Select "+" > "WT Data Catalog"
  13. Add "Belmot Forum..." dataset to Tale
  14. Run Tale and confirm contents on file system match Zenodo

The register tests the following cases.

  • Datasets with multiple files

  • Datasets with multiple versions

  • Concept DOI

  • Zenodo dataset via AiWT

  1. Open https://dashboard.local.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.820575&name=Automotive%20Sensor%20Data
  2. Confirm Source Data URL and Title match above URL and "READ ONLY" is selected
  3. Select JupyterLab environment
  4. Select 'Create New Tale'
  5. Confirm dataset is mounted read-only, citation and related ids are as expected,
  6. Run IDE and use terminal to ls ../data/Auto* to confirm zipfile is not extracted

The register tests the following cases.

  • Dataset with single file (zipfile)

  • Zenodo Binder via AiWT

  1. Open https://dashboard.local.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.3242073&environment=JupyterLab&name=Simple%20requirements.txt%20based%20example%20for%20repo2docker&asTale=true
  2. COnfirm Source Data and Title match URL and "READ WRITE" is selected
  3. Select 'Create Tale'
  4. Confirm image builds and notebook can run

Notes:

  • If not logged in, the tale title is still URL encoded (Simple%20requirements.txt%20based%20example%20for%20repo2docker). If logged in it isn't.

  • Minor, but do we need to flush logs? Container says exporting image but UI shows lines before.

  • Zenodo Tale via AiWT

  1. Open https://girder.local.wholetale.org/api/v1/integration/zenodo?doi=10.5281%2Fzenodo.490449&resource_server=sandbox.zenodo.org
  2. Confirm Water Tale is created.
  3. Confirm citation on Metadata view
  4. Confirm DOI matches above
  5. Use the Swagger UI to confirm the publishInfo field on the tale object contains pid, repository, repository_id, uri
  6. Run the tale and and notebook, confirm it works as expected (produces glorious map of Texas)

Notes:

  • Date created and updated are based on import action, not what was imported.

  • We could add date published to differentiate

  • Error: AttributeError: 'DataONEVerificator' object has no attribute 'headers'

  • Publish/import to/from Zenodo Sandbox

  1. Follow instructions above to configure your API key for Zenodo sandbox
  2. Create your own Tale and publish to sandbox.zenodo.org
  3. Confirm publishInfo contains Sandbox entry
  4. Delete tale
  5. Import tale from Zenodo
  6. Confirm publishInfo
  7. Modify the tale
  8. Re-publish to sandbox.zenodo.org
  9. Delete tale
  10. Import tale from Zenodo
  11. Confirm publishInfo
  12. Download and extract the exported Tale
  13. Execute the run-local.sh command to run the Tale locally

Notes:

  • We can just do this as an end-to-end test with a recorded run, version, etc.

Versioning Tests

  • Tale Versioning
  1. View a Tale that you own
  2. Expand Tale Version panel (on the right)
  3. Create a New Version: init
  4. Modify metadata, upload a file, and attach a dataset to the Tale
  5. Create another new version: modified
  6. While viewing Run > Files > External Data, restore from init - you should see no datasets
  7. Rename the modified Version to something else
  8. In the same view, restore back to something else - you should see the dataset you added
  9. Run the tale. Confirm the versions are available under ../versions and that the contents including metadata are as expected
  10. View Info for a version
  11. Rename a version. Confirm rename in container.
  12. Export a version. Confirm exported version matches selected version.
  13. Delete a version. Confirm deletion in container.
  • Copy versions
  1. Login as a user with view-only permissions of a tale with versions.
  2. Run the tale and confirm versions are copied
  • Publish/export version
  1. Create a new tale
  2. Upload files but do not create a version
  3. Publish the tale
  4. Note that there is a new version in the version panel
  5. Confirm that the data files in the published artifact are correct
  6. Repeat for export
  7. Confirm manifest contents including vocabulary
  • Notes:
    • Needs automated test for export validation

Git Integration Tests

  • Compose Tale from Git Repository
    Covered by other tests
  1. Select the Create New Tale button on the 'Browse' page
  2. Select the Create Tale from Git Repository option
  3. Paste a link to a Git repository in the correct field
  4. Add a name and pick a compute environment (JuptyterLab)
  5. Create the Tale
  6. Confirm notification progress and success
  7. Confirm that the git repository was cloned to the workspace/ root
  8. Run the tale
  9. Confirm interaction via git tools (may require environment customization)

  • Note:

    • Rewrite as part of RRun?

  • Connect to Git

  1. Create a new empty tale
  2. Select "Connect to Git Repository" menu item from Tale menu
  3. Confirm notification progress and success
  4. Once complete, navigate to the Tale workspace and confirm that the repository has been cloned

Other tests

  • Citations
  1. Add files from multiple external datasets (e.g.: doi:10.5281/zenodo.3381658, doi:10.5065/D6862DM8, doi:10.5063/F10K26VR)
  2. Confirm Datasets Used citations and related identifiers are updated
  • Note: Update docs to reflect which providers support citations (i.e., not HTTP or DERIVA)

Event Notification Tests

  • Tale sharing/unsharing
  1. Login as user1 and user2 in separate browser sessions
  2. As user2, select "Shared with me"
  3. As user1, share a tale with user2; as user2 confirm the tale appears
  4. As user2, select "View" on the shared tale
  5. As user1 and 2 edit the tale metadata
  6. As user1 save changes; as user2 confirm Tale edited modal
  7. As user1, unshare the tale; as user2 confirm Tale unshared modal
  8. As user1, re-share with user2
  9. As user2, select "Shared with Me"
  10. As user1, delete the shared tale; as user2 confirm that the tale disappears
  • Import tale
  1. Import a tale from Github, Zenodo, and as Binder
  2. Confirm metadata updates accordingly after import completes

Vocabulary Tests

  • Vocabularies
  1. Go to https://vocabularies.stage.wholetale.org/wt/1.0/index.html
  2. Confirm that it matches https://whole-tale.github.io/serialization-format/wt/1.0/

Forward Authentication Test

  • Forward authentication
  1. Start a tale
  2. Confirm that the tale is accessible via iframe and popout
  3. Right click popout icon and copy URL (this is necessary to get JupyterLab token)
  4. Open an incognito window and paste the copied URL
  5. Confirm that you are prompted for authentication and can access the tale once authenciated
  6. Login as a user who does not have permission to access the tale
  7. Confirm that they receive an access denied errror
  • Notes: Access denied error would be nicer if non-rest...

Changing the Environment

  • Changing the environment
  1. Create a new JupyterLab tale
  2. Run the Tale, confirm JupyterLab IDE displays
  3. Edit metadata, change environment to "RStudio"
  4. Confirm icon changes in title bar and Environment section
  5. Stop/run tale or rebuild/restart and confirm RStudio IDE displays

Idle Instances

  • Idle instance culling
  1. Create and start a MATLAB tale and a JupyterLab tale
  2. Confirm both instances start and IDEs are accessible
  3. Close all IDE tabs/windows
  4. Wait for 2 hours and 24 hours respectively
  5. Confirm both instances are stopped automatically
@craig-willis
Copy link
Collaborator Author

craig-willis commented Feb 16, 2022 •
edited
Loading

Summary

UI issues/comments

  • AinWT title remains URL encoded if going through login flow, but not if logged in. AinWT URL encoding issue when user isn't logged in ngx-dashboard#250
  • TOS URL not working (https://dashboard.local.wholetale.org/undefined/tos)
  • Tooltips on icons could be helpful. This was also feedback from verifiers
  • Suggest changes to copy on launch modal wording
  • Pop-out and tale history icons are not obvious (per verifier feedback). Also need tooltips or make more prominent (text).
  • There's no reason the pop-out icon should be restricted to the interact tab -- just show if the tale is running.
  • Issues renaming files and folders in home. Create folder test1. Copy the folder (test 1 (1)). Rename new folder to test1. Returns "OSError: OSError(39, 'Directory not empty')" and page is not refreshed -- folder appears to be renamed test1. Create folder test2. Rename to test3. File menu no longer works without refresh? Happens also with files.
  • Move within a tale is overly complicated
  • Workspace: Renaming to existing name results in Folder 'test2' already exists in wtlocal:L3RtcC9kYXRhL3dvcmtzcGFjZXMvNi82MjBiZGI3Y2ZkYmZkMjc0YTRlOGIwZDgvdGVzdDF8NjIwYmRiN2NmZGJmZDI3NGE0ZThiMGQ5. Context menu no longer works without refresh.
  • Workspace: Select Data -- Copy and move to workspace fails without error
  • Not reproducible, but I have noticed not-infrequently strange behavior in the file manager where either tab contents aren't changed (i.e., workspace files show in external data view) or appear merged. Possibly due to network/latency.
  • Starting a recorded run could take me to the RR panel; as could creating a version take me to the version panel

Backend issues/comments:

  • Search is case sensitive and seems to only search a subset of metadata (need to clarify)
  • "Cached image exists for this Tale" in log -- possibly indicate which image was used (just me)
  • Update citation fails for DERIVA due to Minor updates to BDBag/Deriva girder_wholetale#519
  • Run-local could rely on image in registry if we push to Dockerhub
  • Consider validating bag on import including profile
  • Remove dev2 from list for deploy-dev
  • Need to remove password from Jupyter URL at some point (requires migration...)
  • Minor, but do we need to flush logs? Container says exporting image but UI shows lines before.
  • AinWT: Date created and updated are based on import action, not what was imported. We could at least add date published to differentiate.
  • Running Water Tale notebook results in error: AttributeError: 'DataONEVerificator' object has no attribute 'headers' [DataONE] add missing headers prop to DataONEVerificator girder_wholetale#522
  • ORCID auth problem 500 on AiWT When Authenticating with ORCID & Globus girder_wholetale#459 -- appears to be limited to dev (works for me on stage).
  • ~Get manifest fails with DERIVA dataset registered File "/girder/plugins/wholetale/server/lib/deriva/provider.py", line 44, in getDatasetUID return doc["meta"]["identifier"] KeyError: 'identifier'. If this happens during recorded run, the run fails silently -- no status, no nothing. ~
  • Docker stats doesn't need to have container name
  • Add system information to recorded run

Test plan issues:

  • Revise test plan for overall coverage, reduce redundancy
  • Add DERIVA test cases
  • Expand Globus test cases to include file access
  • Expand run local test cases to confirm each provider
  • Have test case cover registering and accessing data from each provider
  • Update test plan to include recorded run (consolidate tests)
  • Update test plan per Create valid BDBags with external data girder_wholetale#518
  • Need have test script for bag validation that includes metadata/environment validation and valid RDF
  • Update sharing test case for versions/runs (view cannot delete/rename, edit can, etc)
  • Correct CN in D1 test cases (cn-sandbox not cn-stage2)
  • Add test cases related to CORE2 features?

Documentation issues:

  • Do we need to update docs to define which capabilities need to be added for a token?

@craig-willis
Copy link
Collaborator Author

Closing for actual RC testing

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@craig-willis