Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

v1.2rc2 testing (Chrome/MacOS) #173

Open
76 of 83 tasks
craig-willis opened this issue Dec 21, 2022 · 1 comment
Open
76 of 83 tasks

v1.2rc2 testing (Chrome/MacOS) #173

craig-willis opened this issue Dec 21, 2022 · 1 comment

Comments

@craig-willis
Copy link
Collaborator

craig-willis commented Dec 21, 2022

Test Plan

Test plan walkthrough on Staging

End-to-End Test

  • End-to-End Test
  • Open private session as user1
  • https://girder.stage.wholetale.org/api/v1/integration/zenodo?doi=10.5072%2Fzenodo.1021503&resource_server=sandbox.zenodo.org
  • # when prompted
  • Start the interactive environment (Run Tale)
  • sh run.sh, confirm no errors
  • Start a recorded run using run.sh, confirm SUCCESS
  • Export tale
  • Run locally
  • Execute run.sh, confirm success
  • Share the tale view-only with user2
  • As user2, copy-on-launch. Confirm versions and runs were copied
  • Start a recorded run using run.sh, confirm SUCCESS
  • Export the tale
  • sh run-local.sh and access the interactive environment
  • In the interactive environment sh run.sh, confirm no errors

Unauthenticated access

  • Unauthenticated access
  1. Open a new private/incognito window
  2. Goto https://dashboard.stage.wholetale.org
  3. Confirm that you are able to access the dashboard without signing in
  4. Confirm only "Public Tales" display
  5. Confirm "#" button appears in upper right
  6. Select "View" on a public tale (e.g., LIGO)
  7. Confirm that you are able to view metadata; browse and download files in workspace, external data, versions, and runs.
  8. Confirm that tale menu displays only docs link.
  9. Confirm that "Run Tale" button is disabled and that a tooltip displays "You must log in to run this tale"
  10. Confirm that tale history panel displays versions and runs read only
  11. Confirm navigation links are functional: Whole Tale logo, Tale Dashboard, Docs, File and Issue, Return to Dashboard

Signing in

  • Basic login flow
  1. As unauthenticted user, go to https://dashboard.stage.wholetale.org
  2. Note that you are redirected to /public
  3. Select "#" and complete auth flow
  4. Confirm that you are taken to the /public panel as a logged in user. All features for signed-in users should be functional (My Tales, Shared Tales, Create Tale, etc).
  5. Logout, confirm "#" button appears and only "Public Tales" is accessible
  6. Open a public tale (e.g., LIGO)
  7. Select "#", confirm that you are accessing the tale as a signed-in user
  8. Logout, confirm that you are accessing the tale as an unauthenticated user
  • Confirm auth providers
  1. Select "#" button
  2. Select "Access Whole Tale" button. You should be prompted to login via Globus Auth.
  3. Confirm institutional login
  4. Confirm Google login
  5. Confirm ORCID login
  • Basic logout flow
  1. As an authenticated user, select the "User" menu
  2. Select the "Log out" menu item. You should be taken to the Public Tales catalog.
  3. Select "#"
  4. Note that you are not prompted to login to Globus again
  5. Select User menu > Logout.
  6. Go to https://app.globus.org/logout to logout from Globus
  7. Go to https://dashboard.stage.wholetale.org
  8. Select "#", you should be promted to login
  • Return-route for non-logged in users
  1. Open private window
  2. Attempt to access https://dashboard.stage.wholetale.org/mine
  3. You should be redirected to Globus auth
  4. Enter your credentials
  5. You should be redirected to the "My Tales" page.
  • Forward auth
  1. Create and start an empty tale instance
  2. Copy the link
  3. Start a new private session
  4. Attempt to access the link
  5. Login as user with permission on the tale. Confirm that the IDE displays.
  6. Login as user without permission. Confirm instance access is denied.
  • Analyze in WT (unauthenticated user)
  1. Open new private session
  2. Go to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.820575&name=Automotive%20Sensor%20Data
  3. You should prompted to #
  4. Confirm "Create new Tale" dialog opens with title "Automotive Sensor Data" and datasource populated

Navigation

  • General navigation
  1. #
  2. Select "Tale Dashboard" link. "My Tales" page should display
  3. Click "Create New Tale" > "Create New Tale". The Create Tale modal should display. Click "Cancel" to close the modal
  4. Select the "i" icon at the upper right corner; it should open the User Guide in new tab. Hovering should display tooltip.
  5. Select the "Notification" button next to the "i" icon; it should display the notification panel. Hovering should display tooltip.
  6. Select the "bug" icon; it should open https://github.com/whole-tale/whole-tale/issues in new tab (after authentication). Hovering should display tooltip.
  7. Mouse over a Tale and select "View". The Run page should display
  8. Click the "user" icon at the top right
  • You should see a dropdown expand
  • The dropdown should contain the user's first/last name and gravatar image
  • The dropdown should provide a link to the Settings page
  • The dropdown should provide an option to Logout
  • Select the "Settings" option; it should bring you to the Settings page

Tale Dashboard

Preconditions:

  • Authenticated user

  • Assumes production Tales present (e.g., LIGO, Materials, etc).

  • No running instances

  • General

  1. Select "My Tales" link. Confirm "My Tales" page displays
  2. Confirm that no Tales are running
  3. Select "Shared with Me" link. Confirm "Shared with Me" page displays,
  • Search
  1. Select "Public Tales" link. Confirm "Public Tales" page displays
  2. Enter "LIGO" as search term. Confirm only LIGO tale displays (by K. Kowalik)
  3. Clear search term. Confirm all tales display
  • View tale
  1. Select "View" button on LIGO Tale
  2. Confirm that Metadata page displays by default
  3. Confirm that you cannot delete or edit Tale properties
  4. Select Back button. Confirm that you are taken back to Public Tales page
  5. Select "View" button on Tale you own
    1. Confirm that the Run and Close buttons are present
    2. Confirm that you can edit Tale properties
    3. Select "Close"
    4. Confirm that you are taken back to "My Tales"
  • Launch instance
  1. Select "Run Tale" button on "Ligo" tale
    1. Confirm you are prompted to Copy and Run
    2. Select "Confirm"
    3. Confirm that you are redirected to the Tale's "Metadata" page
    4. Confirm "Copy" label appears under title
    5. Confirm that the "Edit" button is present
    6. Select "Run Tale"
    7. Confirm that the "Interact" page is displayed with spinner
    8. Confirm that the notification dialog appears
    9. Confirm that the step progression is correct
    10. Confirm that the "View Logs" button shows the build logs in a modal
    11. Navigate to the "My Tales" page
    12. Confirm that the Tale is shown under "Currently Running"
    13. Select "View" on the Tale card
    14. Confirm that "Interact" page is displayed
  2. Select "Stop" to end the Tale instance
    1. Confirm that the Tale has stopped
    2. Confirm that the button has changed to Run Tale
  3. Navigate to "My Tales"
    1. Confirm that there are no Tales under "Currently Running"
    2. Select the "Run Tale" button on a Tale you own
    3. After the instance is launched, navigate back to the My Tales page
    4. Confirm that you cannot select the delete button on the My Tales page
    5. Confirm that the "Run Tale" button has been replaced with "Stop Tale"
    6. Confirm that the Tale is shown at the top of the page under "Currently running"

Managing Data

Preconditions

  • Empty home directory

  • No registered data

  • Register General

  1. Open an owned Tale
  2. Select 'External Data' > "+" > Web (DOI or URL)
  3. Confirm list of providers matches list of supported providers (Globus, DataONE, Dataverse, Zenodo)
  • Register DataONE data
  1. Open an owned Tale
  2. Select 'External Data' > "+" > Web (DOI or URL)
  3. Confirm modal displays
  4. Search for DOI 10.5065/D6862DM8
  5. Confirm "Humans and Hydrology" dataset is found on DataONE.
  6. Select Register
  7. Confirm dataset appears in the Data Catalog
  8. Confirm additional URL formats:
    1. https://search.dataone.org/view/doi:10.5063/F1Z60M87
    2. https://search.dataone.org/view/https://pasta.lternet.edu/package/metadata/eml/knb-lter-nwt/115/1
    3. https://search.dataone.org/view/doi:10.18739/A2CN6XZ7H
    4. https://raw.githubusercontent.com/whole-tale/dashboard/master/.travis.yml
  • Dataverse
  1. Register dataset doi:10.7910/DVN/SZ9YXZ
  2. Confirm "Replication Data for: Direct Democracy and Women's Political Engagement" from Dataverse
  • Globus/MDF
  1. Confirm DOI resolution for http://dx.doi.org/doi:10.18126/M2301J results in https://petreldata.net/mdf/detail/pub_30_shahani_twinmediated_v1.2
  2. Register dataset https://petreldata.net/mdf/detail/pub_30_shahani_twinmediated_v1.2
  • DERIVA
  1. Register dataset https://pbcconsortium.s3.amazonaws.com/wholetale/5ad7cdf55b0d5007601015b7ff1ea8d6/2021-11-09_21.47.58/Dataset_1-882P.zip

Run Tale

Preconditions:

  • No running Tale instances

  • General

  1. On "Public Tales", click "Run" on LIGO tale (Copy on Launch)
  2. Select "Run Tale" to start the instance
  3. Confirm you are redirected to the Run page
    1. Confirm that the Run panel displays the Tale image, title, and author
    2. Confirm that the Run page displays the Interact, Files, Metadata and Share tabs
    3. Confirm that Interact tab is displayed by default
    4. Confirm notification panel displays and progression is correct
  4. Select the "..." menu.
    1. Confirm that menu options exist for "Rebuild Tale", "Restart Tale", "Save Tale Version", "Recorded Run", "Duplicate Tale", "Publish Tale", "Export Tale", "Connect to Git Repository..."
    2. Confirm that the "Read the docs" link displays https://wholetale.readthedocs.io/en/latest/users_guide/run.html in a new tab or window (stable is for production)
    3. Confirm that the "View Fullscreen" link displays the Run panel in full screen
  • Interact tab
  1. Select the "Interact" tab
  2. Confirm Jupyter environment is displayed in iframe
  3. Hover over popout icon, confirm tooltip displays.
  4. Select "popout" icon. Confirm Jupyter is opened in new tab,
  • Files tab
  1. Select "Files" tab
  2. You should see a vertical navigation menu with two sections CURRENT and ARCHIVED.
  3. Under CURRENT you should see three tabs: "Home", "External Data", and "Tale Workspace".
  4. Under ARCHIVED you should see Saved Versions and Recorded Runs under ARCHIVED
  5. Confirm Tale Workspace is selected by default
  • Metadata tab - Owned Tale
  1. Open a Tale that you own (e.g., LIGO Tale copy)
  2. Navigate to the Metadata tab
  3. Confirm that your Tale has the CC4 license assigned to it
  4. Confirm that the Tale's Published Location reads This Tale has not been published
  5. Note "Datasets Used: No citable data"
  6. Add multiple dataset via External Datasets (Humans and Hydrology, Replication Data for Direct Democracy...)
  7. Navigate to Run > Metadata
  8. Confirm that there are citations for the external datasets
  9. Click the "Edit" button to edit the metadata fields
  10. Confirm editable fields (Title, Authors, Category, Environment, License, Description, Illustration)
  11. Valid environments should be listed in the Environment dropdown menu
  12. Valid licenses should be listed in the License dropdown (CC4 and CC0)
  13. Change the name of the Tale, Category, Description, Illustration, Public, License, and add multiple authors
  14. Select Save. Confirm that you see a notification telling you that the Tale was saved.
  15. Confirm that changes are reflected in the page.
  16. Refresh the page. Your changes should be preserved between page refreshes, confirming that they are being properly persisted.
  17. Confirm that the new Tale authors are displayed on the Tale card
  • Metadata tab - Non-Owned Tale (e.g., Public LIGO Tale)
  1. Open a Tale that you don't own
  2. Navigate to the metadata page
  3. Note that there isn't an 'Edit' button present
  4. Confirm read-only fields (Title, Created By, Authors, Category, Environment, License, Description, Datasets Used, Related Identifiers, Publised Location, Date Created, Last Updated)
  • Home
  1. View a tale that you own
  2. Select Files > Home. Confirm that your home directory displays
  3. Create a folder and upload a file to the folder
  4. Confirm file operations: Copy, rename, remove, download, Move To...
    1. Copy file/folder
    2. Rename file/folder
    3. Download file/folder
    4. Upload file / create folder with duplicate name
    5. Upload folder
  • External data
  1. Precondition: Register one or more Datasets (e.g. doi:10.18739/A29G5GD0V)
    1. Confirm notification progress and success
  2. Create a new tale
  3. Confirm "External Data" list is initially empty
  4. Confirm (+) button ("Select Data modal") behavior
    1. Confirm that "WT Data Catalog" shows all registered datasets
    2. Confirm that "My External Data" filters to only show your registered datasets
    3. Select a dataset
    4. Confirm that clicking "Add Selected" adds the dataset to the "Selected Data" column
    5. Confirm that "Remove Selected" data to dissociate it with a Tale
    6. Using the dataset above, add the dataset to the Tale, by clicking "Save"
    7. Confirm that the Tale's "External Data" list matches what was saved
    8. Remove the dataset from the Tale
    9. Confirm "External Data" list matches what was saved
    10. Select a dataset but then click out of the modal or click "Cancel"
    11. Confirm that the changes weren't saved
  • Tale Workspace
  1. Confirm create folder
  2. Confirm upload file
  3. Select "+" button has options "New folder", "Upload file", "Select Data..."
  4. Folder operations
    1. Confirm context menu Copy, Rename, Download, Remove
    2. Confirm rename folder
    3. Confirm download folder (this should result in a zip file)
    4. Confirm remove folder
  5. File operations
    1. Upload a file
    2. Confirm context menu Copy, Rename, Download, Remove
    3. Confirm rename file
    4. Confirm download file
    5. Confirm remove file
  6. Select "Select Data..."
    1. Note vertical navigation with Tale Workspaces
    2. Copy to Workspace from another workspace
    3. Move to workspace from another workspace
  7. Upload folder (TODO)
  • Files - Non-Owned Tale
  1. Open a Tale that you don't have write permissions to
  2. Attempt to add data to the workspace and external data
  3. Note that the "+" icons are greyed out, preventing you from doing so
  4. Note that Home directory isn't present
  • Export Tale
  1. Open your copy of the LIGO Tale or import https://girder.stage.wholetale.org/api/v1/integration/zenodo?doi=10.5281%2Fzenodo.490449&resource_server=sandbox.zenodo.org
  2. Select "Export Tale" from Tale menu
  3. Confirm that a download starts for the ZIP archive
  4. Open the archive
  5. Confirm that exported ZIP is a valid Bag
    1. Must contain bagit.txt, bag-info.txt
    2. Optionally pip install bdbag and bdbag --validate full .
  6. Run the exported tale and confirm that it builds and runs as expected
    1. Follow instructions in README.md

Versions

  • Tale Versioning
  1. Create a new tale or open a tale that you own
  2. Expand Tale Version panel (on the right)
  3. Create a New Version: init
  4. Modify metadata, upload a file, and attach a dataset to the Tale
  5. Create another new version: modified
  6. While viewing Run > Files > External Data, restore from init - you should see no datasets
  7. Rename the modified Version to something else
  8. In the same view, restore back to something else - you should see the dataset you added
  9. Run the tale. Confirm the versions are available under ../versions in-container and that the contents including metadata are as expected
  10. View Info for a version
  11. Rename a version. Confirm rename in container.
  12. Export a version. Confirm exported version matches selected version.
  13. Delete a version. Confirm deletion in container.
  • Publish/export version
  1. Create a new tale
  2. Upload files but do not create a version
  3. Publish the tale
  4. Note that there is a new version in the version panel
  5. Confirm that the data files in the published artifact are correct
  6. Repeat for export
  7. Confirm manifest contents including vocabulary

Recorded Runs

  • Create tale from Github
  1. Create tale from Github repo https://github.com/whole-tale/rrun-test
  2. Select JupyterLab environment
  3. Create and start the tale
  4. Select "..." > "Recorded Run" with script run.sh
  5. Confirm recorded run completes successully
  6. Select history icon > "Perform Run" with script notfound.sh
  7. Confirm recorded run fails
  8. Select Files > Recorded Runs. You should see two runs, one COMPLETED and one FAILED
  9. Select Files > Saved Versions. You should see a single version. Confirm that the version name matches the history panel.
  10. Rename the version, confirm that it updates in the File manager and history panel for runs.
  11. Attempt to remove the version, confirm error "ersion is in use by a run and cannot be deleted."
  12. Rename one of the runs. Confirm it is renamed in the history panel and in-container.
  13. Remove both of the runs. Confirm they are removed in the history panel and in-container.
  14. Confirm that you can now remove the version
  • Copy versions and runs
  1. Using the above tale, select "..." > "Recorded Run" with script run.sh
  2. Share the tale with a user with view-only permissions
  3. As the view-only, run the tale and confirm versions and runs are copied
  4. Remove copy of the tale
  5. Share the tale with edit permissions
  6. Confirm that versions and runs can be renamed but not deleted.
  7. Share the tale with owner permissions
  8. Confirm that versions and runs can be deleted

Settings

  • Default State
  1. Navigate to the Settings page
  2. Confirm there are empty settings for DataONE, Dataverse, and Zenodo
  • Connect to Zenodo
  1. Click "Connect Account" in the Zenodo section
  2. Confirm that the connection modal opens
  3. Confirm that sandbox.zenodo.org and zenodo.org are listed
  4. Select sandbox.zenodo.org from repository list
  5. Select "Get from Zenodo"
    1. Login to Zenodo if necessary
    2. Create token
    3. Copy token
  6. Enter your API key into WT modal
  7. Click "Connect"
  8. Confirm that you see "Authorized on "
  9. Select disconnect, confirmation window should appear
  10. Repeat test with invalid key, confirm error message appears
  11. Repeat for each Zenodo server
  12. Confirm "Get from" link matches selected repo
  • Connect to Dataverse
  1. Click "Connect Account" in the Dataverse section
  2. Confirm that the connection modal opens
  3. Enter your API key for one of the repositories
  4. Click "Connect"
  5. Confirm that you see "Authorized on "
  6. Select disconnect, confirmation window should appear
  7. Repeat test with invalid key, confirm error message appears
  8. Repeat for each Dataverse server
  9. Confirm "Get from" link matches selected repo
  • Connect to DataONE
  1. Click "Connect Account" under the DataONE repository
  2. Confirm that the connection modal opens
  3. Enter your JWT for one of the repositories
  4. Confirm that you see "Authorized on "
  5. Select disconnect, confirmation window should appear
  6. Repeat test with invalid key, confirm error message appears
  7. Repeat for each DataONE server
  • Confirm tokens retained across logins
    #370
  1. Connect one or more accounts as outlined above
  2. Logout and login again
  3. Using Swagger UI, hit the /user/me endpoint to confirm tokens are still present

Tale Creation

  • Create JupyterLab Tale
  1. Select the Create New Tale button on the 'Browse' page
  2. Enter title "Test Tale"
  3. Select JuptyerLab environment
  4. Select "Create New Tale"
  5. Confirm that you're brought to the 'Metadata' page and the "Run Tale" button is visible
  6. Click "Run Tale"
  7. Confirm notification dialog opens and progress order is correct
  8. Confirm that you are redirected to "Interact" tab with Tale launching spinner
  9. During build image step, select View Logs. Confirm logs are accessible.
  10. Wait for launch to complete
  11. Select "Return to Dashboard"
  12. Confirm the Tale is shown under "Currently running"
  13. Select "View", confirm that the interact tab is displayed with JupyterLab IDE
  14. Select popout icon and confirm that JupyterLab IDE displays
  15. Stop and delete the tale
  • Create RStudio Tale
  1. Create tale from Git repo https://github.com/whole-tale/rstudio-example/
  2. Select environment RStudio 4.0.2
  3. Run tale and confirm RStudio IDE displays in iframe and works in popout
  4. Open Code/1_credit_permissiveness.R
  5. Confirm outputs are created in Outputs without error
  • Compose Jupyter Notebook Tale
  1. Create tale from Git repo https://github.com/whole-tale/jupyter-example/
  2. Select environment Jupyter Notebook
  3. Run tale and confirm Jupyter Classic IDE displays in iframe and works in popout
  4. Add usco2005.xlsx from Humans and Hydrology dataset
  5. Run notebook, confirm it produces most glorious map of Texas
  • Compose JupyterLab Tale
  1. Create tale from Git repo https://github.com/whole-tale/jupyter-example/
  2. Select JupyterLab environment
  3. Run tale and confirm JupyterLab IDE displays in iframe and works in popout
  4. Confirm JupyterLab IDE displays
  5. Add usco2005.xlsx from Humans and Hydrology dataset
  6. Run notebook, confirm it produces most glorious map of Texas
  • Compose JupyterLab with Spark
  1. Create tale from Git repo https://github.com/whole-tale/spark-example/
  2. Select Jupyter with Spark environment
  3. Run tale and confirm JupyterLab IDE displays in iframe and works in popout
  4. Run notebook and confirm version is 2.4.3
  • Compose MATLAB Desktop Tale
  1. Create tale from Git repo https://github.com/whole-tale/matlab-example/
  2. Select MATLAB Desktop environment
  3. Run tale and confirm MATLAB Web Desktop IDE displays in iframe and works in popout
  4. Open and run multiplicative_arima_example_script.m, confirm outputs
  5. Open and run multiplicative_arima_example_livescript.m, confirm outputs
  • Compose MATLAB Jupyter Kernel Tale
  1. Create tale from Git repo https://github.com/whole-tale/matlab-example/
  2. Select MATLAB Jupyter Kernel environment
  3. Run tale and confirm JupyterLab IDE displays in iframe and works in popout
  4. Open and jun multiplicative_arima_example.ipynb, confirm outputs match multiplicative_arima_example.html
  • Compose MATLAB Linux Desktop Tale
  1. (Only test if present)
  2. Create tale from Git repo https://github.com/whole-tale/matlab-example/
  3. Run tale and confirm Xpra-based MATLAB IDE displays in iframe and works in popout
  4. Open and run multiplicative_arima_example_script.m, confirm outputs
  • Compose STATA Desktop Tale
  1. Create tale from Git repo https://github.com/whole-tale/stata-example/
  2. Run tale and confirm Xpra-based STATA IDE displays in iframe and works in popout
  3. Run example.do, confirm outputs
  • Compose STATA Jupyter Kernel Tale
  1. Create tale from Git repo https://github.com/whole-tale/stata-example/
  2. Run tale and confirm JupyterLab IDE displays in iframe and works in popout
  3. Run example.ipynb, confirm outputs match `example.html
  • Too many instances
  1. Run 2 tales
  2. Attempt to run another Tale
  3. Confirm that you receive an error that you have the max limit

Analyze in Whole Tale

These test cases cover potential situations that can occur when importing datasets from Dataverse.

  • Import dataset from Dataverse
  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataverse?datasetPid=doi%3A10.7910%2FDVN%2F3MJ7IR&siteUrl=https%3A%2F%2Fdataverse.harvard.edu
  3. Confirm that the Tale title reads as Replication Data for: "Agricultural Fires and Health at Birth"
  4. Confirm that the only item in the Input data section matches the uri with Data Source appended
  5. Confirm that no environment is selected
  6. Confirm that the Create New Tale button is disabled
  7. Select an environment
  8. Confirm that READ/WRITE is selected
  9. Click Create New Tale
  10. Confirm registration notification progress and success
  11. Confirm that you are redirected to the Metadata page and "Run Tale" is enabled
  12. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  13. Confirm that the data exists under Tale Workspace
  14. Confirm that the Tale's category is science
  • Import dataset from DataONE: READ-ONLY
  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataone?uri=https%3A%2F%2Fsearch.dataone.org%2Fview%2Fdoi%3A10.18739%2FA2VQ2S94D&title=Fire%20influences%20on%20forest%20recovery%20and%20associated%20climate%20feedbacks%20in%20Siberian%20Larch%20Forests%2C%20Russia&environment=RStudio
  3. Confirm that the Tale title reads as Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, Russia
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that Rstudio is selected in the Environments widget
  6. Confirm that the Create New Tale button is enabled
  7. Confirm READ ONLY is selected
  8. Click Create New Tale
  9. Confirm that you are redirected to the Metadata tab
  10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  11. Confirm that the data exists in the Tale under External Data
  12. Confirm that the Tale's category is science
  13. Confirm that the Datasets Used and Related Identifiers are updated to reflect the dataset
  • Import from DataONE: READ-WRITE
  1. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataone?uri=https%3A%2F%2Fsearch.dataone.org%2Fview%2Fdoi%3A10.18739%2FA2VQ2S94D&title=Fire%20influences%20on%20forest%20recovery%20and%20associated%20climate%20feedbacks%20in%20Siberian%20Larch%20Forests%2C%20Russia&environment=RStudio
  2. Confirm that the Tale title matches the dataset
  3. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  4. Confirm that the Create New Tale button is disabled
  5. Select READ/WRITE
  6. Click Create New Tale
  7. Confirm that the notification bar appears & properly updates
  8. Confirm that you are redirected to the run page
  9. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  10. Confirm that the data exists in the Tale Workspace
export GIRDER_TOKEN=<your token>
curl -X POST -H "Content-Type: application/zip" --data-binary "@<path-to-zip>.zip" --header 'Girder-Token: ${GIRDER_TOKEN}' 'https://girder.local.wholetale.org/api/v1/tale/import'
  1. Confirm tale imports, builds and run
  2. Try with invalid token: (You must be logged in)
  3. Try with invalid zipfile (Bad bag)

Tale metadata tests

The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.

  • manifest.json
  1. Launch a Tale that you own
  2. Add a dataset to the Tale
  3. Add at least one author
  4. Add at least one file to the workspace
  5. Create a version
  6. Export the Tale (as BagIt)
  7. Open the manifest.json file under metadata/
  8. Confirm wt context present
  9. Confirm @id references the correct Tale ID
  10. Confirm that the author is listed under schema:author
  11. Confirm that you are listed under createdBy
  12. Confirm that the file(s) in the workspace are properly listed in aggregates
  13. Confirm that the external dataset is referenced in aggregates
  14. Confirm that the dataset is also listed under wt:usesDataset
  15. Confirm dct:hasVersion is present and matches your created version
  16. Optionally confirm rdflib can parse:
from rdflib import Graph

import urllib.parse
urllib.parse.uses_relative.append('arcp')
urllib.parse.uses_netloc.append('arcp')

g = Graph().parse(source="manifest.json", format="json-ld")
print(g.serialize(format='turtle', indent=2).decode())
  • environment.json
  1. Open the archive from the manifest.json test
  2. Navigate to metadata/environment.json
  3. Open the girder web API
  4. Locate and GET the Image that the Tale used
  5. Confirm that the config section from girder matches the config section in environment.json
  • LICENSE
  1. Open the archive from the manifest.json test
  2. Open the LICENSE file
  3. Navigate to Run > metadata
  4. Confirm that the opened LICENSE file matches the selected license on the metadata page
  5. Change the Tale's license
  6. Export the Tale
  7. Confirm that the LICENSE changed appropriately

Tale Sharing Tests

These tests are centered around the feature for sharing Tales with other Whole Tale
users. To run these tests, register at least one other account, referred to as User B.

  • Case 1: Sharing with 'Can View' permissions

    1. Create a new Tale
    2. Add data to the workspace and add at least one dataset
    3. Navigate to the 'Share' tab
    4. Share the Tale with User B
    5. Confirm that the default permission is 'Can View'
    6. Switch the account to User B
    7. Navigate to the 'Shared with Me' tab on the Browse page
    8. Confirm that you see the Tale from step 1
    9. Confirm that you can't edit the Workspace, External Data, or metadata
    10. Unshare tale with User B
    11. Note modal displays that tale has been unshared.
  • Case 2: Sharing with 'Can Edit' permissions

    1. Create a new Tale
    2. Add data to the workspace and add at least one dataset
    3. Navigate to the 'Share' tab
    4. Share the Tale with User B
    5. Set the permissions to 'Can Edit'
    6. Switch the account to User B
    7. Navigate to the 'Shared with Me' tab on the Browse page
    8. Confirm that you see the Tale from step 1
    9. Remove one of the files from the Workspace
    10. Add a dataset to External Data
    11. Change a field in the metadata (title, description, etc)
    12. Switch to User A
    13. Confirm that the changes made to the Tale persisted
  • Case 3: Edit sync modal

    1. Login as User A and User B in separate browser sessions
    2. As User B, select "Shared with me"
    3. As User A, share a tale with User B; as User B confirm the tale appears
    4. As User B, select "View" on the shared tale
    5. As User A and B edit the tale metadata
    6. As User A change title, author, category, description; save changes
    7. As User B confirm Tale edited modal, select Yes, confirm updates to metadata

Publishing tests

  • Case 1: No Accounts Configured

    1. Disconnect all of your external accounts
    2. Open a Tale that you own
    3. Click the ellipsis dropdown on the Run page
    4. Click Publish tale...
    5. Confirm that your are notified to link accounts on the settings page
    6. Click Go to Settings
    7. Confirm you are re-directed to settings
  • Case 2: Publishing to DataONE

    1. Connect to DataONE (dev server)
    2. Open a Tale that you own
    3. Add some data to it
    4. Click Publish tale...
    5. Confirm that the publishing modal opens
    6. Confirm that the third party that you are connected to appears in the dropdown
    7. Select the third party
    8. Click Publish
    9. Once complete, navigate to the metadata page
    10. Confirm that the published location is present
    11. Visit the published location
    12. Confirm that you see the following files plus yours
      1. metadata.xml
      2. LICENSE
      3. README.md
      4. manifest.json
      5. environment.json
    13. After publishing to DataONE, use girder to query the Tale
    14. Ensure that the 'pid' field in the 'publishInfo' object has an identifier in a doi format
  • Case 3: Re-publish Tale to DataONE

  1. Launch a Tale that has been published before
  2. Take note of the Published Location in Run > metadata
  3. Re-publish the Tale to the third party
  4. Confirm that the Published Location has changed to the correct package landing page

Zenodo integration tests

  • Register Zenodo data
  1. On a new or existing tale, select "Files" > "External Data"
  2. Select "+" > "Web (DOI or URL)"
  3. Search for Concept DOI doi:10.5281/zenodo.16384
  4. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_1
  5. Search for URL https://zenodo.org/record/1172960#.XidIvlNKh24
  6. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
  7. Search for DOI doi:10.5281/zenodo.1172960
  8. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
  9. Search for Concept DOI doi:10.5281/zenodo.608984
  10. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
  11. Select Register
  12. Select "+" > "WT Data Catalog"
  13. Add "Belmot Forum..." dataset to Tale
  14. Run Tale and confirm contents on file system match Zenodo

The register tests the following cases.

  • Datasets with multiple files

  • Datasets with multiple versions

  • Concept DOI

  • Zenodo dataset via AiWT

  1. Open https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.820575&name=Automotive%20Sensor%20Data
  2. Confirm Source Data URL and Title match above URL and "READ ONLY" is selected
  3. Select JupyterLab environment
  4. Select 'Create New Tale'
  5. Confirm dataset is mounted read-only, citation and related ids are as expected,
  6. Run IDE and use terminal to ls ../data/Auto* to confirm zipfile is not extracted

The register tests the following cases.

  • Dataset with single file (zipfile)

  • Zenodo Binder via AiWT

  1. Open https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.3242073&environment=JupyterLab&name=Simple%20requirements.txt%20based%20example%20for%20repo2docker&asTale=true
  2. COnfirm Source Data and Title match URL and "READ WRITE" is selected
  3. Select 'Create Tale'
  4. Confirm image builds and notebook can run
  • Zenodo Tale via AiWT
  1. Open https://girder.stage.wholetale.org/api/v1/integration/zenodo?doi=10.5281%2Fzenodo.490449&resource_server=sandbox.zenodo.org
  2. Confirm Water Tale is created.
  3. Confirm citation on Metadata view
  4. Confirm DOI matches above
  5. Use the Swagger UI to confirm the publishInfo field on the tale object contains pid, repository, repository_id, uri
  6. Run the tale and and notebook, confirm it works as expected (produces glorious map of Texas)
  • Publish/import to/from Zenodo Sandbox
  1. Follow instructions above to configure your API key for Zenodo sandbox
  2. Create your own Tale and publish to sandbox.zenodo.org
  3. Confirm publishInfo contains Sandbox entry
  4. Delete tale
  5. Import tale from Zenodo
  6. Confirm publishInfo
  7. Modify the tale
  8. Re-publish to sandbox.zenodo.org
  9. Delete tale
  10. Import tale from Zenodo
  11. Confirm publishInfo
  12. Download and extract the exported Tale
  13. Execute the run-local.sh command to run the Tale locally

Git Integration Tests

  • Compose Tale from Git Repository
    Covered by other tests
  1. Select the Create New Tale button on the 'Browse' page
  2. Select the Create Tale from Git Repository option
  3. Paste a link to a Git repository in the correct field
  4. Add a name and pick a compute environment
  5. Create the Tale
  6. Confirm notification progress and success
  7. Confirm that the git repository was cloned to the workspace/ root
  8. Run the tale
  9. Confirm interaction via git tools (may require environment customization)
  • Connect to Git
  1. Create a new empty tale
  2. Select "Connect to Git Repository" menu item from Tale menu
  3. Confirm notification progress and success
  4. Once complete, navigate to the Tale workspace and confirm that the repository has been cloned

Other tests

  • Citations
  1. Add files from multiple external datasets (e.g.: doi:10.5281/zenodo.3381658, doi:10.5065/D6862DM8, doi:10.5063/F10K26VR)
  2. Confirm Datasets Used citations and related identifiers are updated

Event Notification Tests

  • Tale sharing/unsharing
  1. Login as user1 and user2 in separate browser sessions
  2. As user2, select "Shared with me"
  3. As user1, share a tale with user2; as user2 confirm the tale appears
  4. As user2, select "View" on the shared tale
  5. As user1 and 2 edit the tale metadata
  6. As user1 save changes; as user2 confirm Tale edited modal
  7. As user1, unshare the tale; as user2 confirm Tale unshared modal
  8. As user1, re-share with user2
  9. As user2, select "Shared with Me"
  10. As user1, delete the shared tale; as user2 confirm that the tale disappears
  • Import tale
  1. Import a tale from Github, Zenodo, and as Binder
  2. Confirm metadata updates accordingly after import completes

Vocabulary Tests

  • Vocabularies
  1. Go to https://vocabularies.stage.wholetale.org/wt/1.0/index.html
  2. Confirm that it matches https://whole-tale.github.io/serialization-format/wt/1.0/

Forward Authentication Test

  • Forward authentication
  1. Start a tale
  2. Confirm that the tale is accessible via iframe and popout
  3. Right click popout icon and copy URL (this is necessary to get JupyterLab token)
  4. Open an incognito window and paste the copied URL
  5. Confirm that you are prompted for authentication and can access the tale once authenciated
  6. Login as a user who does not have permission to access the tale
  7. Confirm that they receive an access denied errror

Changing the Environment

  • Changing the environment
  1. Create a new JupyterLab tale
  2. Run the Tale, confirm JupyterLab IDE displays
  3. Edit metadata, change environment to "RStudio"
  4. Confirm icon changes in title bar and Environment section
  5. Stop/run tale or rebuild/restart and confirm RStudio IDE displays

Idle Instances

  • Idle instance culling
  1. Create and start a MATLAB tale and a JupyterLab tale
  2. Confirm both instances start and IDEs are accessible
  3. Close all IDE tabs/windows
  4. Wait for 2 hours and 24 hours respectively
  5. Confirm both instances are stopped automatically

Upload folder

  • Upload folder to home
  1. Create local folder containing files, folders, etc.
  2. Upload folder to home. Confirm contents match local folder.
  3. Download folder. Confirm contents match.
  4. Repeat in workspace

Advanced settings

  • Advanced settings
  1. Create a new tale
  2. Edit Metadata > Advanced, enter:
{
    "environment": [
        "MY_ENV=value"
    ],
    "memLimit":"12gb"
}
1. Start tale
1. `cat /sys/fs/cgroup/memory.max`, confirm 12gb
1. `echo $MY_ENV`, confirm value

Create Tale from DOI

  1. #
  2. Create Tale > Create Tale from DOI
  3. Enter DOI doi:10.7910/DVN/TJCLKP: Title should populate with the DVN dataset title
  4. Enter URL https://sandbox.zenodo.org/record/1059441: Modal should convert to import tale
  5. Enter a non-existent DOI, confirm that Failed to find DOI/URL message displays
  6. Confirm that radio buttons work as expected
  7. Logout

New test cases

  • Confirm runtime.txt
  1. Create new JupyterLab tale
  2. Create runtime.txt
r-4.2-2022-07-01
  1. Start tale and confirm R version is 4.2.x
  • OpenICPSR Integration
  1. Go to Settings, connect to OpenICPRS
  2. API Key is OpenICSPR password. Note that your OpenICSPR account email must
    match your Globus account email for this to work.
  3. https://dashboard.stage.wholetale.org/mine?name=AEAREP-3198-Stata&environment=STATA%2017%20%28Desktop%29&uri=https%3A%2F%2Fwww.openicpsr.org%2Fopenicpsr%2Fproject%2F132081%2Fversion%2FV1%2Fview&asTale=true
  • Export and run Stata Tale
  1. Create tale from Git repo https://github.com/whole-tale/stata-example/
  2. Export tale without building the image
  3. docker manifest inspect <image from run-local.sh> confirm no such manifest
  4. sh run-local.sh, confirm image builds
  5. Run tale and confirm Xpra-based STATA IDE displays
  6. Run example.do, confirm outputs
  • As New Tale
  1. Create a new tale
  2. Create version v1
  3. Make changes
  4. Create a new version v2
  5. Files > Saved Versions > v1 > As New Tal
  6. Confirm new tale is created with tags Copy, Private, Restored with contents of v1
  • Canceling builds
  1. Create tale from https://github.com/whole-tale/rrun-test
  2. Start recorded run with run-infinite.sh
  3. Confirm notification panel displays cancel button
  4. Select cancel button
  5. Confirm run state changes to cancelled and job is cancelled in Girder
  6. Update apt.txt, add package
  7. Select "Rebuild Tale", cancel during build, confirm build is canceled
  8. Select "Stop Tale" then "Run Tale", cancel during build, confirm instance isn't created
@craig-willis
Copy link
Collaborator Author

craig-willis commented Dec 22, 2022

Issues found:

Observations:

  • Spark test case is failing, unclear why at the moment
  • MATLAB Desktop fails to start completely. Should probably fall back to XPRA
  • Rename modal should include original name
  • Moving file in the current workspace is annoying because you have to traverse from the list of tales
  • Downloading a folder doesn't include the selected folder name?
  • Why do we still have the force checkbox on create version/
  • We should populate the user's ORCID if we have it. If I don't have an ORCID configured and try to publish a version to Zenodo, it fails.
  • It would be great to have a way to specify external data in a Git repo (datasets.txt)
  • It would be nice to be able to dismiss individual notifications
  • Lots of HTTP requests in Girder logs
  • I wish I had more ways to sort my tales
  • Need to update MATLAB proxy?
  • What happens if I duplicate a tale while an RR is in process?
  • Need automated test cases for metadata

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant