Redesign the filter-panel (research)

[gogonzo]

Please provide the balsamic

[gogonzo]

Please consider following:

• possibly enable hierarchical filtering. Discuss how the dataset related filters should be changed when other filter is applied. For example imagine two filters SEX and AGE. If SEX filter is limited only to say F, how the AGE should be affected:
  • AGE slider entire range should be limited to the values presented in the filtered-data?
  • AGE slider entire range should remain the same (as unfiltered data) but the selected range should be modified? What happens if user tries to extend this selected range? Can user select AGE range to outside of the filtered-data range but within unfiltered data range?
• Inclusion of the filter labels - should there be two unlabeled and labeled filter selections? What if user select labeled filter and unlabeled which involves the same variable
• global and local filters. Can user select global and local filter for the same variable?

[donyunardi]

I will be using this ticket to track FilterPanel redesign for sprint 62.
Updating SP to 21 to match Kendis.

[nikolas-burkoff]

So rather than thinking about general filtering I wonder if we should be understanding what types of filters are most useful and making the filter panel easier to use for those cases (for example some datasets/variables are much more important to filter on) - if as an app users you are not aware of all the details of all your datasets it can be daunting to set the filters you want.

So you could imagine a "subgroup filter" which allows users/app developers to specify a set of subjects based on some filters in say ADSL which filters the rest of the data accordingly - each of these subgroup filters is named and then in the filter panel you can easily turn on/off a subgroup filter

You could imagine a paramcd filter, an endpoint filter, a visit filter and maybe others which provide other intuitive ways of filtering the data - you then can of course have a completely general filter so more obscure cases can still be done. Certain modules (in say goshawk) which require say a paramcd filter could require an app developer to include one in their filters argument etc.

When creating/editing/viewing details of these filters a modal is used and say for a paramcd filter the information/UI could be tailored to that type of filter and in the filter panel you see (maybe with a mouse over) some details and the different filter types could be coloured and easy to turn on and off.

You could even have a mark on the filter which donates whether the filter is "turned on" for all modules or only the current one sort of handling whether the filter is (currently) global or not.

You could then imagine the (teal) reporter being able to do smarter things with the filters (i.e. defining a set of filters at the beginning of the report and then saying in the cards "Output for Subgroup XXX, for details see filter list"

The above of course would also need to work for MAE data (i.e. select a set of genes/experiments) and also require some analysis to work out what filters would most help the user base and is a big step away from here is a list of columns of your data to filter and so may not be feasible without a huge amount of effort.

I'd also hope there's opportunity for specialism of the filter panel as say if you have helios data how you handle filtering may be different (and that also opens up app developers/module developers to create their own filter panel if what we provide isn't sufficient)

You could imagine a reset to default option which resets the filter panel to exactly how the app developer created it

[danielinteractive]

One important feature request that I hear from e.g. dashdis folks sometime is to allow for predefined filters. So would it be possible for teal to support a simplified filter panel as alternative option, so that the user does not need to select stuff from dropdown menus etc. but can just click on simple on/off buttons, to select efficacy population etc.

[Polkas]

KEYWORDS: SPECIALISM, SIMPLIFY, SHARED-ENGINE
Summing up we need to have a FIlter Panel factory (design pattern) which will provide different panels for specific scenarios.
Some of these panels will be highly simplified (@danielinteractive suggestion). From the code maintenance perspective it will be great to have as much as possible shared engine for all type of filter panel.

[kumamiao]

A quick summary of current user requests here

[gogonzo]

A quick summary of current user requests here

Summary

From above document I conclude that we need couple of things which require consistent API and UI for app-developer (and user) to set them up. I can distinguish so far independent filter-settings which can be combined together to achieve specific effect:

1. filterable - option to limit possible filter-item choices
2. global/local - option to assign filter to particular modules (local) or to all modules (global)
3. filter-labels (custom filters) - unlike currently supported one-column-filter-state we need a way to specify the filter-condition(s) which can be then selected in the app. This means that except choosing add-filter-variable we will have add-filter-label where user can select some predefined filters. teal users expectation is to have simple filter-flags which de facto are single TRUE/FALSE choices, but this can be easily extended with the & operator and also doesn't prevent us to extend this to multiple-conditions (more below).
4. fixed-filters - for chevron there is a need to set fixed-local-filter-label so that this filter can't be removed or modified.

Above requirements are made on the users requests, but I can see the need for more if we want to discuss this together with data-merge refactor. Considering filter-spec being specified only once per dataset it might be wort to move this functionality into filter-panel, which can be extended by:
5. multiple/single - option to have a single select (multiple = FALSE)
6. locked - option to make a filter unremovable but alterable.

Above can be achieved with consistent API and new UI which can extend filter-panel to data-panel where user can add filters, create a labels/variables. Please have a look on the wireframe below, key notes:

• Having one panel instead of three (summary, active filters, add filter variable).
• Adding new states is done through popup-modal instead of having fixed add-filter-variable panel.
• We gain more space and ease.

---

Details

0. Basic option to initialize and modify filter states (available already)

API CALL

# I want to start the app with filters on: ADSL.gender{M, F} and ADRS.avisit{baseline, week 1}

filters = list(
  ADSL = list(
    gender = list(c("M", "F"), keep_na = TRUE)
  ),
  ADRS = list(
    avisit = list("baseline", "week 1")
  )
)

UI element

1. filterable - Option to specify possible filterable variable choices (done here)

# I want to limit filter variables to: adsl.{gender, age, sex, race, country, arm} and adrs.{avisit, paramcd, aval}

filters = list(
  ADSL = list(
    ...,
    filterable(c("gender", "age", "sex", "race", "country", "arm"))
  )
)

2. Global and local filters.

# I want to add module-filter: adrs.paramcd{besrspi, invet}

filters = list(
  ADRS = list(
    avisit = list(...),
    paramcd = list(
      c("basrspi", "invet"), 
      modules = c("KM Plot", "Demographic table"),
    custom_variable = list()
  )
)

3. filter-labels (custom filters)

As described in the summary, initial expectation was about having filter-flag but this can be easily extended to multiple conditions/choices.

# I want to add custom-filter (including filter-flag):
 - adult-male: adsl.gender == m & adsl.age >= 18
 - adult-woman: adsl.gender == f & adsl.age >= 18
 - minor

filters = list(
  ADSL = list(
    genderage = custom_filter(
      label = "Gender and age category",
      `adult-male` = gender == "m" & age >= 18,
      `adult-woman` = gender == "f" & age >= 18,
      `minor` = age < 18
    ),
    adult = custom_filter(
      label = "Adulthood",
      `is adult` = age >= 18
    )
  )
)

Above only intitializes the app with the specific filters, but doesn't provide the set of labels possible to select from. It means we need a consistent way to specify a filterable as in (1). This can be illustrated with

filter = list(
  "ITT" = list(), # default selection
  "AP" = list(), # default selectio
  filterable(read_citrix_labels("path/to.yaml")) # possible labels
)

4. fixed filter - unchangable and unremovable (for all modules or specific modules)

# I want gender filter to be fixed (not changeable):

filters = list(
  ADSL = list(
    gender = list(c("M", "F"), fixed = TRUE)
  )
)

5. Option to set single-selection filter

# I want gender filter to be fixed (not removable):

filters = list(
  ADSL = list(
    gender = list(c("M", "F"), multiple = FALSE)
  )
)

UI is obvious here radioButton/selectizeInput - illustration not provided.

6. Option to specify unremovable (but editable) filter

# I want gender filter to be fixed (not removable):

filters = list(
  ADSL = list(
    gender = list(c("M", "F"), permanent = TRUE)
  )
)

Another things to consider to include in API:

• do we want to limit possible choices
• I'm not a fan of having herarchical-filter which seems too compicated to me. Why should we change the counts in the specific order (filter1 > filter2 > filter3 > ...) while we can update filter-counts everytime when something changes. Counts can be updated in filter1 when filter4 is changes end so on. This could be confusing for the other users than these supporting hierarchy. If it's only about seeing counts I think we might need some other tools to do this, maybe "count tree module"?

---

Not related to API directly is to:

• make a filter-state cards reactive to changes in the data. So the counts will be updated when data changes.
  

• Make a proper UI design for all scenarios to make filter-panel visually attractive

[gogonzo]

WIP: I'll continue in this post with some UML focused on backend.
Key implementation questions:

1. For global/local - if we decide to have local filters then we either have two filter-calls (apply global states and local states separatelly) or single filter-call (applies both on the local level). Both approaches have significant consequences. Two filter calls will complicate the process as we will need to process two queues once in teal and secondly in nested_tabs. If we have only one filter call then filters will be resolved on the "local" level which means that same filter might be computed multiple times across the modules.
2. How to source filter-states with unfiltered and filtered data to keep counts updated.

[nikolas-burkoff]

I'm not a fan of having herarchical-filter which seems too compicated to me. Why should we change the counts in the specific order (filter1 > filter2 > filter3 > ...) while we can update filter-counts everytime when something changes. Counts can be updated in filter1 when filter4 is changes end so on. This could be confusing for the other users than these supporting hierarchy. If it's only about seeing counts I think we might need some other tools to do this, maybe "count tree module"?

@gogonzo it's not just the counts - more importantly it's the ranges

If you have 2 biomarkers X and Y with the range of values of X in [0, 1] and the range of Y in [100, 200] then if you want to filter on only X values > 0.5 then at the moment you filter only on biomarker X but then the range filter still shows you a range of [0, 200] which is very unhelpful - this is the reason why goshawk has its own filtering in the encoding panel I believe @npaszty ?)

Maybe a special "filter" option (hierarchical or I prefer Tableau's name context) to allow a couple of linked variables to be filtered together to handle ranges nicely? Without everything being in a complex hierarchy? This concept seems to be called "context filters" https://help.tableau.com/current/pro/desktop/en-us/filtering_context.htm which I think I prefer (and the general behaviour of filters changing is also called "cascading filters" in other places)

• In some sense the "custom filter" is a "filter-group" and it's almost as if we could let people take existing filters and group them into one filter which can be turned on/off rather than having to get people to specify filters in code and us to parse it?

• Any thoughts on how to get the filter information back to ADSL for filtering for subgroups? It sounds like that is quite an important piece of functionality ( @kumamiao ?) - even if it's not implemented now we should make sure it could be implemented in future?

[kumamiao]

re: hierarchical filter, I think the issue is that currently in our filter panel, the filtering statistics (counts, ranges, etc.) cannot reflect the last filtered data, but only the original unfiltered data, which could be confusing and make it hard to fulfill some needs (i.e., accurate AVAL range based on certain filtered PARAM in ADLB). The concept of the cascading/context filters sound good, will also take a look at some Spotfire examples.

Any thoughts on how to get the filter information back to ADSL for filtering for subgroups? It sounds like that is quite an important piece of functionality ( @kumamiao ?) - even if it's not implemented now we should make sure it could be implemented in future?

User request is reasonable given the analysis scenarios (i.e., subgroups who have a certain AE [based on ADAE], subgroups who have abnormal glucose [based on ADLB], etc.) and is marked to be urgent, although I am also debating on the generalization vs. clinical use for this feature. My initial thought is when users apply filters for non-parent data, there will be a checkbox next to the data name, for users to choose whether to create a subgroup. If so, a new flag will be created in the parent data (user can fill in flag name and filter label via a pop-up window) if the the primary keys in the parent data also exist in the filtered non-parent data. Happy to hear other possibilities as well.

Another thought re: the subgroup, I agree with Pawel as this subgroup creation (with relationship to other non-parent data) can complicate the filter panel. What if there is a way to create a separate module just for the purpose of the subgroup creation/data exploration, in that case we can leave the filter panel to be cleaner, and more generalizable to wider adoption beyond clinical trials.

[npaszty]

@nikolas-burkoff

the goshawk screening and baseline data constrain filtering is very specific. it allows users to filter out subjects who have baseline values, for example, between a specifically set range. this can't be done effectively with current filter design. if paramcd was filtered to let's say ALT then it would be very difficult if not impossible to filter base because the slider would include the range of all biomarkers. so along the lines of what was noted above but with a much broader range.

no paramcd filtering

filtered to one paramcd and no change in base range.

a much more simple concept is if you filter on gender "F" for a demographics subset then you don't want to see data in the filters that belong to "M".
SAS viewtable creates hierachical filtering and to me that's what you would expect from a data filtering.

[npaszty]

had follow up discussion on dynamically building UI elements into the encoding panel. this could be done using a couple of additional arguments to the modules. one argument to indicate if dynamic UI element code should execute (logical) and the other being a list of lists instruction. list element would be type of ui and values. so list(radio, c("Ocular AESIs = aesi", "Ocular Serious AEs")), etc.

this would need to be implemented via a renderUI in the server function along with a code block to do the filtering of the reactive data. found this article on dynamic UI elements in shiny. not exact match but concept indicates this would work.

with that said a little bird told me...
the proper way to filter data is to enable filter label based filtering in the filter panel. For example if the filter label ITT stands for the filter (subset) operation ADSL$ITTFL == "Y" and AESER stands for ADAE$SERIOUSAE == "Y" then one could have a select box where the user can select different filter labels, e.g. ITT and AESER and the resulting subset expression would be ADSL$ITTFL == "Y" & ADAE$SERIOUSAE == "Y"
this would match the current SPA thinking of subsetting data

[donyunardi]

@kumamiao and I discussed how the slider should behave when ranges recalculated during hierarchical filtering activity.

For example:

1. We have demographic data with AGE overall ranges from 1-50
2. User filters AGE and updates the filter slider range to 10(min)-20(max)
3. User makes other filter changes which affects the AGE overall ranges to 30-50.
4. Ranges recalculated, but what would happen to slider on step 2 since the min/max is now out of range?

For this scenario, we think it makes sense to reset the filter slider range. We also suggest that this could always the default behavior for ranges/slider related for recalculating ranges.

[donyunardi]

Linking #102
We want to expose FilterStates object from teal.slice so that user can extend it in their own package.

[gogonzo]

@kumamiao and I discussed how the slider should behave when ranges recalculated during hierarchical filtering activity.

For example:

1. We have demographic data with AGE overall ranges from 1-50
2. User filters AGE and updates the filter slider range to 10(min)-20(max)
3. User makes other filter changes which affects the AGE overall ranges to 30-50.
4. Ranges recalculated, but what would happen to slider on step 2 since the min/max is now out of range?

For this scenario, we think it makes sense to reset the filter slider range. We also suggest that this could always the default behavior for ranges/slider related for recalculating ranges.

This would lead to recalculation of each active FilterState. Because each FilterState (selected) value will be updated it will cause data filtering n-times. It means that if we have 5 filters, changing one FS can trigger recalculation of the module 5 times. To sequential retriggering we can apply some caching comparison mechanism, but nothing is for free (cache will affect size of the app). This is why I suggested to update counts only (density chart in case of slider).

[nikolas-burkoff]

I think there's been talk recently @mhallal1 @pawelru about having the option for much more locked down filter panels as certain use cases may require this

[pawelru]

I think there's been talk recently @mhallal1 @pawelru about having the option for much more locked down filter panels as certain use cases may require this

Let's don't bring that up yet. This discussion is still very much up in the air. Don't want to introduce some another level of complexity now

[donyunardi]

Note from collaboration:

• They use frequency where we use density.
• The frequency got updated when filter is modified (add/edit/delete).
• The two-tone color for filtered data for filtered vs non-filtered.

[gogonzo]

Concept

Currently fiilter panel works in the way depicted on the diagram below. Filters are added by the user or through the API by add_filter_variable. FilterState is created from the selected column of a data.frame and populated to the ReactiveQueue. Based on the changes in ReactiveQueue different filter-call is evaluated (and thus different filtered data). All FilterState in the ReactiveQueue are shared by all modules and imediately applied to them all.

New proposition is slighty different. ReactivseQueue where active filters are kept needs a mapping interface to match the slot (module) with it's filters. The mapping system enables that modules will be able to use different set of filters. Because of the new mapping system UX/UI, API need necessary changes which will affect significant code refactor (but not rewriting from scratch)

Before going further, it's important to distinguish few terms:

1. Filterable - name of the columns in datasets which can be picked to create a filter.
2. Labeled or available filter - a single filter with reactive values and label stored in the filters container. Equivalent of the current FilterState class.
3. Enabled filter - same as (2) but assigned to particular slot (module).

Filter types:

• interactive filter - equivalent of the current filter (by column)
• fixed filter - fixed condition which can't be changed by the app user.
• filter group - a collection of filters (interactive or fixed). Combines the informations of the components

API

Comming soon...

UX design

I'd like to present the UX prototype (first iteration). The prototype focuses on the interaction with the filter cards, filter manager, changing labels, and adding a new filter. Please see the wireframe with entire prototype

1. Filter cards

Please see interactive wireframe. Enabled filters are displayed in the filter-panel as "non-interactive" cards presenting summary of applied labeled filters. Clicking "gear" icon (or card itself) will show the interactive control of the filter. One can change the selected values and assign the label to the filter. By default labels are following a selection, so that:

• for numeric/interger/date - $varname: $min-$max
• for factors/characters: $varname: $choice[1], ..., $choice[n] cut on some character limits.
  When user assigns the value to the filter it no longer follows the selection.
  Icon x on the filter-card removes the filter from the slot (module).

Default	Interactive
	cters -

2. Enable/disable filters

In the module one can apply or unapply filters. By clicking "plus" on the top of the filter-panel menu should show up where one can select/deselect labeled filters for this module. One can also disable filters for the current module by clicking switch on the bottom of the dropdown.

Default	Enable/disable filters

3. Filter manager

Opened by clicking "gear" icon on the top of the filter-panel. New modal will pop up

3.1 Add filter

If you look on the list in "Enable/disable filters " (2), you might ask "Ok, how to apply filter outside of the list". To do this, app user has to first add a labeled filter from the filterable (column). There are two dropdowns, first to select a dataset and second to select a filter. In the second dropdown one can select multiple options:

• labeled interactive filters
• labeled fixed filters
• filterable columns

Selecting two values automatically create a group. Please see the wireframe

Default	Select filters to add	Save choice

After clicking save filter is saved to the list of labeled filters and then can be applied to the model.
Notes:

• selecting a single column adds to a list interactive filters
• selecting only fixed filter(s) is not allowed
• selecting interactive group + interactive column
• selecting interactive group + fixed filter
• selecting multiple interactive groups
• one can select and save existing filter - this will make a duplicate

3.2 Edit labeled filter

3.2.1 Edit filter column

Similar to the "Filter cards" (1), here one can also select values of the labeled filter, change the label. But also can change other properties of the filter like:

• "single value only" (equivalent of multiple in the selectizeInput)
• possible choices
• reset to default values
• apply to multiple modules (probably not needed if we have "filters map")

3.2.2 Edit filter group

Similar to (3.2.1) but on a group level. One can change label and selected values the labeled group and additional properties of labeled components.

3.3 Filter map

In progress. To have a control over the slot/filters map. Possibly drag and drop or a checkbox matrix

Further notes

• filters specified by the app developer on init are not removable
• @nikolas-burkoff rangeslider should have option to zoom in range to see more detailed histograms. This is important for vectors which have kurtic ("peaked") distribitions.
• choices/ranges should be initialized on "unfiltered" data, but the distribution should present the values from the filtered data
• TBD what reactive value of the label should follow (TBD):
  • Default to reflect the value of interactive selection but when the label input is applied by the user, it shouldn't change anymore when interacting with value.
  • Should label follow interactive value selection and interactive label.
  • Or not depend on the interactive value at all

[danielinteractive]

@gogonzo looks very comprehensive!
but also quite complex. Does this proposal include a an optional "low-complexity" filter version, that the app developer can activate?

[gogonzo]

@danielinteractive many things can be controlled by the app developer when initializing FP. I'm happy to continue discussions especially regarding MAE objects

[asbates]

When setting which filters can be applied to which modules, I would consider an option to not allow a given filter to apply to certain modules. Let's say I as an app developer do not want users to be able to apply the Age 20-60 filter for the demographic table module. Then I would like a way to specify that so users are prevented from applying the Age 20-60 filter to the demographic table.

[gogonzo]

Thanks @asbates , good point. Result of such configuration could look like this in the filter manager. Where some filters will be disabled to apply them in the module

[gogonzo]

I'm moving discussions to the document here
Let's update this issue if we conclude on something because comments here can make this messy.

[gogonzo]

@donyunardi I'm happy to close this one ;)