Monitoring: No more need to send all data points to a device

[janagombitova]

Context

One of our goals is to ensure users can continuously monitor their data points on large scale.

Current monitoring in Flow

In Flow you can define a registration form and its connected monitoring forms in one survey.

The registration form defines the object of your data collection. If you are interested in learning about the status of a water point in communities in your country, your registration form will hold the basic information about a water point - the information that defines it, that does not change (location, ID, photo, type). If you are interested in learning about the quality of education in schools in your programme, your data point will be a school and in the registration form you will ask questions as location, photo, when was it built, by who, etc. It you are interested in learning about farmers in your programme, then your data point will be a farmer and you will ask him questions in the registration form as her name, date of birth, photo, etc.

The monitoring forms (one or many) will hold questions that you want to ask your subject of your data collection repetitively over the course of time to see change. In our case of water points one monitoring form can hold questions about the water quality, another monitoring form about the overall status of the water point as if it is functioning, clean, how many people are there, etc, all depending on what you want to analyse. In the school case your monitoring form can be an internal inspection form, another form can be about the satisfaction of the pupils, etc. In the farmer case a monitoring form can hold questions about his crops and yield, where another form on his nutrition.

When in the field you first submit the registration form, this creates your data point. Now you can submit the monitoring form to that data point. Your first monitoring form submission will create your baseline and the following submissions will help you see change over time. The next time you come back to your data point, you will select the data point from the list of points on your device and submit another monitoring form submission.

Current issue

If you are monitoring all water points in a region or country wide we are imagining thousands of data points to monitor. Currently in Flow once a registration form is submitted and a data point created any enumerator who is assigned the survey, can download the created data points to his device. This makes it possible that if Al registered point A, Isha can download also his point A and monitor it, not having the data points only available to Al.

But if there are thousand points registered, this means once Al and Isha hit download on their Flow app all thousand points will start downloading to their devices.

• takes a lot of time, bandwidth and storage on the device to download all data points (tested to download 6k data points on Amsterdam internet, took me over 3 hours with multiple retries)
• in areas with low connectivity is it impossible
• Al does not need to have all the data points as he is responsible to monitor only a certain group of them
• having so many data points, finding the one to monitor becomes painful for Al and Isha and possibly resulting in them selecting the wrong data point causing the submission to link to the wrong previous data creating an analysis headache and risk
• having so many data points, it happens that Al sometimes by mistake makes two monitoring form submissions to the one data point in the round due to all the data points on his phone as it is hard to see which he already monitored

Opportunity

How might we enable Al to only get the data points he needs to monitor and for Isha the points she need to monitor.

We see that there are multiple issues when monitoring data in the field:

• Al does not select the right data point submitting information to the wrong one
• Al submits a monitoring form twice in that round even though he should make one submission per point but finds it hard to see which he already worked on
• Downloading data points to the device on low connectivity is impossible

And we guess that by limiting the number of data points Al receives on his device will if not address then at least decrease the other problems.

What we are not doing

When we researched monitoring we also learned about cases when monitoring was done with random sampling in this case monitoring is not done by going back to the same location, person or school to see the change for the specific data point but is done on a group as a whole. A random sample is defined, there people are interviewed and in the next round another random sample is defined and data is collected. In this case the data is compared for the aggregated group over the course of time. Flow forms can be used here, without the monitoring feature.

We will not be investigating opportunities around supporting random sampling at this moment of time.

Documentation

Notes from user research:
https://docs.google.com/spreadsheets/d/1qK13keQZ1rchlhLCBX3SNSudf5IXjnSdlQKRp5GrG8k/edit#gid=0

User journey and story map:
https://app.mural.co/t/akvo8194/m/akvo8194/1519979617595/9e87113d9211e9ead6ab12b98ea8ad51e8b8f079

[janagombitova]

Today we agreed that:

• supporting sampling as a processes is an interesting opportunity but we will not pursue it for now
• we will focus on helping Al get the points he needs on his device
• Salim created a group of points and assigns these to Al
• the group is based on criteria that come from the registration form - answers to the registration form

We also still need to think of:

• what is the role of the dates for assignments? (we are playing with the idea to remove them fully)
• what happens once Al is removed from an assignment? what happens once an assignment is deleted? What implications does this have?

Separate issue:

• if there are rules on which forms Al needs to submit now, how do we enforce it?

[janagombitova]

Mock up review

Today we reviewed the current mock ups made by @Kiarii and suggested to:

• look at better supporting Salim in making the decisions, so what does he need to do first and what next in the UI
• clarify more that the filters relate to data points
• explore the fact that after selecting the survey Salim might first want to see the available data points before he selects or filters them out to add to the assignment as a whole or per device
• add the concept of device groups

We also agreed that Salim will only assign data points and see any functionality related to this, when he selects a monitoring survey and monitoring forms to assign. If he selects a non-monitoring survey, he has no data points to assign thus does not see the functionality. If he selects a monitoring survey but only the registration form, he is not going to monitor yet thus also has no data points to assign, so does not see the functionality.

The filters will only be based on the registration form. We agreed to limit what the filters can be based on further, like only option and cascade question answers and some meta data columns as submission date. We still need to define the filters in more detail.

What is an assignment

When discussing the question of what role do dates play in an assignment and what do we sync in in term of previously made submissions - what is assigned or everything, we realised we need to first answer to big question: What is an assignment? We found 2 answers:

1. A way of getting survey forms to devices
2. A task to collect data.

Assignment = task

An assignment as a task means that an assignment is something Salim gives to Al to do. So Al is given an assignment that he needs to complete. This assignment holds survey forms, data points and has a timeframe of completion. It has its rules. It holds What forms Al has to collect data for, Which data points (if that applies) he needs to update, When must he do this work. In this case then Al should not see surveys on his device, but assignments. Then the assignment date can have a stronger role also enforcing open/closed submission dates.

Decision

We agreed to delay the decision as:

• we do not want to implement two big changes at once and the monitoring change has more priority right now
• we want to learn more from users on how they understand assignments and how they manage the data collection process
• if we want to enforce the task concept then we should move to app users first. Currently one device can have more than one user thus enforcing tasks per user in the current concept will complicate the workflow on the app

Other questions

Currently, you can add more different survey forms to one assignment (survey A is monitoring and I assign form 1, survey B is not monitoring and I assign its form too, survey C is monitoring and I assign form 1 and 2). How will we support this?

Eventually, how will we break down the implementation to small chunks we can get out to users quickly?

[janagombitova]

Mock ups: https://projects.invisionapp.com/share/TZP8HSGRB5N#/screens

Next we will review them with the team, define the 1st slice, roll up our sleeves and build.

[janagombitova]

Today we discussed with the team how to make the implementation happen and agreed to go about it in 2 steps:

1st step

• one assignment per survey rule (but one survey can be used in multiple assignments)
• redesign current Assignments UI (how an assignment is set up and how the list of existing assignments will look like - what info is handy to have there in the overview)
• no changes to how dates work
• only allow assigning published forms for the survey (as today)
• re-write frontend to React
• add tests

2nd step will be

• add the option to assign data points by searching by point name and ID
• the search will not support autosuggest or autocomplete
• Salim types in a word or number and hits search -> see a list of data points that in their data point name or ID have what he searched for -> Salim decides which of these data points he assigns (one by one or by 'assigning all')
• might change the backend to move away from our current datastore on Google to a Postgres one depending on how we move forward with Flow-Lumen work

Future changes

• eventually, once the two steps are completed we will gather input on how things are working and what other changes need to be made
• we will also look at adding assigning data points by form data (option or cascade answers)
• we will also research later what the role of an assignment is as a task not only for Salim but also for Al collecting the data

[janagombitova]

Updates mockups from @Kiarii are here: https://projects.invisionapp.com/share/NHQGVLB2EKX#/346093699_0-0_Assignments_ES

[janagombitova]

The final mockups are here: https://projects.invisionapp.com/share/5YUUUPXF7DA#/screens

[muloem]

After reviewing the mockups with @iperdomo @stellanl and @Kiarii, we concluded that implementing the Assign all datapoints functionality has a number of problems associated with it

• First it creates the issue where if we have a survey with a large number of datapoints, we will run into the same performance issues that we want to solve by selective datapoint assignment. i.e. if there are too many datapoints, the backend will cancel the request halfway through and the device user doesnt get all the datapoints they expect.

• The usability of very large number of datapoints on the devices is questionable.

• We can potentially end up with an exponential explosion of entities on the backend related to datapoint assignments, which will also be a problem for the performance of this feature.

Based on these issues, we concluded that we will put on hold the part of assigning all datapoints to a single device. For now we will only implement that part where the user is able to search for datapoints by display name or by datapoint ID.

[Kiarii]

Its amazing to have this working even though still a work in progress :) Notes and comments on current state of implementation:

• perhaps we should unify the folder structure dropdown navigation with the rest of flow

• Assignment names in the Assignments page should be left aligned for ease of scanning and reading, also to keep consistency with the rest of asset lists in Flow. A side note; not sure why we refer to Assignments and Devices as Assignment List and Device List.. it is an unnecessary redundancy and only increases the number elements on the interface that the user has to pay attention to
• Enable only the selection of Device groups when adding devices to assignment; this should simplify the task. Lets reserve device group management to Devices page. I had also commented about this in Slack

• lets keep the survey structure browsing drop downs on a single line as with other pages as well instead of breaking it when these exceed two;

• Seems like after adding a device, the empty state message "no devices added to the assignment yet" is still showing

This also appears even when the device is there as a clickable item and has datapoints assigned to it

• It seems strange that an assignment cannot be saved until all the components - duration, forms, devices - have been defined. I think the user should be able to save an assignment at any point in its editing
• When successful, Saving should not take the user back to the Assignment's list, the action is discreet from navigating back now that it can be applied at any time during an Assignment editing..
• we can drop the ghost search icon that appears on the right of the datapoint search field

• Enable partial data name search to give results
• All datapoint showing by pressing [enter] is a handy keyboard shortcut even though it is a hidden one. We could make this more obvious by adding "press Enter to view all datapoints" to the search datapoint default text. This shortcut might not be as useful for the infamous 20k datapoint projects :) We need to figure out how to search datapoints by some other aspect e.g. location or provide such as a way showing specific batches of datapoints. I think we shd give it a thought and @muloem, @marvinkome and myself should try to find some consensus for the first iteration..
• keep the Assignment name field's border in view by default - but with a lighter shade than when active - to make it obvious that this is an editable element..
• sort device groups and devices names within Alphabetically so its easier for the user to scan and find devices (groups) of interest
• I find it a weird that when you create an assignment and save it, it is automagically pushed to devices. I think there should be an explicit action on the part of the user to do that; even though a user saves an assignment perhaps they are not done editing it and might want to delay the push till they are done - this would be another point of consideration @muloem. It is at the push to devices action that the components shd be validated (I think) not at save..
• for registration assignments, the 'assign datapoints' action should not be present. The datapoints assigned 'counter' and 'Edit' should also be removed
• lets drop "by name or id" from the assign datapoints button;

[Kiarii]

Since assigning datapoints to a device is not obvious, the idea had been that by adding devices, the first device in the list would automatically be selected for the purpose, to make it clear to the user what more they can do..e.g.

[Kiarii]

@marvinkome there is a bug with the published form; tricky to describe but I captured it:

[Kiarii]

Following up with some minor notes for improvement:

• On assignment options;

lets shorten the first option to the following, there is no need to be repetitive when these two are just next to each other

• Lets make the keyboard shortcut to reveal all datapoints more obvious; This requires that once the user selects the "By name or ID" option and the sidebar for this is opened, we place the cursor into the search field by default

this is a first step, for huge lists we will need to figure out a way to present datapoints gracefully

[Kiarii]

• long device list breaks the screen layout; the side bar should scroll independently so that the essential actions such as Save, Navigating back, the Device and Form elements are in view..

@muloem
*As a side note I have questioned the necessity of the presence of the Footer in every screen; I think it should be folded into a main nav item e.g. support or the like.. Atm it breaks every screen since it appears in different locations - in the vertical dimension. I am also not convinced that its omnipresence provides that much added value to users..

[Kiarii]

• similar issue with several forms; the screen layout breaks

[marvinkome]

This has been reported here #3475

[muloem]

Datapoint assignemts

[janagombitova]

This epic intended to get assigning data points to devices out in the open. The initial implementation is out there. The implementation is that by default all devices in an assignment get all data points assigned and Salim needs to explicitly assign specific data points if he wants to.

The next step (now in progress) is to improve this current implementation. We have created separate issues to manage the work, they are labelled '1 - Assignments' and added to the 'To do' column.

We will keep this iteration live for a certain period of time to learn about it and then pivot. The pivot-steps will be captured in separate issues linking to this one.

So it is time we close this monster. Well done team!