image and heatmap traces jump/flash on Firefox and Safari

[jonmmease]

The new image trace seems to have a flashing/jumping behavior when zoomed in and panning on Firefox and Safari. (Firefox version 69.0, Safari version 13.0.3)

Example from https://plot.ly/python/imshow/

This jumping does not happen on Chrome but I do see it on Firefox/Safari. Not sure if it's related, but in all three browsers, the tooltip also seems to jump to another location when the pan is completed.

[antoinerg]

@jonmmease I think the issue also occurs on heatmap (https://plot.ly/python/heatmap/). Could you rename the issue to reflect that?

The problem stems from reinjecting an image when pan operation is completed. I thought this pattern was safe since it's been used for a long time in heatmap. Maybe this issue only affects newer versions of Firefox 🤔 ?

[jonmmease]

Could you rename the issue to reflect that?

Done, thanks!

[etpinard]

Hmm. I can't replicate in FF 70

[jonmmease]

Hmm, I am seeing it in FF 70 on Ubuntu. @antoinerg were you able to reproduce?

[antoinerg]

Hmm, I am seeing it in FF 70 on Ubuntu. @antoinerg were you able to reproduce?

I can reproduce it in FF 70.0.1 (64-bit) on Linux (NixOS distro).

[firasm]

I can also reproduce in Safari 13.0.3 on macOS 10.15.1.

On the same system, the problem does not appear in Chrome Version 78.0.3904.108.

[etpinard]

@antoinerg do you think there's an "easy" way to fix this bug?

Or do you think our canvas -> pixels -> img inside the _paper SVG is fundamentally flawed and FF does behaves as expected?

[antoinerg]

Or do you think our canvas -> pixels -> img inside the _paper SVG is fundamentally flawed and FF does behaves as expected?

It's hard to say which one behaves right and I suspect the specification may not be clear in this regard.

do you think there's an "easy" way to fix this bug?

Idea for an easy fix: have two images (A and B) and alternate between the two. Once you pan, image B content is updated and moved in front of image A (so no flash in principle). Then if you pan again, image A content is updated and moved in front and so on. Hopefully, this approach works.

If we only target a fix for FF: leverage image-rendering: pixelated, draw the image only once and things will fly 🐎.

[alexcjohnson]

So we're reverting the drag, then redrawing with the new view... and it seems like even if we do this synchronously, some browsers will do two repaints (if that's it, there will be async edge cases in other browsers too). I wonder if we can just not revert the drag, and tweak the redraw process if necessary so it still works right. I think it's this call that's doing it:

plotly.js/src/plots/cartesian/dragbox.js

Lines 770 to 774 in 4ab3b49

	function dragTail() {
	// put the subplot viewboxes back to default (Because we're going to)
	// be repositioning the data in the relayout. But DON'T call
	// ticksAndAnnotations again - it's unnecessary and would overwrite `updates`
	updateSubplots([0, 0, pw, ph]);

updateSubplots seems to modify some internal state as well as adjusting viewboxes, we still may need the internal state even if we don't change anything visible.

[antoinerg]

This may be a problem with Firefox itself: Images are flashing when updates by img.src[]=url in Firefox >= 8.0

Potential solution: https://stackoverflow.com/questions/14704796/image-reload-causes-flicker-only-in-firefox/14711272#14711272

[etpinard]

FYI: using some navigator.userAgent-based solution here would be 👌 with me.

---

It would be interesting to see how image-rendering: pixelated performs in comparison to the current algorithm. Using that in browsers that support it could be a big win and allow us to fix this problem in FF.

The solution in https://stackoverflow.com/questions/14704796/image-reload-causes-flicker-only-in-firefox/14711272#14711272 may be a quick and easy way to fix this bug in Safari (albeit with a potential performance cost).

[antoinerg]

The issue is twofold:

1. Image is flashing (can be fixed by not redrawing the image)
2. Tooltip appears at the position of the last click

EDIT: I opened an issue to tackle 2) since it affects other trace types (#4570)

[antoinerg]

@etpinard at the moment, when an axis has a scaleanchor attribute (the default for image), the calc step is called on every pan operation. For the planned improvement to the image trace type, this is somewhat undesirable. We'd like to produce the image in calc once and then simply move/scale it in the plot phase when a pan operation is completed. The goal is to save ourselves from the sometimes expensive operation of looping over all the pixels in view (and then some). Do you see potential roadblocks or flaws in trying to achieve what I just described? How would you go about skipping the calc phase when it's being triggered by a pan?

[etpinard]

Here's the relevant logic that makes zoom interactions go through the 'calc' pipeline:

plotly.js/src/plot_api/plot_api.js

Lines 2274 to 2293 in 15d75db

	// figure out if we need to recalculate axis constraints
	var constraints = fullLayout._axisConstraintGroups || [];
	for(axId in rangesAltered) {
	for(i = 0; i < constraints.length; i++) {
	var group = constraints[i];
	if(group[axId]) {
	// Always recalc if we're changing constrained ranges.
	// Otherwise it's possible to violate the constraints by
	// specifying arbitrary ranges for all axes in the group.
	// this way some ranges may expand beyond what's specified,
	// as they do at first draw, to satisfy the constraints.
	flags.calc = true;
	for(var groupAxId in group) {
	if(!rangesAltered[groupAxId]) {
	Axes.getFromId(gd, groupAxId)._constraintShrinkable = true;
	}
	}
	}
	}
	}

By the looks of it, all the range "constraints" operations are taken care of in

plotly.js/src/plots/cartesian/constraints.js

Lines 198 to 352 in 15d75db

	exports.enforce = function enforce(gd) {
	var fullLayout = gd._fullLayout;
	var constraintGroups = fullLayout._axisConstraintGroups || [];
	var i, j, axisID, ax, normScale, mode, factor;
	for(i = 0; i < constraintGroups.length; i++) {
	var group = constraintGroups[i];
	var axisIDs = Object.keys(group);
	var minScale = Infinity;
	var maxScale = 0;
	// mostly matchScale will be the same as minScale
	// ie we expand axis ranges to encompass *everything*
	// that's currently in any of their ranges, but during
	// autorange of a subset of axes we will ignore other
	// axes for this purpose.
	var matchScale = Infinity;
	var normScales = {};
	var axes = {};
	var hasAnyDomainConstraint = false;
	// find the (normalized) scale of each axis in the group
	for(j = 0; j < axisIDs.length; j++) {
	axisID = axisIDs[j];
	axes[axisID] = ax = fullLayout[id2name(axisID)];
	if(ax._inputDomain) ax.domain = ax._inputDomain.slice();
	else ax._inputDomain = ax.domain.slice();
	if(!ax._inputRange) ax._inputRange = ax.range.slice();
	// set axis scale here so we can use _m rather than
	// having to calculate it from length and range
	ax.setScale();
	// abs: inverted scales still satisfy the constraint
	normScales[axisID] = normScale = Math.abs(ax._m) / group[axisID];
	minScale = Math.min(minScale, normScale);
	if(ax.constrain === 'domain' || !ax._constraintShrinkable) {
	matchScale = Math.min(matchScale, normScale);
	}
	// this has served its purpose, so remove it
	delete ax._constraintShrinkable;
	maxScale = Math.max(maxScale, normScale);
	if(ax.constrain === 'domain') hasAnyDomainConstraint = true;
	}
	// Do we have a constraint mismatch? Give a small buffer for rounding errors
	if(minScale > ALMOST_EQUAL * maxScale && !hasAnyDomainConstraint) continue;
	// now increase any ranges we need to until all normalized scales are equal
	for(j = 0; j < axisIDs.length; j++) {
	axisID = axisIDs[j];
	normScale = normScales[axisID];
	ax = axes[axisID];
	mode = ax.constrain;
	// even if the scale didn't change, if we're shrinking domain
	// we need to recalculate in case `constraintoward` changed
	if(normScale !== matchScale || mode === 'domain') {
	factor = normScale / matchScale;
	if(mode === 'range') {
	scaleZoom(ax, factor);
	} else {
	// mode === 'domain'
	var inputDomain = ax._inputDomain;
	var domainShrunk = (ax.domain[1] - ax.domain[0]) /
	(inputDomain[1] - inputDomain[0]);
	var rangeShrunk = (ax.r2l(ax.range[1]) - ax.r2l(ax.range[0])) /
	(ax.r2l(ax._inputRange[1]) - ax.r2l(ax._inputRange[0]));
	factor /= domainShrunk;
	if(factor * rangeShrunk < 1) {
	// we've asked to magnify the axis more than we can just by
	// enlarging the domain - so we need to constrict range
	ax.domain = ax._input.domain = inputDomain.slice();
	scaleZoom(ax, factor);
	continue;
	}
	if(rangeShrunk < 1) {
	// the range has previously been constricted by ^^, but we've
	// switched to the domain-constricted regime, so reset range
	ax.range = ax._input.range = ax._inputRange.slice();
	factor *= rangeShrunk;
	}
	if(ax.autorange) {
	/*
	* range & factor may need to change because range was
	* calculated for the larger scaling, so some pixel
	* paddings may get cut off when we reduce the domain.
	*
	* This is easier than the regular autorange calculation
	* because we already know the scaling `m`, but we still
	* need to cut out impossible constraints (like
	* annotations with super-long arrows). That's what
	* outerMin/Max are for - if the expansion was going to
	* go beyond the original domain, it must be impossible
	*/
	var rl0 = ax.r2l(ax.range[0]);
	var rl1 = ax.r2l(ax.range[1]);
	var rangeCenter = (rl0 + rl1) / 2;
	var rangeMin = rangeCenter;
	var rangeMax = rangeCenter;
	var halfRange = Math.abs(rl1 - rangeCenter);
	// extra tiny bit for rounding errors, in case we actually
	// *are* expanding to the full domain
	var outerMin = rangeCenter - halfRange * factor * 1.0001;
	var outerMax = rangeCenter + halfRange * factor * 1.0001;
	var getPad = makePadFn(ax);
	updateDomain(ax, factor);
	var m = Math.abs(ax._m);
	var extremes = concatExtremes(gd, ax);
	var minArray = extremes.min;
	var maxArray = extremes.max;
	var newVal;
	var k;
	for(k = 0; k < minArray.length; k++) {
	newVal = minArray[k].val - getPad(minArray[k]) / m;
	if(newVal > outerMin && newVal < rangeMin) {
	rangeMin = newVal;
	}
	}
	for(k = 0; k < maxArray.length; k++) {
	newVal = maxArray[k].val + getPad(maxArray[k]) / m;
	if(newVal < outerMax && newVal > rangeMax) {
	rangeMax = newVal;
	}
	}
	var domainExpand = (rangeMax - rangeMin) / (2 * halfRange);
	factor /= domainExpand;
	rangeMin = ax.l2r(rangeMin);
	rangeMax = ax.l2r(rangeMax);
	ax.range = ax._input.range = (rl0 < rl1) ?
	[rangeMin, rangeMax] : [rangeMax, rangeMin];
	}
	updateDomain(ax, factor);
	}
	}
	}
	}
	};

which is called during

plotly.js/src/plot_api/subroutines.js

Lines 669 to 721 in 15d75db

	exports.doAutoRangeAndConstraints = function(gd) {
	var fullLayout = gd._fullLayout;
	var axList = Axes.list(gd, '', true);
	var matchGroups = fullLayout._axisMatchGroups || [];
	var axLookup = {};
	var ax;
	var axRng;
	for(var i = 0; i < axList.length; i++) {
	ax = axList[i];
	cleanAxisConstraints(gd, ax);
	doAutoRange(gd, ax);
	axLookup[ax._id] = 1;
	}
	enforceAxisConstraints(gd);
	groupLoop:
	for(var j = 0; j < matchGroups.length; j++) {
	var group = matchGroups[j];
	var rng = null;
	var id;
	for(id in group) {
	ax = Axes.getFromId(gd, id);
	// skip over 'missing' axes which do not pass through doAutoRange
	if(!axLookup[ax._id]) continue;
	// if one axis has autorange false, we're done
	if(ax.autorange === false) continue groupLoop;
	axRng = Lib.simpleMap(ax.range, ax.r2l);
	if(rng) {
	if(rng[0] < rng[1]) {
	rng[0] = Math.min(rng[0], axRng[0]);
	rng[1] = Math.max(rng[1], axRng[1]);
	} else {
	rng[0] = Math.max(rng[0], axRng[0]);
	rng[1] = Math.min(rng[1], axRng[1]);
	}
	} else {
	rng = axRng;
	}
	}
	for(id in group) {
	ax = Axes.getFromId(gd, id);
	ax.range = Lib.simpleMap(rng, ax.l2r);
	ax._input.range = ax.range.slice();
	ax.setScale();
	}
	}
	};

from a few places in plot_api.js including in the "fast" 'axrange' edit pipeline:

plotly.js/src/plot_api/plot_api.js

Lines 1970 to 1976 in 15d75db

	seq.push(
	clearSelect,
	subroutines.doAutoRangeAndConstraints,
	drawAxes,
	subroutines.drawData,
	subroutines.finalDraw
	);

---

So perhaps zoom/pan interaction just work already under the fast 'axrange' edit pipeline? @antoinerg have you tried commenting the above block in _relayout to see in any tests fail?

We'll have to be extra careful though. I wouldn't want to trust 100% our test coverage for constraints.

But yeah, to answer your question, making zoom/pan on constrained axes not have to go through the 'calc' pipeline would be a huge win for image traces. Maybe 'axrange' works already, maybe we'll need a new flavour of that 'axrange' pipeline, but yeah we can do better than 'calc'!

[etpinard]

(Ping, I updated my comment above)

[jackparmer]

This issue has been tagged with NEEDS SPON$OR

A community PR for this feature would certainly be welcome, but our experience is deeper features like this are difficult to complete without the Plotly maintainers leading the effort.

Sponsorship range: $10k-$15k

What Sponsorship includes:

• Completion of this feature to the Sponsor's satisfaction, in a manner coherent with the rest of the Plotly.js library and API
• Tests for this feature
• Long-term support (continued support of this feature in the latest version of Plotly.js)
• Documentation at plotly.com/javascript
• Possibility of integrating this feature with Plotly Graphing Libraries (Python, R, F#, Julia, MATLAB, etc)
• Possibility of integrating this feature with Dash
• Feature announcement on community.plotly.com with shout out to Sponsor (or can remain anonymous)
• Gratification of advancing the world's most downloaded, interactive scientific graphing libraries (>50M downloads across supported languages)

Please include the link to this issue when contacting us to discuss.