Skip to content

Commit

Permalink
Fix wrong percentile values returned during calibration (#10847)
Browse files Browse the repository at this point in the history
* Use numpy.percentile to get the lookup value.

* Use 1.0 as float value rather than integer.

* Add missing cdf parameter for `np.percentile`.

* Use 100. instead of 1.0

* Remove print.

* Update from @yufenglee
  • Loading branch information
mfuntowicz authored and chilo-ms committed Mar 16, 2022
1 parent 2db707d commit 791462c
Showing 1 changed file with 6 additions and 5 deletions.
11 changes: 6 additions & 5 deletions onnxruntime/python/tools/quantization/calibrate.py
Original file line number Diff line number Diff line change
Expand Up @@ -599,19 +599,20 @@ def compute_percentile(self):

print("Number of tensors : {}".format(len(histogram_dict)))
print("Number of histogram bins : {}".format(self.num_bins))
print("Percentile : {}".format(percentile))
print("Percentile : ({},{})".format(100.0 - percentile, percentile))

for tensor, histogram in histogram_dict.items():
hist = histogram[0]
hist_edges = histogram[1]
total = hist.sum()
cdf = np.cumsum(hist/total)
if self.symmetric:
idx_right = np.searchsorted(cdf, percentile/100)
thresholds_dict[tensor] = (-float(hist_edges[idx_right]), float(hist_edges[idx_right]))
idx_right = np.searchsorted(cdf, percentile / 100.0)
thresholds_dict[tensor] = (-float(hist_edges[idx_ringht]), float(hist_edges[idx_right]))
else:
idx_right = np.searchsorted(cdf, percentile/200)
idx_left = np.searchsorted(cdf, (1.0 - percentile/200))
percent_to_cut_one_side = (100.0 - percentile) / 200.0
idx_right = np.searchsorted(cdf, 1.0 - percent_to_cut_one_side)
idx_left = np.searchsorted(cdf, percent_to_cut_one_side)
thresholds_dict[tensor] = (float(hist_edges[idx_left]), float(hist_edges[idx_right]))

# Plot histogram for debug only
Expand Down

0 comments on commit 791462c

Please # to comment.