Clustergram not reordering heat-map rows

[plpla]

Describe the bug
Clustergram is not using the order provided by the clustering method and dendrogram to display the heat-map.

To Reproduce
Steps to reproduce the behavior:

import numpy as np
import dash_bio
from plotly.offline import iplot, init_notebook_mode

init_notebook_mode()
data = np.array([[1,1,1,1], 
        [3,3,3,3],
        [1,1,1,1],
        [3,3,3,3],
        [1,1,1,1],
        [3,3,3,3]])

iplot(dash_bio.Clustergram(
    data=np.array(data),
    center_values=False
))

Expected behavior
A clear and concise description of what you expected to happen.
Heat-map rows reordering based on the clustering result.

Screenshots
If applicable, add screenshots to help explain the issue.

Python version: [e.g., 3.7.2]
3.7.4
Python environment (all installed packages in your current environment):

Additional context
Add any other context about the problem here.

[mkcor]

Hi @plpla,

Thank you very much for reporting.

Indeed, this is a bug in the read_soft() function from dash-bio-utils, more specifically when setting argument return_filtered_data=True. This in turn calls internal function _get_selected_data, which gets the dataframe's values (https://github.com/plotly/dash-bio-utils/blob/7b9b43a84fe6f091a3eff9a8b6637793dc6a53dc/dash_bio_utils/gene_expression_reader.py#L209), discarding the dataframe's labeled, ordered structure -- preserving it is the entire point of using dataframes though!!

I noticed this bug when writing tests for the read_soft() utility function with return_filtered_data=True. The order isn't deterministic, so I couldn't actually write a test for it! We have to fix the function first. I'll be on it next week. If you want to give it a try, I'll be happy to review your work and iterate.

Best,
Marianne

[mkcor]

Hi @plpla,

Sorry, I got completely mixed up with a bug in the Clustergram demo app! Here, we are talking about the Clustergram component. I'm not sure what you mean by "the order provided by the clustering method and dendrogram..."

By default, the Clustergram() method clusters along columns first:

dash-bio/dash_bio/component_factory/_clustergram.py

Lines 72 to 75 in 74e5173

	- cluster (string; default 'all'): The dimension along which the data will
	be clustered: 'row', 'column', or 'all'; 'all' means data will be
	clustered along columns, then clustered along rows of
	column-clustered data.

In your example, data is

array([[1, 1, 1, 1],
       [3, 3, 3, 3],
       [1, 1, 1, 1],
       [3, 3, 3, 3],
       [1, 1, 1, 1],
       [3, 3, 3, 3]])

so its transpose matrix is

array([[1, 3, 1, 3, 1, 3],
       [1, 3, 1, 3, 1, 3],
       [1, 3, 1, 3, 1, 3],
       [1, 3, 1, 3, 1, 3]])

and we can see that all 'observations' (rows) are identical, so there is zero distance. In the computation of the clusters and dendrogram, this distance matrix comes first:

dash-bio/dash_bio/component_factory/_clustergram.py

Lines 749 to 752 in 74e5173

	# cluster along columns
	if self._cluster in ['col', 'all']:
	tmp = np.transpose(self._data)
	dcol = self._dist_fun(tmp, metric=self._col_dist)

I'm not a dendrogram expert, but what kind of dendrogram were you expecting in this trivial case?

[plpla]

Hi @mkcor ,
I would expect the result to be highly similar to Searborn Clustermap since they are the same type of figure.
Using the data from my very basic example, the rows with identical values (ones or threes) should be next to each other. See the example below.

import seaborn as sns
import numpy as np

data = np.array([[1,1,1,1], 
        [3,3,3,3],
        [1,1,1,1],
        [3,3,3,3],
        [1,1,1,1],
        [3,3,3,3]])

sns.clustermap(data, cmap="mako")

To be clear, the clusering in the Clustergram works as the dendrogram on the left side shows 2 groups. Although the rows in the heat-map are not reordered to represent these clusters. In the end, I would expect to get a block of red rows and a block of green rows.

[mkcor]

Hi @plpla,

Thank you very much for the detailed answer. This is definitely a bug on our end. Our Clustergram builds on top of scipy.cluster.hierarchy just like seaborn's clustermap and we should definitely get two clusters when running dash_bio.Clustergram(data) or dash_bio.Clustergram(data, cluster='row') (in this example).

I'll try and fix it at the beginning of 2020!

/cc @shammamah

[mkcor]

I have finally found the bug, yay! Fixed in 611a1b3. PR coming soon...