[BUG] grnboost2 reports msgpack._cmsgpack.unpackb ValueError: 3159985209 exceeds max_bin_len(2147483647)

[xyang2uchicago]

Describe the bug
How to handle big dataset? It is suggested to feed pySCENIC raw counts. I have a dataset of the shape(19386, 20483). It stops when running grnboost2().

Steps to reproduce the behavior

1. Command run when the error occurred:

tf_names = load_tf_names(MM_TFS_FNAME)

# the ranking databases are loaded:
db_fnames = glob.glob(DATABASES_GLOB)
def name(fname):
    return os.path.splitext(os.path.basename(fname))[0]

dbs = [RankingDatabase(fname=fname, name=name(fname)) for fname in db_fnames]
dbs


## load the scRNA-seq raw counts to be anbalyzed 
RESOURCES_FOLDER="/project/xyang2/heart_dev/IbarraSoria2018_MouseE8.25/"
SC_EXP_FNAME = os.path.join(RESOURCES_FOLDER, "rawcounts_filtered_wSymbols.tsv") # raw counts, mm10
genome = 'mm10'

ex_matrix = pd.read_csv(SC_EXP_FNAME, sep='\t', header=0, index_col=0).T
ex_matrix.shape
# (19386, 20483)  # cells , genes


# The arboreto package is used for this phase of the pipeline. 
adjacencies = grnboost2(ex_matrix, tf_names=tf_names, verbose=True)

2. Error encountered:

...

Expected behavior
This is codes following the tutorial at https://pyscenic.readthedocs.io/en/latest/tutorial.html#

Please complete the following information:

• pySCENIC version: 0.12.1
• Installation method: Pip
• Run environment: HPC script
• OS: Linux
• Package versions: conda list

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aiohappyeyeballs          2.4.3                    pypi_0    pypi
aiohttp                   3.10.8                   pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
anndata                   0.9.2                    pypi_0    pypi
arboreto                  0.1.6                    pypi_0    pypi
async-timeout             4.0.3                    pypi_0    pypi
attrs                     24.2.0                   pypi_0    pypi
bokeh                     3.1.1                    pypi_0    pypi
boltons                   24.0.0                   pypi_0    pypi
bzip2                     1.0.8                h4bc722e_7    conda-forge
ca-certificates           2024.8.30            hbcca054_0    conda-forge
certifi                   2024.8.30                pypi_0    pypi
cffi                      1.17.1                   pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
cmake                     3.18.2                   pypi_0    pypi
contourpy                 1.1.1                    pypi_0    pypi
ctxcore                   0.2.0                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
cytoolz                   0.12.3                   pypi_0    pypi
dask                      2023.5.0                 pypi_0    pypi
dill                      0.3.9                    pypi_0    pypi
distributed               2023.5.0                 pypi_0    pypi
fonttools                 4.54.1                   pypi_0    pypi
frozendict                2.4.4                    pypi_0    pypi
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2024.9.0                 pypi_0    pypi
get-annotations           0.1.2                    pypi_0    pypi
h5py                      3.11.0                   pypi_0    pypi
idna                      3.10                     pypi_0    pypi
importlib-metadata        8.5.0                    pypi_0    pypi
importlib-resources       6.4.5                    pypi_0    pypi
interlap                  0.2.7                    pypi_0    pypi
jinja2                    3.1.4                    pypi_0    pypi
joblib                    1.4.2                    pypi_0    pypi
kiwisolver                1.4.7                    pypi_0    pypi
ld_impl_linux-64          2.43                 h712a8e2_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    14.1.0               h77fa898_1    conda-forge
libgcc-ng                 14.1.0               h69a702a_1    conda-forge
libgomp                   14.1.0               h77fa898_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libsqlite                 3.46.1               hadc24fc_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.3.1                h4ab18f5_1    conda-forge
llvmlite                  0.41.1                   pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
loompy                    3.0.7                    pypi_0    pypi
lz4                       4.3.3                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.7.5                    pypi_0    pypi
msgpack                   1.1.0                    pypi_0    pypi
multicoretsne             0.1                      pypi_0    pypi
multidict                 6.1.0                    pypi_0    pypi
multiprocessing-on-dill   3.5.0a4                  pypi_0    pypi
natsort                   8.4.0                    pypi_0    pypi
ncurses                   6.5                  he02047a_1    conda-forge
networkx                  3.1                      pypi_0    pypi
numba                     0.58.1                   pypi_0    pypi
numexpr                   2.8.6                    pypi_0    pypi
numpy                     1.23.5                   pypi_0    pypi
numpy-groupies            0.9.22                   pypi_0    pypi
openssl                   3.3.2                hb9d3cd8_0    conda-forge
packaging                 24.1                     pypi_0    pypi
pandas                    2.0.3                    pypi_0    pypi
partd                     1.4.1                    pypi_0    pypi
patsy                     0.5.6                    pypi_0    pypi
pillow                    10.4.0                   pypi_0    pypi
pip                       24.2               pyh8b19718_1    conda-forge
psutil                    6.0.0                    pypi_0    pypi
pyarrow                   17.0.0                   pypi_0    pypi
pycparser                 2.22                     pypi_0    pypi
pynndescent               0.5.13                   pypi_0    pypi
pyparsing                 3.1.4                    pypi_0    pypi
pyscenic                  0.12.1                   pypi_0    pypi
python                    3.8.20          h4a871b0_2_cpython    conda-forge
python-dateutil           2.9.0.post0              pypi_0    pypi
pytz                      2024.2                   pypi_0    pypi
pyyaml                    6.0.2                    pypi_0    pypi
readline                  8.2                  h8228510_1    conda-forge
requests                  2.32.3                   pypi_0    pypi
scanpy                    1.9.8                    pypi_0    pypi
scikit-learn              1.3.2                    pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
seaborn                   0.13.2                   pypi_0    pypi
session-info              1.0.0                    pypi_0    pypi
setuptools                75.1.0             pyhd8ed1ab_0    conda-forge
six                       1.16.0                   pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
statsmodels               0.14.1                   pypi_0    pypi
stdlib-list               0.10.0                   pypi_0    pypi
tblib                     3.0.0                    pypi_0    pypi
threadpoolctl             3.5.0                    pypi_0    pypi
tk                        8.6.13          noxft_h4845f30_101    conda-forge
toolz                     0.12.1                   pypi_0    pypi
tornado                   6.4.1                    pypi_0    pypi
tqdm                      4.66.5                   pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
tzdata                    2024.2                   pypi_0    pypi
umap                      0.1.1                    pypi_0    pypi
umap-learn                0.5.6                    pypi_0    pypi
urllib3                   2.2.3                    pypi_0    pypi
wheel                     0.44.0             pyhd8ed1ab_0    conda-forge
xyzservices               2024.9.0                 pypi_0    pypi
xz                        5.2.6                h166bdaf_0    conda-forge
yarl                      1.13.1                   pypi_0    pypi
zict                      3.0.0                    pypi_0    pypi
zipp                      3.20.2                   pypi_0    pypi