How Do I Run STUMPY on Coiled Infrastructure

[seanlaw]

To run STUMPY on Coiled's infrastructure, let's start with a fresh local miniconda installation, update the Python version, and then install all of the minimum STUMPY requirements:

bash ./Miniconda3-latest-MacOSX-x86_64.sh 
conda install -c conda-forge python=3.7.8
pip install numba numpy scipy stumpy

Next, we install coiled and set up our token

pip install coiled
coiled login --token <insert_your_token_here>

Finally, now that everything is set up, we can use the following stumpy-coiled.py Python script to run STUMPY on Coiled:

import coiled
import numpy as np
import stumpy
from dask.distributed import Client

coiled.create_software_environment(
    name="stumpy-software-env",
    conda={"channels": ["conda-forge"],
           "dependencies": ["python=3.7.8",
                            "dask",
                            "distributed"]},
    pip=["numba", "numpy", "scipy", "stumpy"]
 )

coiled.create_cluster_configuration(
    name="stumpy-cluster-config",
    software="stumpy-software-env",
)

cluster = coiled.Cluster(n_workers=10, configuration="stumpy-cluster-config")

client = Client(cluster)

T = np.random.rand(1000)
m = 50

mp = stumpy.stumped(client, T, m)

# coiled.delete_cluster(name="stumpy-cluster-config")
# coiled.delete_software_environment(name="stumpy-software-env")

print(mp)

If successful, depending on your random seed, this should print out something like:

[[7.487199436853299 179 -1 179]
 [7.6694344979315625 180 -1 180]
 [7.59570606433141 497 -1 497]
 ...
 [6.90167799200012 660 660 -1]
 [6.907953945280755 661 661 -1]
 [7.17327595895547 662 662 -1]]

[nsankar]

Cool !

[seanlaw]

@nsankar Would you mind reporting back and let us know if this solved your problem?

[nsankar]

@seanlaw It works locally (Windows - Python 3.7.7 local environ)
However , It didnt work from Colab for the same code as above. (colab is 3.6.x python ) I got numba serialize missing error. Any idea on this ?

[seanlaw]

@nsankar, as suggested by the error message, it is a Dask requirement that the client environment must match the cluster environment. This means that the python version as well as all of the installed dependencies (i.e., dask, distributed, numba, numpy, scipy, stumpy) must all be the same. A serialization error typically points to the python version being different in the client (i.e., colab) and cluster (i.e., coiled) so one needs to take care and ensure that the python version (and other dependencies) is the same on both the client and the cluster. So, the following combination of stumpy and coiled should work on Google Colab:

!pip install --upgrade coiled
!pip install --upgrade tornado
!pip install --upgrade tbb>=2019.5
!pip install --upgrade numpy scipy numba stumpy
!coiled login --token <insert_your_token_here>

With these local package updates, you'll want/need to restart the runtime for the Colab notebook. Once that is done, then, in another Colab cell, you can execute:

import coiled
import numpy as np
import stumpy
from dask.distributed import Client
import sys

version = sys.version_info
colab_python_version = f"{version.major}.{version.minor}.{version.micro}"

coiled.create_software_environment(
    name="stumpy-software-env",
    conda={"channels": ["conda-forge"],
           "dependencies": [f"python={colab_python_version}",  # This needs to match the Colab Python version!
                            "dask",
                            "distributed",
                            "tbb>=2019.5",
                            ]},
    pip=["numba", "numpy", "scipy", "stumpy"]
 )

coiled.create_cluster_configuration(
    name="stumpy-cluster-config",
    software="stumpy-software-env",
)

cluster = coiled.Cluster(n_workers=10, configuration="stumpy-cluster-config")

client = Client(cluster)

T = np.random.rand(1000)
m = 50

mp = stumpy.stumped(client, T, m)

# coiled.delete_cluster(name="stumpy-cluster-config")
# coiled.delete_software_environment(name="stumpy-software-env")

print(mp)

[nsankar]

Thanks @seanlaw Earlier I had set the colab and the target coiled python versions to be the same which is 3.6.9 .and with this only, I got the error. However, I will look in to your instructions on this , try it again and revert.

[seanlaw]

Earlier I had set the colab and the target coiled python versions to be the same which is 3.6.9 .and with this only, I got the error. However, I will look in to your instructions on this , try it again and revert.

Yeah, there are a few other dependency issues but this worked for me on Colab so please report back if it does solve your issue.

[nsankar]

@seanlaw It works now. Thanks for your time and guidance on this.

I have a question. What is the real world data scenario for mstump ? Please give me a few examples of a real world multi-dimensional time series data and to some public datasets . Also in this regard, for instance, let us say there is a multivariate time series data produced by a monitoring system like prometheus which is measuring the cpu load, memory, network ingress, egress , disk IO etc from an Application Server with a time series sampling frequency of 1 sample per minute and this data may contain anomolous events. Would we consider this type of time series data a suitable candidate for mstump analysis ? Thanks.

[seanlaw]

What is the real world data scenario for mstump ? Please give me a few examples of a real world multi-dimensional time series data and to some public datasets .

@nsankar If you haven't done so already, I strongly recommend reading the original mSTOMP paper (mstump faithfully reproduces this work) as it is your best bet for learning more about the applications. Note that a multi-dimensional matrix profile is different from taking individual 1-D matrix profiles and stacking them together so one needs to fully understand this and it is covered in detail in the paper above.

Also in this regard, for instance, let us say there is a multivariate time series data produced by a monitoring system like prometheus which is measuring the cpu load, memory, network ingress, egress , disk IO etc from an Application Server with a time series sampling frequency of 1 sample per minute and this data may contain anomolous events. Would we consider this type of time series data a suitable candidate for mstump analysis ?

Yes, this would be the ideal candidate for mstump. Note that multi-dimensional matrix profile analysis can be quite expensive to compute and finding anomalies is even harder (in general) and is still an area of research. So you'll want to explore cautiously as there may not be a ton of information out there. Additionally, you may consider reaching out to the original researchers for more information.

[nsankar]

@seanlaw Got it. Thank you again.

[seanlaw]

@nsankar Closing this for now as it seems to be resolved. Please feel free to re-open or start a new issue if you have any further questions.