[Question] Unable to create index using pyspark

[UlrikChristensenGit]

When trying to create an index on a simple parquet file, i get the error below. Hyperspace-log gets created, but status of index remains "CREATING".

Error message:

Py4JJavaError: An error occurred while calling o44.createIndex. : java.lang.NoSuchMethodError: org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(Lorg/apache/spark/sql/SparkSession;Lorg/apache/spark/sql/execution/QueryExecution;Lscala/Function0;)Ljava/lang/Object; at com.microsoft.hyperspace.index.DataFrameWriterExtensions$Bucketizer.runCommand(DataFrameWriterExtensions.scala:78) at com.microsoft.hyperspace.index.DataFrameWriterExtensions$Bucketizer.saveWithBuckets(DataFrameWriterExtensions.scala:65) at com.microsoft.hyperspace.actions.CreateActionBase.write(CreateActionBase.scala:151) at com.microsoft.hyperspace.actions.CreateAction.op(CreateAction.scala:75) at com.microsoft.hyperspace.actions.Action.run(Action.scala:92) at com.microsoft.hyperspace.actions.Action.run$(Action.scala:83) at com.microsoft.hyperspace.actions.CreateAction.run(CreateAction.scala:30) at com.microsoft.hyperspace.index.IndexCollectionManager.create(IndexCollectionManager.scala:43) at com.microsoft.hyperspace.index.CachingIndexCollectionManager.create(CachingIndexCollectionManager.scala:77) at com.microsoft.hyperspace.Hyperspace.createIndex(Hyperspace.scala:42) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Unknown Source)

Code producing error:

from pyspark.sql import SparkSession
from pyspark.sql.types import IntegerType

"""setup spark session"""
spark = SparkSession
.builder
.appName("test")
.master("local[*]")
.config("spark.driver.memory", "4g")
.config("spark.executor.memory", "2g")
.config("spark.jars.packages", "com.microsoft.hyperspace:hyperspace-core_2.12:0.3.0")
.getOrCreate()

from hyperspace import *
hyperspace = Hyperspace(spark)

"""create test set"""
df = spark.createDataFrame([1,2,3,4,5], IntegerType())
df.write.parquet("test")

"""re-load test set"""
df = spark.read.parquet("test")
idx_conf = IndexConfig("idx", ["value"], [])
hyperspace.createIndex(df, idx_conf)

Setup:

• pyspark v 3.0.1
• python v 3.7.8
• hyperspace-core_2.12_0.3.0.jar downloaded manually

[imback82]

Can you try passing --packages com.microsoft.hyperspace:hyperspace-core_2.12:0.3.0 when you run spark-submit instead of setting it at the spark session creation time? Also, I see that you edited the error message? (The current error message seems a bit weird).

[UlrikChristensenGit]

I tried passing it with spark-submit as well, but no luck (same error message).

About the edit, I copy-pasted from the wrong vscode window at first, so that was a typo.

[imback82]

Oh looks like you are using Spark 3.0.1.

Hyperspace currently supports only Spark 2.4 and support for Spark 3.0 is on the way: #85.