Support the new minhash 25.02 api

[praateekmahajan]

Description

Solves #426 and piggy backs on https://github.com/NVIDIA/NeMo-Curator/pull/442/files

Version	API	Speed / Implementation
24.10	minhash(a)	Slow (Old)
24.12	minhash_permuted(a,b)	Fast (new)
25.02	minhash(a,b)	Fast (new)

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[sarahyurick]

Wait I'm confused, why are we doing it for 24.10 and 24.12 instead of for 24.12 and 25.02?

[praateekmahajan]

Maybe the code block here helps

if MINHASH_DEPRECATED_API:
     # this is 24.10, the variable is named minhash deprecated api, 
     # because now cudf only needs four args, rather than two args
    return ser.str.minhash(seeds=seeds, width=char_ngram)
else:
    if MINHASH_PERMUTED_AVAILABLE: 
        # this is 24.12 because in this version the four arg function is called minhash_permuted
        return ser.str.minhash_permuted(
            a=seeds_a, b=seeds_b, seed=seeds[0][0], width=char_ngram
        )
    else:
        # this is the 25.02 case where it's neither the deprecated case neither permuted is available
        return ser.str.minhash(a=seeds_a, b=seeds_b, seed=seeds[0][0], width=char_ngram)

[sarahyurick]

Ah ok I see, thanks! So are we waiting until the next NeMo Curator release (which will have 24.12 as the stable RAPIDS version) before removing MINHASH_DEPRECATED_API?

[praateekmahajan]

Yes that's correct! And 25.02+ we wouldn't need any of these variables

[sarahyurick]

LGTM, thanks! I think you just need to update this branch, then feel free to merge.