Importing terms from .csv containing foreign Japanese characters results in gibrish

[marleyjaffe]

We have a csv containing foreign characters as part of the search terms (in this case they are Japanese characters/terms). After loading the terms using the csv import function, the Japanese characters are converted into gibrish characters. I've attached a screenshot of the issue below, as well as included a screenshot of the csv loaded into Notepad++ to show the file encoding settings (UTF-8-BOM & CR LF). We have tested and identified this issue in SearchModule v3.14 through v3.18

[JuicyDragon]

What version of Nuix are you using when this happens? Are you starting Nuix with the following argument?

-Dfile.encoding=UTF-8

[JuicyDragon]

Hello @marleyjaffe I went ahead and modified the CSV import code to enforce UTF-8 encoding. Looks like before it was relying on system encoding which on a windows machine is likely CP1252 (AKA Window-1252).

3f56040

Link to download updated release below. Please let me know if this resolves the issue for you and I'll make it a full fledged release. Thanks!

SearchModule_v3.19.0-preview-20220307.zip