Harmonise response categories #58

[tongyu0924]

Description

This update addresses issue #58 by introducing functionality to calculate and integrate response options similarity into the clustering process. A method to compute semantic similarity between response options has been added, using SentenceTransformer (paraphrase-MiniLM-L6-v2), while ignoring ordering differences. The final similarity score combines question text similarity and response options similarity, with adjustable weights for flexibility.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

Testing

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• Test A
• Test B

Since the Harmony Python package is used by the Harmony API (which is itself used by the R library and the web app), we need to avoid making any changes that break the Harmony API. Please also run the Harmony API unit tests and check that the API still runs with your changes to the Python package: https://github.com/harmonydata/harmonyapi

Test Configuration

• Library version: 1.0.3
• OS: Windows 11 (64-bit)
• Toolchain: Python 3.10.6

Checklist

• My code follows the style guidelines of this project. I have applied a Linter (recommended: Pycharm's code formatter) to make my whitespace consistent with the rest of the project.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings
• The Harmony API is not broken by my change to the Harmony Python library
• I add third party dependencies only when necessary. If I changed the requirements, it changes in requirements.txt, pyproject.toml and also in the requirements.txt in the API repo
• If I introduced a new feature, I documented it (e.g. making a script example in the script examples repository so that people will know how to use it.

Optionally: feel free to paste your Discord username in this format: discordapp.com/users/yourID in your pull request description, then we can know to tag you in the Harmony Discord server when we announce the PR.