Unable to install

[philm001]

I tried running the command

pip install nemo-curator and this is the output that I recieved.

Defaulting to user installation because normal site-packages is not writeable
Collecting nemo-curator
  Using cached nemo_curator-0.6.0-py3-none-any.whl.metadata (16 kB)
Collecting awscli>=1.22.55 (from nemo-curator)
  Using cached awscli-1.37.10-py3-none-any.whl.metadata (11 kB)
Collecting beautifulsoup4 (from nemo-curator)
  Using cached beautifulsoup4-4.12.3-py3-none-any.whl.metadata (3.8 kB)
Collecting charset_normalizer>=3.1.0 (from nemo-curator)
  Using cached charset_normalizer-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (35 kB)
Collecting comment_parser (from nemo-curator)
  Using cached comment_parser-1.2.4.tar.gz (8.3 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [60 lines of output]
      /path/to/user/.local/lib/python3.10/site-packages/setuptools/dist.py:493: SetuptoolsDeprecationWarning: Invalid dash-separated options
      !!
      
              ********************************************************************************
              Usage of dash-separated 'description-file' will not be supported in future
              versions. Please use the underscore name 'description_file' instead.
      
              By 2025-Mar-03, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
      
      !!
        opt = self.warn_dash_deprecation(opt, section)
      /path/to/user/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py:270: UserWarning: Unknown distribution option: 'test_suite'
        warnings.warn(msg)
      /path/to/user/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py:270: UserWarning: Unknown distribution option: 'tests_require'
        warnings.warn(msg)
      running egg_info
      creating /tmp/pip-pip-egg-info-yi5dl0ec/comment_parser.egg-info
      writing /tmp/pip-pip-egg-info-yi5dl0ec/comment_parser.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-yi5dl0ec/comment_parser.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-pip-egg-info-yi5dl0ec/comment_parser.egg-info/requires.txt
      writing top-level names to /tmp/pip-pip-egg-info-yi5dl0ec/comment_parser.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-yi5dl0ec/comment_parser.egg-info/SOURCES.txt'
      reading manifest file '/tmp/pip-pip-egg-info-yi5dl0ec/comment_parser.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 983, in run_commands
          self.run_command(cmd)
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/dist.py", line 999, in run_command
          super().run_command(command)
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
          cmd_obj.run()
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 312, in run
          self.find_sources()
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 320, in find_sources
          mm.run()
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 548, in run
          self.prune_file_list()
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/command/sdist.py", line 162, in prune_file_list
          super().prune_file_list()
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 380, in prune_file_list
          base_dir = self.distribution.get_fullname()
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/_core_metadata.py", line 272, in get_fullname
          return _distribution_fullname(self.get_name(), self.get_version())
        File "/path/to/user/.local/lib/python3.10/site-packages/setuptools/_core_metadata.py", line 290, in _distribution_fullname
          canonicalize_version(version, strip_trailing_zero=False),
      TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[philm001]

I feel like that this should have been caught in testing

[ryantwolf]

Hello! Could you please provide some more details on what your environment setup is? The following would be helpful:

• Python version
• OS version
• GCC version
• setuptools version

[philm001]

Hello Ryan, thank you for the feedback.

Python 3.10.12
OS - Ubuntu 22.04
GCC - 11.4.0
setuptools version - I am not sure how to get this one.

[philm001]

Just for the record, the cluster that I am running Curator under is a CPU only cluster

[ryantwolf]

This command should get you the setuptools version:

python -c "import setuptools; print(setuptools.__version__)"

[philm001]

Thank you. I get back v75.8.0

[ryantwolf]

Interesting, I was able to install fine in a docker image with that environment. I'm curious if this is related to this issue: pypa/setuptools#4483. Could you paste the version of packaging you are using?

python -c "import packaging; print(packaging.__version__)"

And if it's lower than 22.0 can you try manually upgrading? I'm using 24.2 and it's fine.

[philm001]

v 21.3. Pardon my ignorance but do you have the command on hand to update the tool?

[ryantwolf]

No worries at all, try this

pip install packaging==24.2

[philm001]

Great! Thank you. It worked.

Would it be possible for me to edit the readme to include this step as a sanity check to ensure noone else has to go through this like I did?

[ryantwolf]

Sure, go ahead and add it as a sub-bullet under the requirements like so:

• Python 3.10 or higher
  • packaging>=22.0
• Ubuntu 22.04/20.04
• NVIDIA GPU (optional)
  • Volta™ or higher (compute capability 7.0+)
  • CUDA 12 (or above)

Also please update docs/user-guide/image/gettingstarted.rst in a similar way too:

* Python 3.10 or higher
  * packaging>=22.0 
* Ubuntu 22.04/20.04
* NVIDIA GPU
  * Volta™ or higher (compute capability 7.0+)
  * CUDA 12 (or above)

I would suggest that a better way would be to add it to our dependencies, but I can't seem to reproduce the failure myself in a Dockerfile so I'm not sure if adding it as a dependency would work. A documentation update is the best way to go now. Make sure you follow our contributing guidlines. In particular, make sure you both sign and sign-off your commits by doing git commit -sS. Thank you!