Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NCCL_GRAPH The INFO of the logo cannot be printed to the console or to a specified file #1556

Open
lmhahatest opened this issue Dec 26, 2024 · 1 comment

Comments

@lmhahatest
Copy link

Please, which NCCL_DEBUG setting needs to be configured in order for the information identified by NCCL_GRAPH to be printed to the console or a specified file?
eg:
INFO(NCCL_GRAPH, "init.cc 1 Ring 2-1 LM test %02d : %d -> %d -> %d", c, comm->channels[c].ring.prev, comm->rank, comm->channels[c].ring.next); can't work when run "mpirun -np 4 -x NCCL_ALGO=ring -x NCCL_DEBUG=TRACE all_reduce_perf -b 8G -e 8G -f 2 -g 1 2>&1 |tee output_file.txt"

@marksantesson
Copy link

Hi lmhahatest, In general, to see any INFO log you need to have "NCCL_DEBUG=INFO" or more (i.e., ABORT or TRACE will also work), and then also indicate "NCCL_DEBUG_SUBSYS=GRAPH" to get those settings. If you do not set NCCL_DEBUG_SUBSYS explicitly then you will get the default, which in v2.23 is INIT, BOOTSTRAP, and ENV as per the default for ncclDebugMask in src/debug.cc.

Does that answer your question?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants