ISAM2 segfaults when number of factors exceed ~50k

[kjcantrell9]

Description

Using Unary and Odometry measurements (Between Factor) with ISAM2 will fail once ~50k (not deterministic) factors are added to the graph.
Thread 1 "python3" received signal SIGSEGV, Segmentation fault. 0x00007ffff6dfc079 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal(std::function<std::pair<boost::shared_ptr<gtsam::GaussianConditional>, boost::shared_ptr<gtsam::GaussianFactor> > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4 (gdb) bt #0 0x00007ffff6dfc079 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal(std::function<std::pair<boost::shared_ptr<gtsam::GaussianConditional>, boost::shared_ptr<gtsam::GaussianFactor> > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4 #1 0x00007ffff6dfc278 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal(std::function<std::pair<boost::shared_ptr<gtsam::GaussianConditional>, boost::shared_ptr<gtsam::GaussianFactor> > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4 #2 0x00007ffff6dfc278 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal(std::function<std::pair<boost::shared_ptr<gtsam::GaussianConditional>, boost::shared_ptr<gtsam::GaussianFactor> > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4 #3 0x00007ffff6dfc278 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal(std::function<std::pair<boost::shared_ptr<gtsam::GaussianConditional>, boost::shared_ptr<gtsam::GaussianFactor> > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4 #4 0x00007ffff6dfc278 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal(std::function<std::pair<boost::shared_ptr<gtsam::GaussianConditional>, boost::shared_ptr<gtsam::GaussianFactor> > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4 #5 0x00007ffff6dfc278 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal(std::function<std::pair<boost::shared_ptr<gtsam::GaussianConditional>, boost::shared_ptr<gtsam::GaussianFactor> > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4 #6 0x00007ffff6dfc278 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal(std::function<std::pair<boost::shared_ptr<gtsam::GaussianConditional>, boost::shared_ptr<gtsam::GaussianFactor> > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () --Type <RET> for more, q to quit, c to continue without paging-- #7 0x00007ffff6dfc278 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal--Type ... #26163 0x00007ffff6dfc278 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal(std::function<std::pair<boost::shared_ptrgtsam::GaussianConditional, boost::shared_ptrgtsam::GaussianFactor > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4
#26164 0x00007ffff6dfc278 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal(std::function<std::pair<boost::shared_ptrgtsam::GaussianConditional, boost::shared_ptrgtsam::GaussianFactor > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4
#26165 0x00007ffff6dfc278 in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::separatorMarginal(std::function<std::pair<boost::shared_ptrgtsam::GaussianConditional, boost::shared_ptrgtsam::GaussianFactor > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4
#26166 0x00007ffff6dfcb1f in gtsam::BayesTreeCliqueBase<gtsam::ISAM2Clique, gtsam::GaussianFactorGraph>::marginal2(std::function<std::pair<boost::shared_ptrgtsam::GaussianConditional, boost::shared_ptrgtsam::GaussianFactor > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)>) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4
#26167 0x00007ffff6de62ef in gtsam::BayesTreegtsam::ISAM2Clique::marginalFactor(unsigned long, std::function<std::pair<boost::shared_ptrgtsam::GaussianConditional, boost::shared_ptrgtsam::GaussianFactor > (gtsam::GaussianFactorGraph const&, gtsam::Ordering const&)> const&) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4
#26168 0x00007ffff6de42f7 in gtsam::ISAM2::marginalCovariance(unsigned long) const () from gtsam-4.2/lib/libgtsamRelWithDebInfo.so.4 <--Type <RET> for more, q to quit, c to continue without paging--

Steps to reproduce

1. Setup 1 UnaryMeasure (i.e., gps) and add to the graph.
2. Setup ~50-100k Between Factors and add to the graph (these correspond to a 20Hz INS output).
3. Run ISAM2 update
4. Segfault

Expected behavior

Environment

Ubuntu 22.04, GTSAM 4.2 with TBB turned on AND off.

Additional information