sr3 v3.0.44 (Gamma?)

[petersilva]

Sarracenia version 3 (metpx-sr3) is the best version of sarracenia to use, going forward.
For anyone new to sr3, it is at the point that v2 is a waste of time. It is better to start with sr3.
v2 will remain in maintenance mode for years (as it has been for the past three years.)

Documentation: https://metpx.github.io/sarracenia

While there has been considerable evolution during this Beta period, it is expected that things
will substantially stabilize going forward, as all the changes were caused by considering all existing use cases. Until now, we were avoiding cluttering up github with interim sr3 releases. Going forward,
Releases will include Github. We'll call this a Gamma release. More Stable than Beta...

For this particular release:

What's Changed (from 3.0.43)

• Flowcb accept fixes by @gcglinton in Flowcb accept fixes #756
• Unit tests for all Flowcb/Accept plugins by @gcglinton in Unit tests for all Flowcb/Accept plugins #757
• fix Cannot Run Sarracenia Due to Missing Magic, Part 2 #753, use try import rather than importlib to test module presence. by @petersilva in fix #753, use try import rather than importlib to test module presence. #755
• V03 issue752 2 flowcb/scheduled was not working with the scheduled_hours setting. re-written to address that and other issues. by @petersilva in V03 issue752 2 flowcb/scheduled was not working with the *scheduled_hours* setting. re-written to address that and other issues. #754
• 758 adding wiskis query support by @petersilva in 758 adding wiskis query support #759
• Add timeout to basic_publish and improve logging Message posting retry for tx_commit() hang #732 by @reidsunderland in Add timeout to basic_publish and improve logging #732 #761
• match new PR for python library. https://github.com/eclipse/paho.mqtt… by @petersilva in match new PR for python library. https://github.com/eclipse/paho.mqtt… #763
• 625 re enable reporting left aside for sr3 by @petersilva in 625 re enable reporting left aside for sr3 #762

Full Changelog: v3.00.43...v3.00.44

Installation

Guide: https://metpx.github.io/sarracenia/Tutorials/Install.html

packages available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
python: https://pypi.org/project/metpx-sr3/

---

This discussion was created from the release Gamma Release of sr3.

[petersilva]

Gamma? Release of sr3

Sarracenia version 3 is a deep re-factoring that changes the way the entire code is built, the internal algorithms, all the internal APIs. Work started in 2019, and is nearing the finish line now. It's been around 950 commits and 28 interim releases since the beta announcement last year and things are firming up nicely:

v2 to sr3 introduction: #507

Since that time only interim releases were done (reserving formal releases on github for the v2 stable version.) Sr3 is edging towards being the only evolving version. We are still mostly on v2 internally, but since the beta announcement, we have ported over 200 plugins and reviewed about 600 configurations for transfer to sr3 and some deployments on are sr3. There is also a lot more documentation for sr3 than for v2 nowadays.

While the Beta announcement was supposed indicate that we were just stabilising for release,
the work in the interim proved that we were still missing a number of important features needed
for sr3 to replace v2 (and hopefully, Sundew as well) in our
deployments.

Functionality added since the Beta announcement:

• substantial progress on fault tolerance/ error recovery, now superior to v2 when faced with every known failure.
• post_format ... support for alternate message encoding became necessary as our path diverged from the WMO pub/sub work. It would likely have been necessary for ICAO/AMHS work anyways.
• flows ... a generic component that just implements the main flow algorithm without subclassing.
• AM protocol support (ECCC Internal AlphaManager bulletin transfer over sockets.) there is an flow/amserver component for serving AM connections, as well as a sender.
• flowcb/scheduled... The normal "poll" component connects to a pollURL, lists the announced resources, and generates a new list of resources available for download. While porting plugins it turned out that some polls are just specialized gets done on a schedule,... ther is no "listing" phase, just a "get" of pre-determined URL's done on a schedule. That pattern is much easier/simpler to work with as it's own thing, rather than subclassing poll. They are now called "scheduled flows"
• Optional Redis backend for duplication suppression and retry queues. (part of cloud enablement.)
• miscellaneous cloud enablement: logStdOut, s
• Poll Re-factored during porting it became evident that polls were not completely working in sr3. They now work, and have been tested by porting dozens of polling plugins.
• metrics sr3 status gives much more status messages about rates of message transfer etc... future work may involve reporting metrics to external tools that can consume such things.
• features sr3 features give better feedback about missing dependencies and has a framework for plugins to express their dependencies as well.
• documentation entirely translated to French.
• simulation (--dry_run) mode added to aid debugging when porting flows from v2.
• sr3 convert configuration conversion added.
• changed defaults so that they are no longer component dependent (more consistent)
  • acceptUnmatched now always true by default.
  • Windows binary generation is working again.
• migration from v2 to sr3 is complex, so it is best to get people on sr3 ASAP, and not have any future deployments with v2.
• For anyone new to sr3, it is at the point that v2 is a waste of time. It is better to start with sr3.
• v2 will remain in maintenance mode for years (as it has been for the past three years.)

While there has been considerable evolution during this Beta period, it is expected that things
will substantially stabilize going forward, as all the changes were caused by considering all existing use cases. There are no more existing use cases to investigate... although actually commissioning of those use cases is mostly still to come.

It should be bug-fixing from now on, as we head towards a stable release.

~