Announcing Sr3 Beta

[petersilva]

Announcing Sr3

After two years of development, on 2022/04/11, we are pleased to announce the availability of the first beta version of Sarracenia version 3: Sr3. To celebrate the release (v3.00.16), there is a new web-site with in depth information:

https://metpx.github.io/sarracenia

Compared to v2, Sr3 brings:

• Native support for mqtt <https://www.mqtt.org>_ and amqp <https://www.amqp.org>_ (rabbitmq <https://www.rabbitmq.com>_ and MQTT brokers.) with a modular implementation that allows straightforward additional message queueing protocols <https://metpx.github.io/sarracenia/Reference/code.html#module-sarracenia.moth>_ to be supported.

• The Flow Algorithm <https://metpx.github.io/sarracenia/Explanation/Concepts.html#the-flow-algorithm>_ unifies
  all components into slight variations of this single common code. <https://metpx.github.io/sarracenia/Reference/code.html#module-sarracenia.flow>_ This re-factor has enabled the elimination of code duplication and allowed reduction of total lines of code by approximately 30% while adding features.

• A new command-line interface centred on a single entry-point: sr3 <https://metpx.github.io/sarracenia/Reference/sr3.1.html#sr3-sarracenia-cli>_

• Improved, jupyter Notebook-driven Tutorials <https://metpx.github.io/sarracenia/Tutorials/index.html>_

• A new plugin API <https://metpx.github.io/sarracenia/Reference/flowcb.html>_, which allows pythonic customization of default application processing.

• A new python API <https://metpx.github.io/sarracenia/Reference/code.html>_, which gives complete access to the implementation, allowing elegant extension through sub-classing.

• Applications can call Sarracenia Python API from their mainline.
  (In v2, one had to write callbacks to call application code, the application mainline could not be used.)

• Newly added GitHub Discussion module, for questions, and community-driven
  clarification: https://github.com/MetPX/sarracenia/discussions

[petersilva]

with sr3 3.0.20, the latest beta snapshot version, released yesterday, all CI/CD tests pass (we have a green dashboard)
which means shutdown and startup of both application itself and a message broker provokes no message loss in all measured configurations.

detailed of changes in this release:

• dynamic flow CI/CD, for late propagation, use a retry_queue instead of blithely proceeding ( 6012c79 )
• fixing poll behaviour in vip... not working quite right. #566, co-ordination among polls sharing a vip, so no missing files, no double posts.
• fixing poll behaviour in vip... not working quite right. #566, better nodupe choices for performance with the cod integrity checksum case.
• fixing fix restart_server test 6 failure (comparison of downloaded_by_sub_amqp and cfile directories. #555 and clean shutdown #369 clean shutdown assured by having start/stop ignore foreground tasks.
• fixes for postRateLimiting parsing, and invalid broker specifications.
• more documentation/web site information in French.

With this version, we will likely proceeding to internal deployments for validation purposes (to get closer to a stable release.)

compared to the version released in the spring:

• detection/fallbacks for python libraries to get partial functionality.
• Retranslation (translating sr3 Documentation into French) #539 French documentation for sr3 now exists.
• Review v03 integrity/checksums and actions. #547, Make integrity optional. #504 v03 post format revised to separate checksums from non-data file operations.
• If sender does not succeed in posting a message after sending a file, it should retry. #466 retry queue for failed send, also post retry queue.
• improved documentation language fixup: normalize ¨notification message¨ #552
• post/cpost/watch ... post since stopped? #522 honor permLog
• simulation mode. #465 simulation mode for sender (not yet complete.)
• v02 to v3 config translator #438 v2 to sr3 config converter added (still has some problems, but mostly working.)
• running sr3 sanity starts up configs it shouldn't be starting #513 improved shutdown on signal receipt. *

(It's available on pypi ( https://pypi.org/project/metpx-sr3/ ) as a wheel, or launchpad ( https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx ) repository as a debian package. Not producing a release on github so far... will wait until
a stable version is available. Still beta, but progessing well.

[petersilva]

obsolete by this point. See #768