Merge branch 'development' into development_py36

.github/workflows/flow.yml

@@ -40,7 +40,7 @@ jobs:
     timeout-minutes: 40
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Install dependencies
         run: |

.github/workflows/flow_amqp_consumer.yml

@@ -37,7 +37,7 @@ jobs:
     timeout-minutes: 40
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Install dependencies
       # 2023-11-13 RS Added SSH config changes to see if it makes the tests more reliable

.github/workflows/flow_basic.yml

@@ -34,7 +34,7 @@ jobs:
     timeout-minutes: 20
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Install dependencies
         run: |

.github/workflows/flow_mqtt.yml

@@ -37,7 +37,7 @@ jobs:
     timeout-minutes: 45
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Install dependencies
         run: |

.github/workflows/flow_redis.yml

@@ -6,6 +6,7 @@ on:
   push:
     branches:
       - v03_disabled
+      - '*redis*'
 
     paths-ignore:
       - '.github/**'
@@ -37,7 +38,7 @@ jobs:
     timeout-minutes: 40
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       # Enable tmate debugging of manually-triggered workflows if the input option was provided
       # https://github.com/marketplace/actions/debugging-with-tmate

.github/workflows/ghcr-base.yml

@@ -26,7 +26,7 @@ jobs:
 
     steps:
       - name: Checkout branch
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v1

.github/workflows/ghcr-dev.yml

@@ -27,7 +27,7 @@ jobs:
 
     steps:
       - name: Checkout branch
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v1

.github/workflows/ghcr.yml

@@ -25,7 +25,7 @@ jobs:
 
     steps:
       - name: Checkout branch
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v1

.github/workflows/publish-to-pypi.yml

@@ -12,7 +12,7 @@ jobs:
 
     steps:
       - name: Checkout source
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Set up Python
         uses: actions/setup-python@v3

debian/changelog

@@ -1,6 +1,20 @@
+metpx-sr3 (3.00.56rc2) unstable; urgency=medium
+
+  * fix #1261 http performance regression.
+  * fix #1270 invalid mirroring of relative symlinks when realpath is
+    symlinked.  (requires next metpx-sr3c version 3.24.10rc2 ?3?)
+  * fix #1267 queueName before subtopic from this point forward.
+  * fix #1262 mtime referenced before assignment (download check fail.)
+  * fix #1271 renamer for AM socket generalized.
+  * fix #1255 remove malformed useless error message.
+  * incrementing github action version because of plaintive supplications
+  * demoted false-alarm error message about missing subscriptions.json to debug.
+
+ -- SSC-5CD2310S60 <[email protected]>  Fri, 25 Oct 2024 14:47:54 -0400
+
 metpx-sr3 (3.00.56rc1) unstable; urgency=medium
 
-  * NEW option *recusion* option ( #1237 ) (actually restored, was removed.)
+  * NEW option *recursion* option ( #1237 ) (actually restored, was removed.)
   * fix #1156 - NEW option *logDuplicates* 
   * NEW EXPERIMENTAL option: *logJson*
   * fix #1223 Switch from global QoS to per-consumer.

docs/source/Contribution/Release.rst

@@ -19,7 +19,7 @@ To publish a pre-release one needs to:
   * git pull
   * git checkout development_py36
   * git pull
-  * git merge development
+  * git merge --strategy-option=theirs development
 
 - validate that the correct version of C stack will be running when running flow tests.
   on each server::
@@ -84,15 +84,15 @@ To publish a pre-release one needs to:
      * git pull 
      * git checkout pre_release_py36
      * git pull
-     * git merge development_py36
+     * git merge --strategy-option=theirs development_py36
      * git push
      * git push origin o3.xx.yyrcz
 
      * git checkout development
      * git tag -a v3.xx.yy.rcZ -m "pre-release v3.xx.yy.rcz"
      * git checkout pre_release
      * git pull
-     * git merge development
+     * git merge --strategy-option=theirs development
      * git push
      * git push origin v3.xx.yyrcz
 
@@ -172,7 +172,7 @@ the stable release does not require any explicit testing.
      git pull
      git checkout stable
      git pull
-     git merge pre-release
+     git merge --strategy-option=theirs pre-release
      git push
 
      # there will be conflicts here for debian/changelog and sarracenia/_version.py
@@ -191,7 +191,7 @@ the stable release does not require any explicit testing.
      git pull
      git checkout stable_py36
      git pull
-     git merge pre_release_py36
+     git merge --strategy-option=theirs pre_release_py36
      git push
      # same editing required as above.
      git tag -a o3.xx.yy -m "o3.xx.yy"
@@ -256,6 +256,26 @@ ubuntu 18 is not compatible with the current pypi.org.
   * attach redhat 9 rpm 
   * attach windows exe ... see: `Building a Windows Installer`_
 
+Post-Release 
+------------
+
+Sometimes there is *just one bug* that should really be addressed in a stable release.
+Usually build with:
+
+* choose the branch (either pre-release or stable or similar for v2)
+* one (or more) *git cherry-pick* emergency fixes from some other branch.
+* one individual commit for sarracenia/_version.py to change add the post1 branch,
+* one commit to debian/changelog to update that with information about post1 and the changes therein
+
+These changes to debian/changelog and sarracenia/_version that will cause conflicts with 
+future development.  Afterwards, one needs to cherry-pick the debian/changelog back into 
+development to have a record of it.  This will likely cause a conflict, but it's easy to
+resolve.
+
+That's why a lot of the merges have *--strategy-option=theirs* because these postX releases
+cause conflicts.  
+
+
 Details
 -------
 
@@ -376,7 +396,7 @@ A tag should be created to identify the end of the cycle::
 Once the tag is in the development branch, promote it to stable::
 
    git checkout pre-release
-   git merge development
+   git merge --strategy-option=theirs development
    git push
 
 Once stable is updated on github, the docker images will be automatically upgraded, but
@@ -527,6 +547,12 @@ Repositories:
   * metpx-sr3-old -- on demand build sr3 packages from *stable_py36* branch.
   * metpx-sarracenia-release -- on deman build v2 packages from *v2_stable* branch.
 
+* Post-Release ( version.post1, post2 etc...)
+  usually the result of a cherry-pick for an emergency fix, and a commit or two about
+  debian/changelog and sarracenia/_version that will cause conflicts with future development.
+  * metpx-sr3 is the repository to target (using *stable* (or *v2_stable*) branch)
+
+
 for more discussion see `Which Version is stable <https://github.com/MetPX/sarracenia/issues/139>`_
 
 

docs/source/Explanation/CommandLineGuide.rst

@@ -542,6 +542,7 @@ The second row of output gives detailed headings within each category:
 The configurations are listed on the left. For each configuraion, the state
 will be:
 
+* cpuS:  process is expensive in CPU usage (runStateThreshold_cpuSlow)
 * disa:  disabled, configured not to run.
 * hung:  processes appear hung, not writing anything to logs.
 * idle:  all processes running, but no data or message transfers for too long (runStateThreshold_idle)
@@ -552,6 +553,7 @@ will be:
 * run:   all processes are running (and transferring, and not behind, and not slow... normal state.)
 * slow:  transfering less than minimum bytes/second ( runStateThreshold_slow )
 * stop:  no processes are running. 
+* wVip:  process doesn't have the vip (only applies when the vip option is specified in the config)
 
 The next columns to the right give more information, detailing how many processes are Running, out of the number expected.
 For example, 3/3 means 3 processes or instances found of the 3 expected to be found.

docs/source/How2Guides/UPGRADING.rst

@@ -52,7 +52,11 @@ is perhaps complex. This version will read and write both files,
 so as to preserve ability to downgrade. later version will drop
 support for qname files.
 
-
+*CHANGE*: in configuration files: *subtopic* must come after
+the relevant queue naming options (queueName, queueShare)
+in prior releases, the queue naming was a global setting.
+In a future version, one will be able to subscribe to multiple
+queues with a single subscriber.
 
 3.0.54
 ------

docs/source/Reference/sr3_options.7.rst

@@ -1549,6 +1549,9 @@ Instances started on any node with access to the same shared file will use the
 same queue. Some may want use the *queueName* option as a more explicit method
 of sharing work across multiple nodes.
 
+This *subtopic* option should appear after the queueName setting in files
+for the topic bindings to apply to the given queue.
+
 
 queueShare <str> ( default: ${USER}_${HOSTNAME}_${RAND8} )
 ----------------------------------------------------------
@@ -1567,6 +1570,9 @@ will result in a random 8 digit number being appended to the queue name.
 All the instances within the configuration with access to the same state directory
 will use the queue name thus defined.
 
+This *subtopic* option should appear after the queueShare setting in files
+for the topic bindings to apply to the given queue.
+
 
 randomize <flag>
 ----------------
@@ -1973,8 +1979,8 @@ message flows.
 subtopic <amqp pattern> (default: #)
 ------------------------------------
 
-Within an exchange's postings, the subtopic setting narrows the product selection.
-To give a correct value to the subtopic,
+Within an exchange's postings, the subtopic setting narrows the product selection,
+for objects to place in the currently selected queue. To give a correct value to the subtopic,
 one has the choice of filtering using **subtopic** with only AMQP's limited wildcarding and
 length limited to 255 encoded bytes, or the more powerful regular expression
 based  **accept/reject**  mechanisms described below. The difference being that the
@@ -1983,18 +1989,18 @@ to the client at all. The  **accept/reject**  patterns apply to messages sent by
 broker to the subscriber. In other words,  **accept/reject**  are client side filters,
 whereas **subtopic** is server side filtering.
 
-It is best practice to use server side filtering to reduce the number of notification messages sent
-to the client to a small superset of what is relevant, and perform only a fine-tuning with the
-client side mechanisms, saving bandwidth and processing for all.
+Use server side filtering to reduce the number of notification messages sent
+to the client to a small superset of what is relevant, and refine further with the
+client side accept/reject, saving bandwidth and processing for all.
 
-topicPrefix is primarily of interest during protocol version transitions,
-where one wishes to specify a non-default protocol version of messages to
-subscribe to.
+Often, the user specifies one exchange, and several subtopic options.
+**Subtopic** is what is normally used to indicate messages of interest 
+for a given queue. If needed, queueName, and/or queueShare need to be
+earlier in the configuration file for the subtopic to apply to the selected queue.
 
-Usually, the user specifies one exchange, and several subtopic options.
-**Subtopic** is what is normally used to indicate messages of interest.
 To use the subtopic to filter the products, match the subtopic string with
-the relative path of the product.
+the relative path of the product (non Sarracenia pumps may have different
+topic hierarchy conventions.)
 
 For example, consuming from DD, to give a correct value to subtopic, one can
 browse the our website  **http://dd.weather.gc.ca** and write down all directories
@@ -2084,7 +2090,6 @@ Sarracenia has a convention for how topics for products should be organized. The
 a topicPrefix, followed by subtopics derived from the *relPath* field of the message.
 Some networks may choose to use different topic conventions, external to sarracenia.
 
-
 timeout <interval> (default: 0)
 -------------------------------
 
@@ -2134,6 +2139,14 @@ prepended to the sub-topic to form a complete topic hierarchy.
 This option applies to subscription bindings.
 Denotes the version of messages received in the sub-topics. (v03 refers to `<sr3_post.7.html>`_)
 
+topicPrefix is primarily of interest during protocol version transitions,
+where one wishes to specify a non-default protocol version of messages to
+subscribe to.
+
+For example, Sr3 expects v03 messages by default, but there are
+plenty of sources that offer the old version (requiring a topicPrefix of *v02.post*)
+to specify the old version of messages.
+
 users <flag> (default: false)
 -----------------------------
 

docs/source/fr/CommentFaire/MiseANiveau.rst

@@ -51,6 +51,10 @@ est peut-être complexe. Cette version lira et écrira les deux fichiers,
 afin de préserver la possibilité de rétrogradation. La version ultérieure abandonnera
 la prise en charge des fichiers qname.
 
+*CHANGEMENT* : dans les fichiers de configuration : *subtopic* doit venir après
+la spécification de nom de file d'attente pertinentes (queueName, queueShare.)
+Dans les versions précédentes, la dénomination de la file d'attente était un paramètre global.
+Dans une future version, on pourra s'abonner à plusieurs files d'attente avec un seul abonné.
 
 3.0.54
 ------

docs/source/fr/Reference/sr3_options.7.rst

@@ -1539,6 +1539,8 @@ Les instances démarrées sur n’importe quel nœud ayant accès au même fichi
 même fil d’attente. Certains voudront peut-être utiliser l’option *queueName* comme méthode plus explicite
 de partager le travail sur plusieurs nœuds. Il est pourtant recommandé d´utiliser queueShare a cette fin.
 
+l´option *subtopic* devrait apparaître après le paramètre queueName dans les fichiers
+pour que les liaisons de sujet s'appliquent à la file d'attente spécifié.
 
 
 queueShare <str> (default: ${USER}_${HOSTNAME}_${RAND8} )
@@ -1558,6 +1560,8 @@ Ce entraînera l'ajout d'un nombre aléatoire à 8 chiffres au nom de la file d'
 Toutes les instances de la configuration ayant accès au même répertoire d'état
 utilisera le nom de file d'attente ainsi défini.
 
+l´option *subtopic* devrait apparaître après le paramètre queueShare dans les fichiers
+pour que les liaisons de sujet s'appliquent à la file d'attente spécifié.
 
 randomize <flag>
 ----------------
@@ -1965,7 +1969,8 @@ origine. À utiliser uniquement avec des flux de données fiables et organisés
 subtopic <modèle  amqp> (défaut: #)
 -----------------------------------
 
-Dans les publications d’un échange, le paramètre de subtopic restreint la sélection du produit.
+Dans les publications d’un échange, le paramètre de subtopic sert à préciser des messages 
+à placer dans la file d'attente actuellement sélectionnée. 
 Pour donner la bonne valeur au subtopic, on a le choix de filtrer en utilisant **subtopic** seulement avec le
 wildcarding limité d’AMQP et une longueur limitée à 255 octets encodés, ou de manière plus puissante, les expressions régulière
 basés sur les mécanismes **accept/reject** décrits ci-dessous. La différence est que le
@@ -1978,12 +1983,17 @@ Il est recommandé d’utiliser le filtrage côté serveur pour réduire le nomb
 au client et envoyer seulement ce qui est pertinent, et seulement régler les mécanismes côté client,
 économisant du bandwidth et du traitement pour tous.
 
-topicPrefix est principalement utilisé lors des transitions de version de protocole,
-où l’on souhaite spécifier une version de protocole non-commune des messages d'annonce auquel s’abonner.
-
-Normalement, l’utilisateur spécifie un échange et plusieurs options de subtopic. **subtopic** est ce qui est
 normalement utilisé pour indiquer les messages d'annonce d'intérêt. Pour utiliser **subtopic** pour filtrer les produits,
+
+Souvent, l'utilisateur spécifie un échange et plusieurs options de sous-thèmes.
+Le **subtopic** est ce qui est normalement utilisé pour indiquer les messages d'intérêt
+pour une file d'attente donnée. Si nécessaire, **queueName** et/ou **queueShare** 
+doivent apparaître plus tôt dans le fichier de configuration pour que le sous-thème 
+s'applique à la file d'attente sélectionnée.
+
 il faut que la chaîne de caractère subtopic corresponde au chemin relatif du produit.
+(les pompes non Sarracenia peuvent avoir d´autres conventions de hiérarchie des sujets.)
+
 
 Par exemple, en consommant à partir de DD, pour donner la bonne valeur au subtopic, il est possible de
 parcourir le site Web **http://dd.weather.gc.ca** et noter tous les répertoires
@@ -2109,6 +2119,12 @@ rajouté au subtopic pour former une hiérarchie complète de thèmes (topics).
 Cette option s’applique aux liaisons d’abonnement.
 Indique la version des messages d'annonce reçus dans les subtopics. (V03 fait référence à `<sr3_post.7.html>`_)
 
+topicPrefix sert principalement lors des transitions de format de messages.
+Le topicPrefix identifie dans quel version de format les messages sous le thème
+sont créés. Sr3 s´attend a des messages v03 par défault, mais il y plein
+de sources qui offrent l´ancienne version (nécessitant une topicPrefix de *v02.post*) 
+pour spécifier l´ancienned version de messages. 
+
 topicCopy (défaut: False)
 -------------------------
 

sarracenia/_version.py

@@ -1 +1 @@
-__version__ = "3.00.56rc1"
+__version__ = "3.00.56rc2"