Skip to content

Meeting minutes archive

Jump to bottom Edit New page
Sasha Ames edited this page Jul 26, 2018 · 2 revisions

Old minutes from 2016 and earlier

29 September 2016

Luca, Lukasz, Sasha, William, Zed

  • Time to advise sites to put manual updates into production
  • Get information about how to cut releases available; check with IPSL on their progress with Nicolas notes
  • Docker discussion
    • Luca is working on Docker demo
    • Question of how OS security updates are applied to Docker containers: do they automatically update or will node admins need to update containers frequently for patches? Some updates need to be applied in a day's notice
    • Plan for a discussion at the F2F based on progress by then
  • Zed will re-post his notes on plans for the next-gen installer

7 July 2016

Prashanth, Sasha, Katharina, Nicolas

  • 2.4 (Next release)
    • Proposed date for code freeze: friday july 15
    • Proposed date for first RC cut in devel: monday july 18
    • Will include
      • CoG
      • publisher
      • Globus feature to make registration optional
      • New TDS library for search
      • backfill_logs cron job setup in the installer
  • Tier 1 and 2 nodes
    • NCI node is up but still trying to get the publication process working
  • New node manager
    • has been tested across PCMDI and IPSL test nodes

23 June 2016

Dean, Lukasz, Prashanth, Sasha, Alan, Nicolas

  • 2.4 (Next release)
    • Nicolas to be release manager
    • Release to be cut beginning/mid july
    • Will include
      • CoG
      • publisher
      • Globus feature to make registration optional
      • New TDS library for search
      • backfill_logs cron job setup in the installer?
  • Tier 1 and 2 nodes
    • NCI node status?
  • Test fed
    • IPSL test fed is now reachable from the internet. It is always prod like, 2.3.8 right now.

09 June 2016

Luca, Lukasz, Prashanth, Sasha, Alan, Nicolas

  • 2.3.8 (Current master release)
    • Roland and Luca identified the issue with LAS. Rebuild the link to /esg/content/las in /usr/local/tomcat/content/
  • 2.4 (Next release)
    • Nicolas to be release manager
    • Release to be cut beginning/mid july
    • Will include CoG, publisher, Globus feature to make registration optional, New TDS library for search, ... and more?
  • Solr6
    • Luca mentioned the need to upgrade to Solr6. Solr5 index nodes will not be forward compatible.
  • Tier 2 Nodes
    • Prashanth to finalize the tier 2 nodes setup document
  • Test fed
    • Need to have a worldwide test fed on the internet
    • IPSL has one. The firewall will be open in a few days

15 Apr 2016

Luca, Lukasz, Nicolas, Prashanth, Sasha, Katharina

  • 2.2.4
    • Will be cut by Prashanth - End of April
    • Will include new publisher, CoG, AccessLoggingFilter, esgf-security and installer apache conf upgrade
  • Tier 1 nodes status
    • Every node is up and running except NCI which is still testing publication and configuring search
  • Tier 2 nodes installation
    • Prashanth to create a wiki or confluence page to lists tier 2 nodes back to operations checklist (Need to move UNICAN node to this new page)
    • Once ready, Luca to send an email to esgf-devel to ask admins to chose an existing index and idp peer and to install a data/compute only node.

17 Sep 2015

Dean, Luca, Lukasz, Nicolas, Prashanth, Sasha, Katharina, Alan, Sandro

  • RC4.4 out
    • Tomcat 8/Thredds4.3 OK
    • myproxy-logon issue. Prashanth to investigate.
    • wget downloads have "trust anchors" issue. Someone to investigate?
    • Katharina and Sasha to test a data only node installation.
  • RC4.5 to include:
    • AuthorizationTokenFilter removal (Nicolas)
    • TDS5 integration (Luca to send installation information to Nicolas)
    • LAS 8.4 ? (Roland to send installation information to Nicolas and Prashanth)
    • Dashboard/Desktop (Sandro to send installation information to Prashanth )

RC4.5 to be cut by the end of next week and sent to Goddard

-Next meeting: 24 September 2015, 0800 hrs PDT

11 Jun 2015

Luca, Lukasz, Nicolas, Prashanth, Sasha, Kai

  • Globus RPMS: RPMs for both C5 and C6 now ready. They will be served from the Globus RPM repository. Lukasz to work with Nicolas to modify installer.
  • Apache-Frontend: Lighter version of the esgf-ca-bundle tested by Luca; it works!
  • UVCDAT and C5: Since UVCDAT is still not compatible with C5/RHEL5, we've decided to roll out release 1.9.0 with the older version of UVCDAT, with a hack to fix the sslv3 issue with the publisher.
  • 1.9.x schedule: 1.9.0 devel is targeted for release by Wednesday, 17 June. A point release of 1.9.1 is set to closely follow 1.9.0, as soon as a C5 compatible UVCDAT is available.
  • Standalone publisher: Sasha to explore the feasibility of decoupling the Publisher from UVCDAT, for future releases.
  • IDP Peering problem: Since IDP peering in IDP nodes is completely broken, choice of IDP peer for index nodes will be taken out.
  • Next meeting: 25 June 2015

28 May 2015

Dean, Katharina, Luca, Lukasz, Nicolas, Prashanth, Georgi, Sasha

  • Globus RPMs: ought to be ready in two weeks -Lukasz
  • Apache-Frontend: Luca to send recipe for CoG; Prashanth to create slimmer esgf-ca-bundle.crt for Luca to test.
  • 1.9 release: dependent on Globus RPMs and availability of slimmer esgf-ca-bundle that works for Luca.
  • Next meeting: 11 June 2015

02 April 2015

Georgi, Katharina, Luca, Lukasz, Prashanth

  • Apache-Frontend: Package with test install script has been put up on github.
  • Apache-Frontend: Tested by Katharina successfully on Centos6/RHEL6. Failure to install dependencies on Centos5/RHEL5.
  • Apache-Frontend: Tested successfully on Centos6 by Prashanth. Also, solr has been reverse-proxied, to be available on both 80 and 8983 ports. Can be tested here: http://esg-test.nsc.liu.se/solr/datasets/select/ and
    http://esg-test.nsc.liu.se:8983/solr/datasets/select/
  • Apache-Frontend: More testing expected next week by IWT members.
  • 1.9: rpm installation of Apache,mod_wsgi, mod_ssl is to be initiated by esgf-installer, to ensure it uses ESGF python which is needed by CoG.
  • 1.9: Once compatibility with Centos5/RHEL5 is established, cut release in devel, to enable installation testing.
  • 1.9: CoG to be served by Apache
  • 1.9: Solr will be reverse-proxied by Apache, to be served on both http and 8983 ports, to provide smooth transition to tomcat application.
  • 1.9: Luca to provide Apache configuration for CoG deployment, to be included with esgf-httpd configuration.
  • Lukasz to look into ANL node myproxy certificate issue.
  • To be discussed in the next meeting
    • Globus RPM packaging status, and compatibility with Centos5 (Lukasz)
    • Schedule for release of 1.9 in devel
  • Next meeting: 16 April 2015, 0800 hrs PDT

19 March 2015

Dean, Luca, Nicolas, Katharina, Georgi, Sasha, Prashanth, Lukasz

  • 1.8 - Postgresql 8.4.7 used by ESGF is no longer being supported. ESGF stack pgsql has to be upgraded and tested (Nicolas)
  • 1.8 - New OpenSSL patches pointed out (Georgi)
  • 1.9 - Globus RPMs packaged and tested on CentOS6, need to test on CentOS5 (Lukasz)
  • 1.9 - esg-globus subscript need to be modified (Lukasz)
  • 1.9 - Globus RPMs will use openssl from the system. 1.9 would be a good opportunity to upgrade openssl in ESGF
  • 1.9 - Testing Globus Connect CA - Need to update the truststore (Lukasz)
  • 1.9 - Apache in front of ESGF working in test env (Prashanth)
  • 1.9 - Release candidate has to be cut to test @JPL (Prashanth)

-Next meeting: 2 April 2015, 0800 hrs PDT

05 March 2015

Dean, Luca, Nicolas, Katharina, Georgi, Matthew, Sasha, Prashanth, Lukasz

  • 1.8 - Release has been cut in master (Nicolas)

  • 1.8 - CEDA is preparing a hotfix for the wget template: forcing TLSv1 (Kleanthis)

  • 1.8 - Hotfix will include CDAT repo url fix: https instead of http git clone (Nicolas)

  • 1.8 - Hotfix will include LAS archives download from esgf dist mirrors instead of FTP at NOAA (Nicolas)

  • 1.8 - Issue with CoG login on nodes which have disabled SSLv3 reported by Katharina

  • 1.8 - Issue with latest getcert.jar from Karem. Rollback fixed ht issue. (Lukasz to continue devs on getcert?)

  • 1.8 - Issue with wget compiled against GnuTLS instead of openssl (Prashanth to update the wiki)

  • 1.9 - Prashanth to work on Apache in front of tomcat for 1.9

  • 1.9 - Lukasz to work on rpmize auth callout for 1.9

-Next meeting: 19 March 2015, 0800 hrs PDT

23 January 2015

Luca, Nicolas, Katharina, Georgi

  • 1.8 - Release is still in progress and should be cut soon (Prashanth)

  • 1.8 - It has been decided that CoG integration should be part of 1.8 instead of 1.9

  • 1.8 - Zed to write a snippet for CoG and other component subscripts

  • 1.9 - LAS 8.3 sent by Roland has been tested and fixed. Subscript has to be adapted so it can be executed independently (Nicolas)

-Next meeting: 5 Ferbuary 2015, 0800 hrs PDT

9 January 2015

Dean, Luca, Nicolas, Prashanth, Eric, Zed, Katharina, Stephan, Roland

  • 1.8 - Release is still in progress and should be cut soon (Prashanth)
  • 1.9 - CoG has been installed and tested on a test index node (Nicolas)
  • 1.9 - CoG install is not fully operational and has to be debugged (Luca and Nicolas)
  • 1.9 - Zed wrote a subscript template which implements new standard flags (--oldversion, --newversion, --installpath)
  • 1.9 - Eric to fix a bug in globus and adapt the ESGF scripts
  • 1.9 - Roland sent latest version of las for esgf: v8.3. Nicolas to test it and see if it can be pre-built.

-Next meeting: 22 January 2015, 0800 hrs PDT

13 November 2014

Dean, Luca, Nicolas, Prashanth, Georgi

  • Releases

    • 1.8 should be out in devel by next week (Prashanth)
    • 1.8 is being cut in a local git repo at NSC (Prashanth)
    • 1.8 will still include changes that are in github devel branches

  • Issues

    • OpenId resolution issues from older data nodes if sslv3 has been disabled because of poodle on IDP peer. Mail sent out to esgf-devel to ask admins to either upgrade java to 1.7 or ESGF to 1.7.2.
    • Upgrading to Openssl v1 is still not finished, pb with myproxy-logon(Nicolas). Dirty fix exist for publisher issue by Katharina.

-Next meeting: 27 November 2014, 0800 hrs PDT

2 October 2014

Dean, Alan, Stephen, Luca, Nicolas, Prashanth, Kleanthis

  • Releases

    • 1.7.2 Released and installed in production at several sites
    • 1.8 Apache in front of tomcat planned by end of October
    • 1.8 Integration of new development from ceda concerning idp
    • 1.8 RPMization of git and curl
    • 1.8 Pre-compilation of LAS
    • 2.0 Integration of CoG

  • Prashanth to send instructions concerning pam_pgsql.conf issue in 1.7.2

  • Prashanth to update the wiki with instructions to generate signing requests for production nodes

  • Prashanth release manager for 1.8

  • See if existing idp whitelist can be used for new idp version from ceda

  • Nicolas to modify LAS installation. Check with Roland.

  • Nicolas to integrate rpm based installation for git

  • Nicolas and Alan to study Globus RPMs

-Next meeting: 16 October 2014, 0800 hrs PDT

4 September 2014

Dean, Zed, Luca, Matthew, Eric, Nicolas, Prashanth

  • Releases

  • Prashanth to add instructions to install from devel on the wiki

  • Prashanth to look at globus installation problem and consider to add it to 1.7.2 master

  • Nicolas to add esgf_test_suite repo to ESGF github project

  • Nicolas to add esgf_test_suite link to installation from devel procedure

  • Wiki to be updated with instructions to generate signing requests for production nodes.

  • Matthew to sync PCMDI esgf dist mirror

-Next meeting: 18 September 2014, 0800 hrs PDT

21 August 2014

Dean, Luca, Allan, Eric, Prashanth, Nicolas, Katharina

  • Issues

    • unauthorized credential delivery by MyProxy on various index nodes.
    • broken installer due to Tomcat version rolling off the distribution servers

  • Release

    • 1.7.2 to be cut Friday or early next week. Prashanth to do it.
    • 1.8 scheduled for third week of September.
    • esg-node scripts to be maintained according to version, on the distribution servers.

  • Certificate signing

    • Wiki to be updated with instructions to generate signing requests for production nodes.

  • Post meeting follow-up:

    • The unauthorized credential problem was fixed. Details are as follows:
      • Problem cause: PAM is provided a username and password, to verify against the values in the database. When a user registers for an openid on a local node, the password field is filled in with his actual password but is kept empty, if the user has registered on a different IDP. PAM_psql only checks the first row, if there are multiple results. This meant that if in the db, there were entries with a username with a blank password, from a different site BEFORE the entry with the valid password from the local registration, PAM would simply report a successful authentication, and myproxy would return the credentials. Since we have multiple IDPs and many users have the same userids, this has been a problem that's been around for a very long time now.
      • Fix: Edit the /etc/pam_pgsql.conf and add the following line:
        auth_query=select password from esgf_security.user where username = %u and openid like '%%esg-dn1.nsc.liu.se%%'
        (Replace esg-dn1.nsc.liu.se with your myproxy server's name)
        This ensures than only the locally registered openid is selected for the password comparison.

-Next meeting: 4 September 2014, 0800 hrs PDT

10 July 2014

Dean, Luca, Stephen, Georgi, Prashanth, Nicolas

  • Release

    • 1.7.1 has been cut in master. Dist master repo @IPSL, mirror@BADC More details: (https://github.com/ESGF/esgf-installer/wiki/ESGF-Release-Notes)
    • Nicolas to contact Tony and Matthew to add put rainbow back in the loop
    • Nicolas to send the mail and procedure to esgf-devel
    • Prashanth to start cutting 1.7.2
    • Nicolas to send details to Prashanth about compilation process

  • Installation at new sites

    • PNL node up and running. pNl now need to learn how to publish

  • CentOS7 should be supported for 1.8

  • BADC need to join the test federation to test new security features. vesgint-data & vesgint-idx.ipsl.jussieu.fr test fed still operational.

26 June 2014

Alan, Dean, Eric, Katharina, Luca, Nicolas, Prashanth

  • IPSL dist-files mirror: Almost ready. SQLAlchemy download location hardcoded in Publisher code and is an obsolete version. Can this be bumped up to the current release and the hardcoding removed from the publisher code?

    • Prashanth to write to Carla, Stephen, Rachana and Dean, regarding way forward.
    • Nicolas to write mail to Alan/Stephen and Prashanth, with mirroring particulars.

  • Automated test environment: To explore the possibility of using automated test environment to test components on variety of os distributions/versions.

  • Wget issues: Behavior of wget is observed to vary across distributions. User environment not apparent from initial problem reports (Linux or Mac? CygWin? What distro of Linux?)

    • Quickfix: get the wget script to log information which can then be sent to ESGF admins, in case of trouble.
    • Long term: evaluate and develop replacement for get-cert and the wget script itself.

  • Release of 1.7.1:

    • Nicolas reports that main change is to use IPSL repo. Should be ready for release within two weeks.
    • Luca to tag commits for changed esg-search and publisher resources, to ensure inclusion in 1.7.1.

  • CoG integration:

    • Dependent on release of 1.8, with apache proxying for tomcat.
    • CoG integration anticipated timeline: October 2014

  • Truststore pruning and intermediate CA certificates

    • To reduce bloat of truststore, SimpleCAs of index nodes to get intermediate CA certificates, issued by existing federation CAs, eliminating need to have every SimpleCA cacert in the truststore.
    • Prashanth to mail with further details and begin migration exercise.

  • ESGF usage issue with Safari:

    • Luca requests user experiences/feedback with latest version of Safari browser while accessing ESGF. He observes failure when the client presents a user certificate which is not part of ESGF federation.

  • Post meeting comments:

    • Stephen Pascoe:
      • wget issues: Part of the security team roadmap is to develop certificateless wget scripts and eventually to use OAuth (via OpenidConnect). Once the download script doesn't need esoteric certificate features we will have much more freedom to redesign the scripted download use case. A couple of projects in Europe see this as a priority so we will be focussing on it.
      • More generally on security: We are developing upgrades to the security components which include OpenID IdentifierSelect and an HTTPS CA service (the latter would mean we don't call have to call myproxy). We hope to have components ready to try out in test nodes in August and would like to get it live, at least in some nodes, by the end of September. This would be in addition to MyProxy so we don't break existing workflows.

  • Next meeting: 0800 PDT, 10 July 2014

5 June 2014

Stephen, Luca, Matthew, Eric, Prashanth, Nicolas

  • New Online storage system for artifacts and dist files in ESGF 1.7.1 (End of June):

    • It has been decided by the participants to install a new ESGF distrib master repo hosting binaries as static files and as an RPM repo at IPSL. Have to send mail to the list for final GO.
    • IPSL is already an official mirror of most linux distribs
    • Repo will be started from what exist on rainbow
    • Repo will be cleaned using the list provided by Alan at last sprint in Hamburg
    • Mirrors at other sites will be set up in the future
    • STFC commits to be a mirror and will be working within the IWT to enable installation from closest mirror.
    • Rsync tools will be provided for mirroring
    • ESGF Installer will the be adapted to get binaries from the closest mirror

  • Release Management Process has been clarified: Developers push and tag stable and releasable code into devel (https://github.com/ESGF/esgf-installer/wiki/ESGF-Release-Management-&-Installation-Process)

  • Prashanth will work on setting up tomcat behind apache

    • 1.8 to have Apache proxying connections to Tomcat. This is to allow for non java server side code as well as to enable transitioning to CoG
    • Prashanth to seek help from Luca's colleague in this regard

  • LIU has a tool with fine-grained detail extraction and reporting capability for data download analysis, which also provides user details, for messaging purposes. This is currently LIU specific and is being made generic. Prashanth to work with Sandro to discuss integration with available GUI tools

  • RPMization

  • AccessLoggingFilter Issue

    • Downloads count are not always quite accurate - False Negatives; Some downloads are marked as failed because the file size is not equal to the transfer size
    • Some ESGF users perform partial downloads by specifying 'Range' in the header. This leads to HTTP response code 206 (success) but is marked failed by the AccessLoggingFilter because the file size is greater than the transfer size of this piece of file!
    • A better way for the AccessLoggingFilter to set success or failure would be to use the HTTP reponse code
    • Volunteers are welcome to try to fix this issue (https://github.com/ESGF/esgf-node-manager/issues/7)

15 May 2014

  • New Online storage system for artifacts and dist files:

    • Need to check if repos bigger than 1GB can be handled by github (Nicolas)

  • RPMization

    • Git will be the first component installed with a RPM in ESGF 1.8 (Nicolas) (Issue #39)
    • Globus Installation will be done with a RPM in ESGF 1.9 (Eric)

7 May 2014

  • Announcement of 1.7: Nicolas to send out email to devel list

  • Globus RPMS: EPEL has globus RPMS which may contain divergent commits. Eric to communicate with the EPEL maintainer to decide course of action. There also exists ESGF-specific globus code to be RPMed. Since we intend to heavily RPMize the ESGF setup, we need to decide on how/where to host an ESGF specific RPM repository. Stephen's inputs sought.

  • To enable transition to CoG and allow for other non-java server side code, it's been decided to create a new 'feature branch' for ESGF components and have tomcat behind httpd/apache. Prashanth to work on this with inputs from Luca.

  • Test installations: On setting up a test machine, it is advised to peer it with itself, to have a self-contained setup which does not interfere with production environment. If testing federation features is desired, peering may be done with the test federation setup at IPSL.

  • All developers encouraged to visit github tickets and tackle what they can.

  • Prashanth to handle cutting of release 1.8. Will coordinate with Nicolas for binary-building particulars.

  • Next meeting will be in a week, on Tuesday 13 May 2014 9 AM PDT, instead of after two weeks, as yesterday was a rescheduled meeting.

17 April 2014

  • ESGF1.7.0
    • UV-CDAT build failure fixed by Charles
    • Matthew and Jeff will test 1.7.0 installation at PCMDI
    • wget and curl calls should use http and not https. Charles could not install from PCMDI.
    • Final version of 1.7.0 will be cut in master next week
    • Release Note and detailed installation procedure has to be written and given to admins
    • Post installation manual test procedure has to be translated from the IPSL wiki into english on github ESGF-installer wiki
    • Production nodes running ESGF 1.3 or 1.4 will try to upgrade, but they should be warned it might fail. (if yes -> again from scratch)
  • ESGF1.7.1
    • Need to chose online storage system for dist files
    • Need to chose how many past versions of ESGF the installer can handle
    • Need to chose a release frequency: time based instead of feature based (proposed: every 3months is the minimum)
  • ESGF 1.8.0
    • Upgrade slave solr to port 80
    • Need to add node OS version to node ESGF version. (Need to fix the node manager and then ask Sandro)
    • Globus RPM?
    • UV-CDAT RPM?
  • ESGF TEST SUITE
    • Should implement a port testing function
    • Should implement federated search test, and server side tests (are files well deployed, publication test, ...)
    • Has to be moved to ESGF repo, has to use ticket system
  • ESGF IWT
    • Will work with github Milestones (1.7.1 and 1.8.0 have been created)
    • Tickets will be assigned to milestones and people
Add a custom footer
Add a custom sidebar
Clone this wiki locally